All posts by Ethan Schwartz

Watching the temperature outside was clearly a gateway drug to wherever I am now

OK, I’ve probably gone too far now…

Airplane dashboard, or apartment dashboard?
The ultimate in function (mostly) over form.
The night was moist... or was it sultry?
Humidity is rising…

It’s a mix of devices providing the data…

  • For the inside/outside/fridge temperatures and humidities I’m using 433MHz “Acurite” brand probes.  They’re received using a Raspberry Pi with a USB SDR and RTL_433 package.  I’m using a Python script to parse the JSON output from RTL_433 and then write that to a PHP script on my web server which stores the data in a MySQL database.  The Acurite’s transmit every 16s, I only write data if the change is greater than 0.5 from the last value.
  • For the “Storage” info, it’s a 3G cellular connected Particle Electron with an AM2302 and a 4400mAh battery set in my tin-can storage unit where I store all my tools and stuff that doesn’t fit or belong in the apartment–This is a new storage unit, my old one was indoors and somewhat climate controlled.  I’m a bit worried about how some of my stuff will do in this one which is a drive-up and seems to get very hot and very humid.  The Electron will measure and transmit every hour, to preserve the battery it goes into deep sleep between readings.
  • For the “Soil” info, it’s a WiFi connected Particle Photon using an SH-10 stuffed into the spider plant’s pot and a YL-69 which measures resistance between two copper traces.  I take readings every 10 minutes and transmit regardless of changes.  This device is powered by an 18Ah 12V SLA battery so I do sleep the Photon between readings, but the SLA is kept float charged by mains.

Particle’s MVNO has pretty good coverage!

Last year I bought a Particle Electron which is a little module that runs Arduino-like code, but has a cellular modem attached to it as well.  I mated it to a temperature sensor and GPS, put it in a weather tight box and I keep it in the cargo area of my car connected to a switched 12V power point back there–it also has a lithium-ion battery which can keep it running (with judicious standby times) for a couple of weeks between running the car.

The sketch that I have running on it reports the position of my car every ~30s while the vehicle is running (12V is present) and every 1-5 hours when it’s not running (no 12V), there’s also other data reported like speed, course, temperature, cellular signal statistics, etc.  The reporting is through a PHP script that writes it to a database.

This what you’d normally see, a traffic-overlay map with a Subaru icon representing the car’s location.  The InfoWindow opens when the car hasn’t reported a position in more than 15 minutes, “Last Report” is the last report from the car and “Last Check” is the last time this browser pulled data from the server.  Temp is the cargo area temperature in the car and battery percentage is for the lithium ion back up.  The InfoWindow closes itself when the car is moving and the direction of the Subie icon matches the last reported course of the car:

I recently drove all over the place and I decided to see how the cellular signal was along the way so I put together a heatmap where the weights are based on the RSSI of the cellular signal.  I was pretty impressed with how good the signal is, but also the fact that the service works in other countries (like Canada too).

I assume the roaming from USA to Canada was seamless, but I do have a watchdog that may have restarted the device when no cellular was detected for more than 10 minutes.  That might explain the short blips around the border.

Cell Strength on my recent driving, red is stronger

Using Amazon Dash buttons to control TP-Link plugs

I have a couple of fans around the apartment to keep us cool.  Being the lazy type I found that using their manual switch is terrible; I really needed to be able to turn them on or off without getting out of bed or getting off the couch.

To add on/off control to them I decided to buy a couple of TP-Link WiFi-controlled plugs at Amazon.  These aren’t bad, but the only way to control them is with an app that TP-Link produces–it’s cumbersome to pick up your phone, unlock, swipe to find the app, then touch the fan for the room you’re in, especially in the middle of the night.

What I needed was a really simple controller.  Just a single button.  If the fan is on, pushing the button turns it off.  If the fan is off, pushing the button turns it on.

After some quick research it became clear that local network control of these WiFi plugs is really trivial–you make a TCP socket connection to their IP at port 9999 and then send a set of XOR’d JSON commands.  If you poke around online you’ll find the details easily enough.

I have some Particle Photon’s (WiFi enabled Arduino’s), so I wrote up a quick sketch that works fairly well, but it ends up being a switch hung off of hook-up wire, a breadboard, and a 5V power supply–the control may be simple, but the rest is bulky and delicate.  It’s also not cheap, the Photon’s alone are $20.

Then Amazon Prime Day happened and they put their Dash buttons up for 99¢ … that’s the perfect controller for this project! — It’s compact, easy to use (just one button), battery powered, and now it’s under $1 each.

Amazon only allowed buying of one of each brand, so I picked eight random products and ordered some Dash buttons.  At under $1 each Amazon is certainly losing money sending me these, but I figured I spend so much there anyway they owe me one (or eight).

Two of the Dash buttons

There are plenty of posts on the Internet about hardware teardowns of Dash buttons and all the potential from the hardware packed into these guys, but most people have settled on a very simple and practical way of using these in “off label” ways — The gist of it is that each time the Dash button gets pressed the Dash will connect to your WiFi network and send an ARP broadcast to make sure it’s OK to keep using the IP that it has.  That ARP is associated to a mostly-unique MAC address which you can then sniff out using another computer on the network.  Using this technique you can determine when a Dash button is pressed without having to make any changes to the Dash button hardware or software.

The key here of course is that you must get your Dash button on your WiFi network without completing the setup to Amazon–otherwise you’ll be ordering stuff each time the button is pushed!  Luckily it’s surprisingly easy to do this, you pretty much follow all the usual instructions for setup, but then near the final step you simply don’t select a product to associate to the button, instead you quit the app/setup.  This leaves your Dash setup to get on your network and make its request to Amazon, but Amazon will reject it because this Dash is not associated to an order or product.

There are a variety of ways to accomplish the sniffing, some people online are using NodeJS, but I’m more comfortable with Python.  Python also makes quick work of the TP-Link half of this project.

The script below is a quick and lazy cobbling of some Dash button sniffer logic and TP-Link HS-105 controlling logic.  In summary, it sits and listens for an ARP broadcast from the hard-coded MAC addresses, once one is detected it polls the current state of the appropriate TP-Link plug and then sets that plug’s relay to the opposite state.

You can modify the script to attach any number of Dash button MAC addresses to any number of TP-Link plugs–even controlling multiple plugs with a single Dash button press.  In the script I have hardcoded everything for simplicity, but you could get fancy about the associations if you wanted.

First note I’ll make is that the Dash buttons will sometimes broadcast an extra ARP beacon or two, to counter this I added in a short 5-sec back-off time to prevent toggling an outlet more than one time in those cases.

I should also note that at this stage of development the error handling is virtually non-existent, in fact it’ll quit if it can’t reach a plug on the network.

That said, it’s good enough for now!

from scapy.all import *
import json
import socket
import time

onCmd = '{"system":{"set_relay_state":{"state":1}}}'
offCmd = '{"system":{"set_relay_state":{"state":0}}}'
infoCmd = '{"system":{"get_sysinfo":{}}}'
port = 9999
brMAC = '68:37:e9:e6:33:f0' # Airheads
lrMAC = 'ac:63:be:b8:31:d4' # Tide
brIP = '10.0.1.44'
lrIP = '10.0.1.39'
brLC = 0
lrLC = 0

def arp_display(pkt):
 global lrLC, brLC
 if pkt[ARP].op == 1: #who-has (request)
   if pkt[ARP].hwsrc == lrMAC:
     print "Pushed Tide - Living Room"
     if (time.time() - lrLC > 5):
       #print "Executing..."
       lrLC = time.time()
       currentState(lrIP)
      else:
        print "Duplicate, skipping..."
     elif pkt[ARP].hwsrc == brMAC:
       print "Pushed Airheads - Bedroom"
       if (time.time() - brLC > 5):
         #print "Executing..."
         brLC = time.time()
         currentState(brIP)
       else:
         print "Duplicate, skipping..."
     #else:
     # print "ARP Probe from unknown device: " + pkt[ARP].hwsrc

def encrypt(string):
 key = 171
 result = "\0\0\0\0"
 for i in string: 
   a = key ^ ord(i)
   key = a
   result += chr(a)
 return result

def decrypt(string):
 key = 171 
 result = ""
 for i in string: 
   a = key ^ ord(i)
   key = ord(i) 
   result += chr(a)
 return result

def currentState(string):
 try:
   sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   sock_tcp.connect((string, port))
   eString = encrypt(infoCmd)
   sock_tcp.send(eString)
   data = sock_tcp.recv(2048)
   sock_tcp.close()
   rawJson = decrypt(data[4:])
   #print rawJson
   decodedJson = json.loads(rawJson)
   currentState = decodedJson["system"]["get_sysinfo"]["relay_state"]
   if (currentState == 1):
     print "Currently ON, turning OFF"
     turnOff(string)
   else:
     print "Currently OFF, turning ON"
     turnOn(string)
 except socket.error:
   quit("Cound not connect to host")

def turnOff(string):
 try:
   sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   sock_tcp.connect((string, port))
   eString = encrypt('{"system":{"set_relay_state":{"state":0}}}')
   sock_tcp.send(eString)
   data = sock_tcp.recv(2048)
   sock_tcp.close() 
 except socket.error:
   quit("Cound not connect to host")

def turnOn(string):
 try:
   sock_tcp = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   sock_tcp.connect((string, port))
   eString = encrypt('{"system":{"set_relay_state":{"state":1}}}')
   sock_tcp.send(eString)
   data = sock_tcp.recv(2048)
   sock_tcp.close()
 except socket.error:
   quit("Cound not connect to host")

print sniff(prn=arp_display, filter="arp", store=0)

Using SSH to connect a computer on your remote network

So let’s say you expose SSH of your Raspberry Pi to the internet and you’d like to be able to VNC or RDC to a Windows machine on the same network while you’re away.

What you need to do is use SSH to forward your computer’s local port to the port of the remote network computer

Here’s an example:
ssh -L 5900:10.0.1.50:5900 ethanms@ethanms.com

The command above will connect me via SSH to my machine exposed at ethanms.com and use it to forward any requests to my own localhost port 5900 to port 5900 at the remote network address 10.0.1.50.

Assuming that address has a VNC server running I can open VNC viewer on my local machine and try to connect to localhost:5900–it’ll connect me to that remote machine.

This is super useful and doesn’t require you to go through complicated VPN setup, expose a ton of ports to the internet, or use a service that tunnels out and exposes your computer (like Team Viewer or Log Me In)

miniDV capturing in 2017

I have a Sony DCR-HC52 miniDV camera that I’ve made a few tapes with over the years, mostly just family parties and the like–things that at the time you mostly don’t care to rewatch, but suddenly after 10+ years the memories become more precious.

Ever since recording video on your phone has become higher quality, and easier to do for long periods, the standard def DV camera has fallen into disuse, but the other day I came across it and decided I’d break it out to convert those old tapes to a more portable digital format while I still had the working equipment and before the tapes got too old.

These DV cameras came with an interface that let you move the data off tape digitally and on to a computer, in theory there is no quality loss like you had when you converted from analog tape/signal to digital.  Apple calls the interface “Firewire” and Sony called it “i.Link”.  There are other vendor specific names, but the technical (and common) name is “1394”, named for the number assigned to the specification by the IEEE (https://en.wikipedia.org/wiki/IEEE_1394)

My first hurdle, my circa-2014 Macbook Pro doesn’t have Firewire, but it does have Thunderbolt.  After confirming online that this is the way to go I bought a Thunderbolt to Firewire 800 adapter and then a Firewire 800 to Firewire 400 (4-pin) cable.  Plugging the entire thing in I saw the “DVin” icon flash on my camera for a moment, but otherwise no connection… Further poking around online came up with nothing much except a surprisingly short list of supported DV cameras for iMovie from Apple.

I never did a get a definitive “it won’t work” from my searching, but based on the fact that it doesn’t work it appears that modern iMovie has dropped support for older DV cameras.

One other alternate suggestion online was to break out an older Firewire equipped Mac from an era where these cameras were modern and might still be supported by the older copy of iMovie–I do still have my 15 year old G4 PB, but I decided to go another route and look at Linux options instead.

Within a minute off googling I had the answer–there is an open source program called “dvgrab” that will connect to, control, and pull over the video data from DV devices.  A quick question/answer here provided everything I needed:

http://askubuntu.com/questions/1694/is-it-possible-to-import-dv-video-using-firewire

Fortunately I have an old Lenovo T410 laptop which is running Ubuntu 16.04 and happened to have a 4-pin 1394 (firewire) connector on it, I also had a 4-pin to 4-pin firewire cable laying around (see, this is why I never throw old cables away…)

Installing dvgrab was as easy as running sudo apt-get install dvgrab and then the command to start creating files from tapes was equally easy (because some wonderful askubuntu user put it together 5-7 years ago for me!)–

sudo dvgrab -a -format raw -rewind -t prefix-

The above command will start dvgrab with a few useful options–

-a will try to break the video files up by scene (if you had started/stopped your recordings)

-format raw will save the file as .dv which is the native format … this format is easily playable by VLC and can be imported directly into iMovie

-rewind commands the device to rewind first–convenient if you’re cycling through tapes, you pop it in and run the command it will detail with rewinding and then starting playback as well as stopping when the tape is done

-t will add a timestamp to the file name

prefix- will append whatever you replace that word ‘prefix’ with, for example ’70th_birthday-‘ will append that text to each file created

By default dvgrab will break the video up into 1GB files for easier handling and importing (or smaller if -a finds scene splits)

Make sure that you have plenty of space available as it’s about 1GB per 5 minutes.  I have a 250GB SSD in my Lenovo so capture is easy, and then I moved them to a 1TB USB drive for safe keeping.

I was able to copy the files from my Linux machine over to my Mac using ‘scp’ from the Mac command line and then import them into iMovie for editing, titling and saving into a more compressible format (each 1GB .dv file wound up being about 200MB once in .mp4 format).

Now the files can be put up on Dropbox, burned to DVDs, or otherwise spread out to keep the content safe and accessible for more years.

Converting Mac OS X text/hex files to Windows

Mac OS X produces files with just an LF the end of each line.  Windows produces files with both CR and LF.

I ran into a problem building HEX files with MPLAB X IDE on my Mac, the hex files had just LF’s terminating each line but my downloading process required that I have CR and LF marking EoL.

The internet–and I don’t reference one site because I found this in many places–had the answer as usual.

Here is a short Perl incantation that can replace each LF with a CRLF:

perl -pi -e 's/\r\n|\n|\r/\r\n/g' YOURFILENAMEHERE

I use the above as part of a bash script that automatically converts and uploads my hex files to a server:

#!/bin/bash
fwVer=$(date +%y%m%d%H%M%S)
if [ -n "$1" ]
then
  fwVer=$1
fi
perl -pi -e 's/\r\n|\n|\r/\r\n/g' /my/local/path/myprojectfile.hex
scp /my/path/myfile.hex myuser@my.server.com:/my/remote/path/$fwVer.hex

If I supply an argument when calling the script it will use that name when uploading the file, otherwise it will name the file based on the current time using YYMMDDHHMMSS format which allows the latest files to sort up to the top of a list and gives you some clues about when you built one vs. the other.

A quick note, there is no line break in the scp command, but it may appear that way due to formatting on this page.

NASA: Never Assume Something’s Apparent

The Voyagers’ Galactic Trek [NASA.gov]

Thank you NASA for specifically letting me know that this ~40 year old probe will not be functioning in tens of thousands of years.

That said, here’s to hoping that one day we offer cruises along the historic paths these probes took and they become moving historical artifacts that we can visit.

Setting up RPI3 for RTL_433

Install RTL-SDR

apt-get install rtl-sdr

Install gpsd and clients

apt-get install gpsd gpsd-clients python-gps

Install the parts needed for rtl_433

apt-get install libtool libusb-1.0.0-dev librtlsdr-dev

Install cmake

apt-get install cmake

Clone the rtl_433 git repository and install

git clone https://github.com/merbanan/rtl_433.git
cd rtl_433/
mkdir build
cd build
cmake ../
make
sudo make install

Install the parts needed to write your own gpsd programs

apt-get libgps-dev

Go off and do stuff that needs doing…

Star Trek: The Return

I felt like I had to write something about this… I know this won’t be read by ANYONE outside myself but that’s OK.

Fair warning–Here there be spoilers.

This book is some of the worst sci-fi trash I’ve ever read.

Ostensibly penned by Shatner himself, most of the people I’ve mentioned it to start with the same line: “Well, he didn’t really write it did he?”, implying ghostwriting or collaboration… you know, I thought that myself before I started reading, but afterwards I have no doubt it was him.  No one else could write Kirk quite like this while also turning the rest of the cast into bumbling idiots.

The basic premise is that we’re a few months out from the point where the movie Star Trek: Generations has ended.  Enterprise-D is being cut up for scrap and Kirk’s body is still buried on the planet, but soon to be moved to a more permanent resting place.

A mutinous group of Romulan’s have secretly allied with the Borg and using a mysterious device they’ve managed to not only re-animate Kirk’s months old corpse using Borg nanites, but perform a temporal-Katra transfer to restore Kirk’s mind from the moment before his death.

The Romulan is hell bent on Kirk’s destruction because Kirk is responsible for her grandfather’s death (see some TOS episode where they fight Romulan’s).  The Borg are hell bent on Picard’s destruction (or at least return to the collective) because he apparently unknowingly retains some linchpin bit of information from his time as Locutus that could take them all down.  They decide to get together by reanimating Kirk and programming him to assassinate Picard.  The book even questions the sanity of this plan a few times, but it’s brushed aside by showing us that this Romulan is performing her best Ahab impersonation (and the Borg basically don’t give a damn and are apparently largely inept and manipulatable despite their massive successes in conquering species after species across the galaxy).

The entire book writes about Kirk as though he’s a mythic god, complete with describing his “iron muscled forearms” and having him outwit pretty much everyone he deals with.  Just to express how far into the absurd this writing goes, Kirk defeats Worf in a bat’leth competition… WORF.  Also, to be clear, at no point do they ever insinuate that Kirk’s physical or mental abilities are being enhanced, actually quite the opposite they imply he’s fighting through constant pain caused by the nanites.

Every character used–and they pretty much just reuse our old favorites, as though there is no one else in Starfleet–is either written out of character, or as a worst cliche of their character.

Picard is written as an aging, and somewhat dopy, semi-coward.  Riker is written as a series of assertive, though not always well thought out, actions and one-liners.  140 year old McCoy might as well have “Dammit Jim” written in every 6-7 words.  The Borg are written as comically inept at times, as well being able to be manipulated easily with a few obvious logical arguments, or just plain old acting emotional.  Data is written as child-turned-adult comic relief.  Even Spock, who I think is arguably the most well written, is frequently forced into out of character dialog and actions.

The worst part is that the hook that originally brought me here–That the Borg’s origin is actually V’ger following its melding with Commander Decker at the end of Star Trek: The Motion Picture–turns out to be a “Oh by the way” sort of mention within the last 20-30 pages with no real depth.

The further I got into this book the more I realized that it’s written in the style of TOS episodes and movies–full of deus ex machina and contrived situations for the sole purpose of supporting one-liners or some cliched set of actions by a character.  It’s rife with “wait, why did this happen?” or “how could they have done that?” moments.  Your only choices are to stop reading, or suspend the analytical part of your brain that highlights the plot holes and enjoy the predictable, often groan-worthy, lines and scenarios.

In some ways that actually makes this book more endearing, but it also makes it frustrating because you don’t receive the depth you might have hoped for in most of the story points and you find yourself agonizing over dialog or actions that simply don’t fit the character.

As best I can determine, the reason this book exists is that Shatner was pissed by his limited role in Star Trek: Generations and luke warm, not quite-a-real-hero, death; so he decided to re-write that death by coming back to life as a combination action hero, master tactician, technology guru, and victim of circumstance.  He can best Worf in Klingon combat, he can beat up Picard, outwit Romulan’s and Borg, intuitively use 24th century technology against its own occupants despite his being a century out of date, and all this while saving the galaxy.

In conclusion, this is absolutely Star Trek schlock, but hell who am I to judge?  I’m sitting here bitching about a 20 year old Star Trek novel.  Go take it out of your local library and spend a few days reading it, it’s not like you have better things to do.

I’ll just leave you with a picture of the backcover of this book… no accolades or glowing reviews, just a huge ass picture of Shatner.

He also couldn’t resist filling the backcover with his gorgeous mug.