Friday, April 19, 2019

So what's up with the name change?

I've been trying out just using my first and middle initials instead of my full name. It's not that I don't like my name, I just wanted to try something new. Plus, it puts me in good company.

Now if you'll excuse me, I have to go yell at my engineers, who so far have completely failed to create the combustible lemons I requested...

Monday, April 01, 2019

Photos from Texas Furry Fiesta 2019

Well, TFF is over, and I had a blast! Thanks to @Skywind_Kitsune for driving me down there and back. This was my first furry convention, and I look forward to going to many more.

Here are a few photos that I got. Unfortunately, I didn't get names of everybody that I got photos of, so if you recognize someone, please let me know!

My friend Hexadoodle in the fursuit playground.
Lies!
Fursuit Improv
On the observation deck of Reunion tower.
@CRT_QT
Me with CandyPaws
Playing with a pizza frisbee

Thursday, March 21, 2019

My first fursuit

Yesterday, I received my first fursuit for my character Jace Pendry from GEEKpaw productions, and I love it! It fits pretty well, although it's a little tight. It gets hot inside, as expected, which is why I bought an EZcooldown cooling vest.

I can't wait to show it off at TFF next week!

Friday, March 15, 2019

"Have you tried turning it off and on again?" "Yes, my robot did"

Ok, so it's not quite a robot, but it is automated...

Yesterday while I was at work, there was a power failure at home that lasted longer than my UPS could handle. My server shut down, and when it came back up after the power was restored, it couldn't get online. The problem that I discovered after I got home was that the cable modem had stopped responding to DHCP requests, pings, or anything. The solution to this was simple - unplug it and plug it back in.

But what if something like this happens when I'm out of town? I need some way of resetting it without being physically present. I realized I already had the components to hack together a solution - I had a relay module from an Arduino kit, an Arduino already up and running controlling my air conditioner via an IR LED, and jumper wire. I even had a script monitoring the internet connection already running on the server, but all it could do at that point is force the server to re-request DHCP. All I needed to do was cut the low-voltage wire for the power adapter, connect the cut ends to the Common Normally Closed terminal of the relay, then wire it to my Arduino. That proved a bit challenging, because the Arduino has to be in a spot where its LEDs have line-of-sight to my air conditioner, but I don't want to put the modem right there. I had to chain together several individual jumper wires to get it to reach, and I may re-visit that design in the future.

So then it's just a matter of writing the software. I modified my existing Arduino sketch to pull a GPIO high most of the time, but when it receives a command over USB serial, pulls it low for 5 seconds. The relay module has a P-channel MOSFET controlling the relay, so it is active-low. When the relay activates, it disconnects the Normally Closed circuit and the modem reboots. I set up my monitoring software so that if it can't ping the modem for a period of 90 seconds, it resets. While the modem is connecting, it hands out a local IP address of 192.168.100.11, so I can still ping it and grab the status. It also resets the whole thing if it boots up, but doesn't come online within 4 minutes.

It's not the most hacky solution I've come up with so far, but it's close.

Sunday, March 03, 2019

Volt mod: Forcing brake lights on in "L" mode

The Volt has a standard 5-position gear selector lever, with Park, Neutral, Reverse, Drive, and "Low". It doesn't make much sense for a hybrid or electric transmission to have a "Low" position, but like other manufacturers, GM has chosen to implement it in a way that simulates the same kind of engine braking that you would get from a traditional transmission. Unlike a traditional powertrain, there is no issue with leaving it in this position while driving at high speeds. It does this by changing the mapping on the accelerator pedal. In "D", it will apply a small amount of regenerative braking (about 2.5 kW) to slow the car down a bit. In "L" mode, it applies a larger amount of braking (30 kW). This can quickly slow the vehicle almost to a stop.

The only issue is that it doesn't turn the brake lights on when you do this! This can lead to a situation where the car is decelerating quickly, but but the vehicles behind don't notice.

So I had an idea... I can certainly detect the conditions under which the car is slowing down, but how can I turn on the brake lights? It turns out there are diagnostic commands that can be sent over the OBD2 port to turn each individual light on for testing. So the conditions I need to check are:

  1. Gear selector is in "L" position,
  2. Accelerator pedal is less than 5% pressed, and
  3. Cruise control is not active.

I feel a lot safer driving in "L" mode with this hack in place. Although this would not have stopped that kid from rear-ending me - that happened at a stoplight with the brakes pressed.

Thursday, February 28, 2019

Cursing at ncurses

I've just spent the last hour and 45 minutes debugging a really strange problem I encountered after updating my server to Ubuntu 18.04. Whenever I use ncurses-based programs within tmux, sometimes the text would display strangely, as if the cursor were in the wrong place. Now, it's not immediately obvious who is at fault here - there are three completely different terminal emulators at play here. The program I am running is ncdu (ncurses disk usage), which sends its output to tmux, which is running under mosh, and is finally rendered by xterm.

I was quickly able to determine that the problem must lie between ncdu and tmux, because switching tmux to a different window and switching back did not fix the problem. I also verified this by doing tmux capture-pane -p, and it indicated that everything I was seeing was exactly now tmux was drawing it.

So what changed? Well, ncdu and ncurses both got upgraded. I'm still using my old custom-compiled tmux, though, so something must have changed in the way ncdu communicates. I suspected one of two things: either something else was writing to the terminal, or ncurses had been upgraded to a version that is more "clever" about the way it draws the screen. Luckily, tmux has a feature to log all terminal output to a file, so I was able to replay it. I wrote a quick Python script to slowly dump the log to the terminal, byte by byte, so I can pinpoint what it is sending when the problem occurs:

What I found out was that when I press Page Down, it was sending a command to scroll a region of the screen: \033[11S. However, tmux didn't seem to understand this instruction, and just ignored it. I verified this by looking at the source itself.

So what could I do? I could just use the normal version of tmux from the Ubuntu repository. It's a newer version that does support the scroll command, but the reason I switched back to the old version is because the newer version was crashing. I'd rather use an older, stable version than deal with crashes that take out all my sessions. So the only chance is to convice ncdu not to send the scroll command. Since ncdu uses ncurses, which reads a database of command codes for different terminal types. What I ended up having to do was:

  1. Download ncurses.
  2. Edit misc/terminfo.src, and remove indn=\E[%p1%dS, from the screen section.
  3. Run tic terminfo.src to convert it to the binary format read by ncurses.

This is the kind of problem that I enjoy solving. A subtle bug, which doesn't endanger my data, but is just annoying enough to fix, and has a root cause that is easy to understand - and not just a stupid mistake on my part.

Sunday, February 24, 2019

Time for an upgrade

Last evening, at 21:00, my server (marvin) powered off for no reason. Well, it happens. No reason to be worried, right? Well, earlier this morning, at 00:38, it did it again. So it's dying. It's time for an upgrade, anyway... It's using a motherboard I bought in 2010 and an AMD Phenom II X4 CPU that I bought in 2011. I bought an AMD Ryzen 7 2700X processor, 16GB DDR4 memory, and an MSI X470 Gaming Plus motherboard. It's not going in marvin, though - I don't need that much power for what I'm using it for. No, the new stuff is for arthur, my gaming desktop. Like I did 4 years ago, marvin is getting arthur's old motherboard. In fact, I ended up just sticking marvin's boot drive in arthur's case, since that was much easier that swapping a motherboard, and I don't actually need to run arthur most of the time. The only thing I was using it for most of the time was to watch videos, and I don't need a super powerful CPU and video card for that. In the meantime, I've set up marvin to run X so I can at least do what I've been doing until the new stuff arrives.

I actually like this arrangement because it is more efficient - I don't need to have two power-hungry systems running all the time. A few months ago, I actually tried replacing arthur with the Raspberry Pi 3 B+ that I ended up using in my Volt mod project. The only reason it didn't work is because it struggled to play 1080p videos. I don't know why I didn't think of just using marvin as a media player - it already does so much other stuff, surely it can handle one more task. I may keep arthur turned off more often even after I get it set up.

One complication is the fact that arthur's old motherboard doesn't have a header for a parallel port, which I was using to run my super-old automation system that I built in 2008, using the data pins of the parallel port as 5v GPIO. I could just wire it up to my Arduino, but that would be more than I'm willing to invest in that project. I'd rather build something from scratch than put any more work into it. So I just ordered a PCIe parallel port card. Hopefully, it will work just the way it did before.

I'm also in the process of upgrading marvin to Ubuntu Server 18.04. I've got another (bigger) SSD boot drive that I've installed a clean image of 18.04 onto, and I'm slowly getting services up and running on it using a VM. It's still not ready, though, so I'm still stuck on 14.04 for now. It's going to be EOL soon, though. I definitely understand the upgrade fears - it used to be that I was always eager to try out the latest stuff, even if I had to compile it from source, because fixing it when it broke was an adventure. Now, I want it to just work the same way it always did before.

Saturday, February 23, 2019

How I remember all my passwords

Most password managers are based on the same basic idea - you remember one password, you put that into your password manager, and it spits out your plaintext passwords for all the sites you visit. This is fine for sites you don't visit a lot and don't need high security for.

But I wanted to be able to just remember my passwords. So I created Brain Password Manager, which allows me to practice entering passwords on a schedule, so I don't forget them.

BPM doesn't actually store my passwords. Instead, it runs the passwords through a very large number of calculations (800,000 rounds of PBKDF2) to arrive at a number between 0 and 4095. This number is the only information it stores.

When you practice entering your passwords, it does the same calculations. If the resulting number is different, then your definitely typed your password wrong. But, if you enter any password at random, there is a 1 in 4096 chance that it will still say you are correct. This isn't very likely to happen due to a simple typo in your password, though.

What this means is that any attacker trying to extract your passwords will find that lots of passwords match, and has no way of proving which one is the real password, other than trying it out on the site that the password is for. In fact, BPM has a built-in feature which finds false positives for a password to demonstrate just how many passwords an attacker would have to try.

I even created a utility to bruteforce false positives, which demonstrates just how many passwords an attacker would have to try:

All of these passwords will be accepted as correct.

I still use a regular password manager for sites which I rarely log into. BPM is mainly for passwords that I enter frequently enough that a password manager is a hassle, but not frequently enough that I don't forget the password.

Sunday, February 17, 2019

Hacking my Volt with a Raspberry Pi - Part 4: Software

Code repositories

Since I started this series, I have had many requests for the code behind it. A majority of the code is on Gitlab already:

Some of the other code (like the Android app) I cannot release yet because it's intertwined with a bunch of other irrelevant code (like my home automation system). I may release this code in the future at some point.

Overview of services on the Raspberry Pi

Serial Monitor

serial_monitor.py

This is the primary service that interfaces with the Macchina M2 and the phone. It is responsible for decoding frames from the serial port, connecting to the phone app via Bluetooth, and receiving and routing messages between the 3 systems. It is also responsible for managing the GPIO poller process (gpio_poll.c). It opens a UDP socket bound to the loopback interface that it receives messages on. There are several other services that send it commands this way.

monitor_hotload.py

This is loaded as a module by serial_monitor.py, which defers quite a bit of logic to it. It is a bit unusual in that most of the functions defined in it have a 'self' argument, even though they are not actually part of a class. These are treated as "methods" of the SerialMonitor class in serial_monitor.py. The reason it is done this way is so that it can be easily reloaded on the fly, as seen in my demonstration video.

This module is responsible for:

  • Layout of the display
  • Parsing and interpreting data frames and events from Macchina
  • Responding to button press events (from gpio_poll)
  • Data logging (cardata and managing can bus dumper)
  • Idle queries - sends an info packet to my server every 10 minutes while the car is off (replaces OnStar Vehicle Status)
  • Keeping the system clock updated based on clock frame from vehicle
  • Controls power to the screen
  • Controls the beeper (via gpio_poll)
gpio_poll.c

This C program polls the GPIO pins for the buttons. It maps the registers directly so that it can control GPIO without going through the kernel. It also sets up the internal pull-up resistors for the button GPIOs, and controls the PWM output for the beeper. The six buttons are connected to 3 GPIO pins (A, B, and C) and ground (G). The buttons are connected to the following pins:

  • B & G - White
  • A & B - Black
  • A & G - Dial
  • A & C - Blue
  • B & C - Green
  • C & G - Red

To poll the buttons, it goes through 4 cycles with a delay of 5ms in between: one with all of the pins in input mode, one with A pulled low, one with B pulled low, and one with C pulled low. From this, it can determine when any one of the six buttons is pressed. Unfortunately, it cannot detect more than one button being pressed at the same time, but that hasn't been a problem. Whenever a button is pressed or released, an event is sent over a pipe to serial_monitor.py. This process also detects "long presses" (250ms).

The rotor is also polled at the same time through its two dedicated pins. There are 4 codes it can return (in Gray code) which is repeated 6 times around the dial for a total of 24 pulses per rotation.

The beeper is controlled through a ring buffer in shared memory consisting of (frequency, duration) pairs. At the end of the buffer, or when frequency = 0, then the beeper is turned off.

canlog_shmem.c

This program is optional, and runs as two processes: In "source" mode, it connects to the Macchina via USB and tells it to forward every CAN frame it recieves. It then reads those frames from the Macchina and places them in a shared-memory buffer. In "sink" mode, it reads from this shared buffer, and writes the frames to a compressed file. This ensures that even if the I/O to the SD card stalls for whatever reason, or the compression takes too long, then the source process can still read frames from USB and doesn't get backlogged - it simply dropps frames that can't go into the ring buffer.

hud_shm.py

This is a helper module which contains the necessary data structures for communicating with the HUD service.

bitstream/*, cardata_shmem.py

This is an auxilliary Python extension to assist with parsing data frames from the Macchina. It is not strictly necessary, since the data can be parsed in Python code, but it is more efficient to do it here.

connect_spp.sh

This is a helper script which determines the the RFCOMM channel that the phone is listening to the SPP service on, then associates rfcomm0 with the phone.

Dashcam Monitor

dashcam_monitor.py

This service is responsible for managing the dashcams. It is state-machine driven, transitioning between states based on the presence of flag files. The states all have 3-letter codes which are shown in the corner of the display:

  • IDL (IdleState) - Default state, cameras are off.
  • REC (RecordState) - Cameras are powered on and recording.
  • PWR (PoweroffWaitState) - Cameras are being powered off, and the monitor is waiting for the capacitors to drain.
  • WFS (WifiWaitState) - Cameras possibly have new videos, waiting to connect to home Wifi network.
  • USB (WaitUSBState) - Cameras and hub are powered on, waiting for cameras to go into storage mode and show up in /dev/disk/by-label.
  • MNT (MountState) - One of the cameras is being mounted.
  • CPY (RunCopyState) - Videos are being copied from camera.
  • UMT (UnmountState) - Camera is being unmounted.
  • MAN (ManualMountState) - User requested manual poweron and mount (want-mount).
  • HUB (ResetHubState) - User requested hub be turned off for 2 seconds to reset it. This is necessary because sometimes the cameras don't show up over USB.

There are several flag files that control the state machine:

  • want-record - Something wants the cameras to record. Causes IdleState to transition to RecordState, and causes RunCopyState to abort any copy in progress.
  • need-check - Set whenever the camera has been powered on in record mode, indicating that there may be new videos. Causes IdleState to transition to WifiWaitState
  • want-lcopy - Indicates the user wants to do a local copy.
  • abort-all - Causes the current operation to be aborted and returns the monitor to IdleState.
  • copy-restart - Causes the current copy operation to be restarted.
  • reset-hub - Causes the USB Hub to be reset.
do_copy.py

This script is responsible for actually copying videos to my servers. It gathers metadata about each video, determines which videos are contiguous, then sends metadata to the server, which responds with which files still need to be uploaded. It then uploads each video in chunks of 2MB at a time.

do_mount.sh, do_unmount.sh

These scripts are responsible for actually mounting the cameras. One quirk of the A119S camera I use in the front is that it will start recording for a second even when going into storage mode. This leaves a lot of very short videos on the card unless they are deleted. The do_mount.sh script deletes these files, then remounts the card read-only so that even if power is lost to the camera while copying, it won't damage the filesystem.

HUD

hud.c

This program is responsible for actually drawing text on the screen. It receives layout instructions through shared memory, and updates text asynchronously. It uses Cairo to do the drawing to a memoroy buffer, then copies changed portions of that buffer directly to the Raspberry Pi's framebuffer. This is more efficient than running an instance of X.org.

Wifi monitor

wifi_monitor.py

This service nudges the Wifi interface to "encourage" it to try its hardest to connect whenever we have videos we want to upload. It does this by pinging the upload host, and if it can't be reached, it will call wpa_cli scan, and if that doesn't work, it will remove and reinsert the driver for the interface. It reports its status back to dashcam-monitor by writing the local address to wifi-addr.

Hologram monitor

hologram_monitor.py

This service keeps PPPD running whenever the Hologram Nova modem is plugged in. It also maintains the alternate routing table, which ensures that sockets only use the modem when the specifically bind to its address. This prevents background services from consuming data over Hologram, which is expensive.

ppp-chat-script

This file is used by pppd to set up the modem to connect.

Hologram Command service

hologram_command_listener.py

This service listens for command packets on the Hologram interface, using port 4011. The only way to send packets to this address is to use the Hologram REST API. The commands are verified using HMAC-SHA256 and are timestamped to protect against replay attacks.

Log service

logmgr.py

This service receives the output from all the other services and saves it to a log file.

RFCOMM Listen service

rfcomm_read_commands.sh, run_rfcomm_listen.sh

I haven't actually run this service since setting up Hologram - it is intended to recieve commands over bluetooth, but it wasn't secure since it would accept connections from any device.

Support modules

utils.py

This module contains common functions and classes used by most of the Python services

hotload.py

This module contains the code to detect live changes to the source code of a module and reload it.

Service management

run_*.sh

Any shell script starting with run_ is simply a wrapper that runs the corresponding service and ensures its output is connected to the Log manager service.

etc/*.server

These are systemd unit files for each of the services that needs to run.

etc/install

This script installs the .service files into /etc/systemd after updating the paths in them to point to the correct location.

Utility scripts

obd.sh

This script causes the Macchina to send an OBD2 query, then it prints the result.

send_command.py

This script does not run on the Pi. It is used to send a command to the Pi using the Hologram API.

log_receiver.py

This script runs on my server, and receives and logs the info packets sent by the Pi.

update_cardata_fields.py

This contains the master list of cardata columns, how many bits each one takes in the frame, and whether the value is signed. It auto updates all the relevant sections of the parser and generator in each source file so everything stays in sync.

Macchina M2 firmware

bt_interface.cpp

This file contains most of the custom code I have written for the Macchina. It contains the code to extract data from received CAN frames, communicate with the Pi via the serial port on the back, and communicate directly with the phone if a bluetooth module is present (it currently isn't). It also sends OBD2 queries in a specific sequence designed to sample all PIDs at even intervals, with more important PIDs being sampled more often.

calc_pids.py

This script contains the master list of PIDs to query, as well as the sequence in which they are queried. It is organized as a tree - the top level has 5 elements, and so time is divided equally 5 ways between each subtree. The first 3 elements are dedicated to sampling the amperage of the HV battery and the two motors, which consumes 3/5 of all queries sent. The remaining 2/5 is divided between voltage, range remaining, temperatures, and trouble code queries.

obd2_query_sequence.h, obd2_switch.h

These files are generated by calc_pids.py.

update_m2ret_cardata_fields.py

This script useds the master list from update_cardata_fields.py to update bt_interface.cpp with the current list of fields.

Other software

There are a few other scripts and programs that interface with the system or otherwise use the data, but I haven't had time to organize it. I may go into more detail in a future post.

An open letter to Josie B. Smith

Your email address is not јѕрenguіn@gmail.com! Stop signing up for stuff with it. If I knew your real email address, I would tell you directly, but since you don't even seem to know what it is, there's not much I can do. Until you figure it out, I will keep resetting passwords on your accounts and deleting them.

Saturday, February 16, 2019

Goodbye, Peter: 2007 - 2019

Today, I said goodbye to my beautiful friend, Peter. He was one month shy of being 12 years old. He has kept me company through the good and the bad times. He was always very friendly and always ready to cuddle.

I will miss you, Peter.


The first thing he was curious about when I brought him home (May 12, 2007)
Hiding under the bed (May 12, 2007)
May 12, 2007
May 14, 2007
May 14, 2007
May 25, 2007
Learning to climb onto my messy workbench (June 4, 2007)
June 4, 2007
Playing outside for a bit (September 22, 2007)
September 22, 2007
September 22, 2007
First time encountering snow (December 23, 2007)
December 23, 2007
March 29, 2008
October 2, 2011
December 30, 2014
August 5, 2015
January 7, 2017
March 17, 2017
October 4, 2018
Even as an old man, he loved to play (October 7, 2018)
January 10, 2019