Sunday, February 17, 2019

Hacking my Volt with a Raspberry Pi - Part 4: Software

Code repositories

Since I started this series, I have had many requests for the code behind it. A majority of the code is on Gitlab already:

Some of the other code (like the Android app) I cannot release yet because it's intertwined with a bunch of other irrelevant code (like my home automation system). I may release this code in the future at some point.

Overview of services on the Raspberry Pi

Serial Monitor

serial_monitor.py

This is the primary service that interfaces with the Macchina M2 and the phone. It is responsible for decoding frames from the serial port, connecting to the phone app via Bluetooth, and receiving and routing messages between the 3 systems. It is also responsible for managing the GPIO poller process (gpio_poll.c). It opens a UDP socket bound to the loopback interface that it receives messages on. There are several other services that send it commands this way.

monitor_hotload.py

This is loaded as a module by serial_monitor.py, which defers quite a bit of logic to it. It is a bit unusual in that most of the functions defined in it have a 'self' argument, even though they are not actually part of a class. These are treated as "methods" of the SerialMonitor class in serial_monitor.py. The reason it is done this way is so that it can be easily reloaded on the fly, as seen in my demonstration video.

This module is responsible for:

  • Layout of the display
  • Parsing and interpreting data frames and events from Macchina
  • Responding to button press events (from gpio_poll)
  • Data logging (cardata and managing can bus dumper)
  • Idle queries - sends an info packet to my server every 10 minutes while the car is off (replaces OnStar Vehicle Status)
  • Keeping the system clock updated based on clock frame from vehicle
  • Controls power to the screen
  • Controls the beeper (via gpio_poll)
gpio_poll.c

This C program polls the GPIO pins for the buttons. It maps the registers directly so that it can control GPIO without going through the kernel. It also sets up the internal pull-up resistors for the button GPIOs, and controls the PWM output for the beeper. The six buttons are connected to 3 GPIO pins (A, B, and C) and ground (G). The buttons are connected to the following pins:

  • B & G - White
  • A & B - Black
  • A & G - Dial
  • A & C - Blue
  • B & C - Green
  • C & G - Red

To poll the buttons, it goes through 4 cycles with a delay of 5ms in between: one with all of the pins in input mode, one with A pulled low, one with B pulled low, and one with C pulled low. From this, it can determine when any one of the six buttons is pressed. Unfortunately, it cannot detect more than one button being pressed at the same time, but that hasn't been a problem. Whenever a button is pressed or released, an event is sent over a pipe to serial_monitor.py. This process also detects "long presses" (250ms).

The rotor is also polled at the same time through its two dedicated pins. There are 4 codes it can return (in Gray code) which is repeated 6 times around the dial for a total of 24 pulses per rotation.

The beeper is controlled through a ring buffer in shared memory consisting of (frequency, duration) pairs. At the end of the buffer, or when frequency = 0, then the beeper is turned off.

canlog_shmem.c

This program is optional, and runs as two processes: In "source" mode, it connects to the Macchina via USB and tells it to forward every CAN frame it recieves. It then reads those frames from the Macchina and places them in a shared-memory buffer. In "sink" mode, it reads from this shared buffer, and writes the frames to a compressed file. This ensures that even if the I/O to the SD card stalls for whatever reason, or the compression takes too long, then the source process can still read frames from USB and doesn't get backlogged - it simply dropps frames that can't go into the ring buffer.

hud_shm.py

This is a helper module which contains the necessary data structures for communicating with the HUD service.

bitstream/*, cardata_shmem.py

This is an auxilliary Python extension to assist with parsing data frames from the Macchina. It is not strictly necessary, since the data can be parsed in Python code, but it is more efficient to do it here.

connect_spp.sh

This is a helper script which determines the the RFCOMM channel that the phone is listening to the SPP service on, then associates rfcomm0 with the phone.

Dashcam Monitor

dashcam_monitor.py

This service is responsible for managing the dashcams. It is state-machine driven, transitioning between states based on the presence of flag files. The states all have 3-letter codes which are shown in the corner of the display:

  • IDL (IdleState) - Default state, cameras are off.
  • REC (RecordState) - Cameras are powered on and recording.
  • PWR (PoweroffWaitState) - Cameras are being powered off, and the monitor is waiting for the capacitors to drain.
  • WFS (WifiWaitState) - Cameras possibly have new videos, waiting to connect to home Wifi network.
  • USB (WaitUSBState) - Cameras and hub are powered on, waiting for cameras to go into storage mode and show up in /dev/disk/by-label.
  • MNT (MountState) - One of the cameras is being mounted.
  • CPY (RunCopyState) - Videos are being copied from camera.
  • UMT (UnmountState) - Camera is being unmounted.
  • MAN (ManualMountState) - User requested manual poweron and mount (want-mount).
  • HUB (ResetHubState) - User requested hub be turned off for 2 seconds to reset it. This is necessary because sometimes the cameras don't show up over USB.

There are several flag files that control the state machine:

  • want-record - Something wants the cameras to record. Causes IdleState to transition to RecordState, and causes RunCopyState to abort any copy in progress.
  • need-check - Set whenever the camera has been powered on in record mode, indicating that there may be new videos. Causes IdleState to transition to WifiWaitState
  • want-lcopy - Indicates the user wants to do a local copy.
  • abort-all - Causes the current operation to be aborted and returns the monitor to IdleState.
  • copy-restart - Causes the current copy operation to be restarted.
  • reset-hub - Causes the USB Hub to be reset.
do_copy.py

This script is responsible for actually copying videos to my servers. It gathers metadata about each video, determines which videos are contiguous, then sends metadata to the server, which responds with which files still need to be uploaded. It then uploads each video in chunks of 2MB at a time.

do_mount.sh, do_unmount.sh

These scripts are responsible for actually mounting the cameras. One quirk of the A119S camera I use in the front is that it will start recording for a second even when going into storage mode. This leaves a lot of very short videos on the card unless they are deleted. The do_mount.sh script deletes these files, then remounts the card read-only so that even if power is lost to the camera while copying, it won't damage the filesystem.

HUD

hud.c

This program is responsible for actually drawing text on the screen. It receives layout instructions through shared memory, and updates text asynchronously. It uses Cairo to do the drawing to a memoroy buffer, then copies changed portions of that buffer directly to the Raspberry Pi's framebuffer. This is more efficient than running an instance of X.org.

Wifi monitor

wifi_monitor.py

This service nudges the Wifi interface to "encourage" it to try its hardest to connect whenever we have videos we want to upload. It does this by pinging the upload host, and if it can't be reached, it will call wpa_cli scan, and if that doesn't work, it will remove and reinsert the driver for the interface. It reports its status back to dashcam-monitor by writing the local address to wifi-addr.

Hologram monitor

hologram_monitor.py

This service keeps PPPD running whenever the Hologram Nova modem is plugged in. It also maintains the alternate routing table, which ensures that sockets only use the modem when the specifically bind to its address. This prevents background services from consuming data over Hologram, which is expensive.

ppp-chat-script

This file is used by pppd to set up the modem to connect.

Hologram Command service

hologram_command_listener.py

This service listens for command packets on the Hologram interface, using port 4011. The only way to send packets to this address is to use the Hologram REST API. The commands are verified using HMAC-SHA256 and are timestamped to protect against replay attacks.

Log service

logmgr.py

This service receives the output from all the other services and saves it to a log file.

RFCOMM Listen service

rfcomm_read_commands.sh, run_rfcomm_listen.sh

I haven't actually run this service since setting up Hologram - it is intended to recieve commands over bluetooth, but it wasn't secure since it would accept connections from any device.

Support modules

utils.py

This module contains common functions and classes used by most of the Python services

hotload.py

This module contains the code to detect live changes to the source code of a module and reload it.

Service management

run_*.sh

Any shell script starting with run_ is simply a wrapper that runs the corresponding service and ensures its output is connected to the Log manager service.

etc/*.server

These are systemd unit files for each of the services that needs to run.

etc/install

This script installs the .service files into /etc/systemd after updating the paths in them to point to the correct location.

Utility scripts

obd.sh

This script causes the Macchina to send an OBD2 query, then it prints the result.

send_command.py

This script does not run on the Pi. It is used to send a command to the Pi using the Hologram API.

log_receiver.py

This script runs on my server, and receives and logs the info packets sent by the Pi.

update_cardata_fields.py

This contains the master list of cardata columns, how many bits each one takes in the frame, and whether the value is signed. It auto updates all the relevant sections of the parser and generator in each source file so everything stays in sync.

Macchina M2 firmware

bt_interface.cpp

This file contains most of the custom code I have written for the Macchina. It contains the code to extract data from received CAN frames, communicate with the Pi via the serial port on the back, and communicate directly with the phone if a bluetooth module is present (it currently isn't). It also sends OBD2 queries in a specific sequence designed to sample all PIDs at even intervals, with more important PIDs being sampled more often.

calc_pids.py

This script contains the master list of PIDs to query, as well as the sequence in which they are queried. It is organized as a tree - the top level has 5 elements, and so time is divided equally 5 ways between each subtree. The first 3 elements are dedicated to sampling the amperage of the HV battery and the two motors, which consumes 3/5 of all queries sent. The remaining 2/5 is divided between voltage, range remaining, temperatures, and trouble code queries.

obd2_query_sequence.h, obd2_switch.h

These files are generated by calc_pids.py.

update_m2ret_cardata_fields.py

This script useds the master list from update_cardata_fields.py to update bt_interface.cpp with the current list of fields.

Other software

There are a few other scripts and programs that interface with the system or otherwise use the data, but I haven't had time to organize it. I may go into more detail in a future post.

An open letter to Josie B. Smith

Your email address is not јѕрenguіn@gmail.com! Stop signing up for stuff with it. If I knew your real email address, I would tell you directly, but since you don't even seem to know what it is, there's not much I can do. Until you figure it out, I will keep resetting passwords on your accounts and deleting them.

Saturday, February 16, 2019

Goodbye, Peter: 2007 - 2019

Today, I said goodbye to my beautiful friend, Peter. He was one month shy of being 12 years old. He has kept me company through the good and the bad times. He was always very friendly and always ready to cuddle.

I will miss you, Peter.


The first thing he was curious about when I brought him home (May 12, 2007)
Hiding under the bed (May 12, 2007)
May 12, 2007
May 14, 2007
May 14, 2007
May 25, 2007
Learning to climb onto my messy workbench (June 4, 2007)
June 4, 2007
Playing outside for a bit (September 22, 2007)
September 22, 2007
September 22, 2007
First time encountering snow (December 23, 2007)
December 23, 2007
March 29, 2008
October 2, 2011
December 30, 2014
August 5, 2015
January 7, 2017
March 17, 2017
October 4, 2018
Even as an old man, he loved to play (October 7, 2018)
January 10, 2019

Monday, February 11, 2019

Hacking my Volt with a Raspberry Pi - Part 2: History

Before the Volt

The project actually began in April 2017 (for my previous car, a 2013 Ford Fusion Energi), when I received my Macchina M2 beta unit. It started with just the Macchina, with a Bluetooth module (a Microchip RN42XVP) to connect to my phone. I added code to my all-encompassing Android app to display data gathered from sniffing the CAN bus and OBD2 queries. It also logs this data so I can review it later, and I can even display it along with my dashcam videos. I used one of the 12 volt I/O drivers on the back of the Macchina to power my dashcam - that way, I could not only turn the camera on and off from my phone app, I could record in the log the exact time that power was applied, and use that timestamp to synchronize the recorded data with the video. Over the next year I made a few minor improvements to the firmware, but didn't add any additional hardware.

Of course, when I got my Volt, I had to redo a lot of the reverse-engineering that I had done, because other than the standard OBD2 queries, everything going over the CAN bus was completely different. Eventually, I was able to pull all the same data that I could pull from my old car, and more.

Dashcam video auto-copy

Shortly after I got the Volt, I had an idea - what if I could automatically upload videos from my dashcam to my server when I get home instead of having to swap out the microSD card all the time? I had a plan. My dashcam will power on and start recording if it doesn't detect that it is plugged into a computer, otherwise it will go into mass storage mode, allowing the computer to access the microSD card.

I had a C.H.I.P. single-board computer that I got in December 2015, but had never found a use for, and it had a USB port, built-in WiFi, and a 3.3v TTL serial port on the GPIO pin block that I could use to communicate with the Macchina. All I had to do was figure out a way to connect and disconnect the data lines to control whether the camera would record or copy.

Well, it turned out that the C.H.I.P. is a piece of C.R.A.P. that wouldn't even power on after sitting in a box for 2 years. So, I decided to get a Raspberry Pi Zero W instead. This turned out to be a good thing in the end, because the Raspberry Pi is much better supported and documented.

In order to control the power to the camera, I just need a circuit that could control the +5v line of USB power. I built a circuit that connected a USB B port to a USB A port, with power going through a P-channel MOSFET controlled by a GPIO pin. To trick the camera into thinking that it wasn't plugged into a computer, I just had to effectively disable the USB port on the Pi by switching it into USB Device mode instead of Host mode. This required a change to the kernel module, since it wasn't designed to switch modes without changing the boot configuration.

Since the Raspberry Pi had built-in Bluetooth, I eventually realized I no longer needed the buggy Bluetooth module on the Macchina, which tended to drop characters and sometimes fail to power on. I could just have the Pi connect directly to my phone, using the same protocol I had developed.

The original power control board for the dashcam.

Climate controls

One thing that always bugged me about the Volt, and a lot of other cars in general, is that there are no physical buttons for the climate controls. Everything is controlled through the touchscreen, or flat-labeled touch-sensitive buttons on a panel. I had an idea - there are plenty of GPIO pins on the Raspberry Pi, and they even have built-in pull-up resistors, so I can easily add a few buttons that I can program to do whatever I want. All I had to do was watch the CAN bus and see what frames get sent when I press the virtual buttons, and I have full control over the heating and cooling system.

I came up with a design using 5 of these buttons, and a Rotary encoder (which also can be pressed as a 6th button). I managed to connect the entire board using only 5 GPIO pins, plus ground. The two pins of the rotary encoder are connected to dedicated GPIO pins, and the 6 buttons are connected in a tetrahedral pattern to the other 3 GPIO pins and ground. Everything is soldered to a solderable breadboard, which is cut down to size. I originally mounted it right below the center stack, beneath the volume control, but I found it was a lot easier to use if I mounted it horizontally just to the left of the gear selector (covering the PRNDL indicator - but the same indicator is visible on the main screen).

I assigned the blue button to switch between "Fan Only" and "Comfort/ECO" mode, like I used to be able to easily do in my old 2007 Ford Fusion. The green button would toggle an internal flag selecting between "ECO" and "Comfort" mode. The dial was multi-purpose - by default, it would control the radio volume, but I could press it once to switch it to controlling fan speed, or twice to make it control temperature.

Of course, there's no reason to stop at just controlling stuff on the car. Since the Raspberry Pi was connected was connected to my phone via Bluetooth, I could also have it control that, as well. The black button pauses and plays music on my phone, and I could switch the dial into a mode that would scroll through my music tracks. The red button would insert a marker into the timeline of the recorded data (in case I saw something interesting that I wanted to note later), and holding it down would cause the app to focus itself and bring the data view to the foreground. Holding the white button would cause the Pi to connect to my phone even when the car was off (for diagnostic purposes).

The button board before soldering and cutting to size
The button panel in its original location

A dedicated screen

One day, I was in an auto parts store and I happened to see a product intended to be used to add a heads-up display to a car. It's a cheap plastic unit that you can place a phone into, and it has a pop-up half-silvered mirror to project information directly in front of the driver. This got my attention - the mount for my phone in this car was in a spot that was too low and hard to see. I had an idea - if I could find a cheap LCD screen, I could plug it into the HDMI port on the Pi Zero, and write a program to display the data directly instead of going through my phone. I purchased one of these screens, and tried to use it with the cheap-o HUD adapter.

Unfortunately, it was a disaster. The screen had a poor vertical viewing angle, with light leakage causing another, brighter image of the screen to be visible in the windshield, making it difficult to see at night. In addition, during the day, it was extremely difficult to see at all. Rather than just give up entirely, I decided to compromise and just mount the screen directly between the steering wheel and the infotainment screen, where my phone mount had been on my old car. This way, the extra data would still be easily visible, even if I did have to glance over at it.

Now that I had an actual display, I could expand the functionality of the controls. I made the white button toggle an on-screen menu, allowing me to activate shortcuts that aren't important enough to have dedicated buttons, and pressing the red button now brings up a virtual keyboard that I can use to make a quick note. In keyboard mode, the dial selects a column on the virtual keyboard, and the white, black, dial, and blue buttons select a character from the current row. The green button accepts the current entry and closes the keyboard, while pressing the red button temporarily closes the keyboard while keeping the current text entry. I also added a beeper from an old multimeter, connected to a GPIO port, which could give me some feedback without having to look at the screen.

The improvised mount for the custom display. I re-used a plastic piece from the HUD adapter, since I had already drilled it and couldn't return it. The mount itself is a modified VoltPhone 3.

Dual dashcams

One thing I had been thinking about for a while was adding a rear-facing dashcam. This would complicate my setup, though - would the dashcams still record if I plugged them into a USB hub? Would the Macchina 5 volt output be able to power both of them? I decided to build a completely new power/interface board. Instead of hacking the USB driver kernel module, I would build a board with 3 data passthrough / power control circuits. I would plug the Pi into the B port on first one, then connect the A port to a USB hub. Then, two of the ports on the hub would be connected to the B ports on the other two, and each dashcam would be plugged into one of the A ports. That way, if I wanted to record, I can just cut power to the hub before I power on the cameras, and when I want to access the files, I just turn the hub on.

To power the dashcams, I bought a 3 amp 12v-5v converter, and connected it to a fuse tapper that I bought at an auto-parts store. I also added one more USB power-control circuit (without data) which I could use to turn the screen on and off. I added a manual override switch to this one, so I could turn the screen on to watch the Pi boot up.

Soon after, I realized that with all the extra stuff I was adding, the Pi Zero wasn't quite powerful enough. I bought a full-size Raspberry Pi Model 3 B+, and a case with a cooling fan. I also bought another dedicated 3-amp 12v-5v converter with a micro-USB plug to power it. It turns out my WiFi upload speeds were being hindered by the underpowered Pi Zero, limiting it to 2MB/s, but the Pi 3 B+ could upload at 8MB/s.

Current power control board
The power board with everything plugged in
Close-up of the Raspberry Pi 3 B+ in the case with everything plugged in

Remote control

Around June, my 3-month trial of OnStar expired. I was told that to access the Vehicle Status page on my phone, I would have to pay $20 per month for their cheapest plan. Well, I wouldn't stand for that! All I needed was a 3G/LTE modem, and a sim card for another line on my cell phone plan, which would cost... $20 per month. Well, shoot. Luckily, I found a cheaper option - Hologram Maker Edition. With as little data as I send, I pay next to nothing. I can use their API to send packets directly to the Pi, even though it does not have a public IP address. I ended up not using their standard SDK - I wrote my own monitor program to connect using PPPD, then set up the routing tables so that it doesn't attempt to send normal traffic over the very limited connection.

I had managed to sniff all of the commands sent by the OnStar module, so I could replicate all of its functionality, and more. I set it up to send a series of queries every 10 minutes while the car is off, then it sends a single binary packet to a listening server, which logs the data in a location where I can view it, and even sends me alerts when certain things happen (e.g. charging complete).

Up Next: Demonstration

Saturday, February 09, 2019

Hacking my Volt with a Raspberry Pi - Part 1

In March 2018, I bought a 2015 Chevrolet Volt. This is, by far, the best car I've ever owned. It has 38 miles of all-electric range, which is more than enough to get me to work and back every day. I rarely have to run the engine at all. In fact, when I bought it, the tank was 75% full (about 7 gallons), and that lasted until the end of January 2019 - almost 11 months.

There are a few annoying things about it, though - the lack of physical buttons for climate controls, the scattered information that you have to keep switching screens to see, the fact that OnStar Vehicle Status costs $20 per month and only gives the most basic stats... There are lots of other improvements that could be made.

So I decided to do something about it...

An extra screen gives me real-time data about battery state of charge, kilowatts going into/out of the battery (+2.4 means it's charging at 2.4 kW), kilowatts/RPM for each drive motor, engine RPM and fuel consumed (when engine is running), trip odometer, coolant temperature, battery temperature, and much more.
The brains of the project: the Raspberry Pi itself. On the left is the button panel I use to control the system. Behind the shift lever is the power control board, which not only supplies the Pi with power, it also allows the Pi to control power to other components.
The Macchina M2 that actually interfaces with the Volt's systems, plugged into the OBD2 port.

This is just the first part in a series of posts about this project. I started writing it all as one post, but it got way too long.

Up next: History of the project

Thursday, February 07, 2019

Giving this another shot

My last post on this blog was in August 2012. I stopped blogging because I figured nobody was reading it, so why should I bother writing? Well, funny thing about that... Writing doesn't have to be about other people reading it. It can just be a way for me to organize my thoughts without worrying if anyone is ever going to see it.

So, what's happened since 2012? I've come out as bisexual, for a start. I've joined the furry community, and created a fursona. I'm now driving a 2015 Chevrolet Volt - the best car I've ever owned - and have been working on a project for it involving a Raspberry Pi (which I will be blogging about soon, hopefully). I'm still living in Wichita, and I'm trying to meet new friends.

I'm going to try to post as much as I can here. If nothing else, I can document all the personal projects I've worked on over the years.

Saturday, August 04, 2012

SATA hotplug speed fix for Linux

Like many Linux users, I'm fairly paranoid about backups. I back up all of the important data on my desktop and laptop to the RAID on my main fileserver. I have several offline backup disks which I keep unplugged most of the time. Once or twice a week, I insert one into the trayless hot-swappable SATA bay on my server and run a backup to it.

Linux has always had great support for hot-swapping drives. However, there is one problem: if a new drive is inserted too quickly after the old one is removed, the error handler may limit the link speed to 1.5 Gbps:

[7270695.660961] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xe frozen
[7270695.661416] ata4: irq_stat 0x00000040, connection status changed
[7270695.662327] ata4: SError: { RecovComm PHYRdyChg CommWake DevExch }
[7270695.663518] ata4: limiting SATA link speed to 1.5 Gbps
[7270695.663524] ata4: hard resetting link
[7270705.684030] ata4: softreset failed (device not ready)
[7270705.684498] ata4: hard resetting link
[7270712.632035] ata4: SATA link up 1.5 Gbps (SStatus 123 SControl 300)
[7270712.640555] ata4.00: ATA-7: Hitachi HDS721010KLA330, GKAOAB0A, max UDMA/133
[7270712.640560] ata4.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[7270712.641499] ata4.00: configured for UDMA/133
[7270712.656024] ata4: EH complete

For the longest time, the only solution was to unplug the drive, wait 5 minutes, then plug it back in. I found the function responsible for limiting the speed (sata_down_spd_limit), but since Ubuntu kernels include libata in the base image, changing it would mean patching and recompiling the kernel every time it updates.

Today, I discovered that Linux has a feature called kretprobe which allows a module to patch a running kernel without a reboot. Using this, I created a module which patches libata to prevent it from limiting SATA link speeds after link errors occur. I've tested it on all three of my systems with no problems.

If you have problems with link speeds after hot-swapping, you can download the module from here.