How I added Automatic License Plate Reading to my home surveillance system.

I have a Ubiquiti UniFi G3 surveillance camera watching my property. It streams video 24 hours a day into a Network Video Recorder running on a Linux server in my garage. The NVR is configured to email me when it sees motion occur in a zone within the camera’s view. I can follow links from the email notifications to view recorded video or a livestream from the camera when I’m not at home. This also works reasonably well over LTE when I don’t have wifi.

OpenALPR is an open source Automatic License Plate Reader. It processes images or video and attempts to read license plates. If found it turns them into a textual representation suitable for text matching or recording in a database.

Until now I had no automated way of doing this. If I wanted to see the license plate on a car that pulled into my driveway the easiest and quickest way to do that would be to watch the video and pause it to read the plate. If I could automatically run each of these videos through ALPR I knew it could send me an email with the plate number long before I could take the time to review the video myself. Not only did this sound useful to me, but I thought it would be an interesting technical problem to solve.

As with any good software development effort it helps to start with a goal: Automatically process new camera video to extract license plate numbers and email them to me.

The first order of business was to just make it work. The second task would then be to make it faster.

Resources

I have two servers. Each of them contain 24 2.27Ghz Intel Xeon CPUs. On one of them I have Ubuntu 20.04. On the other I have Ubuntu 18.04 because, as of this writing, OpenALPR is not yet supported on 20.04. I am using both of these servers to implement this ALPR project.

Reverse Engineering

I’d read some discussion on the Ubiquiti forums about using their API to download videos from the NVR. Unfortunately, that API is undocumented and unsupported for 3rd party use. I knew that the NVR uses MongoDB for video storage so I decided to see if I could get the videos from that instead. I found that the videos are broken into ~750kB pieces before being stored on disk. There is a ‘meta’ directory which contains information about each video stored in a JSON file. Using the inotifywait utility in Linux my script waits until it sees a new metadata file added and then finds the relevant video files. Once the video files have been found ffmpeg is used to glue them together into a single video.

Enhancing the commercial software’s capabilities

At this point I realized I could do something that Ubiquiti’s software doesn’t. Each email notification that the NVR sends contains a frame grab from the video. This is useful, but if I want to view the video I have to follow the link to login to the NVR. I thought it would be much quicker if the email contained the whole video. I use ffmpeg to turn the video into an animated GIF, which plays on my iPhone as well as Thunderbird on macOS and Linux.

Since OpenALPR only runs on the second server my script copies it onto a Samba share there. On the second server a script is using inotifywait to wait for new video files to arrive.

OpenALPR reports its findings with a confidence score. I have the script email me results when that score is greater than 80% along with the frame with the highest confidence score.

It works!

In the end I receive an email with the plate number along with the frame grab in which the plate was read.

Making it faster

“…premature optimization is the root of all evil…”

- Donald Knuth

Now that I’ve accomplished the goal above it’s time to optimize this. OpenALPR is single threaded so it only uses one CPU core to process the entire video. Since my server has 24 CPUs I thought that I could use ffmpeg to split the video into 24 equally sized pieces, each being no more than a couple seconds long. It turns out that someone else had this need and wrote a Python script called ffmpeg-split.py. You might expect this to be roughly 24x as fast as running it singlethreaded, but in fact it’s much faster than that. This is because we can send our email with a plate match before a segment of video is done being processed by alpr. It writes output in JSON for each frame as they’re being processed so if it finds a license plate the script can immediately send that email. In my example video with the Amazon van it would take a single alpr process a few minutes to reach the frame in the video with the readable plate. Chopping the video into 24 pieces first lets one of the alpr processes find it within 5 seconds.