Openpilot is an opensource advanced driver assistance system (ADAS) that can perform lane keeping, adaptive cruise control, etc. Native openpilot only works on very particular pieces of hardware like LeEco pro 3, comma 3, etc.
Flowpilot is a project that aims to run openpilot on as many devices as possible (android phones, desktop computers, jetsons and other SBCs) while aiming to modify the existing openpilot codebase as less as possible.
The aim of this blog post is to convey what changes does flowpilot make to openpilot's codebase and how it is magically able to support thousands variations of android phones and other platforms.
How android phones are different from standard linux devices ?
There are too many differences to go into detail. But majorly, android runs bionic rather than glibc. Bionic is an implementation of the standard C library, developed by Google for its Android operating system. It differs from the GNU C Library (glibc) (which runs on standard linux distros) in being designed for devices with less memory and processor power than a typical Linux system. Nor does android include all of the GNU libraries you’d find on a typical Linux distribution. It also doesn’t include an X server like Xorg, so you can’t run standard graphical Linux applications.
This means that all the libraries that are compiled for standard linux cannot be used with android unless you cross compile them with the android NDK to use bionic. This already seems to be to much work and there is no gaurantee at all that all the libraries that openpilot uses can be compiled using bionic for the arm architecture. The problem is not the arm architecture, it is widely supported and almost all the libraries that are available on standard x86 linux distro are available for arm too. The main problem is the bionic. Almost none of the libraries come pre compilied with bionic.
So instead of cross compiling the libraries for android, flowpilot bypasses this by using chroot or proot. think of chroot as a kind of a 'naive virtual machine'. With this you can emulate running a standard linux distribution.
Now you can compile openpilot fully in the chroot. But wait, what about display/ui ? How would you access android cameras, gpu, other hardware ?
The Flowpilot App
Each android phone has different hardware component vendors. And each of this component mostly has a proprietary driver associated with it. To talk to any of the hardware, you need the drivers. And android phones have all sorts of combinations of hardware components.
Luckily you dont need to mess at driver level for any of this since the AOSP requires the phone manufacturers to expose working APIs to these hardware components, mostly via java. And this is what how it should be. Grabbing imae bytes from a camera should be as easy as calling a camera.read() function. This is possible via the java api that andoid exposes.
This is where the real magic comes in. We segregate the openpilot project into two categries, the one that talks to hardware like gpu, camera, screen, gyro, accelerometers, etc and one that dosent. The code that dosent interacts with any of this hardware is not modififed and is left as it is, but the other part requires to be re-written to use the android APIs.
Mostly, the neural network runners (requires access to gpu), camerad (access to camera), ui (access to screen), sensord (access to sensors) have been re-written. This approach has worked surprisingly well in terms of being generalized, though we sometimes do face vendor specific issues which become edge cases.
Forks
Most of the parts that forks focus upon tend to be things that are untouched in flowpilot from the openpilot codebase, so its just a matter of copy pasting files and code. The only thing that forks maintainers will have to re-write is the ui specific changes.
We use libgdx as our ui framework that is cross platform and same code runs on desktop as well as android phones. Messing with UI with user friendly java APIs is much more easier and effiecient for the developers. Using a dektop PC, to develop, run and test code quickly and fast and then generating an APK that runs on android out-of-the box is a big privillege. Try it.