For anyone who reads my posts regularly or follows me on twitter, you’ll be aware I’ve worked a lot with Raspberry Pi’s in a production capacity. The devices I’ve helped work with have ended up in shops and restaurants all around the world.
Recently I came across some security issues with debian 7 – Wheezy, oldstable. Due to the nature of where these Raspberry Pi’s are being deployed they have to comply with minimum requirements for PCI, which essentially means having specific versions of certain packages installed and certain configurations must be met for those packages.
The problem originally came about last February, when the minimum version for both openssh and openssl was increased beyond the version that was available for Wheezy. The devices being used were Raspberry Pi 2’s running Raspbian on Wheezy, this was decided because at the time of original inception Raspbian on Jessie was not available. There were also some concerns about the Linux kernel debian 8 – Jessie, stable – was built on, and whether or not certain other packages were going to work correctly, i.e. the libapache2-mod-mono for apache2.
Previously we had installed the latest openssh from the Jessie repository, which seemed to cause no ill-effects and satisfied the PCI scan for a further six months: the version installed was 6.7. A similar fix was applied for
Some months later more vulnerabilities were found in openssh and apache2. This time though upgrading apache2 to the latest version available for Jessie did not work, furthermore the update for openssh was only available for debian 9 – Stretch, testing.
The workaround for the vulnerabilities in apache2 was to permanently disable the apache2 daemon just before the device entered the final production environment, as the web portal was only used for initial configuration it didn’t need to be on after setup. This worked fine in the short term, but there are only so many workarounds you can put in place before it gets too much for an end-user or even the developer.
Debian 8 – Jessie, Stable
An obvious solution here was to upgrade to Jessie, move forward from Wheezy and embrace the changing Linux landscape with the movement that is systemd. This caused a number of concerns, some mentioned earlier about the Linux kernel version, but mostly because of the issues around rewriting the scripts around the daemons being used.
These devices run a set of programs that were originally written for Windows CE, the details of which can be found starting here. These devices are running Windows services as console applications via mono, which have required a number of interesting resolutions for prolonged use in production environments. Most of these resolutions reside around the daemon scripts, which moving from Wheezy to Jessie would require a rewrite.
This an automatic way to convert a sysvinit script to systemd, however because of the complicated nature of these scripts this failed. There are however lots of people writing about how to convert them, so if this is a problem you’re facing there might be help yet: just search for convert sysvinit script to systemd in Google.
Devuan 1 – Jessie, Beta
Another solution, albeit quite out there, is to go sideways, or fork if you will, and upgrade to Devuan instead. There are lots of article out there, as well as the official Devuan website, which explains what this fork of Debian is all about, however to summarise it’s a systemd free fork of Debian.
When tested we were able to install everything required for the device successfully, including the most recent packages for openssl, openssh and apache2 without having to reconfigure the apt repository lists and creating pinnings for them. This did not fix the PCI issue with openssh, however this could be fixed by either installing it from a Testing repository, or waiting until it moves into Stable.
Strangely however this fixed an unforeseen pain point when installing the apache2 website. Because the devices are Raspberry Pi 2’s, previously running Raspbian Wheezy, apache2 requried a very specific version of libapache2-mod-mono installed for the RPi – a package that would only be found on an obscure FTP server, then rehosted elsewhere for ease. As you can imagine this caused a whole host of issues, however when moving to Devuan the regular libapache2-mod-mono package could be installed without issue, and from that the entire installation process went without issue.
Upgrading to Devuan was a very pleasant experience, and installing a clean version on the Raspberry Pi was simple as the images are available here. If you want to upgrade from an existing Debian installation, either Wheezy or Jessie, you can do that too. There didn’t appear to be an image available for the ODroid C2, however from experience I can say if you install Wheezy on the device you can use the previous instructions to upgrade to Devuan.
Devuan seems like a very nice operating system, and the upgrade process was surprisingly smooth with some unexpected benefits. However it’s still in beta, and as much as I want to roll Devuan out I can’t in all good conscience recommend to a business they roll out a beta operating system to hundreds of devices in production environments. Debian 8, although frustrating and concerning, appears to be the lesser of the evils.
That being said, I will continue to deploy Devuan into my own infrastructure, and any issues I find will be troubleshooted and blogged about.