Saturday, March 26, 2016

Troubleshooting a crashing bitlbee container in Docker

I've been experimenting with running containers on different flavors of Linux.  I've been noticing that what builds/runs on Hypriot is a bit buggy on Ubuntu or Debian (or just won't build).  Same problem for the other direction.

One of things I've been doing to teach myself Docker is to build a non-purple bitlbee container, with all of the available plugins enabled.  Wanting the bitlbee container to start on boot, I added Monit to the mix (wanting the additional features that Monit has when compared to Docker's restart capabilities).  On Hypriot (running on a RPi2), the kit runs without issue.

The same build on a "normal" Docker server periodically disconnects the IRC client.  In looking at the container's running processes, it shows a second instance of bitlbee running.  This shouldn't happen when using bitlbee's "-D" switch, which is what I'd employed in the container's start script.

Bitlbee's "-D" switch is supposed to cause bitlbee to service all clients via the same instance (i.e., one running binary).  Without looking at the code, I'm guessing that there's some sort of HUP signal or function that interrupts operation of PID 1 just long enough to annoy Docker, restart bitlbee with a second instance, and cause the problem I'm seeing.

I proceeded to play with each of the three programs' settings and think (hoping!) I may have a solution.  It's non-intuitive, but replacing the "-D" switch with the "-F" appears to fix it.

Bitlbee's "-F" switch intentionally creates a new instance of bitlbee for each new client that connects.  What's not documented is that it appears that the instance at PID 1 handles the initial connection and, when the client authenticates, the client is passed to a newly spawned bitlbee process. This appears to leave PID 1 to continue listening for new clients.

Fingers crossed.  Time will tell...