Today I learned that, under certain circumstances, Docker and an IPsec VPN can conspire to make your life as a developer miserable, by eating outgoing HTTPS connections started from inside a container.
The first symptom that something is amiss is usually being unable to go past the “TLS Client Hello”
message during the handshake process, or having the connection stall shortly after that. For
curl from inside a container would just hang, even though it would work just fine
on the host machine itself.
The scenario is the following: I have a standard Ubuntu 16.04 machine with Docker and other tools coming straight from the official repository, quite boring. An L2TP over IPsec VPN connects me to the remote site with a split-tunneling configuration.
Said VPN is configured client-side with StrongSwan and
xl2tpd, two of the most evil pieces of
software. Especially the latter, which will often crash unless planets are aligned correctly as the
author wanted. At the other end of the VPN is a Meraki box that shuts itself down if you just so
happen to sneeze around it.
All network interfaces have an MTU of 1500, except for the L2TP tunnel that sits around 1400 since
x2ltpd/pppd duo configures the
ppp0 interface like that, for whatever reason.
Here’s what an imaginary packet would encounter if it had to travel from inside a container to a machine at the other end of the VPN tunnel (in reality it’s more complicated than that so, please, bear with me):
It seems that there are several issues with the way Docker does its networking on Linux and the way Linux itself handles bridge interfaces.
It appears that the issue stems from Docker’s use a bridge interface and the fact that Linux won’t generate the “Fragmentation Needed” ICMP message that would allow for Path MTU Discovery (PMTUD) to work when IP packets have the “Don’t Fragment” bit set (which should be typical for TCP streams). Now, I’m no network engineer so take my layman’s explanation with a grain of salt.
In my case the fix was simple: start the Docker daemon passing the
--mtu=1400 parameter. On Ubuntu
I only had to edit the value of the
DOCKER_OPTS variable present in
systemctl restart docker.