We are transitioning our traditional servers to a Kubernetes cluster, so for our north<>south traffic, we are using Project Contour to configure our routing inside the Kubernetes cluster. But our traditional server setup includes HAProxy to route the traffic. Since we haven’t migrated all our services to the Kubernetes cluster, we still need it to steer traffic to the new Kubernetes cluster until we finished the entire migration.
Traditionally we didn’t use https for our backend servers in HAProxy, so when we switched over the backends to point to our Project Contour Load Balancer, we got stuck in a redirect loop when using plain HTTP. Because by default, Project Contour redirects plain HTTP traffic, using a 301, to HTTPS automatically.
There were to approaches we consider
- Disable the redirect in Project Contour, using
- Configure the HAProxy backends to use HTTPS instead
We’ve chosen to go with the latter option since it’s better not to expose any plain HTTP traffic now anymore nowadays. And it prepares us better for the future where we will remove HAproxy from the setup.
Because we couldn’t use domain names to point to our Project Contour Loadbalancer and use static IP’s instead, we had to dive deep into HAProxy documentation to find a couple of solutions for the problems we encounter.
First, after enabling the
ssl directive for our backend server, we received errors from our health checks, hence the
backend couldn’t serve any traffic.
Layer6 invalid response, info: "Connection error during SSL handshake (Connection reset by peer)"
After banging my head against the wall for quite some time, I found the
check-sni option, which HAProxy introduced
in version 1.8. This option allowed me to configure an SNI domain used by the health checks 🎉.
check-sni discovery is a step in the right direction. However, we still encounter issues during the health
check, though other errors then we originally received:
reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>"
We got a step further along, and this time around, we also saw request logs in our envoy pods from Project Contour.
Host header to the
http-check health check, using the
http-check send directive, solved these errors.
http-check send hdr Host example.domain.com
At this point, we had a healthy backend in HAProxy. Unfortunately, when we try to reach our website, we encounter the
Layer6 invalid response errors the health check encounter earlier. To fix these errors, we use the
to configure the domain of our https certificate for our regular traffic. And to be safe, we also configure a
header to make sure we pass along the right domain to our Project Contour Load Balancer.
Eventually, we ended up with the following HAProxy configuration for our backend:
backend example http-request set-header Connection keep-alive http-request set-header Host example.domain.com http-request set-header X-Forwarded-Proto https option httpchk GET / http-check send hdr Host example.domain.com http-check expect status 200 server contour 10.0.0.10:443 ssl verify none check-sni example.domain.com sni str(example.domain.com) check