Configure SNI for HAProxy Backends

We are transitioning our traditional servers to a Kubernetes cluster, so for our north<>south traffic, we are using Project Contour to configure our routing inside the Kubernetes cluster. But our traditional server setup includes HAProxy to route the traffic. Since we haven’t migrated all our services to the Kubernetes cluster, we still need it to steer traffic to the new Kubernetes cluster until we finished the entire migration.

The “problem”

Traditionally we didn’t use https for our backend servers in HAProxy, so when we switched over the backends to point to our Project Contour Load Balancer, we got stuck in a redirect loop when using plain HTTP. Because by default, Project Contour redirects plain HTTP traffic, using a 301, to HTTPS automatically.

There were to approaches we consider

1. Disable the redirect in Project Contour, using permitInsecure
2. Configure the HAProxy backends to use HTTPS instead

We’ve chosen to go with the latter option since it’s better not to expose any plain HTTP traffic now anymore nowadays. And it prepares us better for the future where we will remove HAproxy from the setup.

The “solution(s)”

Because we couldn’t use domain names to point to our Project Contour Loadbalancer and use static IP’s instead, we had to dive deep into HAProxy documentation to find a couple of solutions for the problems we encounter.

First, after enabling the ssl directive for our backend server, we received errors from our health checks, hence the backend couldn’t serve any traffic.

Layer6 invalid response, info: "Connection error during SSL handshake (Connection reset by peer)"

After banging my head against the wall for quite some time, I found the check-sni option, which HAProxy introduced in version 1.8. This option allowed me to configure an SNI domain used by the health checks 🎉.

The check-sni discovery is a step in the right direction. However, we still encounter issues during the health check, though other errors then we originally received:

reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>"

We got a step further along, and this time around, we also saw request logs in our envoy pods from Project Contour. Adding a Host header to the http-check health check, using the http-check send directive, solved these errors.

http-check send hdr Host example.domain.com

At this point, we had a healthy backend in HAProxy. Unfortunately, when we try to reach our website, we encounter the same Layer6 invalid response errors the health check encounter earlier. To fix these errors, we use the sni option to configure the domain of our https certificate for our regular traffic. And to be safe, we also configure a Host header to make sure we pass along the right domain to our Project Contour Load Balancer.

Eventually, we ended up with the following HAProxy configuration for our backend:

backend example
	http-request set-header Connection keep-alive
	http-request set-header Host example.domain.com
	http-request set-header X-Forwarded-Proto https

	option httpchk GET /
	http-check send hdr Host example.domain.com
	http-check expect status 200

	server contour 10.0.0.10:443 ssl verify none check-sni example.domain.com sni str(example.domain.com) check