Revert Google Managed Certificates on Kubernetes Ingress

Last Friday, we migrated a couple of our Kubernetes Ingresses to use Google Managed Let’s Encrypt Certificates. The managed certificates are a nice new feature offered by GCP. You can configure the certificate using a short resource definition and single annotation.

apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: test-example-com
spec:
  domains:
    - test.example.com
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: api
  annotations:
    networking.gke.io/managed-certificates: "test-example-com"
spec:
...

ps: if you are migrating to managed certificates like we do, you can keep the spec.tls.secretName together with your annotation. Keeping both configurations at the same time makes sure your Ingress is always configured with a proper certificate. Once the Managed Certificate becomes active, your Ingress will serve traffic using the new certificate, and you can remove the old TLS secret without having any downtime.

Reverting the changes 😢

Unfortunately, something didn’t go right for us, and we had to revert our changes which means configuring a TLS (spec.tls.secretName) manually.

So we removed the networking.gke.io/managed-certificates annotation and configured the spec.tls.secretName again. But we had no luck. The certificate didn’t change after we applied the update. 🤔

We noticed two additional annotations on the Ingress ingress.kubernetes.io/ssl-cert and ingress.gcp.kubernetes.io/pre-shared-cert. These annotations aren’t configured by us but are part of the GKE addon, which manages the certificates. When we reverted to manually configuring the TLS secret, these annotations weren’t removed. So we removed them manually from the Ingress, and after a few minutes, the GLBC was configured properly again, serving traffic with the correct certificate. When we had a closer look at the documentation, they mentioned it’s the responsibility of the users to create/delete them.