Access private GKE nodes over SSH

lately, we’ve been busy setting up a new GKE cluster, and we decided to protect ourselves from any unwanted guest we determined to set them up as private nodes.

This means that these nodes don’t get a public IP assigned to them hence they aren’t accessible from the outside world without any extra work of using a VPN or bastion host.

Luckily Google Cloud has a product which helps you out called Cloud Identity-Aware Proxy which makes it very easy to still gain access to your nodes without any extra work, well almost no extra work.

So we thought we were all set to go and when the moment came to investigate our nodes we didn’t get access but got the following error:

> gcloud compute ssh gke-node-xxxx
gcloud.compute.start-iap-tunnel) Error while connecting [4003: u'failed to connect to backend’].

So after googling this specific error, I couldn’t find anything immediately, but the documentation does mention something about whitelisting a particular block of IP’s 🙈

Firewall rules that are configured to allow access from Cloud IAP’s TCP forwarding netblock, 35.235.240.0/20, on all ports of your machine. This ensures that connections are allowed from Cloud IAP’s TCP forwarding IP addresses to the TCP port of the admin service on your resource. Note that you might not need to adjust your firewall rules if the default-allow-ssh and default-allow-rdp default rules are applied to ports used for SSH and RDP.

See https://cloud.google.com/iap/docs/using-tcp-forwarding

So the final solution to get access to your private GKE nodes is whitelisting the 35.235.240.0/20 netblock.

resource "google_compute_firewall" "allow-iap-traffic" {
  allow {
    ports    = [22]
    protocol = "tcp"
  }

  description = "Allows TCP connections from IAP to any instance on the network using port 22."
  direction   = "INGRESS"
  disabled    = false
  name        = "allow-iap-traffic"
  network     = google_compute_network.${network_name}.self_link
  priority    = 1000
  project     = 'project_id'
  source_ranges = [
    // Since we have private IP's for our GKE nodes we need to use Google IAP to access them
    // We need to allow this specific range to have access
    "35.235.240.0/20" // Cloud IAP's TCP netblock see https://cloud.google.com/iap/docs/using-tcp-forwarding
  ]
}

That’s all folks!