2

I'm running a private GKE cluster and want to add a NAT Gateway so I can create firewall rules from requests coming from the cluster (by default, the public IP's for all nodes are ephemeral, thus hard to keep track of).

I'm trying to follow the instructions described here, and here (to create a highly available NAT, a variation of the first link).

I'm able to install terraform, create terraform.tfvars on my working directory and run terraform init. When I run terraform plan, I receive a lot of errors like: 

Error: Unsupported attribute

  on ../../main.tf line 51, in module "nat-gateway":
  51:   module_enabled        = "${var.module_enabled}"

An attribute named "module_enabled" is not expected here.


Error: Unsupported attribute

  on ../../main.tf line 52, in module "nat-gateway":
  52:   project               = "${var.project}"

An attribute named "project" is not expected here.


Error: Unsupported attribute

  on ../../main.tf line 53, in module "nat-gateway":
  53:   region                = "${var.region}"

An attribute named "region" is not expected here.


Error: Unsupported attribute

  on ../../main.tf line 54, in module "nat-gateway":
  54:   zone                  = "${local.zone}"

An attribute named "zone" is not expected here.
...

I get an error for almost every input in this module (this is what ../../main.tf points to). 

module "nat-gateway" {
  source                = "GoogleCloudPlatform/managed-instance-group/google"
  version               = "1.1.14"
  module_enabled        = "${var.module_enabled}"
  project               = "${var.project}"
  region                = "${var.region}"
  zone                  = "${local.zone}"
  network               = "${var.network}"
  subnetwork            = "${var.subnetwork}"
  target_tags           = ["${local.instance_tags}"]
  instance_labels       = "${var.instance_labels}"
  service_account_email = "${var.service_account_email}"
  machine_type          = "${var.machine_type}"
  name                  = "${local.name}"
  compute_image         = "${var.compute_image}"
  size                  = 1
  network_ip            = "${var.ip}"
  can_ip_forward        = "true"
  service_port          = "80"
  service_port_name     = "http"
  startup_script        = "${data.template_file.nat-startup-script.rendered}"
  wait_for_instances    = true
  metadata              = "${var.metadata}"
  ssh_fw_rule           = "${var.ssh_fw_rule}"
  ssh_source_ranges     = "${var.ssh_source_ranges}"
  http_health_check     = "${var.autohealing_enabled}"

  update_strategy = "ROLLING_UPDATE"

  rolling_update_policy = [
    {
      type                  = "PROACTIVE"
      minimal_action        = "REPLACE"
      max_surge_fixed       = 0
      max_unavailable_fixed = 1
      min_ready_sec         = 30
    },
  ]

  access_config = [
    {
      nat_ip = "${element(concat(google_compute_address.default.*.address, data.google_compute_address.default.*.address, list("")), 0)}"
    },
  ]
}

I also noticed that the errors I get are the fields that are declared here but not here, so it looks like some kind of version miss-match.

I tried to change this line as follows, to get the latest from github instead of terraform registry, but that did't solve the issue.

// before:
module "nat-gateway" {
  source                = "GoogleCloudPlatform/managed-instance-group/google"
  version               = "1.1.14"

// after:
module "nat-gateway" {
  source                = "github.com/GoogleCloudPlatform/terraform-google-managed-instance-group"

My working directory is examples/gke-ha-nat-gateway/, in case that matters. The whole environment is the same as freshly cloned from github, with the exception of terraform.tfvars.

Unfortunately I'm don't have any experience with Terraform, so I don't really know how to debug this errors.

Thank you for your help!

ButterDog
  • 121
  • 3
  • Have you declared each variable with a `variable "module_enabled" {}`, `variable "project" {}` block? – bodgit Nov 12 '18 at 17:39
  • If you are running on GCP you should consider a [managed NAT](https://cloud.google.com/nat) configuration rather than creating your own NAT gateway. – kasperd Nov 12 '18 at 19:21
  • @bodgit: Yes, they are provided in the example source: https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway/blob/master/variables.tf – ButterDog Nov 13 '18 at 08:44
  • @kasperd ideally yes. My issue at the moment is that it's still in Beta, and I only got it to work for having my nodes reach the internet. You can attach a static IP to the NAT, but when any of my nodes connect to a site, the source IP appears to be a random IPv6, instead of the static IPv4 I create. This is a problem since I want to add these IPs to a firewall, which can't do if every node is different. If this part was working, that would be the preferred solution. I already have a support ticket with Google for that one. – ButterDog Nov 13 '18 at 08:48
  • @Xocoatzin Which sites are reporting an IPv6 source address? And is that a routeable IPv6 address or an RFC 4193 address? – kasperd Nov 13 '18 at 09:08
  • 1
    @kasperd Good question. The IP's look like this `fda3:e722:ac3:10:30:dXXX:aXX:YYY` (with `YYY` being different depending on the node used). I have a web server (running on app engine) I setup to test the firewall, so I get access to the logs. The app is pretty much 5 lines of python logging each request with their remote address, see [this SO answer for reference](https://stackoverflow.com/a/4231085/575085) – ButterDog Nov 13 '18 at 09:19
  • 1
    @Xocoatzin The only public mention I could find of that IP range is this https://qiita.com/web_se/items/c4a7dbbdbdb1f7eb1f0a. I don't understand the language, but I see that it mentions Private Google Access. That's a feature which allows your VMs to use their internal IP addresses to reach Google services. You can read more details about that here: https://cloud.google.com/vpc/docs/private-access-options. That feature is always turned on when using the managed NAT. I don't know if that IP showing up in app engine is intentional or a bug. But above link suggests that was always the case. – kasperd Nov 13 '18 at 10:16
  • 1
    @Xocoatzin In short you'll only see `fda3:e722:ac3::/48` addresses when your VMs are accessing services hosted on Google. If you access any service outside of Google it will see your NAT IP addresses. Where is the firewall you are trying to configure hosted? – kasperd Nov 13 '18 at 10:17
  • @kasperd That's a pretty interesting observation. My application is hosted in App Engine, so it might be connecting internally using the google private access. The firewall is part of App Engine. I will set up a different test to verify if that is the case when reaching a site not hosted on Google. How did you come up with the `/48`? I haven't been able to find anything in any documentation, but it seems to make sense! – ButterDog Nov 13 '18 at 10:36
  • 1
    @Xocoatzin I know [RFC 4193](https://tools.ietf.org/html/rfc4193). And two things to notice about that RFC is that the addresses start with `fd` followed by a 40 bit random value. And those addresses cannot be used to communicate across the internet. So Google has generated a random 40 bit string to produce the prefix `fda3:e722:ac3::/48` and is free to use the last 80 bits as they see fit. With that knowledge and the example IP you posted it was easy to come up with a search leading to the above URL. The meaning of the `10:30:dXXX:aXX:YYY` part is something Google hasn't published. – kasperd Nov 13 '18 at 10:43
  • Thinking about that, it makes a lot of sense. I tried `curl 'https://api.ipify.org?format=json'` and I read my reserved static IPv4 instead of the internal one. I added `fda3:e722:ac3::/48` to my firewall rules and it's working now. Since using Cloud NAT is preferred over managing my own, I think the original question is irrelevant now. If you are interested in posting this as an answer I'll make sure to get it accepted and rephrase the question to add the Cloud NAT option. Thanks again for your help, been struggling with this for days! – ButterDog Nov 13 '18 at 11:05
  • @Xocoatzin Adding `fda3:e722:ac3::/48` is going to allow all Google Compute Engine customers to access your service. You might to want to specify a more specific prefix. From your comment above it sounds like `fda3:e722:ac3:10:30:dXXX:aXX:0/112` would work for you. Also allow the static IPv4 address in case Google changes it to make the IPv4 address visible rather than the RFC 4193 address. – kasperd Nov 13 '18 at 12:35
  • from testing in different environments it looks like /96 is the safest bet to cover your project – Patrick W Nov 15 '18 at 17:43

1 Answers1

1

Since you are running on GCP, you may consider Cloud NAT configuration rather than creating your own NAT gateway.

It appears that 'fda3:e722:ac3::/48' addresses are used when VMs are accessing services hosted on Google. If you access any service outside of Google, it will see your NAT IP addresses. Regarding the issue of the source IP being a random IPv6 (when nodes connect to a site) instead of the static IPv4 created, you can add ‘fda3:e722:ac3:10:30:dXXX:aXX:0/96’ to the firewall rules.

Rahi
  • 196
  • 6