Unable to update Node Groups in place with cluster placement group strategy without EFA

[Josephuss]

Unable to update Node Groups in place with cluster placement group strategy without EFA

[bryantbiggs]

cc @james-masson ref #2959

[james-masson]

@bryantbiggs

1. This is a general problem we've seen with using cluster placement groups (EFA potentially included)
2. This could actually be seen as a more general enhancement where you want a different AZ layout on a single node group.

Consider an EKS cluster deployed across 3 availability zones.

A nodegroup that makes use of a "cluster" placement group will only be able to deploy into a single one of these AZs. That's the point of the "cluster" placement group, to put the instances into the same physical rack.

It turns out that on initial deployment, this configuration is not a problem. The nodegroup with cluster placement successfully gets deployed into a single AZ of the 3 subnets configured with no errors. Arguably this itself is a bug.

However, when it comes time to replace or upgrade the nodegroup, you get the error listed by my colleague @Josephuss
For some reason the A/Z or subnet deployment problem only happens on replacement.

This PR tries to fix this issue by adding the concept of a AZ filter into the nodegroups, to allow nodegroups to be deployed into a subset of the subnets that the rest of the cluster is configured with.

We're aware that this can be currently worked around by overriding subnet ids.

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/node_groups.tf#L308

eg.

  eks_managed_node_groups = {
    my_custom_nodegroup = {
      name            = "customer1"
      subnet_ids = [ "subnet-abc12345" ]
    }
  }

But this is quite fragile, because:

1. We frequently do full create/destroy cycles - so these references change.
2. We expose nodegroup config to our internal customers as a variable - so they can declare their own requirements. This precludes allowing direct terraform references like module.vpc.my_subnets_in_az_x

Hence seeing the general value in an interface like this - it's more practical, understandable and portable, and it has value outside of placement groups too.

  eks_managed_node_groups = {
    my_custom_nodegroup = {
      name            = "customer1"
      subnet_az_filter = "eu-west-1a"
    }
  }