Cfrgtkky

I need my Composer environment to reach some on-prem resources over the VPN tunnel established between GCP and my network. I have my custom VPC network setup and running with a series of /20 subnets.

The problem is I can't spin up a new Composer instance using any of those subnets. The creation job spins for a long, long time with no apparent error, but the K8S cluster creation shows me what's happening:

Apparently it requires a /14 subnet, but I haven't found anything mentioning that in the documenta available.

I just want to make sure this is in fact a requirement before I go ahead and plan to reconfigure all our VPC setup to accommodate such thing.

The required size depends on your cluster size, and needs to account for any growth in the number of nodes, pods, or services run on the cluster.

Some minimum CIDR ranges are documented in the Cloud Composer Shared VPC guide. For a 3-node cluster, the minimum ranges are:

• Pods composer-pods: /22
  


• Services composer-services: /24
  


• Primary address range: /29
  

Cloud Composer uses Alias IPs, whose docs provide more guidance on sizing:

• Nodes: Node IPs are taken from the subnetwork associated with the cluster. ...


• Pods: Each node in the cluster is currently allocated a /24 (2(32-24) = 28 = 256) block of Pod IPs. ...


• Services: Every cluster needs to reserve a range of IPs for Kubernetes Service cluster IPs. ...

The error you're seeing where the underlying GKE cluster fails to create due to being unable to allocate a /14 address range happens because, in the current state of Composer, all environments are provisioned with non-VPC-native GKE clusters. This means that GKE will attempt to allocate a contiguous /14 address range in the provided subnet. This works fine if you don't have much else competing for addresses in the subnet but not so well otherwise.

From the looks of it, you're not creating a Composer environment using a Shared VPC network. Composer environments connected to Shared VPC networks use so-called VPC-Native clusters in which it's possible to specify smaller secondary address ranges for pods and services. As Tim Swast pointed out, the minimum secondary address range sizes that you need to allocate are /22 for pods and /24 for services, but you should make the pods range large enough to accommodate any future node count upscaling.

To summarize, if you are unable to setup Shared VPC, you'll need to clear out some address space in your network. However, it seems like if you've gone through the trouble of setting up Cloud VPN that it might not be a huge hassle for you to setup another project so that you can use a Shared VPC setup which will allow you to configure smaller address ranges.

EDIT:
A bit more detailed info: For non-VPC Native clusters, GKE searches the routing table for an unused contiguous /14 subnet under 10.0.0.0/8. When it cannot find one, it reports this error. You need to work with your network admin to shuffle allocated address blocks around to free up a /14.