Google Autopilot and Gitlab failed builds

Problem

You want to use Google’s Autopilot for your gitlab runners, but your job/builds fail because of low resources (ie ephemeral storage).

Solution

You can use a limit range to increase the limits for ephemeral storage or/and memory that will make Google’s autopilot to use them and scale them appropriately.

Create a limit range file like:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-ephemeral-storage
spec:
  limits:
  - default:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    defaultRequest:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    type: Container

And then apply it to your cluster

kubectl -n namespace apply -f limit_range.yaml

googlecloudsdk.calliope.exceptions.HttpException: ResponseError: code=400, message=Autopilot clusters must be regional clusters.

Problem

Trying to create an auto-cluster either using terraform or gcloud cli, and specifying the region name returns the error that ‘Autopilot clusters must be regional clusters.

So with gcloud this is the command and output

kosmas: (master %%)$ gcloud container clusters create-auto test-cluster --region=europe-west6-b
Note: The Pod address range limits the maximum size of the cluster. Please refer to https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr to learn how to optimize IP address allocation.
ERROR: (gcloud.container.clusters.create-auto) ResponseError: code=400, message=Autopilot clusters must be regional clusters.

Solution

Using the actual region name (that can be taken from the list of available zones/regions)

gcloud compute zones list

NAME                       REGION                   STATUS  NEXT_MAINTENANCE  TURNDOWN_DATE
us-east1-b                 us-east1                 UP
us-east1-c                 us-east1                 UP
...
europe-west6-b             europe-west6             UP
...

And using the correct region name (without the b)

gcloud container clusters create-auto test-cluster --region=europe-west6 --verbosity debug

...
Created [https://container.googleapis.com/v1/projects/gitlab-runner-343714/zones/europe-west6/clusters/test-cluster].
...
NAME          LOCATION      MASTER_VERSION   MASTER_IP     MACHINE_TYPE  NODE_VERSION     NUM_NODES  STATUS
test-cluster  europe-west6  1.21.6-gke.1503  xxx.xxx.xxx.xxx  e2-medium     1.21.6-gke.1503  3          RUNNING

Required ‘compute.networks.create’ permission Google/Terraform Cloud

Problem

You want to create a GKE cluster in Google using Terraform Cloud and the instructions provided https://registry.terraform.io/providers/hashicorp/google/4.3.0/docs/guides/getting_started#using-terraform-cloud-as-the-backend

You are creating a service account in Google Service Accounts and you have set up the environment variable GOOGLE_CREDENTIALS (after removing the new lines from the json file with tr -d '\n' < original_google_file.json > no_newlines_google_file.json.

But when trying to create the GKE cluster you get the following error in Terraform

Required 'compute.networks.create' permission

Solution

The default permissions when creating the Service Account are not enough.

Add the Editor role to your service account, by going to IAM, Permissions for Project, as it does not seem possible to add this permission by editing the Service Account.