Google Autopilot and Gitlab failed builds

Problem

You want to use Google’s Autopilot for your gitlab runners, but your job/builds fail because of low resources (ie ephemeral storage).

Solution

You can use a limit range to increase the limits for ephemeral storage or/and memory that will make Google’s autopilot to use them and scale them appropriately.

Create a limit range file like:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-ephemeral-storage
spec:
  limits:
  - default:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    defaultRequest:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    type: Container

And then apply it to your cluster

kubectl -n namespace apply -f limit_range.yaml

WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrating-helper-image-to-registrygitlabcom

To change this in your gitlab-runner configuration, depends on the version of your gitlab-runner as described in https://docs.gitlab.com/runner/configuration/feature-flags.html

If you version is 13.9.0 (gitlab-runner –version), then add the following in ~/.gitlab-runner/config.toml

[[runners]]
  name = "gitlab-runner-name"
  request_concurrency = 1
  url = "https://git.url.test/"
  token = "xckaxxxxxxxx"
  executor = "kubernetes"
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]

Setup your own gitlab runner

To add your own gitlab runner, follow the steps below

  • Create your runner, in EC2, Google, or any other cloud provider and have ssh access to it.
  • Login to your runner from the previous step and install the gitlab-runner (there are various methods described here: https://docs.gitlab.com/runner/install/. We install using Gitlab’s repositories)
  • Register the gitlab-runner by following the instructions here: https://docs.gitlab.com/runner/register/index.html . You would need to get the token for each project you want to register your runner with (Gitlab -> Settings -> CI/CD -> Runners -> Expand). You can use different executors for your running during the registration such as docker, shell etc.
  • Go to your Gitlab project and find the runner you register above and enable it for the project. You should also add the tags: name_of_your_runner in your .gitlab-ci.yml file.