Downloading gitlab database backup from s3

You want to download a copy of your gitlab backup (ie when upgrading to version 7.0.0 as described in [here](https://docs.gitlab.com/charts/installation/database_upgrade.html#steps-for-upgrading-the-bundled-postgresql) ).

Login to the toolbox pod and get the credentials file for s3

cat /etc/gitlab/.s3cfg

Create a new file locally (ie ~/.s3cfg_gitlab) and then download the file locally with

s3cmd get s3://gitlab-backups/database_upgrade_6_gitlab_backup.tar  20230531_gitlab_backup.tar -c ~/.s3cfg_gitlab

Removing gitlab’s container registry ‘repository scheduled for deletion’ message

If you want to remove the ‘repository scheduled for deletion’ message from a self hosted gitlab’s container registry installation, you can do the following.

Login to the toolbox pod

kubectl --kubeconfig /path/to/gitlab/kubeconfig -n gitlab-system exec -it gitlab-toolbox-pod-name -- bash

Start the rails console

cd /srv/gitlab
bundle exec rails console

Find the repository with it’s id and then get the registry for it

project=Project.find(project_id)
registry=project.container_repositories.first

The message is displayed when the status of the container is set to ‘delete_scheduled’, so change this to be null

registry.status=''
registry.save

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Problem

You are trying to install the gitlab kubernetes agent (kas) after getting the installation instructions from the Gitlab registration, but you are getting the following error:

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Solution

This is caused by the helm version used, so upgrading the helm version works with no issues.

 helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

so trying to install gives the previous error.

Changing the helm version (ie with asdf)

asdf global helm 3.70


Checking the version

helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}

And running the helm installation again

helm upgrade --install gitlab-kas-agent gitlab/gitlab-agent \

--namespace gitlab-agent --set image.tag=v15.6.0 \

--set config.token=token_from_gitlab-registration \

--set config.kasAddress=wss://kas.domain_name.tld


Release "gitlab-kas-agent" does not exist. Installing it now.
NAME: gitlab-kas-agent
LAST DEPLOYED: Mon Dec  5 13:38:33 2022
NAMESPACE: gitlab-agent
STATUS: deployed
REVISION: 1

TEST SUITE: None

Exporting Gitlab project through the API

To export a project through Gitlab’s API, first create a Personal Access Token (PAT) and give api and repository permissions.


Export PAT as an environment variable.

Then find the project id (main page of the project – 1111).

To start an export use

curl --request POST --header "PRIVATE-TOKEN: $PAT" "https://your.gitlab.repo/api/v4/projects/1111/export" --data "upload[http_method]=PUT"

Then check the status and get the download link when it has finished

curl --header "PRIVATE-TOKEN: $PAT" "https://your.gitlab.repo/api/v4/projects/1111/export"
{"id":1111,"description":"Project description","name":"project","name_with_namespace":"MyOrg / project","path":"project","path_with_namespace":"myorg/project","created_at":"2021-11-01T10:57:23.195+02:00","export_status":"finished","_links":{"api_url":"https://your.gitlab.repo/api/v4/projects/1111/export/download","web_url":"https://your.gitlab.repo/myorg/project/download_export"}}

Request Entity Too Large when trying to import Gitlab project

Problem

You would like to import an existing Gitlab project, through an export file, to a new self hosted instance of Gitlab, but using the Web UI, even after changing the max-body-size in the ingress deployment you end up having the error message

Request Entity Too Large

Solution

There is another way to import the exported file, but is not documented anywhere as it is classed as EXPERIMENTAL from Gitlab.

You can copy the exported file to the gitlab-toolbox pod

kubectl --kubeconfig ~/.kube/gitlab_config cp local_export.tar.gz gitlab-toolbox-xxx-xxx:/tmp/

You can then login to the gitlab-toolbox pod

kubectl --kubeconfig ~/.kube/gitlab_config -n gitlab-system exec -it gitlab-toolbox-xxx-xxx -- bash

get to directory with the application

cd srv/gitlab

and finally use the rake task gitlab:import_export:import to import your project

git@gitlab-toolbox-xxx-xxx:/srv/gitlab$ bundle exec rake gitlab:import_export:import[your_new_gitlab_username,namespace_path,project_path,/tmp/2022-06-14_14-53-007_export.tar.gz]

Google Autopilot and Gitlab failed builds

Problem

You want to use Google’s Autopilot for your gitlab runners, but your job/builds fail because of low resources (ie ephemeral storage).

Solution

You can use a limit range to increase the limits for ephemeral storage or/and memory that will make Google’s autopilot to use them and scale them appropriately.

Create a limit range file like:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-ephemeral-storage
spec:
  limits:
  - default:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    defaultRequest:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    type: Container

And then apply it to your cluster

kubectl -n namespace apply -f limit_range.yaml

WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrating-helper-image-to-registrygitlabcom

To change this in your gitlab-runner configuration, depends on the version of your gitlab-runner as described in https://docs.gitlab.com/runner/configuration/feature-flags.html

If you version is 13.9.0 (gitlab-runner –version), then add the following in ~/.gitlab-runner/config.toml

[[runners]]
  name = "gitlab-runner-name"
  request_concurrency = 1
  url = "https://git.url.test/"
  token = "xckaxxxxxxxx"
  executor = "kubernetes"
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]

Setup your own gitlab runner

To add your own gitlab runner, follow the steps below

  • Create your runner, in EC2, Google, or any other cloud provider and have ssh access to it.
  • Login to your runner from the previous step and install the gitlab-runner (there are various methods described here: https://docs.gitlab.com/runner/install/. We install using Gitlab’s repositories)
  • Register the gitlab-runner by following the instructions here: https://docs.gitlab.com/runner/register/index.html . You would need to get the token for each project you want to register your runner with (Gitlab -> Settings -> CI/CD -> Runners -> Expand). You can use different executors for your running during the registration such as docker, shell etc.
  • Go to your Gitlab project and find the runner you register above and enable it for the project. You should also add the tags: name_of_your_runner in your .gitlab-ci.yml file.