Downloading gitlab database backup from s3

You want to download a copy of your gitlab backup (ie when upgrading to version 7.0.0 as described in [here](https://docs.gitlab.com/charts/installation/database_upgrade.html#steps-for-upgrading-the-bundled-postgresql) ).

Login to the toolbox pod and get the credentials file for s3

cat /etc/gitlab/.s3cfg

Create a new file locally (ie ~/.s3cfg_gitlab) and then download the file locally with

s3cmd get s3://gitlab-backups/database_upgrade_6_gitlab_backup.tar  20230531_gitlab_backup.tar -c ~/.s3cfg_gitlab

Removing gitlab’s container registry ‘repository scheduled for deletion’ message

If you want to remove the ‘repository scheduled for deletion’ message from a self hosted gitlab’s container registry installation, you can do the following.

Login to the toolbox pod

kubectl --kubeconfig /path/to/gitlab/kubeconfig -n gitlab-system exec -it gitlab-toolbox-pod-name -- bash

Start the rails console

cd /srv/gitlab
bundle exec rails console

Find the repository with it’s id and then get the registry for it

project=Project.find(project_id)
registry=project.container_repositories.first

The message is displayed when the status of the container is set to ‘delete_scheduled’, so change this to be null

registry.status=''
registry.save

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Problem

You are trying to install the gitlab kubernetes agent (kas) after getting the installation instructions from the Gitlab registration, but you are getting the following error:

Error: parse error at (gitlab-agent/templates/observability-secret.yaml:1): unclosed action

Solution

This is caused by the helm version used, so upgrading the helm version works with no issues.

 helm version
version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}

so trying to install gives the previous error.

Changing the helm version (ie with asdf)

asdf global helm 3.70


Checking the version

helm version
version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}

And running the helm installation again

helm upgrade --install gitlab-kas-agent gitlab/gitlab-agent \

--namespace gitlab-agent --set image.tag=v15.6.0 \

--set config.token=token_from_gitlab-registration \

--set config.kasAddress=wss://kas.domain_name.tld


Release "gitlab-kas-agent" does not exist. Installing it now.
NAME: gitlab-kas-agent
LAST DEPLOYED: Mon Dec  5 13:38:33 2022
NAMESPACE: gitlab-agent
STATUS: deployed
REVISION: 1

TEST SUITE: None

Exporting Gitlab project through the API

To export a project through Gitlab’s API, first create a Personal Access Token (PAT) and give api and repository permissions.


Export PAT as an environment variable.

Then find the project id (main page of the project – 1111).

To start an export use

curl --request POST --header "PRIVATE-TOKEN: $PAT" "https://your.gitlab.repo/api/v4/projects/1111/export" --data "upload[http_method]=PUT"

Then check the status and get the download link when it has finished

curl --header "PRIVATE-TOKEN: $PAT" "https://your.gitlab.repo/api/v4/projects/1111/export"
{"id":1111,"description":"Project description","name":"project","name_with_namespace":"MyOrg / project","path":"project","path_with_namespace":"myorg/project","created_at":"2021-11-01T10:57:23.195+02:00","export_status":"finished","_links":{"api_url":"https://your.gitlab.repo/api/v4/projects/1111/export/download","web_url":"https://your.gitlab.repo/myorg/project/download_export"}}

Request Entity Too Large when trying to import Gitlab project

Problem

You would like to import an existing Gitlab project, through an export file, to a new self hosted instance of Gitlab, but using the Web UI, even after changing the max-body-size in the ingress deployment you end up having the error message

Request Entity Too Large

Solution

There is another way to import the exported file, but is not documented anywhere as it is classed as EXPERIMENTAL from Gitlab.

You can copy the exported file to the gitlab-toolbox pod

kubectl --kubeconfig ~/.kube/gitlab_config cp local_export.tar.gz gitlab-toolbox-xxx-xxx:/tmp/

You can then login to the gitlab-toolbox pod

kubectl --kubeconfig ~/.kube/gitlab_config -n gitlab-system exec -it gitlab-toolbox-xxx-xxx -- bash

get to directory with the application

cd srv/gitlab

and finally use the rake task gitlab:import_export:import to import your project

git@gitlab-toolbox-xxx-xxx:/srv/gitlab$ bundle exec rake gitlab:import_export:import[your_new_gitlab_username,namespace_path,project_path,/tmp/2022-06-14_14-53-007_export.tar.gz]

Changing gitlab-runner configuration options

It is possible to change the configuration of a running gitlab-runner by editing the file ~/.gitlab-runner/config.toml.

For example to be able to switch the log_level from ‘info’ to ‘debug’ and back again, you can do so by logging into the gitlab-runner and then editing the file.

The file is reloaded without the need for rebooting the gitlab-runner.

Google Autopilot and Gitlab failed builds

Problem

You want to use Google’s Autopilot for your gitlab runners, but your job/builds fail because of low resources (ie ephemeral storage).

Solution

You can use a limit range to increase the limits for ephemeral storage or/and memory that will make Google’s autopilot to use them and scale them appropriately.

Create a limit range file like:

apiVersion: v1
kind: LimitRange
metadata:
  name: limit-ephemeral-storage
spec:
  limits:
  - default:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    defaultRequest:
      ephemeral-storage: "10Gi"
      memory: "16Gi"
    type: Container

And then apply it to your cluster

kubectl -n namespace apply -f limit_range.yaml

WARNING: Pulling GitLab Runner helper image from Docker Hub. Helper image is migrating to registry.gitlab.com, for more information see https://docs.gitlab.com/runner/configuration/advanced-configuration.html#migrating-helper-image-to-registrygitlabcom

To change this in your gitlab-runner configuration, depends on the version of your gitlab-runner as described in https://docs.gitlab.com/runner/configuration/feature-flags.html

If you version is 13.9.0 (gitlab-runner –version), then add the following in ~/.gitlab-runner/config.toml

[[runners]]
  name = "gitlab-runner-name"
  request_concurrency = 1
  url = "https://git.url.test/"
  token = "xckaxxxxxxxx"
  executor = "kubernetes"
  environment = ["FF_GITLAB_REGISTRY_HELPER_IMAGE=1"]

ssh-add, Gitlab CI/CD with alpine image

Problem

Trying to use the usual way of adding Gitlab’s key to the ssh agent, is not working in alpine image (but it works in debian/ubuntu), and you get the following error message:

 - ssh-add <(echo "$GITLABCICD_PRIVATE_KEY")
/bin/sh: syntax error: unexpected "("

Solution

Try to use the following instead:

  - echo "$GITLABCICD_PRIVATE_KEY" | ssh-add -