Using exo cli to display your vm templates in Exoscale

To display your vm templates, as opposed to all available templates in Exoscale using the exo CLI, use the following (you should have already the environment variables set up to be able to use exo):

exo vm template list --mine
?????????????????????????????????????????????????????????????????????????????????????????????????????????
?                  ID                  ?     NAME      ?      CREATION DATE       ?  ZONE   ? DISK SIZE ?
?????????????????????????????????????????????????????????????????????????????????????????????????????????
? a58f0180-689d-4663-aa53-91111111111  ? my-ubuntu     ? 2020-08-12T13:44:01+0000 ? ch-dk-2 ? 50 GiB    ?
?????????????????????????????????????????????????????????????????????????????????????????????????????????

Error: Invalid provider configuration (terraform plan locally with TerraformCloud)

Problem

You are trying to run locally the terraform plan, while using the Terraform’s Cloud remote backend, before commiting your changes to your repo and running it through Terraform’s Cloud UI, and you are getting the previous error message:

Error: Invalid provider configuration

Solution

Remember to add the environment variables needed for your provider (AWS, Google, Exoscale etc) in the Variables section of your Terraform Cloud setup.

Even you have added them before, using a new Workspace means you will need to add them to each new workspace.

asdf and terraform (or vault or packer)

Using the asdf version manager to manage versions of various binaries like terraform, vault or packer is easily done by following the instructions below:

  1. Download and install asdf if you haven’t got it already (https://asdf-vm.com/#/core-manage-asdf)
  2. If you want to install the latest version of terraform for example do the following:
asdf plugin add terraform
asdf install terraform latest
asdf global terraform 0.14.9
asdf list
terraform
  0.14.9
terraform --version
Terraform v0.14.9

Get/Set platform information

To get or set in environment variables the platform information of a system you can use the following:

export ARCH=$(case $(arch) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(arch) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')

Taken from the installation instructions of the Operator SKD here: https://master.sdk.operatorframework.io/docs/installation/

Using port number in Terraform output

Problem

You would like to output some additional information (ie port number) in addition to the variable provided by Terraform.

Solution

Use interpolation of the Terraform variable using ${var} together with the rest of the string you would like to output, like:

output "kibana_endpoint" {
  value       = "http://${aws_instance.elasticsearch.public_ip}:5601"
  description = "The Kibana endpoint"
}

More information about the language https://learn.hashicorp.com/tutorials/terraform/outputs?in=terraform/configuration-language

Failed to start docker.service: Unit is masked

Problem

Trying to start the docker service after some upgrades fails with the following message:

Failed to start docker.service: Unit is masked.

Solution

It turns out that after upgrading or more specifically removing and then upgrading the docker installation in ubuntu (in this particular case in raspberry 4 with Ubuntu 20.04 installed), results in this error.

A search brings up the following:

https://forums.docker.com/t/failed-to-start-docker-service-unit-is-masked/67413

and from that the following bug post:

https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1844894

So the solution is to run the following to be able to start the docker service (described in the first link above):

sudo systemctl unmask docker
sudo systemctl start docker

HAProxy load balancing with ssl pass_through in front of Vault

Problem

You would like to set up HAProxy as the load balancer in your Vault cluster configuration. Vault is set up with internal storage (Raft), and you would also like to be able to pass the connection encrypted from the client to the Vault cluster member, without having HAProxy decrypt/encrypt the message.

Solution

Looking and searching around in the internet provides a few examples of how to do this. During this process there are different errors that you can see happening. There are certain things that you will need to do in order to debug and find the correct setup.

One of the main points is that HAProxy’s frontend and backend mode should be tcp instead of http.

Another thing that will help with debugging would be having access to both HAProxy’s and Vault (active) member logs.

Using curl with the -v flag is also very helpful.

One of the attempts of having HAProxy and Vault server communicate through HTTP/2 was resulting in the following errors:

kosmas:$ curl -v https://proxy_dn:8200/v1/sys/health
*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to proxy_dn (xxx.xxx.xxx.xxx) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=proxy_dn
*  start date: Sep 15 08:30:43 2020 GMT
*  expire date: Dec 14 08:30:43 2020 GMT
*  subjectAltName: host "proxy_dn" matched cert's "proxy_dn"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55c06f25e580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /v1/sys/health HTTP/2
> Host: proxy_dn:8200
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (21):
* TLSv1.3 (IN), TLS alert, Client hello (1):
* Unexpected EOF
* Connection #0 to host proxy_dn left intact
curl: (56) Unexpected EOF

on an external client (not HAProxy, or Vault).

the following on the HAProxy log:

Sep 25 13:07:27 localhost haproxy[33169]: xxx.xxx.xxx.xxx:38196 [25/Sep/2020:13:07:26.985] vault_https~ vault_https_backend/vault1 46/1/57 0 -- 1/1/0/0/0 0/0

and the following on the Vault log

Sep 25 13:07:27 vault1 vault[607]: 2020-09-25T13:07:27.031Z [INFO]  http: TLS handshake error from xxx.xxxx.xxxx.xxxx(proxy_ip):56632: tls: first record does not look like a TLS handshake

Changing the configuration on HAProxy to use http1.1 solves the issue (/etc/haproxy/haproxy.cfg

frontend vault_https
  bind *:443 ssl crt /etc/haproxy/certs/proxy_dn_certificate.pem alpn http/1.1

Which results in the correct response now by using the HAProxy as a ssl pass_through load balancer.

kosmas:$ curl -v https://proxy_dn:8200/v1/sys/health
*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to proxy_dn (xxx.xxx.xxx.xxx) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=proxy_dn
*  start date: Sep 15 08:30:43 2020 GMT
*  expire date: Dec 14 08:30:43 2020 GMT
*  subjectAltName: host "proxy_dn" matched cert's "proxy_dn"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /v1/sys/health HTTP/1.1
> Host: proxy_dn:8200
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 200 OK
< Cache-Control: no-store
< Content-Type: application/json
< Date: Fri, 25 Sep 2020 14:12:59 GMT
< Content-Length: 294
< 
{"initialized":true,"sealed":false,"standby":false,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"disabled","server_time_utc":1601043179,"version":"1.5.0","cluster_name":"vault-cluster-123455","cluster_id":"xxxx-xxxx-xxxx-xxx-xxxx"}
* Connection #0 to host proxy_dn left intact