HAProxy load balancing with ssl pass_through in front of Vault

Problem

You would like to set up HAProxy as the load balancer in your Vault cluster configuration. Vault is set up with internal storage (Raft), and you would also like to be able to pass the connection encrypted from the client to the Vault cluster member, without having HAProxy decrypt/encrypt the message.

Solution

Looking and searching around in the internet provides a few examples of how to do this. During this process there are different errors that you can see happening. There are certain things that you will need to do in order to debug and find the correct setup.

One of the main points is that HAProxy’s frontend and backend mode should be tcp instead of http.

Another thing that will help with debugging would be having access to both HAProxy’s and Vault (active) member logs.

Using curl with the -v flag is also very helpful.

One of the attempts of having HAProxy and Vault server communicate through HTTP/2 was resulting in the following errors:

on an external client (not HAProxy, or Vault).

the following on the HAProxy log:

and the following on the Vault log

Changing the configuration on HAProxy to use http1.1 solves the issue (/etc/haproxy/haproxy.cfg

Which results in the correct response now by using the HAProxy as a ssl pass_through load balancer.

SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

Problem

Trying to configure and use an nginx server that uses intermediate certifcates, you get the error about values mismatch and nginx does not start

Solution

It seems that this is a common mistake and it is mentioned here: https://nginx.org/en/docs/http/configuring_https_servers.html. In order to fix this you will need to change/reverse the original order that you have concatenated the chain, as in:

Tail Vault auth logs

Problem

You would like to tail the Vault auth log files, but if you use the vault tag for auditing and your server is also named vault, you cannot filter out the vault auth logs only.

Solution

Use the client_login as the filter in grep:

Check haproxy configuration file for errors

Problem

You would like to check your haproxy.cfg file for errors when making changes.

Solution

Use the following command to validate/check your HAProxy configuration file:

Decode access secret key from Terraform IAM user creation

Problem

You want to use the pgp encryption when using Terraform to create an AWS IAM user, and you have the secret access token returned as an output but encoded.

Solution

Use the following to get the actual secret key decoded (after copying your encoded key to a file encrypted_key.txt:

Terraform Cloud – ‘operations’ attribute is deprecate, and cannot be used in conjuction with ‘execution’. Use the latter only

Problem

You are trying to add a new workspace in your organization in Terraform Cloud, using the same connected VCS (Gitlab) as the one you already have setup, but you get the above error when trying to add it in Firefox (Linux – Ubuntu – 79).

Solution

Use Chromi(um) to create the workspace as it works there. You can then use it in Firefox as normal.

Error initializing storage of type raft: failed to create fsm: failed to open bolt file: open /home/vault/data/vault.db: permission denied

Problem

Trying to start the vault server following the guides that specify creating the data directory in /home/vault/data results in the above error and service cannot be started.

Solution

According to the guide here: https://learn.hashicorp.com/tutorials/vault/raft-deployment-guide?in=vault/day-one-raft, changing the data directory to /opt/raft and the hcl file to reflect that the server can be started without any errors.

Failed to initialize build ‘qemu’: post-processor type not found: exoscale-import

Problem

Trying to follow along the example in the Exoscale article about creating custom templates, you get the error about exoscale-import not found, when you are trying to validate the packer template.

Solution

Make sure that you use a current packer installation as the exoscale-import was not supported in the older versions.

So after installing the latest version the above error goes away

qemu: Error launching VM: Qemu failed to start. Please run with PACKER_LOG=1 to get more info.

Problem

You are trying to build a QEMU image with Packer, but you are getting the error suggesting to use the PACKER_LOG=1 option. But there is no indication where to place it in the command line.

Solution

Put it at the begging of the command line as folows: