Configure your Gitlab CI with docker-machine against keystone v3

No Comments

We are running our Gitlab CI infrastructure on top of OpenStack. To not use a fixed number of VMs, we use Gitlab CI with docker-machine to create VMs as needed by the build jobs. This blog post will describe how to enable docker-machine to properly talk to the keystone v3 API in this setup.

The old configuration looked something like this:

[[runners]]
name = "Your Runner”
url = "https://your.gitlab.url/"
token = "your-gitlab-ci-runner-token"
executor = "docker+machine"
[runners.machine]
MachineDriver = "openstack"
MachineOptions = [
        "openstack-auth-url=https://your.keystone.url:5000/v3",
        "openstack-flavor-id=build-vm-flavor",
        "openstack-image-name=build-vm-image",
        "openstack-tenant-name=your-openstack-project",
        "openstack-domain-name=Default",
        "openstack-net-id=your-ci-project-network",
        "openstack-sec-groups=your-sec-groups",
        "openstack-username=openstack-username",
        "openstack-password=openstack-password",
        "openstack-keypair-name=your-ssh-keypair",
        "openstack-private-key-file=your-ssh-private-key-file",
        "openstack-ssh-user=your-ssh-user"

The above setup uses the keystone v2 API wording and also connects to the keystone v2 API. But starting with OpenStack Version Queens, keystone v2 API is removed and you have to use the v3 API. The problem is: docker-machine does not provide any config parameters to use the v3 API. The version number in the auth-url is not sufficient.

But docker-machine respects environment variables for the OpenStack client used under the hood. Luckily for us, Gitlab gives us the possibility to provide environment variables to the docker-machine process.

We had to change the configuration in two sections:

[[runners]]
environment = ["OS_PROJECT_NAME=your-openstack-project","OS_IDENTITY_API_VERSION=3","OS_USER_DOMAIN_NAME=Default"]
[runners.machine]
MachineDriver = "openstack"
MachineOptions = [
        "openstack-auth-url=https://your.keystone.url:5000/v3",
        "openstack-flavor-id=build-vm-flavor",
        "openstack-image-name=build-vm-image",
        "openstack-tenant-id=your-openstack-project-id",
        "openstack-domain-name=Default",
        "openstack-net-id=your-ci-project-network",
        "openstack-sec-groups=your-sec-groups",
        "openstack-username=openstack-username",
        "openstack-password=openstack-password",
        "openstack-keypair-name=your-ssh-keypair",
        "openstack-private-key-file=your-ssh-private-key-file",
        "openstack-ssh-user=your-ssh-user"

We previously used the --openstack-tenant-name parameter. With this parameter, docker-machine tries to call the removed apis to resolve the tenant name to its id. By using the --openstack-tenant-id parameter the code doesn’t get executed.

We have to confess, we learned it the hard way. We found the problem because of our recent upgrade from OpenStack Pike to Queens. Our Gitlab CI stopped working with this error message in the gitlab-runner logs:

Nov 21 07:31:07 master-scheduler gitlab-runner[20426]: ERROR: Error creating machine: Error in driver during machine creation: Expected HTTP response code [200 204] when accessing [GET https:://your.keystone.url:5000/v2.0/tenants], but got 404 instead  driver=openstack name=runner-b099e1bc-master-privileged-1542787257-7b239e1a operation=create
Nov 21 07:31:07 master-scheduler gitlab-runner[20426]: ERROR: {"error": {"message": "(https://your.keystone.url:5000/v2.0/tenants): The resource could not be found.", "code": 404, "title": "Not Found"}}  driver=openstack name=runner-b099e1bc-master-privileged-1542787257-7b239e1a operation=create

We hope this type of posts helps other people save some time by learning from our experiences.

Christian Zunker

Christian Zunker is creating and operating distributed software systems and infrastructures. In his current role as a System Engineer, he builds OpenStack based clouds for cc cloud GmbH and its customers.

Comment

Your email address will not be published. Required fields are marked *