Putting the fun back in OpenStack CI/CD pipelines with RHOSP and Gitlab

Introduction

As a technical consultant, an important part of my daily activities is being able to develop code or customization for the customer I'm working with. Quite often, this means working extra hours in lab environments to make sure we have a 100% reliable, reproducible and idempotent software change.

If you're worked with 'Red Hat OpenStack Platform' after version 7, you probably know that a Director-based deployment of RHOSP can be quite time-consuming and may involve some decent amount of trial-and-error.

At the same time, if I only wanted to test a specific feature in a lab, it may take me a few hours to launch a complete environment to try that feature out. This was beginning to become very time-consuming, even with powerful hypervisors.

So, I came up with the crazy idea to make my life easier and use a gitlab pipeline to deploy RHOSP so I didn't have to perform the pre-required steps manually.

Defining the Production Chain

Here's the process I came up with :

1. select an hypervisor and boot it if it's turned off.
2. select a RHOSP version (8,9,10,13,15 or 16)
3. select an overcloud deployment size
4. select if the selected undercloud would need to be restored from snapshot.
5. Block until all answers are provided. When all cleared, proceed with deployment.

Here is what the complete workflow looks like in a successful run:

Then:

And finally:

 It wasn't easy to come up with the complete setup and it took many attempts. A few weeks into this project, I got it working and here are some more details.

The GitLab setup

To make my life easier, I decided to run Gitlab Omnibus in a VM on one of my hypervisors and then add runners as docker gitlab-runners (one per Hypervisor).

This works because Gitlab runners come up and go down dynamically and Gitlab handles this fine. Here's a diagram describing the setup:

The Pipeline design

When running a pipeline, the requirements were pretty simple:

The pipeline would block in stage 5 (after the 4 previous stages, one for each option) and wait for the user to provide the required answers:

The end-user would then go and select an answer for each of the previous stages in no particular order.

• Choose a hypervisor to run the virtual deployment on:

Once selected, the stage changes to:

And then later to:

• Next, choose a Red Hat OpenStack Platform version:

Once selected, the stage changes to:

• Choose an overcloud deployment size:

(1 director, 1 controller, 1 ceph and 2 computes or something larger):

• Choose whether the undercloud should be restored from a snapshot or not

When all 4 questions have been answered, the config injection task will terminate and unleash the rest of the stages:

We have a liftoff!

All this is implemented using a single Gitlab CI/CD YAML here which got added to my templates respository.

The whole idea was that I could keep developing RHOSP TripleO Heat Templates and easily spin up a full-fledged environment when I needed to test a change or reproduce a customer issue.

All that was required was to take my existing my git repo...

..and then add a GitLab CI/CD configuration file:

This configuration file is a little long and contains some specific commands but the whole process should be straightforward with the use of the documentation : https://docs.gitlab.com/ee/ci/