GitOps

GitOps is an approach for implementing infrastructure as code. GitOps uses Git repositories as a single source of truth for determining the desired state of an application in a given deployment environment. A GitOps operator running within or outside the target environment takes this desired state and compares it to the actual state inside the environment. It can then reconcile the desired state either automatically or manually triggered.

Ensuring that any deployment to the cluster has to occur through a GitOps workflow results in several advantages:

• There is a unified way of deploying applications to a cluster.
• All deployment configuration changes are documented.
• Drift between documented configuration and actually deployed configuration is eliminated.
• Rolling back to an earlier version of a deployment is relatively easy.

There are different choices available for GitOps operators in Kubernetes. The two most popular are Argo CD and Flux. VCPS uses ArgoCD behind the scenes to deploy service foundations and managed services which are backed by Git repositories containing all relevant configuration files.

Argo CD

Argo CD is a declarative, GitOps operator for Kubernetes. It runs inside a management cluster using multi-tenancy or is installed directly in the target cluster. It then creates Kubernetes resources based on Git deployment repositories in the specified target clusters. It provides interfaces in form of direct API access, CLI tooling and web based UIs.

All Argo CD related data is stored in the management cluster in form of Kubernetes standard and custom resources. Its components and architecture is illustrated in the following image (source):

Argo CD consists of mainly three components:

• API server provides a gRPC/REST based API used by all supported clients. It has a few responsibilities, like managing credentials and invoking deployment operations (sync, rollback, etc.).
• Repository Service reads the configured Git repositories and generates the desired state in form of a list of Kubernetes resources.
• Application Controller compares desired state against actual state and reconciles the two if necessary.

The desired state can be specified in several different ways in the Git repository:

• Kustomize applications
• Helm charts
• A directory of YAML/JSON/Jsonnet manifests

If this is not sufficient it is also possible to configure a custom config management tool to generate Kubernetes resources from files within a Git repository.

Enforcing Policies

Using Git as the single source of truth for all deployment configuration enables accountability, traceability and improved security for deployments. These are achieved by either native Git features or features in the used code hosting solution.

Documenting Changes

Every change to the deployment configuration is reflected as a commit in the deployment Git repository. Looking at the history of the repository it is clear who made what change when. A commit in the deployment repository could look something like this:

git show --format=fuller HEAD

commit d80ea3896eefdc39135e2db8bbec0523f7df6323 (HEAD -> main)
Author:     Seth Matthams <seth.matthams@acme.co.uk>
AuthorDate: Tue Oct 25 10:49:20 2022 +0200
Commit:     Delia Ventura <delia.ventura@acme.co.uk>
CommitDate: Tue Oct 26 11:21:36 2022 +0200

    Increase control plane and worker node instances

diff --git a/base/values.yaml b/base/values.yaml
index 51f83e1..648d5df 100644
--- a/base/values.yaml
+++ b/base/values.yaml
@@ -9,7 +9,7 @@ provider-hcloud:
   controlPlane:
     machineType: "cx21"
     region: "fsn1"
-    replicas: 1
+    replicas: 3
   encryptionSecretRef:
     key: "config"
     suffix: "encryption-config"
@@ -22,4 +22,4 @@ provider-hcloud:
   workerPools:
   - machineType: "cx21"
     region: "fsn1"
-    replicas: 3
+    replicas: 6

The author is the one who created the change. Depending on the details of the Git workflow the author might also be the committer of the change, i.e. the person who added the change to the repository. In this example the committer is different from the author as there was an additional review step in between making the change and adding it to the repository. The commit date shows the moment in time when the change was added to the repository while the diff lists exactly what was changed.

Using all this information it is possible to determine at any point in time who was responsible for what change. This allows for investigation of malicious or erroneous changes after the fact. If a sophisticated approval process is used in addition the risk of such malicious or erroneous changes ever getting deployed is reduced tremendously.

Approving Changes

A good GitOps workflow requires a number of rules to be enforced by the solution used to host the Git repositories, namely:

• No unauthorized commit pushes are allowed to the repository.
• Pushing commits directly to the main branch of the repository is disallowed even for authorized users.
• Any change to the main branch occurs through pull requests.

The following image illustrates a Gitflow based workflow for creating and integrating changes to the repository using branches and pull requests (source):

The main branch in this example is the one that is being used by the GitOps operator to determine the desired state. Nobody is allowed to directly commit or push in this branch. This branch can optionally be sourced from a develop branch although this structure is more common in Git repositories used for application development. The more important aspect is that any changes to the repository are done in feature branches.

A new change is proposed by creating a new feature branch, adding the changes and creating a new pull request. One or more reviewer an look at the changes done in the feature branch and check them for correctness. Once all relevant parties have given their approval the pull request is accepted and the feature branch commits are merged into the main (or develop) branch.