Executing commands while deploying your app’s new release in Kubernetes

Published in

werf blog

8 min readJan 24, 2020

Please note this post was moved to a new blog: https://blog.werf.io/ — follow it if you want to stay in touch with the project’s news!

In Flant, we are often confronted with the challenge of adapting applications for running them in Kubernetes. When dealing with this challenge, we usually encounter many repetitive problems. We have already discussed one of them in the “Migrating your app to Kubernetes: what to do with files?”. In this article, we’ll focus on another problem which relates to CI/CD processes for this time.

Executing custom commands with Helm and werf

The typical application contains not only business logic and data, but also a set of custom commands which need to be run to make upgrading to a newer release successful. These can be some migration scripts for databases, scripts for checking the readiness of external resources, various tools for unpacking or decoding, scripts for registering in the external Service Discovery, etc — all applications have their specifics.

What tools does Kubernetes provide for solving these tasks? Kubernetes shines when it comes to running containers as pods, so the standard solution, in this case, is to run a command from within an image. For this, Kubernetes has a special Job primitive that allows you to start a pod with application containers and watches its completion.

Helm makes one step further and provides the ability to run Jobs on various stages of the deployment process. The so-called Helm hooks allow you to run a Job before or after updating manifests of resources. In our experience, this Helm feature is brilliantly useful and is perfect for solving various deploying tasks.

However, with Helm, you cannot get the actual information about the state of objects during the deployment; that’s why we prefer to use werf. This tool provides the ability to monitor the state of resources during the deployment process right in the CI system and, in the case of a failure, troubleshoot the problem quickly.

As it turns out, these useful features of Helm and werf are sometimes mutually exclusive. Happily, there’s always a way around! Let’s take a look at possible means of monitoring the state of resources and running custom commands (we base our examples on migrations).

Running migrations before the release

Altering the database schema is an integral part of a release of any application that uses databases. The standard deployment process for applications that run migrations by executing a specific command includes the following steps:

updating the codebase;
starting a migration;
switching traffic to the new version of an application.

In Kubernetes, the deployments process has a similar form yet with several additions we need:

We need to start a container with a new codebase which can contain a new set of migrating scripts;
We need to initiate the process of applying migrations before updating the version of an application.

Let’s stick with a case when an application database is up and running, and we do not need to deploy it as part of the release process that deploys an application. The following two hooks can be used for applying migrations:

pre-install is executed after templates are rendered (during the initial Helm release of an application), but before any resources are created in Kubernetes;
pre-upgrade is executed on an upgrade request after templates are rendered, but before any resources are updated in K8s.

Here is an example of a Job that uses Helm and two hooks mentioned above:

---
apiVersion: batch/v1
kind: Job
metadata:
 name: {{ .Chart.Name }}-apply-migrations
 annotations:
   "helm.sh/hook": pre-install,pre-upgrade
spec:
 activeDeadlineSeconds: 60
 backoffLimit: 0
 template:
   metadata:
     name: {{ .Chart.Name }}-apply-migrations
   spec:
     imagePullSecrets:
     - name: {{ required ".Values.registry.secret_name required" .Values.registry.secret_name }}
     containers:
     - name: job
       command: ["/usr/bin/php7.2", "artisan", "migrate", "--force"]
{{ tuple "backend" . | include "werf_container_image" | indent 8 }}
       env:
{{ tuple "backend" . | include "werf_container_env" | indent 8 }}
       - name: DB_HOST
         value: postgres
     restartPolicy: Never

NB: The YAML template provided above is designed for werf. To adapt it to “pure” Helm use, you have to:

replace {{ tuple "backend" . | include "werf_container_image" | indent 8 }} with the required container image;
delete the {{ tuple "backend" . | include "werf_container_env" | indent 8 }} expression specified in the env key.

Now you have to put this Helm template into a .helm/templates directory where the other release resources are stored. When werf deploy --stages-storage :local is invoked, all templates will be processed, and then they will be uploaded to the Kubernetes cluster.

Running migrations during the release

The above option implies applying migrations when the database is up and running already. But what if we want to deploy a branch review for an application, and the database will be rolled out along with an application as part of a single release?

NB: You may encounter such a problem when deploying to the production environment as well if you use Service with an endpoint (that contains the database IP address) to connect to the database.

In this case, you cannot use pre-install and pre-upgrade hooks since an application will be trying to apply migrations to the database that doesn’t yet exist. Thus, you have to wait until the release is complete to apply migration scripts.

However, with Helm it is a perfectly achievable task because it does not track the state of an application. Post-hooks are always executed after all resources are loaded into Kubernetes:

post-install— after all resources are loaded into Kubernetes (within the initial release);
post-upgrade — after all resources are updated (as a part of the release upgrade).

However, as we’ve previously stated, werf has a mechanism for tracking the resources state during the release. Let us dig into that a little more:

werf uses the capabilities of the kubedog library.
This werf feature allows us to determine the state of the release and display information on the successful or unsuccessful completion of the deployment process in the interface of the CI/CD system.
This information is the key to the automation of the release process since the successful creation of resources in the Kubernetes cluster is only a part of the story. For example, an application may not start due to a misconfiguration or due to a network problem. However, it will be hard to troubleshoot the cause of failure after running helm upgrade (meaning you’ll have to perform some additional actions to find out what’s happened).

Okay, let’s get back to applying migrations with Helm post-hooks. Here is a list of problems we have encountered so far:

Before starting, many applications check the state of the database schema in some way. That’s why without the latest migrations, an application may not start.
Since werf by default checks if all objects are in the Ready state, post-hooks will fail to run, and migrations will not be applied.
You can disable tracking objects via additional annotations. However, in this case, it will be impossible to get reliable information about the results of the deployment.

This is what we came down to:

Jobs are created before core resources, so you don’t have to use Helm hooks for applying migrations.
However, the migration Job should be run on every deployment. To make this happen, the Job must have a unique (random) name: this way, for Helm, it would be an entirely new object with every subsequent release that will be created in Kubernetes.
With this approach, you don’t have to worry about the growing number of migration Jobs: their names are unique, and each previous Job gets deleted with every subsequent release.
Each migration Job must have an init container that checks if the database is available. Otherwise, the deployment process will fail (the Job will fail at the init container).

Here is an example of the resulting configuration:

---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ printf "%s-apply-migrations-%s" .Chart.Name (now | date "2006-01-02-15-04-05") }}
spec:
  activeDeadlineSeconds: 60
  backoffLimit: 0
  template:
    metadata:
      name: {{ printf "%s-apply-migrations-%s" .Chart.Name (now | date "2006-01-02-15-04-05") }}
    spec:
      imagePullSecrets:
      - name: {{ required ".Values.registry.secret_name required" .Values.registry.secret_name }}
      initContainers:
      - name: wait-db
        image: alpine:3.6
        сommand: ["/bin/sh", "-c", "while ! nc -z postgres 5432; do sleep 1; done;"]
      containers:
      - name: job
        command: ["/usr/bin/php7.2", "artisan", "migrate", "--force"]
{{ tuple "backend" . | include "werf_container_image" | indent 8 }}
        env:
{{ tuple "backend" . | include "werf_container_env" | indent 8 }}
        - name: DB_HOST
          value: postgres
      restartPolicy: Never

NB: As a matter of fact, you should use init containers for checking the availability of the database anyway.

An example of a “one-size-fits-all” template for various deployment operations

Aside from running migrations, however, other types of operations must also be carried out during the release. You can control the execution queue of a Job not only through the types of hooks but also by assigning weight to each of them via the helm.sh/hook-weight annotation. Hooks are sorted by weights in ascending order. If weights are the same, the sorting is performed by the resource name.

In the case of a large number of Jobs, we suggest creating a universal template for Jobs and putting all the configuration data into a separate values.yaml file. The latter might look like this:

deploy_jobs:
  - name: migrate
    command: '["/usr/bin/php7.2", "artisan", "migrate", "--force"]'
    activeDeadlineSeconds: 120
    when:
      production: 'pre-install,pre-upgrade'
      staging: 'pre-install,pre-upgrade'
      _default: ''
  - name: cache-clear
    command: '["/usr/bin/php7.2", "artisan", "responsecache:clear"]'
    activeDeadlineSeconds: 60
    when:
      _default: 'post-install,post-upgrade'

… and below is an example of the template itself:

{{- range $index, $job := .Values.deploy_jobs }}
---
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ $.Chart.Name }}-{{ $job.name }}
  annotations:
    "helm.sh/hook": {{ pluck $.Values.global.env $job.when | first | default $job.when._default }}
    "helm.sh/hook-weight": "1{{ $index }}"
spec:
  activeDeadlineSeconds: {{ $job.activeDeadlineSeconds }}
  backoffLimit: 0
  template:
    metadata:
      name: {{ $.Chart.Name }}-{{ $job.name }}
    spec:
      imagePullSecrets:
      - name: {{ required "$.Values.registry.secret_name required" $.Values.registry.secret_name }}
      initContainers:
      - name: wait-db
        image: alpine:3.6
        сommand: ["/bin/sh", "-c", "while ! nc -z postgres 5432; do sleep 1; done;"]
      containers:
      - name: job
        command: {{ $job.command }}
{{ tuple "backend" $ | include "werf_container_image" | indent 8 }}
        env:
{{ tuple "backend" $ | include "werf_container_env" | indent 8 }}
       - name: DB_HOST
         value: postgres
      restartPolicy: Never
{{- end }}

Such an approach allows you to add new commands to the release process quickly and makes a list of executable commands more transparent.

Conclusion

In this article, we provided examples of templates for describing general operations that developers face during the process of releasing a new version of an application. Although they are the result of our experience in implementing numerous CI/CD processes, we feel that there is no single bulletproof solution suitable for all tasks. If the examples presented above are too narrow for the needs of your project, please, share your ideas, experience, and comments on how to improve this article.

P.S. And here is the catchy remark from werf developers:

We are planning to introduce custom, user-configurable stages for deploying resources in the future. With these stages, you can effortlessly implement both described scenarios and more.