Monorepo/multirepo support in werf (and what does it have to do with Docker Registry?)

Published in

werf blog

5 min readSep 9, 2019

Please note this post was moved to a new blog: https://blog.werf.io/ — follow it if you want to stay in touch with the project’s news!

The topic of monorepos is quite popular on the Internet and, as a matter of fact, actively discussed. In Flant, developing werf as an Open Source tool aimed at improving processes of assembling Git-sourced application code into Docker images (and their subsequent deployment to Kubernetes clusters), we prefer to avoid the very question of which approach is more suitable. Our ultimate goal is to supply all the necessary tools for proponents of different paths (if they do not contradict common sense, obviously).

The recent support for monorepos in werf is a perfect case in point. But first, let’s examine how this support relates to werf usage and what does it have to do with Docker Registry…

The problem

Imagine this situation: the company has several development teams involved in separate projects. The majority of applications are run in Kubernetes and, hence, are being containerized. The company employs Docker Hub with a single COMPANY account as a registry to store containers images. Similar to other source code storage systems, Docker Hub does not support multilevel image names, a nested hierarchy of repositories, in the form of COMPANY/PROJECT/IMAGE. So how could you host non-monolithic applications in the registry in this case without creating a separate account for each project?

Someone might have the first-hand experience of the described situation… However, we prefer to look at this problem in a more generalized way, that is, without reference to the above example and Docker Hub.

Possible solutions

If an application is monolithic and delivered as a single image, then no questions asked: we store all images in the single project repository.

When an application consists of several components or microservices, you have to pick a specific approach. Using a typical example of a web application consisting of two images — frontend and backend — we have the following options:

You can store images in the individual nested repositories:

2. You can store images in the same repository and reference them via tags, e.g., in the following manner:

NB: As a matter of fact, there is an option to store images in multiple repositories such as PROJECT-frontend and PROJECT-backend. However, we won’t consider it due to complexities associated with setting it up, maintaining, and managing users’ rights.

Implementing support in werf

Initially, werf supported nested repositories only — fortunately, the majority of registries have this feature. Starting with the v1.0.4-alpha.3 version, we have added support for registries lacking the nested functionality (including Docker Hub). Since then, the user can choose how to store application images.

You can choose the preferred approach by setting up an option --images-repo-mode=multirepo|monorepo (multirepo, i.e. nested repositories, is the default value). It defines patterns used for storing images in the registry. Just pick the right mode when using basic commands — everything else stays the same.

Since the majority of werf options can be set via environment variables, generally you can easily define global storage mode for the whole project in CI/CD systems. For example, in the case of GitLab, it is enough to set an environment variable in the settings: Setting -> CI / CD -> Variables: WERF_IMAGES_REPO_MODE: multirepo|monorepo.

Regarding publishing images and deploying applications (you can learn more in the relevant doc pages: Publish process и Deploy process), mode merely defines the pattern for interacting with the image.

Devil is in the details

The major complication when adding the new storage method is the registry cleaning process (to learn more about cleaning options supported by werf, see Cleaning process).

When cleaning, werf takes into account images used in the Kubernetes clusters as well as policies defined by the user. Policies are based on combining tags into strategies. Currently, werf supports the following strategies:

3 strategies related to Git primitives, such as tags, branches, and commits;
1 strategy for custom tags defined by the user.

We put the tag’s strategy type in the labels of the final image when publishing it. The value itself — the so-called meta tag — is required for applying some policies. For example, it is used when deleting a branch or a tag in the Git repository (it makes sense to remove related unused images from the registry, which is what some of our policies do).

When storing images in a single repository (monorepo), in addition to meta tag, the image tag can also include the image name: PROJECT:frontend-META-TAG. To separate them, we have chosen not to use a specific delimiter but simply added the necessary value to the label of the final image when publishing.

NB: Those interested in the implementation of described features in werf source code could use PR 1684 as a starting point.

Summary

Lack of support for non-nested registries did not prevent us or other users from applying werf given that you can always create a separate image registry (or switch to a ready-to-use Container Registry from Google Cloud and other providers). However, the removal of the above restriction seemed quite logical because it enhances the usability of our tool for the wider DevOps community.

While implementing it, we have had to overhaul the mechanism of cleaning up the container registry (that was the major challenge). Now, when everything is up and running, it is pleasant to feel that we have made life somewhat easier for some of our users. The other advantage is that we (as the principal developers of the project) can easily and effortlessly maintain this feature in the future.

So stay tuned as we are planning to reveal some handy innovations in werf shortly!