Building scalable Pipeline with Azure DevOps

As we are experiencing exponential growth in IT industry and primary in Cloud and DevOps domain. It is critical to build solutions that meets ever growing demand at scale.

I would like to put my thoughts around key aspect of DevOps role which bring lots of value and create big impact if this new paradigm implemented properly. CI-CD will speed-up your development, testing, monitoring and feedback cycle tremendously and you will have frequent changes, low downtime and better customer experience.

Let’s take a look just one part of Azure DevOps implementation, how we can build scalable and repeatable pipelines.

At high level, If we see schema of Azure DevOps Pipeline it is looks like something below.

Addition to above you would usually have “resources” at top of your yml pipeline, will talk later role of “resources” and how it can help to build scalable pipelines. we also need to defined “trigger”, when do you want to run pipeline, as soon as code pushed to your branch or any branch in your repo?

Later in pipeline we need to define runner to run and it called “pool “ where you will define kind of OS images will be used that is determined by your pipeline code…if you are running AKS or bash then usually you will go with Linux/ubuntu based images.

Param and variable you can use as template if those are mostly static if not you can define in your pipeline itself.

All above is general practice that everyone follows, now let’s talk how we can make scalable and repeatable pipeline.

• Ensure your pipeline(main) is as short as possible as people who will be dealing with it may not be familiar...what we can do is create template or create modules in programming terminology and reference in our main pipeline.
• Create separate Repo only to host your template/module and do not open access to all.
• In separate repo let’s call it as azure-templates and create PARAM, variable and other task based template like, maven build task related, docker task related, helm task related etc. Key thing to note here only create template for static and global contain.
• Once you have separate repo and template for all the reusable task, jobs, PARAM, variable it make life much more easy to build pipeline for multiple component of application.
• Last part is to leverage “resources” as we talked earlier in this post in your pipeline to link your pipeline to your template/module repo. you can also add PARAM, variable that is pipeline specific.
• This way we will be able to reference same template/module in multiple pipelines without re-inviting wheels in corporate teams and improve time to market or deployment.

You final main pipeline will like like as below…

If you notice, I have repo called azure-template and added in “resources” in above pipeline in first block. This has all the template/modules that I can now reference in my pipeline as single line(check line number 29).

Note: This need to be in same project to have above implementation and I have not validated if or how we can leverage if template repo in other projects repo.

This is one of way to implement scalable pipelines, if you have implemented and know how it can be improved, please post in comment happy to learn and improve.