Field notes

7 min read

Stop reinventing your CI/CD

Stop reinventing your CI/CD

Any engineer can write a pipeline in an afternoon. The hard part is who keeps dozens of them aligned for years. How we centralised the decisions with a config per project, shared templates and a cron job, and how AI changes that.

Any engineer can write a pipeline in an afternoon. The hard part is who keeps dozens of them aligned for years. How we centralised the decisions with a config per project, shared templates and a cron job, and how AI changes that.

A row of identical light-grey blocks on a long white table in a bright studio, one block red

BY Seppe Rousseau

Frontend Engineer & Tech Lead

Seppe is a frontend engineer and tech lead with nearly 15 years of experience building web and mobile apps. He has a strong interest in 3D, animation and CI/CD, and likes building tools that automate the repetitive work away.

TL;DR

TL;DR

  • Writing a CI/CD pipeline is the cheap part. Keeping the pipelines of many projects aligned for years is the expensive part, and without a shared answer every project drifts.

  • For ten years each project has had one small config file that describes it, while shared templates and a cron job generate and update the full pipeline.

  • A change in a template reaches every active project on the next run, so deprecations, new tooling and fixes like a signing issue stop being a ticket per project.

  • The cost was flexibility: anything custom had to go through the templates first. That only starts to hurt now that AI makes custom workflows cheap.

  • The team is now testing a mix of shared building blocks for things like iOS signing, guidelines and skill files for the rest, and agents that check projects against the guidelines. Nothing is rolled out yet.

The problem isn't writing the pipeline

Any engineer can write a CI/CD pipeline in an afternoon. With AI, in ten minutes. That was never the problem.

At November Five we build web apps, native apps, cross-platform apps, websites. New projects start every month. Old ones get maintained for years. So picture every engineer writing their own pipeline. Fast on day one 🚀. Then two years later: who updates it? The developer who wrote it moved to another project. Project A fixed a signing issue in March. Project B hits the exact same issue in September and starts from zero. Nobody did anything wrong. It just drifts.

We decided a long time ago we didn't want that. So we stopped writing pipelines and started describing projects.

Where does a CI decision live?

Strip any pipeline down and what's left is a stack of decisions. Do tests run before we build? How do we sign? Where does the build go, and who hears about it when it fails? Every project answers these again, and every project answers them a bit differently.

That made the question for us: where do these decisions live? Each project can make them, with whoever happens to be on it at the time. Or the company makes them once, and every project inherits them.

We chose the second. The rest of this post is about how that worked for ten years and why the answer is shifting now that AI writes the pipeline part for free.

Describe the project, don't write the pipeline

Every project has one small config file in its root. It says what the project is. It doesn't say how to build it.

This is a small limited example on how it looks.

# General: what this project is
template: main
platform: ios
branches: [develop]

# Overwritable: project defaults, each workflow can override them
slack_channel: proj-0001-bots
build_scheme: MyApp

# Workflows: what to build
workflows:
  - type: beta
    name: beta-staging
    build_configuration: Beta-Staging
  - type: release
    name: release
    upload_store: true
# General: what this project is
template: main
platform: ios
branches: [develop]

# Overwritable: project defaults, each workflow can override them
slack_channel: proj-0001-bots
build_scheme: MyApp

# Workflows: what to build
workflows:
  - type: beta
    name: beta-staging
    build_configuration: Beta-Staging
  - type: release
    name: release
    upload_store: true
# General: what this project is
template: main
platform: ios
branches: [develop]

# Overwritable: project defaults, each workflow can override them
slack_channel: proj-0001-bots
build_scheme: MyApp

# Workflows: what to build
workflows:
  - type: beta
    name: beta-staging
    build_configuration: Beta-Staging
  - type: release
    name: release
    upload_store: true

Three parts. The general settings say which template to use, which platform this is and which branches build on every push. The overwritable settings are the project defaults, like the notification channel or the build scheme. The workflows list what we build: a beta for testers, a release for the store.

The templates do the rest. They hold the full pipeline for every platform we support, with placeholders where the project's values go, and a template engine fills those in.

Flowchart: a project repo config and the shared templates feed a cron job that scans all active repos and creates or updates the full pipeline yml

A cron job runs every ten minutes over all our active repos. New repo with a config file? It creates the CI project and applies the pipeline. Ten minutes after the push, the project builds, and nobody set anything up. Known repo with a changed config? It updates the pipeline. Template changed? Every project gets the new pipeline on the next run.

The project describes itself. The company decides how it builds.

What ten years of this bought us

The obvious win is the new project. A few lines of config, push, and the pipeline exists. Nobody copies a yml from the last project and hopes it still applies.

The bigger win is everything that happened after day one.

For example: a step we used in every mobile build got deprecated. In the old world that's a ticket in every project, picked up whenever someone has time, done slightly differently each time. For us it was one change in the template. The cron job regenerated every pipeline and the deprecation was gone before most teams noticed.

The same mechanism works the other way. When we wanted new tooling in every project, static analysis for instance, we added it to the templates once and every active project had it the next morning. When we dropped a tool, same thing. Introducing company-wide tooling stopped being a migration and became a template release.

And remember project A fixing a signing issue in March? Here that fix goes into the template, because anyone could contribute to it. Project B never meets the issue in September. The knowledge doesn't sit with one developer, and it doesn't leave when they switch projects. It sits in the one place every project reads from.

Of course this had cons

Less flexibility, mainly. If a project needed something the templates didn't do, you couldn't just add a step to your own pipeline. It had to go through the templates first.

In practice that was rarely a problem, for two reasons.

First, the templates were conditional from the start. The same template built a native iOS app, an Android app, a cross-platform app or a website, and switched steps on and off based on the config. Do we need Firebase distribution for this project? One line. Upload to AWS, Vercel or Azure? A couple of lines, and the template picked the right deploy steps. Decisions that would take a meeting elsewhere were a config change here.

Second, when something really was missing, we built it into the templates. Which meant every other project could use it too. A one-off need became a shared capability, and the person who needed it was usually the one who added it.

So flexibility was the con on paper. For years it barely showed up in practice, because the templates kept absorbing what projects asked for. It's only now, with AI making custom workflows cheap, that we're starting to feel it. So that's what we're working on now.

The knowledge moved. The guidelines didn't.

Back when we started this, CI knowledge was specialised. Knowing the syntax and quirks of a CI service wasn't something every developer had, and nobody expected them to. The templates carried that knowledge for them. Describe your project, and someone who knew the platform had already written the rest.

That part is gone. An AI agent knows every CI platform's syntax better than any of us. Ask it for a workflow and you have one in ten minutes, and it's fine.

What it doesn't know is us. It doesn't know how we sign iOS builds, or that every pull request runs tests before anything merges, or which of our rules are non-negotiable. Those never lived in the documentation of any platform. They lived in our templates and in our heads.

So the cheap part and the expensive part switched places. Writing a workflow is cheap now. Making sure five developers on five active projects write it the same way, with the same signing setup and the same test gate, is the expensive part. Kevin wrote about this shift in AI made code cheap, it made everything else expensive. Our CI setup is the concrete case: the knowledge moved into the model, the alignment didn't.

And that's exactly what templates were for. Not the knowledge, the alignment. So the question from the start of this post stays the same. Where do the decisions live? Only the way we answer it has to change.

A mixed approach

Heads-up: this is where we are now, not where we've been. We're testing this. Nothing below is rolled out.

The idea is to split the pipeline in two.

A small set of shared building blocks for the things we want identical in every project. iOS signing is the clearest example. There's no good reason for ten engineers to solve signing ten ways, so that stays a shared action you use, not something you write.

Everything else is open. Engineers write their own workflows, with AI if they like. Instead of templates they get guidelines, plus skill files so the AI agent they're working with knows those guidelines while it writes. Every PR runs tests. Builds get signed through the shared action. Releases follow the agreed flow. Within those rules, a project can have whatever workflows it needs.

That covers writing. The template setup also solved maintaining, through the cron job. The replacement is agents: they go over our active projects and check them against the guidelines. When a guideline changes, the reviewer tells the project what to update instead of a template silently doing it.

Same goal as before. Where the decisions live hasn't moved. How they reach a project has.

Takeaway

Centralise the decisions, not the pipeline. For ten years we did that with templates and a cron job. Now we're doing it with shared actions, guidelines and agents. The tools changed because the knowledge changed hands. The decisions stayed ours.

If every engineer in your company writes their own pipeline, the question to ask isn't whether they can. They can. It's who decides how it works, and what happens to that decision when they move on.

NOVEMBER FIVE

If the product is core to your business, you can't afford to get the team wrong.

NOVEMBER FIVE

If the product is core to your business, you can't afford to get the team wrong.

NOVEMBER FIVE

If the product is core to your business, you can't afford to get the team wrong.