The Last Mile To Production, Part 2

I’m implementing Camunda 8 at a customer, and the hardest part isn’t modeling or coding. It’s keeping acceptance high when changes keep coming.

In part 1, I introduced compatibility as a contract between models, mappings, and runtime code. But rules on paper are not enough.

This part is about enforcing compatibility with automation. Our target setup has three pipelines:

• A connector pipeline that validates connector code and deploys generated templates to Web Modeler.

• A runtime pipeline that runs the latest runtime code against deployed models and deploys and tags code if tests are green.

• A model pipeline that runs the latest models against deployed code and deploys and tags models if tests are green. If it fails, it reruns the tests against the latest runtime code and, if that works, deploys runtime and models together.

We’re currently in the middle of implementation, in the first stage the pipelines check changes against the latest code or model version. The rest will follow in the next weeks. But the theory behind it is still relevant.

Most late surprises are not BPMN errors. They are environment and ownership problems. Most common cases are:

• the runtime code changed and the model didn’t adapt

• the model changed and the runtime code didn’t adapt

• connector secrets are not working

If you only have one environment, every deploy is a gamble. If you have staging but people can still patch things in place, you end up with drift and a false sense of safety.

At the customer, coding and modeling both happen in dev. That keeps one place where new ideas are allowed to break.

Right now we only run two clusters, dev and prod. That works for now, but it keeps the distance short. You can get away with two clusters, but you still need one place to break things before testing starts. We are thinking about a third clusters:

• Dev for fast iteration.

• Test or QA for integration and business testing.

• Prod for stable operations.

And we try hard to prevent changes in test and prod. Modeling happens in dev only. In test and prod, Web Modeler access should be limited to admins for analysis, not editing.

Note: We keep clusters separated by having different stage organizations. Otherwise we’d need to maintain manual RBAC (resource based access control) rules to prevent access to certain models or cluster. This might change with future releases of Camunda 8 and its API.

The first shift is treating models like artifacts, not like “the thing in Web Modeler”.

Camunda’s Web Modeler can sync models into a Git repo. Once they’re in Git, you can handle them like code:

• Version them via commits.

• Test them in CI.

• Promote the same model version through stages.

This matters because a BPMN deployment is an artifact. If you redeploy from a different source in each stage, you are not promoting. You are rebuilding.

The same applies to connector templates. If you generate them in your connector repo, they are already versioned in Git and can be deployed via a pipeline. If you create them in Web Modeler, sync them to Git.

Contracts drift unless they’re checked continuously. For us, process tests are the gate: not unit tests, not “it compiled”, but end to end tests that deploy a model, run a realistic path, and assert variables and mappings.

Like I mentioned in part 1: compatibility breaks in both directions, so the process tests must run in both directions too.

Trigger: a code commit in the connector repo.

Goal: prove the connector code is valid and the generated templates are valid and safe to publish.

High Level Steps:

• Check out the latest code.

• Run unit tests.

• Generate connector templates.

• Copy generated connector templates to Web Modeler.

• Publish them to a project or organization.

If you model templates directly in Web Modeler instead of generating them from code, they belong to the model’s lifecycle and should be handled by the model pipeline.

Trigger: a code commit in the runtime repo (worker updates, connector updates, connector runtime config).

Goal: prove that new code is compatible with what is already deployed in the target stage.

High level steps:

• Check out the deployed model baseline for that stage.

• Deploy those models into your test engine (Camunda offers an in-memory engine for that).

• Start the current runtime with your changes.

• Run process tests.

If this fails, do not deploy the runtime. The new code is not compatible with the models deployed in that stage. You need updated models now, and that handoff goes to the model pipeline.

In the full setup, if this succeeds, deploy the runtime to the next stage and update the runtime stage tag.

Trigger: a commit in the model repo, usually from Web Modeler sync.

Goal: prove that new models are compatible with the runtime currently deployed in the target stage.

High level steps:

• Check out the deployed runtime baseline for that stage.

• Deploy the new models into your test engine.

• Keep the runtime version pinned to the deployed baseline.

• Run the same process tests against that setup.

In the full setup, if the first test succeeds, deploy the models to the next stage and update the model stage tag.

If this fails, the model change is breaking against the deployed runtime. Do one additional step:

• Check out the latest runtime baseline for that stage.

• Deploy the new models into your test engine.

• Run the same process tests again, this time against the latest runtime code.

If the second test succeeds, you now have a compatible pair. In the full setup, deploy the runtime and models together and update both stage tags in one combined deployment.

If this still fails, there are real errors left to fix. Either the runtime change needs adjustment, the model change needs adjustment, or a migration step is missing.

The tests do not need to be huge. They need to be intentional and run against the real runtime.

At the customer, process tests run against the actual workers and connectors. We usually mock downstream services to force edge cases.

For each critical process, we aim for:

• A realistic happy path with stable test data.

• Edge cases that tend to break in production (missing fields, unexpected status codes, timeouts, rate limits).

• Expected failure paths that prove retry and incident behavior.

• Coverage for all relevant paths, not only “the process ended”.

• Assertions on variable shape at key boundaries, not only outcomes.

Testing out of the box connectors is harder. The Camunda testing framework does not cover every scenario yet, so you may need extra harness code.

This is where staging helps. If you run tests only in dev, you miss the things that break in real clusters. If you run them in test or QA, you catch drift earlier.

Once the relevant pipeline is green, promotion should be easy. Today that still involves manual glue. The target is to move the same proven artifacts from dev to the next stage.

We are in the middle of building that full promotion flow. Prod is different. We do not deploy to prod continuously. Production deployment is planned and manual.

To keep this honest, you need to know what is running where. That is why stage tags matter, even if the full flow is still being built.

In the target setup, after a successful deployment, the responsible pipeline updates the stage tag. For a combined deployment, both tags move together. Then every compatibility pipeline can resolve refs/tags/latest-test to a specific revision and test against it.

This also helps during incidents. When test fails, you can answer quickly: did we change the model, the runtime, or both.

Of course a detailed branching strategy is also possible, if you’re not into using Git tags. You can use whatever helps pinning deployed versions down.

Staging is not overhead. It’s the price of predictable acceptance.

• Treat models as artifacts and promote the same versions through all stages.

• Use Camunda process tests as the compatibility gate, and run them in both directions, and against different baselines.

• Prevent drift between connector code and element template by generating your templates and publishing them to the Web Modeler for everyone to use.

• Tag deployed versions per stage so everyone knows what is running where.

• Lock down access to test and prod so trust does not decay through drift.

If you already do some of this, pick one gap and close it this week. A single stage tag and one process test can already make acceptance feel less random.