Say CIAO to the DEVOps blues: Orchestrated

TL;DR Humans are very bad at repeating a set of instructions, computers are very good at repeating a set of instructions, use computers to ensure everything is called in the right order.

This blog expands on Say CIAO to the DEVOps blues to discuss what an orchestrated deployment is and why you should be doing them.

I’ll start with a true story from a couple of weeks ago.

We provide AppDevOps in a big organisation and as result we work with a lot of third parties. This third party had a ‘simple’ deployment, so simple it didn’t need a pipeline but because we insisted they would supply scripts rather than written instructions, but really how hard could it be, they’d done it multiple times.

So we got the scripts and a readme file and on the appointed day we ran it. Then we called the project sponsor, as they were also the test resource.

The system didn’t work. What had we done wrong? We had run the scripts as requested so back to the third party. After about half an hour they came back to say the readme file was wrong we had to run them in the reverse order.

Wait a minute how come it had worked in the test environment? They didn’t know but after a while their developer said that he hadn’t realised that the order was important.

Despite pressure to ‘just fix it’ we ran both scripts in the test system in the ‘right’ order, got the tester to verify them and then ran them on the production system.

So a fifty fifty chance and we got it wrong.

The lesson is don’t orchestrate a deployment yourself, as you’ve seen even a two step process can go wrong.

What we do is:

• Create an environment orchestration pipeline that runs the comprehensive deployment calling other pipelines and scripts as needed.
• Make all environment specific values parameters.
• Create a comprehensive orchestration pipeline that runs your environment orchestration pipeline for your development environment.
• When it works add a call to your environment orchestration pipeline for your test environment.
• When it works add a call to your environment orchestration pipeline for your production environment.

Now we know that nothing is being run out of sequence or forgotten.

This three tier way of working can be summarised as:

• Orchestration pipeline calls
• Environment pipelines calls
• Deployment pipelines

The use of development, test and production means that by the time you deploy to production you know that the environment pipeline does not have anything environmental hard coded in it, because if it did it would have failed in test.

TL;CR Humans are very bad at repeating a set of instructions, computers are very good at repeating a set of instructions, use computers to ensure everything is called in the right order.