This is how we achieved true CI-CD in our project in 2015
Year 2012: Our FooBar product has following characteristics
Year 2015: Our FooBar product has following characteristics
This article is all about things we did between late 2012 and 2015.
Disclaimer: The article is a practical account of what we did and shies away from creative pursuits.
Step 0: Mindset Change — Most important pre-requisite
Trust me THIS is the crucial and most DIFFICULT step in the DevOps journey.
Developers and leaders cannot embark on this journey without a mindset change. Everyone should place unadulterated faith in the DevOps philosophy and rigorously work towards it.
People who have spent considerable amount of time like 15+ years or so are habituated to working ‘in a particular way’ — classic waterfall, agile, XP or ‘scrumfall’ (combination of waterfall and scrum), and having monthly releases or so. In my experience, changing this attitude is the hardest. More often the comfort of working in the old-way and general reluctance to change is the biggest hindrance. It is in these times when leadership ecosystem need to step up and lead by example. And this can happen only when they themselves develop solid understanding, and demonstrate agility and acceptance to new ways of working.
Leaders need to understand that DevOps is a JOURNEY and that it would be folly to expect magic overnight. The end result of this journey should be crystal clear in their minds.
Note: Here leaders are the key decision makers like VP Engineering, CTO, Head of Engineering and the ilk.
Step 1: Unit Tests — Often Ignored or significantly misunderstood area
When we aspire to release ‘on-demand’ to production, clearly we cannot do things manually. Doing automation and more automation holds the key. It holds true in case of new code as well. Practices like TDD should be followed religiously by the engineering team to make sure that every bit of code is covered with a test case. We started by introducing ‘gates’ to make sure that any new code added meets 80% coverage to begin with. Then gradually we increased the percentage. Also, whenever a P1/P2 issue was reported from production, we first reproduced that issue via test case. If no test case existed, we wrote a unit test case first, reproduced the issue and then added production code to make test case pass. By doing this, we gradually increased the coverage of code. Having a robust set of tests achieves 2 objectives -
a) Unit Tests act as a safety net in case undesired changes are made to other parts of the application. In such situations, our unit tests will fail alerting us of undesired changes.
b) In a true real time release pipeline, the tests will help us to verify new features and release to production like the way it happens in Toyota Production system
Edit 1: We used JUnit framework to write unit tests. Other options can be NGUnit, ScalaTest (for Scala) or Testing package (for GO-lang).
Edit 2: We used Selenium WebDriver and Protractor for Integration Tests since some of our front end services was in Apache Wicket and other was in Angular JS.
Having Code and Unit Tests together need NOT be TDD.
Read this sentence again and again until the meaning is clear
Note: Refrain from ‘hacky’ mindset to make tests green :-)
Step 2: Discipline — its hard but mandatory. No substitute.
In order to realize the shared dream, we laid down some ground rules that every developer should adhere to-
a) Encourage frequent checkin of code — minimum once a day. No loss of work due to inadvertent hardware failures.
b) No branches. We enforced Trunk-Based-Development and everyone worked on the master and made sure that no one left home with the master being RED. No issues of merging and re-basing.
c) If you use Agile, make sure to create stories as granular as possible. This forces developer to think-through all the minute details and brings clarity to the work at hand. No surprises in the end.
d) While following TDD, if at any point developer is unable to write test case, that is a big RED Flag which most likely means that the requirement is not clear. No point in progressing.
Step 3: Code Inspection Tools — some helping hands
To avoid surprises at the end, its best practice to allow only clean code in the repository. This mandates to use static analysis tools like — SonarQube, PMD, FindBugs and Veracode. You might have to refer to language-specific tools like Infer (developed by Facebook for Java,C,C++,etc.) or RuboCop (for Ruby).
At times you may find tools like CheckStyle that enforces uniformity in style of code written by all developers in the group useful. Any non-conformance to this style may be flagged as build error.
Tools like Veracode do static and dynamic analysis of your code with security as focus. I would highly encourage engineering teams to adopt Shift-Left-Approach as far as security is concerned.
Step 4: Use of build script — delegate the mundane task to a script
We wrote several scripts for mundane tasks like deleting some temp folders or copy files from A to B or taking some backups. Try to employ tools like Maven, Ant, Bash or Powershell for such purposes so that you can check-in the scripts.
Step 5: Pipeline — make your production line
As we worked on the features and tests of various kinds (Unit Tests, Integration Tests, Smoke Tests), our next step was to prepare a pipeline that released our features to production with no manual intervention. Having strong affinity to Atlassian suite of products, we chose Bamboo to build the pipeline. We ran the tests in the following environments
When code is pushed to repo, it first goes through the checkstyle rules. Any issues in this needs to be fixed by the developer. At the same time unit tests can be run by developer locally and make sure coverage is passing. Once all this is green, the code is promoted to Dev where the static analysis, unit tests and code inspection tools are run. If there are no issues, the build is promoted to QA where Integration suite is run. If there are no issues, the build is promoted to Stage where Smoke and UAT are run. If there are no issues, the build is released to Prod where a brief Sanity is done to ensure everything is intact.
Some tools like Azure DevOps (aka VSTS) provide several mechanisms like manual approval for promoting the code, pre-release checks and post-release checks that can be used for refinements.
Step 6: Feedback System — show me some action
We added the tests, put-in the tools like Checkstyle & SONAR and created the pipeline. But how do I ‘visualize’ all the action? Here came the Feedback system. We used the services’ wallboard available in Bamboo to check the real time build status of the services. In fact we projected that on a big TV screen in the work area for everyone to get immediate feedback. In addition, we also sent out build failures via emails to the respective members. Both these feedback systems helped the team immensely in fixing bugs and keeping Master Green (as discussed above)
Today, you can have Slack or Teams bots to relay the build status if you wish. And don’t restrict yourselves to build status. Experiment and automate things :-)
Note: Please DO NOT use these feedback systems as a tool to judge the capability of your engineers. Strict NO-NO !
Step 7: Continuous DB integration — sometimes its easy to forget about DBs
As you work towards your dream of building most sophisticated devops pipeline, there may be times when you might have to rollback a release. Believe me. Even in the best of pipelines, rolling back a release is still a possibility. As engineers we need to have a plan for this.
How do we rollback a database?
We had written a custom solution. Whenever there was a DDL/DML change associated with code, we wrote the equivalent rollback portion. This rollback portion was applied if we ever had to do a rollback.
Today there are tools like Liquibase that version the database script. Think of using tools like Liquibase.
Step 8: Feature Toggles — hide-and-sick game
When teams are in agile development, the typical sprint duration is 1/2/3 week(s). We used 1 week. More often the feature is never ready in this 1 week and usually spans several months. How could you then create a pipeline and release half-cooked feature? Welcome to Feature Toggles.
Like db rollback, we created a custom solution for Feature Toggles. We had a features table where we defined features with True/False values based on which the UI reacted. We coded behind the feature flag then.
Today, there are specialized tools for Feature Toggles — Split.IO and Launch Darkly to name a few. I would highly encourage teams to adopt off-the-shelf tools to realize the dream sooner. Of course do the necessary due diligence.
Edit 3:
Business Impact
Adopting the DevOps mindset and implementing a true CI-CD pipeline had impact on multiple aspects of the company -
- The biggest impact was the process and operations-related. Nobody thought of weekly releases anymore. Feature-trials for customers was available even before their release. We could gather customer feedback well ahead of time and if any feature did not work, it was easy to discontinue it
- Employees saw a tremendous boost in morale. Nothing encourages employees more than the release/usage of their features in production. DevOps helped meet this goal. No employee left owing to quality-of-work
- We were able to deliver way ahead of our competition. This helped us win new deals with OEMs and in some cases renew old deals.
This is a classic example of how technology enabled business to flourish. Never again they were thought orthogonal to each other.
Conclusion
All these steps depict the practical problems you might encounter and the potential solutions to those problems in your DevOps journey. I do not prescribe that this is the only way to go forward. There could be other alternatives out there in the world. It simply attempts to share one probable way.
In short, these are all the ingredients towards cooking a tasty DevOps recipe.
The article mentions the year simply because things might have matured today and there could be better ways of doing certain things.
Congratulations if you made this far ! Do consider giving a few claps if you liked my article. It encourages me to write more.
I leave you with Adidas DevOps case study on how they went from 6 weeks release to releasing 3–4 times in a day — Link
How did you undertake the journey? Write down in comments. I will be happy to read and respond.
