How to establish quality gates in a Microservices architecture

Note: This is in continuation to the blog – Best testing strategies in a Microservice architecture – contributed by our SME, Chandra Kiran Mamidala. 

Let us assume that you have implemented the strategy for testing microservices as discussed and written all the test suites as suggested. Where do you run them? How do you make sure all your test suites are executed at right time and at right place (environment)? When do you run your regression packs? Do all these test suites fit in CI/CD? 

I will try to address most of these questions with an approach that we call “Quality Gates model as a strategy to test microservices”. This approach involves construction of quality gates into the CI/CD pipeline for which we take the help of our development teams, build teams, deployment, and infrastructure teams. As the construction of quality gates is a test architecture activity that requires highly technical QA resources, it is essential that all the members across different teams work very closely. More importantly, you should first propose the Quality gates strategy to your stakeholders and obtain the required support from them. 

The following flow diagram offers a bird’s eye view to the overall architecture with Quality gates established within the CI/CD pipelines. I will explain each stage of the Quality gate strategy one by

one: 

Quality Gate 1: Execute Unit Tests 

At this stage, developers commit and push the code to their branches where it undergoes reviews and merges. When the code is merged to a master copy, we ensure that a build pipeline is triggered to run all the unit tests against the master copy. 

Things to consider: 

• This must be a part of the CI process 

• A CI environment is the best place to execute 
• Mock data is often used in building the tests 
• Test component is microservice (scope is limited to a microservice) 
• Junit and NUnit are the best examples of tools/frameworks 
• Recommended exit criterion is 100% pass rate 

Quality Gate 2 (Optional): Static code analysis/Code coverage 

This is an optional quality gate that must be a part of your CI process, if you choose to introduce it. The gate is not enforced by QA. It is, in fact, a collaborative decision from the involved teams (build, development, QA etc.). People usually go for commercial solutions that offer great reporting on number of lines, blocks, and methods covered and to analyse the code quality, unit test coverage, and violations. 

Things to consider: 

• This must be a part of the CI process 

• A CI environment is the best place to execute 
• Test component is microservice (scope is limited to a microservice) 
• SonarQube and Kiuwan are best examples tools/frameworks 
• Exit criteria is as per the tool or team recommendations but surely without blocker/critical issues and with fewer violations 

Quality Gate 3: Publish the Consumer Contracts 

This is the first step of contract testing where you will generate the contract as a consumer and publish them to the provider. 

Things to consider: 

• This must be a part of the CI process 
• A CI environment is the best place to execute 

• PACT is the best tool/framework, it acts as a mock 
• Exit criterion: It must undoubtedly achieve 100% pass rate as you cannot let the provider verifying the previous versions of contracts 

Post this Quality Gate, the Docker images or the build and artefacts for the service, and its dependant modules are generated so that the build/image is used to deploy into various environments. CI can be automated such that the Docker image/build is automatically deployed to the other environments. 

Quality Gate 4: Execute Integration Tests 

Once you have the build(s) of a microservice and its dependencies, it is good to deploy all of them for getting integrated into a Dev environment or Integration Test Environment. Here QA teams must ensure that the infrastructure team helps with the setup to run Integration Test Suites. 

Things to consider: 

• This must be a part of CI to CD transition process 
• A Dev or Integration Test environment is the best place to execute 
• Any framework that allows testing these integrations can be used 

• Exit criterion: If some of the integrations fail then a manual intervention is needed before pushing the build to other environments 

Quality Gate 5: Verify the Consumer Contracts as a Provider 

As a provider, you will verify the contracts published by all of your consumers. These contracts are certainly not the ones published by this microservice. They are published by the consuming microservices (internal or external). 

For example, if this microservice is a consumer for 3 other services then they would have published and shared their contract with your team by now and you ought to verify them. 

Things to consider: 

• This is a part of the CD process 
• A Test or Component Test environment is the best place to execute 
• PACT framework allows you to verify the contracts 
• Exit criterion: If some parts of the contract verification fail, you do not need to stop the deployments.  

There are two possible reasons for failures: 

• Due to an expected change to the schema or API of the provider which was not communicated to its consumers properly: The consumers published old contract whose verification failed. In this situation, you can inform the consumers of the change and just continue with further deployments and propagation towards releases. 
• Due to an unexpected change introduced to API (possibly developer’s mistake): In this case, it is a potential threat to its consumers as they may face unhandled exceptions or ‘resource not found’ errors at their APIs and UIs, thereby causing customer complaints and escalations. 

Quality Gate 6: Execute Component Tests 

With the above quality gates propagating into continuous delivery process, it is your best change to perform a full functional testing of a microservice. Setup some automated jobs that executes all of your component test suites by hitting the public APIs provided by your microservice. This is like a typical web service testing. 

Things to consider: 

• This can be a part of the CD process 
• A Test or Component Test environment is the best place to execute 
• Any framework for API testing (SoapUI, ReadyAPI, REST-assured etc.) can be used 

• Mock services and mock data are the key factors for this execution 
• Exit criterion: 90–95% is enough without blockers and critical issues to propagate to other environments. This includes all the edge cases and failures, many expose very minor trivial defects. 

Quality Gate 7: Execute Component Performance Tests 

Component performance tests are nothing but the performance tests on a microservice. Remember that the components are microservices themselves. It is a good practice to quickly analyse the performance of your microservice alone before you concentrate on the entire solution. 

Things to consider: 

• This can be a part of the CD process 
• A Component Performance environment is the best place to execute. If you do not have resources, then just use your Test environment but remember that the performance measurements where the mock services are involved are unrealistic. 
• Tools such as Meter, Locust, Load Runner, etc. are good examples 
• Exit criterion: Just follow the usual standards of performance testing 

Quality Gate 8: Execute End to End Tests 

With the results from all the above Quality Gates, it gives us a greater level of confidence that our microservice performs well when it is alone, but we are not sure how it works when integrated with all of the microservices to form a complete system. This is where we must setup a job to execute our e2e tests with either a manual trigger or an automated one. As the execution times are long, it is better to choose when you want to trigger this. 

Things to consider: 

• This is a part of the CD process 

• A staging environment is the best place to execute. If you do not have resources, then just use your UAT but remember that the other stakeholders may be working on this at the same time. 
• This environment is production like and expects that all real dependencies are connected and integrated 
• UI framework in combination with API test libraries and tools with data base verification capabilities are useful 
• Exit criterion: It is up to you to define a criteria. You expect many failures as it involves UI and you may want to manually do a test failure analysis before pushing to UAT or Production. 

Quality Gate 9 (Optional): UAT 

This is optional for few organizations. You may give your customers a chance to try their hands on the completely integrated system. The stakeholders may come back to you with their observations, trivial issues, and sometimes with major issues. Just keep listening to them and try reproducing before you log them into your defect management system. 

This means as a QA team, you are still active at this quality gate ensuring the best quality is given to your users. 

Quality Gate 10: System Performance 

Understanding the performance of individual microservices is not sufficient. We must understand the response times, latency, throughput etc. when the complete system is working together with all of its internal and external dependencies.  

It is a good idea to approach specialized performance testing teams for this activity. 

 Quality Gate 11: Production 

What do we do here? QA team’s involvement is still there at the production releases, but it is minimal. Just do a quick sanity check. A manual approach is recommended here before you hand it over to the real users. 

 To Conclude 

Note that the Test First Approach and/or the Test–Driven Development (TDD) approaches in combination with Behavior–Driven Development (BDD) gives the best visibility and the best results in testing microservices. 

Microservices follow a different architecture and have a lot of integrations with other microservices within an enterprise as well as from the outside world. By implementing Quality gates, you can significantly increase the confidence in the quality of each of your microservices, thus, leading to a high performing overall application.