Architecture and Benefits of Continuous Testing Framework

November 06, 2018 

Architecture and Benefits of Continuous Testing Framework

What is Continuous Testing?

Continuous Testing is a procedure of testing early, testing regular, test everywhere, and automate. Continuous means undisrupted testing done on a continuous basis. In a Continuous DevOps process, a software change continuously moves from Development to Testing to Deployment. The Continuous Testing process helps us to find the risk, address them and improve the quality of the product.

Continuous Testing is the procedure of executing automated tests as part of the software delivery pipeline to acquire immediate feedback on the business risks associated with a software release candidate as quickly as possible. Continuous Testing involves executing the right set of tests at the right stage of the delivery pipeline—without creating a bottleneck.

How Continuous Testing Works?

Developers write the code and commit, it goes to version control/SCM. It is the first stage in DevOps. It manages the code written by developers.

Next phase is Continuous Integration. Continuous means rather than waiting for all the developers to commit code and then build and test it, in Continuous Integration approach, the code is merged into the main branch. Then use Jenkins/Git to create an build, and after that, test code in a virtualized environment. It saves a lot of time in moving the code from Dev environment to Test environment without waiting for all the developers to commit.Performing Continuous Integration is of no use, if not testing continuously.

So the next stage is Continuous Testing, where QA automates test by writing scripts like Selenium, Robot Framework and run it through Jenkins/Git in a virtualized environment.

If the test fails then stops here and sends back the issues to the developers. If the test passes, move ahead for the Continuous Deployment.

Continuous deployment stage we have two parts -

  • Configuration management
  • Containerization

Consider there are five servers are up and running having same libraries, tools on them. Many of them are from an older background having shell scripts to do this job. Now to have exact configuration on 100 of servers, using shell scripts, it will be pretty much complicated. Therefore, require a tool to automate the job. Puppet is the most popular tool for Configuration Management.

The second part of Continuous Deployment is Containerization, in this stage as create a test environment to run application and test. It’s used for testing purpose; provides virtualized environment to run script like Selenium scripts to test application. 90% of organizations use Containerization only in the testing stage, only 10% of the organization use on production.

In 2015 Docker 1.9 came stable to run on production as well. For example, to run some application on production for the shorter duration, quickly create a container on prod and run those applications in that container, and when don’t need those applications anymore, destroy the container. Deploy application; monitor to see whether the application is performing as desired and the environment stable.

Continuous Monitoring determines when a service is unavailable and understands the underlying causes, and reports back the issues to the developers. Nagios is one of the most popular tools used in Continuous Monitoring. So this is how all stages are connected.

Benefits of Continuous Testing


Continuous Testing reduces application-bound risks

Faster turnaround time is a crucial benefit expected from Continuous Testing, but ensuring a risk-free output is similarly essential. At whatever point the code changes, there is a right amount of scope for errors. It prompts to a severe effect on the application or software and increases costs while rectifying towards the finish of the development process.

Continuous Testing helps to bring consistency

Continuous Testing is made viable with Test Automation, in keeping consistency with the testing procedure. It helps to keep each system and subsystem on a consistent stream with Continuous Integration. It helps in giving a configured environment that is robust and sustains the required configuration for comparative tests.

Continuous Testing enable faster releases

Continuous Testing and Continuous Integration executed to allow quicker and frequent releases. When tests are automated, code is tested thoroughly and always, enabling the development teams to convey assured results even at speed. With Continuous Testing, teams release cleaner code on a reliable basis, at speed.

Continuous Testing improves Test Coverage

Do you require test coverage for code? Continuous Testing accomplishes more test coverage, as the tests automated and executed right from the earliest of the development process. It guarantees to cover every potential failure, and evaluates all the features. Maximum Test Coverage is a great relief for the testing and development folks.

Continuous Testing enables more transparency

Considering that testing is continually done and right from the earliest starting point; it empowers greater visibility for the testers and the developers. It enhances value to the overall software/application development process. Moreover, the test automation tool enables accurate reporting across the dashboard, increases the straightforwardness of the process. In the end, the team is kept informed on every error, success, and failure of the tests.

Continuous Testing cuts downtime took for Code Review

Continuous Testing empowers quick feedback and encourages more coordinated between the testing and developments teams. In this way, Continuous Integration and Version Control System cooperate and decides when to start a merge request. The system estimates when the tests pass successfully, and related requirements met. It drastically decreases the time taken for Code Review.

Why Continuous Testing Matters?


Early Defects, Less Cost

Continuous Testing helps in the initial authorization & determination of software defects/bugs. As early bugs will fix; the cost to the company gets reduced. A suitable testing process in an initial stage of production allows faster recovery in case of unexpected defects /bugs in the product.

Easy Automation

A Continuous delivery model requires Continuous testing, possible only through test automation. It coordinates Quality Assurance efforts to balance the speed of DevOps, assisting developers to deliver unique software features quicker.

Reduce Testing Efforts

Automation tools reduce the testing efforts. With the focus on Continuous Development, automation tools serve the testing team and overall product development.

How to Adopt Continuous Testing?

Embed quality in applications; teams add more modern testing practices in the form of small quality checks performed throughout the application pipeline. It enables small sections of code tested continuously. Test automation not always the first thing to be tackled. The reason is that even after API tests and GUI tests automated and integrated into a Continuous Integration agent, those tests have dependencies — test data, interfaces, and environments — that have to be satisfied before executed.

Virtualized Environments

Continuous Testing means testing more frequently. That means hitting multiple environments more frequently. And that represents a problem, a bottleneck if those environments are not available all the time. How to coordinate testing through multiple environment owners that may not always keep their environments up and running for you to test against?

Virtualizing those environments allows to test code without having to worry with areas are not changing (i.e., other systems and environments).

Test Data Management

Have the right data and the appropriate diversity of data for positive and negative scenarios. Diversity in production is difficult— that is critical. Therefore, synthetic data generation is the way to go to achieve Continuous Testing with the highest levels of confidence having no personally identifiable information is at risk in the data. Also, cannot bring data from production, condition it and provision it at the speed, application pipeline requires.

Test Automation

In an entire application pipeline, today’s scripts fail due to things not related to the application code, no one trusts the results. Need reliable test automation to adopt Continuous Testing.

Pipeline Orchestration

The backbone, everything tied to it, integrated with automation suite. Understand how it works, how to interpret results and how to make it scalable. It sets up Continuous Testing in a manner to integrate Continuous Testing attributes within a pipeline. Make sure it’s transparent, and everyone has full visibility into what’s run through the pipeline. It’s an automated workflow tool which runs all of automated tests and fully integrates with code deployment activities as it moves through the pipeline. As part of any DevOps adoption initiative, it is unrealistic to expect to get to Continuous testing without the reliability and speed of a standardized and automated pipeline.

API Testing

It enables the alignment of the testing pyramid. Organizations strive to test as much as the unit and API levels as possible and minimizes reliance on UI testing. To adopt Continuous Testing, embrace the concept of the testing pyramid properly, strengthen Unit and API testing, and reduce reliance on UI testing, especially for business logic testing.

Performance/Load Testing

Even when applications functionally useful, must radically change the way think about Performance Testing. As shift testing to the left, testing earlier in the life cycle, with less application and infrastructure components, means the tests are smaller by nature. But they are also much higher in volume. Make that capability accessible to everyone across all Agile teams. That means all developers and testers alike create a performance test and run it by themselves, without having to send any requests to anyone.

Security Testing

Developers need to be clear on what the expectations are around the state of the code to be allowed or denied to be released. They must receive proper training and tooling to measure how secure their code is. Education of developers must be ongoing, as the application security area is always evolving. The application pipeline must be set up to run the static analysis automatically, dynamic analysis and software composition analysis, so that any security issues are caught and highlighted, then those must be used as an opportunity to make developers better. Security testing needs to happen at every step of the life cycle.

Acceptance of TDD And BDD

Take the acceptance criteria for each user story and create the tests to ensure those are met. This keeps testing focused within the sprints and ensures developers develop what the business expects. Over time, teams will define much more detailed acceptance criteria, which requires tests also to be more comprehensive.

Automated Test Generation

The activity of manually designing and writing manual tests or automated test scripts is a bottleneck. Automatically generate the new acceptance-level tests for manual and Automated Testing at the beginning of the sprint, developers start checking in and merging the first working pieces of their code, start running tests in an automated fashion. This gives the ability for developers to get immediate feedback on the quality of their code as they check it into source control.

Requirements Engineering

Include all SDLC stakeholders into the Continuous Testing journey. Find a better way to communicate and collaborate on the requirements. Include requirements engineering into Continuous Testing initiative, the smoother the Continuous testing journey will be, because team members work with the same understanding from the beginning.

Feedback Loops

These are critical to successful Continuous Testing. Dashboards in Real-Time, must be automatic and entire team must have access. Feedback loops across the entire SDLC, not just production, as a compass to navigate Continuous Testing transformation.

Best Practises of Continuous Testing


Treat Test Code as Code

Develop test code the correct way from the start, mean the difference between a high-value test suite and a time-consuming, flaky and wasteful one. To empower Continuous Testing, treat test code the same as production code.

Employ a Risk-based Test Automation Strategy

Incorporate a methodology into the Continuous Testing procedure to identify, evaluate and mitigate the causes and consequences of terrible user experience. Consider time, value, affect, history – and don’t ignore developers’ gut instincts.

Provide Fast Feedback

Continuous Testing creates vast amounts of data that give testers and developers new insight into explicit issues and provide managers with dashboard-level perspectives for “go, no-go”essential leadership.

Generate and Maintain Test Data

Banks, insurance companies, healthcare suppliers and anyone with a mission-basic application need to test against real-world conditions. Provide QA testers and developers with high-esteem test data and guarantee that it evolves with the applications.

Ensure Access to a steady Lab and Test Environment 24/7

A steady lab implies that all platforms associated with it, together with supporting test environments, 3rd party tools, and APIs, working every minute of every day with close to zero downtime. Cloud is highly recommended for this purpose.

Improve the relationship between tester and each developer

Keep the teams small, encourage inter-team collaboration, and make reports easy to access and share online.

Make automation a priority

Do not remove all manual testing tasks, but adopt an “automated first” mindset. Focus that on areas that will be run repeatedly. In essence, set up the plumbing first to let the water flow later without any worry or concern about leaks. Map out SDLC and identify automation opportunities.

Get small Design

Break work down into smaller increments that are easier to test after coding and design. This makes easier to automate the tests, and much more readily deployable.

Keep track of everything

Use metrics and set pass-fail criteria. Continuous Testing immediately identifies if things are working or not.

Get the right continuous testing tools

To develop, test and analyze continuously. Pick the best-of-breed tools that work together in a way that incorporates easily into the working environment. Choose tools that have community-based support where everyone shares challenges, solutions and exciting use cases.

Develop a system to display results

Do deep dives into results, to know if code is working and where the gaps are. Define KPIs and acceptance criteria and make them quantifiable. Create dashboards to track the KPIs, including a baseline and subsequent changes.

Best Continuous Testing Tools

  • A planning tool — Wekan board and similar planning tools provide functionality to track user stories, epics, and tasks.
  • A version control system — Version control systems, like GitLab, allows development teams to track, collaborate and communicate on versions of the source code.
  • A build tool — Gitlab and similar tools to build and run Unit and Functional automation tests from the source code.
  • A testing tool — Tools like Robot framework, Selenium insert the tester  into this Continuous Delivery pipeline providing test validation against Wekan board issues and connect into source code repositories to streamline manual to automated test scripts.