What is a Pipeline?
A Automated Testing Pipeline is a group of jobs or tasks that are executed in stages also called batches.
Pipelines comprise –
- Jobs that define what to run. For example, the compilation of code or test runs.
- Stages are defining when and how to run. For example, that test run after compilation of code.
- Jobs in a stage are executed in sequential/parallel by Runners if there are enough concurrent Runners.
If the jobs in a stage –
- Succeeded, then the pipeline moves to the next step.
- Fails, the next stage will not get executed.
Stages in Automated Testing Pipeline
As an example, imagine a pipeline consisting of four stages, administered in the following order –
- Build A job called compile.
- Test – Two jobs called test and test2.
- Staging – A job called deploy-to-stage.
- Production – A job called deploy-to-prod.
Delivery pipeline shall meet the following requirements –
- source code in Gitlab
- there is an automated mechanism triggered by a code change that
- builds an app package
- deploys an app on local or in the cloud (e.g., Heroku)
- runs automated tests and offer a run report
Process of building Testing Pipeline
Step 1 – Write the script and run locally
First of all, we will write test scripts to let our first manual test run automatically. Selenium is a well – known web automation tool.
With the idea of a basic test model, we can quickly identify the three major components in the script –
- Set up
- Run test case
- Tear down
We can run various test_cases with setUp before the test and tearDown after the test.
Step 2 – Setting up a runner with your testing environment
Gitlab runner can be installed in three ways to set up a runner with your testing environment.
- Use a Docker
- Download a binary
- Use a repository for rpm/deb packages.
Step 3 – Set up an image with the testing environment
Build an image with the testing environment
Test running requires a clean environment. To create a clean environment, we don’t want to establish up a real machine each time and wait for hours to install all the necessary software. The concept of the container helps.
Docker helps you build an image of your testing environment.
The image includes all the software that needs to be pre-installed and run like a virtual machine on that container.
You can create a new container with Docker and pull the same image every time, whenever you want to start over from your default environment.
Forex – We want our image to pre-install the following to perform our test with the Selenium Python client –
- Google Chrome
- Chrome driver
Create a Dockerfile, build the image and upload it to our Docker Cloud.
Then you have an environment ready for performing the UI tests.
Setting Up GitLab CI for Automated Testing Pipeline
Create a job template for your product
When commit/new codes are pushed to the repo, will look for directory and triggers and build according to the configured settings.
GitLab Runner uses this file to manage the ‘project’s jobs which define how the project should be built.
image: joyzoursky/python-chromedriver:3.6 before_script: - pip install -r requirements.txt stages: - test - report test_suite_1: stage: test script: - python -u test_suite_1.py test_suite_2: stage: test script: - python -u test_suite_2.py send_report: stage: report script: - python send_report.py
GitLab looks for job template and then installs the required packages and then starts the process.
It pulls the environment image in the first line from joyzoursky / python – chromedriver:3.6-xvfb in this script. Then it installs the required packages, sets the required variables, and then it starts the process.
In this example, there are two stages of the build process: test and report. The jobs at that stage will be run simultaneously at each stage. In the same step, you can define criteria if they can move in sync.
Step 4: Run and report periodically
Pipelines can be set up according to the user requirements. Tests can be made run on every build triggered on the pipeline or can be run periodically by setting up Cron Jobs and scheduling the time by cron syntax.
Two different ways to integrate this into our system –
- Running the tests every time a new code gets pushed to the project.
- Running the tests periodically.
The nightly run requirement depends on the team where they want to run all the tests (smoke & regressions) when there is no deployment. Test runs at night will yield useful test reports. Executing the tests on each change of code as part of the CD pipeline. The reason for this is that E2E tests are slow to run.
We do not want these tests to slow down our pipeline as it will delay our process and cycle and will affect pull requests, merges, and deployments to different environments.
We wanted a set of core End to End tests we can run regularly that let us know if anything is broken or off. So we can run these tests on a nightly basis via a Jenkins cron job.
Reporting Test results can also vary based on the teams, depending on their requirements such as intimating on the slack channel, generating HTML reports, saving test runs in DB and displaying them on dashboards. We can use Slack real-time messaging API to do the reporting so that we can receive notifications in the corresponding project channels.
- Slack Notification
- HTML ReportingAutomated Testing in Agile Enterprise
- Dashboard Reporting
GitLab is one application for the entire lifecycle of DevOps
- Build your application using the GitLab Runners
- Run unit tests and integration tests to check that your code is valid
- Look at the live preview of your development branch with Review Apps before merging it into stable
- Deploy to the multiple environments like staging and production, and support advanced features like canary deployments
- Monitor the performance and status of your application
Automated Testing Pipeline Approach
Setting up an Automated Testing Pipeline makes QA life easier by finding the bugs in the application. Now, whenever code is updated on the production branch or code change in the project, this automated test is triggered and tests are done automatically and generate test reports to notify Developers and QA. Failure results will be pushed back to Slack channel to notify the developers. This makes the automated testing process becomes more manageable.To Adapt Automated Testing as a testing practice we recommend taking the following steps –