Skip to main content
XenonStack

Serverless

XenonStack Blog GitLab CI Automated Testing Pipeline: Enterprise Guide

and | Published | Updated

GitLab CI Automated Testing Pipeline: Enterprise Guide
14:39

Key takeaway

  • Build an automated testing pipeline in GitLab CI: stages, runners, JUnit reports, schedules and merge trains, with the governance choices enterprises need.

An automated testing pipeline in GitLab CI is a set of jobs, defined in a .gitlab-ci.yml file, that GitLab Runners execute in ordered stages every time code changes. Jobs say what to run, stages say when, and if a job in a stage fails the next stage does not normally start, so a broken change is stopped before it reaches staging or production.

For enterprise teams the pipeline is a control point as much as a convenience: it decides what evidence exists before a release, who can see it and how long it takes to get. This guide explains how to build one with GitLab CI, which design choices matter for speed, cost and governance, and where it fits with wider agentic AI for software testing and the role of automated testing in an agile enterprise.

What is an automated testing pipeline in GitLab CI?

GitLab describes the pipeline as the fundamental component of GitLab CI/CD. It can run on a branch push, a merge request or a schedule, or be started by hand. Its building blocks are:

Two-column diagram mapping GitLab CI pipeline stages (build, test, report, deploy) to what happens: clean environment, fail fast, visible results and governed release
  • Jobs carry out one task, such as compiling code or running a test suite. Runners execute them.
  • Stages group jobs. Stages run in sequence, and the jobs inside a stage run in parallel when enough runners are available.
  • Runners are the GitLab Runner application, which connects to GitLab, picks up jobs, runs them and reports results back.

A typical four-stage pipeline has build, test, staging and production stages. The test stage may hold several jobs (for example unit tests and UI tests) that run side by side, and a deploy job in the staging stage runs only if every test job passed.

Which pipeline type fits which need?

GitLab's pipeline documentation lists several pipeline types. The right choice depends on how fast feedback must be and how much risk a bad merge carries.

Pipeline typeWhat it doesWhen it fitsCurrent status and notes
Basic (stage by stage)Finishes each stage before the next startsSmall projects, simple dependenciesDefault behaviour
Pipeline with needsStarts a job as soon as the jobs it depends on finishLong pipelines where stage order wastes timeDocumented keyword: executes jobs earlier than stage ordering
Merge request pipelineRuns when a merge request is created or updatedReview-time testing before code reaches the default branchDocumented pipeline type
Merged results pipelineTests the change as if already merged into the target branchTeams that see "green on branch, red after merge"Documented pipeline type
Merge trainQueues merge requests and tests each against the changes ahead of itBusy repositories where two passing merge requests can break the default branch togetherPremium and Ultimate tiers
Parent-child and multi-projectSplits or combines pipelinesMonorepos and services that must be tested togetherDocumented pipeline types

How do you build the testing pipeline step by step?

1. Write the tests and run them locally

Start by making a manual test repeatable. Selenium WebDriver drives a browser as a user would, locally or on a remote machine, and is a W3C Recommendation. Every test script has three parts: set up, run the test case, tear down. Get them green on a developer machine first, because a pipeline only automates what already works.

2. Set up a GitLab Runner

GitLab offers two options. GitLab-hosted runners on GitLab.com need no installation, and each job runs in its own short-lived virtual machine, though you have limited control over the environment. Self-managed runners run on your own infrastructure and are available on all GitLab installations, which suits teams that need network access to internal systems or full control. Self-managed runners can use executors such as Docker, Kubernetes and shell. Keep the Runner major and minor version aligned with your GitLab version, and keep it updated if you self-host runners for GitLab.com repositories.

3. Build a clean, repeatable test environment

Tests need a clean environment each time. Containers give you that: build an image with the language runtime, browser and driver your tests need, publish it to a registry your runners can reach and reference it with the image keyword. Pin the image to a version you maintain. The sample from the original version of this guide used a Python 3.6 image; Python 3.6 reached end of life on 23 December 2021 and receives no further security fixes, so base new images on a supported Python release.

4. Define the pipeline in .gitlab-ci.yml

When code is pushed, GitLab reads .gitlab-ci.yml from the repository and creates the pipeline. The illustrative example below runs two test suites in parallel in a test stage and then a report stage. The image tag is an example only.

image: python:3.12   # example tag: pin to a version you maintain

stages:
  - test
  - report

before_script:
  - pip install -r requirements.txt

test_suite_1:
  stage: test
  script:
    - python -m pytest suite_1 --junitxml=report_1.xml
  artifacts:
    when: always
    paths: [report_1.xml]
    reports:
      junit: report_1.xml

test_suite_2:
  stage: test
  script:
    - python -m pytest suite_2 --junitxml=report_2.xml
  artifacts:
    when: always
    paths: [report_2.xml]
    reports:
      junit: report_2.xml

send_report:
  stage: report
  script:
    - python send_report.py

Both test jobs run at the same time because they share a stage. Declaring artifacts:reports:junit lets GitLab show results in the pipeline Tests tab and in the merge request test summary, which compares the source branch with a baseline pipeline on the target branch. Reports do not change job status, so the script must exit with a non-zero status to fail the job. Setting artifacts:when: always uploads the report even when the job fails, which is when you need it most.

5. Run on every change and on a schedule

Run fast tests (unit, integration, smoke) on every push or merge request. End-to-end browser suites are slow, so many teams run the full set on a schedule instead, for example overnight, so they do not delay merges and deployments. GitLab pipeline schedules use cron patterns. The schedule owner's permissions are used for scheduled runs, so if the owner is blocked or removed the schedule becomes inactive and needs a new owner. GitLab also advises staggering schedules rather than starting many at the same minute.

6. Report results where people work

Reporting varies by team: the merge request test summary, HTML reports kept as job artifacts, a Slack or Teams message on failure, or a dashboard fed by stored results. Choose the channel that makes the failing test visible to the person who can fix it, and keep failure notifications short and linked to the job log.

What else can the pipeline test beyond functional suites?

GitLab's testing documentation covers code quality, code coverage, accessibility, browser performance, load performance, fail-fast testing and unit test reports. Code quality results can be imported from your own linters through a codequality report. The built-in CodeClimate-based template is deprecated as of GitLab 17.3 and planned for removal in 19.0, so use the report import approach for new work. Feature availability for code quality varies by tier: merge request findings are in Free, pipeline reports need Premium, and the merge request changes view needs Ultimate.

Review apps add a different kind of check. They create a temporary environment for each branch or merge request, with a View app link in the merge request, so reviewers and testers can exercise the change before merging. Stop them with an on_stop job or auto_stop_in so they do not accumulate cost.

How do you keep the pipeline fast, cheap and governed?

ConcernGitLab controlWhat it gives you
Speedparallel, needs, cacheRun copies of a job at once, start jobs earlier than stage order, reuse files between jobs
Costinterruptible, artifacts:expire_in, rulesCancel redundant runs when a newer run starts, delete old artifacts, create jobs only when conditions match
Flaky testsretry, allow_failureAuto-retry a failed job a set number of times, or let the pipeline continue when a non-critical job fails. Use sparingly so real failures are not hidden
SecretsMasked and protected variables, external secretsMasked values are hidden in job logs but this is not a guaranteed protection; protected variables are available only to pipelines on protected branches or tags. Keep secrets out of .gitlab-ci.yml, which everyone with repository access can read
Scheduled-run accessSchedule owner roleRuns use the owner's permissions, so assign schedules to a role-based or service account that your policy approves
Merge safetyMerged results pipelines, merge trainsTest the combined result before it reaches the default branch

When is a GitLab testing pipeline not enough?

A pipeline proves that the tests you wrote pass. It does not tell you whether the tests cover the risks that matter, whether a skipped or retried job was approved, or why an AI-assisted change was accepted. As coding agents begin to open merge requests and generate tests, teams need a record of the decision behind each change as well as the test result. AgentQA from ElixirData is positioned as automated quality assurance for AI, testing agent outputs and workflows continuously, which is useful when the thing under test is an AI agent rather than a conventional application. ElixirClaw is an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control. For data-heavy systems, see our big data testing services, and for testing agent-based automation see agentic process automation testing.

How should an enterprise adopt it in sequence?

  1. Week one: one repository, one runner, unit tests and a JUnit report on every merge request.
  2. Next: add the browser or API suite as a parallel job, with a pinned, supported base image.
  3. Then: move slow suites to a schedule, set owners for schedules and decide which jobs may use retry or allow_failure.
  4. Then: add merged results pipelines or merge trains where several teams merge into one branch.
  5. Finally: add code quality, coverage and accessibility reports, and review artifact retention and secret handling with security.

The same pipeline pattern applies to data and model workflows; our guide to the machine learning pipeline shows how stages, validation and monitoring carry over.

To map one release decision your pipelines and agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop.

Frequently asked questions

What is the difference between a job and a stage in GitLab CI?

A job is one task, such as running a test suite. A stage is a group of jobs. Stages run in sequence, and jobs in the same stage run in parallel.

What happens when a test job fails?

If any job in a stage fails, the next stage is not usually executed and the pipeline ends early. Using allow_failure lets a pipeline continue after a failed job, so apply it only to non-critical jobs.

How do I show test results in a merge request?

Have the test job write a JUnit XML file and declare it under artifacts:reports:junit. Results then appear in the pipeline Tests tab and the merge request test summary. The report does not fail the job by itself, so the script must exit with a non-zero status.

How do I run slow end-to-end tests without slowing every merge?

Run fast checks on each push or merge request and run the full end-to-end suite from a pipeline schedule, for example overnight. Schedules use cron patterns and run with the owner's permissions.

Should I use GitLab-hosted or self-managed runners?

GitLab-hosted runners need no installation but give you limited control of the environment. Self-managed runners run on your infrastructure and give full control, which suits internal network access or custom environments. Keep the Runner version aligned with your GitLab version.

What is a merge train?

A merge train queues merge requests and tests each against the combined changes of those ahead of it, so two requests that pass alone do not break the default branch together. It is available on the Premium and Ultimate tiers.

Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue