Release Engineering Best Practices for Continuous Delivery

What is Release Engineering?

Release engineering is a new and rapidly growing subject of software engineering. It derives the standards for application development and is seriously followed to ensure a reliable product when it is released. A release engineer has excellent knowledge of multiple disciplines, such as source code management, configuration management, development process, and customer support. Release engineering is a flexible process molded based on the product's requirements.

What are the Development Guidelines for Release Engineering?

This is the first part of the release engineering process definition, which lists all the guidelines that developers must follow.

Branching Model

This clause states that a proper git-flow branching model must be followed during development. Git branches are reasonable and affordable to create and maintain in the long run. Respective guidelines for every branch about working the git-flow branching model are listed below.

Master Branch

The source code master branch represents the production-ready state. It contains the most stable code and will be deployed to production. The master branch officially reserves the release history of any software application's source code. All the commits on the master branch must be tagged with a version number. Developers should not commit anything directly to the master branch. The code shouldn't be merged into the master branch without the approval of the QA team.

Develop Branch

Instead of a single master branch, our workflow will use two branches to record the project's history. The development branch serves as a branch that integrates various feature branches. Both developers and maintainers have the privilege of merging feature branches with development branches. While merging branches into development, the developers should briefly describe the features they worked on that branch.

Feature Branch

A feature branch is a child of the developed branch, i.e., instead of branching off of the master, feature branches are used to develop as their parent branch. Using feature branches, the developers can work on multiple features simultaneously. Once new features are designed and tested, the feature branch will merge into the development branch.

1. The code for each new feature for your application should reside in its particular feature branch, which can be pushed to the central source code repository for backup and collaboration.

2. All the feature branches should exist only on developers' repos, not in origin.

3. The feature branches never interact with or directly merge into the master branch. When a feature is complete, its respective branch always gets merged back into the development branch.

UAT Branch

User Acceptance Testing is the pre-production branch, also known as the staging area. Once code is developed and tested in the develop branch, it gets merged into the UAT branch.

1. The UAT branch should be locked so that developers can't commit directly to this branch. Only the maintainer should have permission to accept the merge/pull request in the UAT branch.

2. Once all testing is done, the UAT branch should create a release branch. Bug fixes in UAT shouldn't be directly merged into the UAT branch; all the bug fixes branches should be merged into the develop branch first.

Release Branch

The release branch will be derived from the UAT branch. The creation of this branch is done to start the next release cycle, and it indicates that no new features can be added after this—only documentation, bug fixes, and other tasks related to the release should be done in this branch. Source code tagging and release versioning are also done here. Once the code is shipped, the release branch is merged into the master branch. It is tagged with a version number, too. Additionally, it should be merged back into the development branch, which may have progressed since the release was initiated.

Hotfix/Bugfix Branch

Hotfix branches, also known as maintenance branches, quickly patch the production releases. The bugfix branch is derived from the master branch when there are some bugs in the production environment. As soon as the bug is fixed, the hotfix branch gets merged into both the master and the development branches. And the ongoing release branch, and then the master branch is tagged with an updated version number.

Branch Naming Conventions

This clause states that proper naming conventions must be followed for each branch mentioned below. Branch names should only contain lowercase letters, hyphens, slashes, and numbers, regexp: /[a-z0-9\-]+/.

Feature Branch

You can use a slash or hyphen as a separator while naming a feature branch, but make sure the feature branch naming is homogeneous; don't mix multiple feature branch naming conventions. Before working on a project, decide on one naming convention and stick to that.

1. feature/*

2. feature/gitlab-integration

3. feature/ticket_id

4. feature-*

5. feature-slack-integration

Release Branch

1. release-*

2. release-1.0.0

Hotfix Branch

1. hotfix/*

2. hotfix/broken-link

Merge Process

Developers should follow proper Merge Process/Request and Release Branching Process.

To ensure that a merge goes smoothly, there are a couple of preparation steps before performing it. Before merging the code to any branch, use a short description.

1. Confirm the destination branch

2. Fetch the latest remote commits

3. Apply Merge

Merge Flow

feature/branch_name → develop → UAT→ Release→ Master

Release branching refers to the concept of making a release reside within a particular branch. A branch is created when the team starts working on the new release (e.g., "Release 2.1"), and all work is done until the next release is stored in this branch.

Git Commit Guidelines

This clause states the guidelines for Git Commit. When you're working on a group project or contributing to an open-source project, the commit message is the best way to communicate the context of a change to other developers working on that project and, indeed, to your future self.

But most of the time, developers don't focus on committing messages. They use weird commit messages that don't help anybody, and also, these messages can't convey what changes are done and why. To make commit messages significant, a developer must follow the following guidelines. Commit message should be short and descriptive; try not to exceed 50 chars ( max 72 chars)

-Commit message should be imperative.

1. "Add Dockerfile"
2. "Added Dockerfile"

-Commit messages should describe why the change is made and how it addresses the issue.

-Capitalize your subject in the commit message.

1. "Fix github integration error."
2. "fix github integration error."

-If commit refers to an issue/bug, make sure to add the issue/bug number in the commit message.

-Use characters to make the commit message short.

1. "Add Hindi & English translation."
2. "Add hindi and english translation."

-The subject line should not end with a period/Full stop.

1. “Add new Dockerfile.”
2. “add new Dockerfile.”

What are the CI/CD Guidelines for Release Engineering?

This is the second part of the release engineering process definition, which enlists all the guidelines that must be adhered to by the release engineers. 

Reference to the required process flow diagram.

CI/CD Stages

Every branch mentioned in Part I above should have a CI/CD pipeline that strictly consists of the below-mentioned stages.

Feature/HotFix Branch

1. Unit Testing
2. Integration Testing
3. Build
4. Push
5. Deploy code (if required)

Develop Branch

1. Code Quality Check
2. Unit Testing
3. Integration Testing
4. Functional Testing
5. Build
6. Push
7. Deploy to Dev

UAT Branch

1. SAST
2. DAST
3. Browser Performance Testing
4. Load Testing
5. Fuzz Testing
6. Build
7. Push
8. Deploy to UAT

Release Branch: This branch should not contain any pipeline. This branch only has code ready for release. Only the documentation, final touches, and minor bug fixes to the code are focused on.

Master Branch:

1. Build
2. Push
3. Deploy to Prod

Definition of each stage

1. Code Quality Check: This will help analyze the source code quality and ensure that the project code stays simple, readable, and easier to contribute to. Source code will be automatically analyzed to surface issues to check if the quality of the code is improving or getting worse with the latest commit.

2. Unit Testing: Unit testing is a stage where discrete functions are tested at the source code level. It is a type of testing where we test a software application's units or components. Unit testing fulfills the purpose of validating the performance of each unit of the software code.

3. Integration Testing: The integration testing stage is included to perform a level of software testing where individual units/components are integrated and tested as a whole. It exposes faults in the interaction between the integrated units.

4. Functional Testing: Functional testing is a quality assurance process and a type of black-box software testing that validates the software system against the functional requirements/specifications. The purpose of functional tests is to test each function of the software application by providing appropriate input and verifying the output against the functional requirements. The internal program structure is rarely considered.

5. SAST: SAST stands for Static Application Security Testing. It scans the application source code and binaries to perform static code analysis, which helps spot potential vulnerabilities before application deployment. Every merge request triggers the scanning, collects the results, and presents the list of vulnerabilities in a single report.

6. DAST: Dynamic Application Security Testing analyzes the running web application for known runtime vulnerabilities. It introduces enhanced security since it is a black box testing where communication with the web application is done through the front end to identify potential security vulnerabilities in the web application and architectural weaknesses.

7. Browser Performance Testing: This is a web performance testing where we ensure the performance of the software in-browser using automated browser performance testing. This would measure the performance of a web page using an open-source tool and create a report that should be uploaded as an artifact. This feature is introduced to provide the overall performance score of each web page.

8. Load Testing: Load Testing is an important part of the CI process. Developers can make performance decisions earlier in the process by measuring how the software responds to load and introduced changes, which can be categorized as performance testing.

9. Fuzz Testing: Fuzz testing, or fuzzing, measures the application's response and stability by providing unexpected inputs like some malformed or random data ecosystem. This helps in monitoring unpredictable behavior or application crashes. Fuzz testing is essential because it finds issues that traditional testing methods typically do not. It lets you discover software defects that should be addressed on priority, as these defects may lead to highly exploitable vulnerabilities.

10. Build: Cloud-native software is typically deployed with Docker regardless of the language. By combining the application's source code and dependencies, a runnable instance of our product is built and shipped to the end users. This CI/CD pipeline stage packages the entire code and its dependencies into images and builds the Docker containers.

11. Push: This is a stage where the built docker images are pushed to our secure and private harbor registry with relevant tags to be later retrieved for deployment as per the requirement.

12. Deploy: This is the final stage of any pipeline where the application is deployed to an environment relevant to the branch, i.e., dev, uat, or prod.

 Best Practices for the Design of the CI/CD Stage

With reference to Design CI/CD Stage mentioned above, it must be ensured that:

1. Build and push stages should be inculcated in the pipeline of every branch, namely feature/hotfix, develop, uat, and master, except release.

2. The deployment stage in the feature/hotfix branch pipeline shall be inculcated only if required for testing purposes. Further, the code present in the development branch must be deployed to a dedicated dev environment. Similarly, code present in the UAT branch must be deployed to the UAT environment for manual testing by the QA team. Final deployment to the production environment for end-users must be done within the pipeline of the master branch.

3. Release branches must not have any pipeline, as they should only be dedicated to documentation and the final touches of the release.

4. As far as testing is concerned, the following guidelines should be implemented:

5. The pipeline in the Feature/Hotfix branch must only have unit and integration testing.

6. The pipeline in the Develop branch must only have the unit, integration, and functional testing.

7. The pipeline in the UAT branch must be dedicated to all types of security and performance tests, namely, SAST, DAST, Browser Performance Testing, Load Testing, and Fuzz Testing.

8. The pipeline in the master branch should not have any testing stage.

Defining Environments

This clause states that there should be isolated deployment environments for DEV, UAT, and PROD.

1. Isolation of each environment must be strictly considered to avoid any point of conflict. Ideally, the safest and most secure way to isolate different environments is to use a separate cluster for each environment. This option minimizes the production environment's risk due to potential human mistakes and machine failures. However, this increases the maintenance and administration overhead, reducing utilization and more infrastructure management costs.

2. Alternatively, a single cluster with different namespaces can be used to isolate different environments. The K8s community has evolved and matured a lot, enabling us to address security issues through proper network policies, node selectors, seccomp, access control, etc.

3. Still, to have a strong separation between production and non-production environments, we can have one cluster for DEV + UAT environments and one cluster wholly dedicated to the final production environment.

As a result, this choice ultimately depends upon the need of the hour. Considering all the probable pros and cons, the best option relevant to the use case shall be implemented.

Naming Conventions based on the environment

This clause states that proper naming conventions should be followed for accessing the applications deployed in different environments.

Suppose an application with the hostname 'example.com' is being developed. So, it must be ensured that:

1. The hostname of the application deployed in the DEV environment should be dev.example.com.

2. The hostname of the application deployed in the UAT environment should be uat.example.com.

3. The hostname of the application deployed finally in the PROD environment should be example.com.

Artifactory Management

This clause states that proper satisfactory management must be given prime importance.

Every release in the DevOps CI/CD processes is a collection of artifacts that must be preserved and managed correctly. An artifact is referred to as the by-product of software development. It is considered a deployable component of an application, e.g., docker images, jar files, wheel files, etc.

1. Source code, meeting notes, workflow diagrams, data models, risk assessments, use cases, prototypes, and the compiled application are all considered artifacts. During the planning stage, a list must be drawn up covering all of the required artifacts produced.

2. Stable tagging of artifacts like docker images must be ensured for proper segregation and easy identification. The tag might include the project name, repository name, branch name, pipeline ID, etc.

   For eg: life data/stt-server:develop123456

3. An Artifactory repository must be used for storing and managing artifacts. It is designed to store, version, and deploy artifacts for builds. It is both a source for artifacts needed for a build and a target to deploy artifacts generated in the build process.

How to Release Your Product to End Users?

Product Shipping to end-users

This clause states the process of product shipping to end-users. Once your code is ready to be shipped to the live environment, the product team must follow the following clauses for the smooth release of the product to the end users.

1. The developer should create Changelog files for each product microservice, following the given link for the Changelog file structure.

2. The production-ready code should also contain README files.

3. The QA team should send the QA signoff to the developers and release engineers. Without the QA sign-off, the code should never be merged into production.

4. The product team should be ready with all the user documents and reference documents.

5. The sales and marketing team should have a basic idea of the product before discussing its features with customers.

6. The support team should be appropriately trained to counter any user issues with the product.

7. The release engineers should create a checklist and cover all the essential things for the release.

8. Once everything in the checklist is completed, the developer must create a release branch for the release versioning and the tagging.

9. The developer should follow semantic versioning while creating a tag or a release.

10. Once the release versioning is done, the branch should be merged into the production branch by the developers.

How to Choose the Right Deployment Strategies?

1. Recreate: Version A is terminated; then version B is rolled out.

2. Ramped (also known as rolling-update or incremental): Version B is slowly rolled out, replacing version A.

3. Blue/Green: Version B is released alongside version A, then the traffic is switched to version B.

4. Canary: Version B is released to a subset of users and then a full rollout.

5. A/B testing: Version B is released to a subset of users under specific conditions.

6. Shadow: Version B receives real-world traffic alongside version A and doesn't impact the response.

Recreate

This deployment strategy states that the old pods will be killed all at once and replaced with the new ones. This deployment approach involves downtime while the old versions are being brought down and the latest ones are starting up.

Situations/Considerations 

1. Your application does not support multiple versions and can withstand a short amount of downtime.

2. You have a ReadWriteOnce volume mounted to your Pod, and you cannot share it with a Replica.

3. You want to stop processing old data and run some prerequisites before starting up your new application.

Rolling Update

It's the default strategy in Kubernetes, in which new pods are slowly rolled out by replacing the old pods without causing any downtime. A rolling update doesn't scale down the old pods unless it validates via the readiness probe and confirms that the new pods have become ready. If there is any problem, you can abort the rolling update or deployment without bringing the whole cluster down.

The rolling update configuration consists of:

1. Max surge: Maximum number of pods that can be created over a desired no of pods.

2. Max unavailable: no of pods that can be unavailable during the update process

Benefits/Considerations of Rolling Update Strategy:

1. Releasing the application/service to development/staging environments.

2. Downtime in an application that processes a massive number of transactions per minute can cause many problems. However, in the Rolling update, the application continues to operate with Zero Downtime.

3. Feature of "track/record" deployments, useful in Rollback.

Blue-Green 

In this strategy, the old and new versions of the application are deployed simultaneously. The users will be able to access only the old version, and the new version is firstly available for the QA team to test; once they're done testing, the users' traffic is shifted to the new version.

It helps in testing a production-quality environment before it is made public. In contrast to the rolling and canary deployment approaches, it also enables them to switch all users over to a news release at once.

Benefits/Considerations of Blue Green Strategy:

1. Quick rollbacks and easy disaster recovery.

2. Blue/green deployment is zero downtime, so the team can make the switch and let the load balancing system automatically shift all users to the green version. The old, blue version of the application is ready and waiting if something goes wrong and requires it to be rolled back.

3. Overhead: running two identical environments is expensive.

Canary

In the Canary deployment strategy, we shift a controlled percentage of user traffic to the new version of the application. We use this strategy mostly when we're not confident about the stability of the new version. A key advantage of this app deployment strategy over blue/green deployment is early access to bug identification and feedback. It finds weaknesses and improves the update before the IT team rolls it out to all users.

Benefits/Considerations of Canary Deployment Strategy:

1. Great to try out new features and see how the application and system behave when the traffic is routed to a new version.

2. Canary deployments will only be helpful to you if you can track their impact on your system.

A/B testing

A/B testing is a deployment strategy that routes a subset of users to a new functionality under specific conditions. It is usually a technique for making business decisions based on statistics rather than a deployment strategy.

Benefits/Considerations of A/B Testing:

1. GGG gives full control over traffic distribution, and you can track customer behavior and revenue results.

2. Several versions run in parallel, and It's easy to revert to the older version if the new version does not provide significant benefits.

3. Overhead: Expensive Setup

Shadow

A shadow deployment consists of releasing version B alongside version A, forking version A's incoming requests, and sending them to version B without impacting production traffic. This is particularly useful for testing the production load on a new feature. The application is rolled out when stability and performance meet the requirements.

Benefits/Considerations of Shadow Deployment:

1. It frees you from setting up a dedicated load test environment. The load tests are based on actual traffic in your existing environment.

2. Performance testing of the application can be done with production traffic for more accuracy.

3. Overhead: Expensive and complex setup (duplicate transactions or requests).

What is Rollback and its role in Release Management?

A rollback is an operation that returns the object to some previous state.

We do the Rollback to an earlier Deployment revision if the current state of the deployment is not stable or not giving expected results due to the application code or the configuration.

There are scenarios where even the perfect deployment halts your application when it is integrated with a newer version. At that time, we had to get back to the application version that was running fine with the help of revisions available for the deployments.

Steps to follow while performing a rollback:

1. Firstly, find the cause of the error because of which application is getting into problems. It can be reviewed through the logs and state of the deployment (its pods). Some error references CrashLoopBack, ImagePullBackOff, etc.

2. We follow the rollback process if the problem cannot be resolved and we need to roll back.

3. Check the rollout history for the revisions available. If the—-record flag was used while creating the deployment, you could check the change cause. kubectl rollout history deployment.v1.apps/<deployment-name>

4. Pick the revision that you consider stable and working as expected, and make its(revision) entry in the --to-revision flag kubectl rollout undo deployment.v1.apps/<deployment-name> --to-revision=<number-of-the-revision>

5. You can describe the rolled-back deployment and verify its configuration. - kubectl describe deployment nginx-deployment

Note: By default, Kubernetes stores the last 10 ReplicaSets. But you can change the number of ReplicaSets to be retained by changing the spec—revision history limit key in your Deployment file.

Conclusion

Release engineering is a rapidly expanding field in software engineering that involves a systematic approach from the development phase to the successful release of a dependable product. Organizations must establish a robust release engineering process to deliver their products with speed and exceptional quality.

• Discover more about how to Build Fast and Reliable Pipelines.
• Read more about Release Engineering And Infrastructure Automation Challenges.
• Know more about Platform Engineering Overview and its Best practices.