Software Testing in DevOps – Overview
“Software testing” defined as the variety of methods, tools, and practices used to justify that a software application works at many different levels or not. In the Automated Testing, all of us do some sort of software testing (either it is manual or ad hoc, e.g., refreshing a webpage after making a change to verify the code you just wrote works). In this blog post, I will try to discuss some concepts about testing in DevOps, and give a high-level overview of the different ways that software can be tested, and how continuous testing is different from automated testing.
Software Testing is one of the critical phases of the software development lifecycle. Software testing involves testing various real-life conditions and matching the actual result with the expected result. It ensures that the developed software/application serves the intended purpose. An application can be tested with the intention of finding bugs. It helps to check the stability of the product.
What is Automated Testing and its Types
Automation is a combination of two words that are: Auto and Matos. Auto means self and Matos means moving. Means “move by itself.” Automated systems also achieve significantly superior performance than the performance possible with manual systems, concerning power and speed. In a fundamental sense, automation means — the use of some technology to complete a task.
Types of Automation:
Three Types of Automation:
In this sequence of processing operations are set by the equipment configuration. So each operation is usually straightforward, generally using the combination of linear or rotational motion to achieve results.
In this, a sequence of operations can be changed to accommodate different product configurations. It should do by using a set of coded instructions, which is read and implemented by the system by using a different programmer.
A system can be quickly and easily re-tasked to change the design for both low and high manufacturing.
Automation Testing using Behavior Driven Testing
BDT full form is: “Behavior Driven Testing.” It is not common but a counterpart of BDD (Behaviour Driven Development). BDD is a set of behaviors that a user can expect from the system. BDD improve the quality of the software we develop and followed TVD you can call it is a successor of fertility or an improvement over TVB.
TDD is Test-driven development where we write our test cases first as a developer or as an automation team, and then we start writing our hold. Once the implementation or development is complete, the testing team runs the test against our code of the implementation. All the test should be passed so initially when we write the test we are writing against the requirements that our product owner our business analyst gave us, but in TDD while we write our tests, we make a lot of assumptions as a developer or even as an automation engineer.
There is no direct link between the test we write and the actual requirements. There just for all but BDD be direct links the requirements or the specifications with the tests we write the way it does. It is in another word typically the requirements expressed as stories assess the term we use and these stories usually sit in Jira or similar story repository or requirement tool like Jira each story comprises of some acceptance criteria along with the actual requirements and everything the detailed explanation for the story.
They are very easy to use. There are plugins which can pull this acceptance criterion from zero and then run the corresponding test case once we code. Out of those test cases if we miss a requirement while we are testing immediately we know these plugins will complain will right away show you that this particular requirement does not even have a test. If somebody accidentally changes something in the code and if the test fails we have use aggression suit at some point as we keep writing these tests against our implementation.
We are going to have a sharp regression suit at some point, so that’s all BDD does it links or gives us direct links between other requirements and the actual implementation and we do the test first even before we start implementation it is an excellent practice to do.
Test Automation is not Automation Testing
Testing is not finding the bugs; it is preventing the bugs. Testing is about understood the product, and the problem(s) tries to solve and to find areas where the product/process can be improved. In automation testing, testers will use the tool to execute the test cases. The will write the test scripts and verify whether the application is working as per requirement or not.
Test automation is the automated execution of predefined tests. It gives proof of the completion of testing. It supports for doing performance testing. It ensures the accuracy of the report.
Why Test Automation is Critical to Continuous Testing
No! Test automation is not the same as continuous testing. Can you implement continuous testing without automation? That’s also a no.
Test Automation needs Automated tests and automation testing, whereas Continuous Testing needs Test Automation to deliver on the speed, quality and efficiency principles.
It is essential to know the distinction. Let’s attempt to understand this:
As organizations move from a traditional testing approach towards DevOps practice, Continuous Integration and Continuous Delivery(CI and CD) are keys to achieve the desired speed. In the CD model, software development is an ongoing process, and the deployment-readiness of software, therefore, matters more than ever. Continuous testing allows consistent quality output at every stage of the development.
Testing occurs incrementally through the cycle. To achieve continuous testing, however, test organization, efficiency, and speed are all necessary. Organizing all these testing needs in this context is a challenge, and in that case, Test Automation comes.
Test automation enables testers to focus their energy and efforts on writing better test cases. It simplifies the process by automating the tracking and managing of all these testing requirements. It is designed to take the internal structure into account and have a quick feedback loop for developers to control the system design. It, in turn, helps the overall quality of the software.
It acknowledged that test automation has to start in the early stage of the development cycle.
The speed at which all development and the testing occurs matters quite a lot. That’s because if something in the pipeline breaks down, it holds up everything else and slows down the release of new developments. The need to deliver new releases faster/on a regular basis paved the way for this Continuous Delivery and testing model, that roadblock defeats the purpose of taking this approach.
How to enable Continuous Testing
The primary role of the tester is to give quality software at the end. So, to achieve this goal try to adopt the different approach. The manual method is very time to consume, not reliable.
Now our goal is to try different types of testing, Manual and Automation -continually throughout the delivery process.
We create a deployment pipeline when we have CI and test automation in place. Means after every change in code run a builds that a) create packages that can deploy to any environment b) run unit tests. Packages /modules that are pass will go for automation.
The tester then writes automation test scripts for the code and get the status either pass or fail. The code which is pass going forward for self-service deployment to other environments for activities such as exploratory testing, usability testing, and ultimately release and the fail case move back to the developer for changes.
If testers cannot detect any defects, they should release any module that has gone through it. If testers find defects laters, it means they have to improve pipeline, adding/updating some tests.
So main goal of the tester is to find the bugs/issues as soon as possible. Thus tester wants to parallelize the activities.
Misconceptions About Automation Testing
Automation is a replacement of testers
Nowadays most of the companies are moving towards Automation Testing in which testers can write the scripts and testing the software. Scripts cannot write automatically by automation; there are the testers behind them who investigates the complete criteria and thinks according to it to write the scripts, so a tester’s perspective will always be valuable.
Automation will provide you more free time
The misconception that automated testing is giving you more free time which is true as well as false. In manual testing, most of the time use for exploratory and functional testing where testers would manually search for errors. Once that process will complete, the manual tester will repeatedly go through the same steps over again like Regression Testing whereas, in Automation testing, testers will spend most of the time on writing the scripts and making improvements to these tests repeatedly as adjustments are needed. Once the analysis is complete, automated testing allows for the recycled use of tests so that they should not go through this whole process again.
The Cost of Automated Testing is Too High
It is the big misconception. Nowadays we have the number of options for automation like Katalon Studio, Selenium, Robot Framework, etc. Which are the entirely free tool and if we go for paid tool also that will also be beneficial because at first, the investment in automated testing might seem cost prohibitive, especially if you are in a smaller company. But after analysis, over time, automated testing pays for itself.
Automated testing reduces the cost and need for multiple code revisions, so with the time, the investment pays out. Repeating these tests with manual testing is costly and time-consuming, but with automation can be run again and again at no additional cost.
Automated Testing is Better Than Manual Testing
The reality is that there is no “better” or “worse” between manual and automated testing, there’s just “different.” Both have their advantages and disadvantages. Manual testing is performed firstly before automation. For every new build firstly we have to do manual testing on it. Automated testing is mostly using after the build has developed. Lengthy tests that avoided during manual testing can be run unattended.
In the end, both have their perspectives, especially if the application you’re developing is too complicated to rely just on the manual approach.
Automated Testing Inhibits Human Interaction
Another common misconception about automated testing is that it inhibits human interaction. Automated testing is clear and faster than what testers could do without suffering human errors, so this misconception is understandable. That said, products are designed to facilitate a collaborative approach by including features. It allows co-workers to go through a piece of test coding and then comment on the test script.
It doesn’t replace the face-to-face communication that’s a necessary part of software development. It enhances that aspect by providing another channel through which to communicate.
When and How to Automate Testing
Test automation is utilizing software, and that might be a script, macro or third-party solutions. Such quick test professionals to handle the execution of test cases and the comparison of actual outcomes to expected results. The setting of test preconditions or you might be using of test automation for other test control and reporting functions, but again test automation is the use of a program, a macro or a third party solution to perform specific testing functions even if it’s for test preparation.
Benefits of Automation Testing
Speed of execution
So automated a test should significantly decrease the execution time, and you can also lower the execution time by distributing automated tests running them on multiple machines so with the speed of execution. Lets you have a manual process, and it takes you about an hour to perform that manual tests if you do an excellent job in automating the process it should execute in two to ten minutes of the most.
It might be closer to 10 min only if you have specific situations where the tester has to manually get involved and answer questions things along that way before the automated test kicks off.
Reliability, Repeatability, and Re-usability (3R’s)
With the manual process lets, you have got john tester executing a test, and he performs that test. He implements the actions, the navigations, and checks for particular results but he has not standardized the manual test.
He goes off on vacation and mary the tester take over she may end up performing that test in the entirely different way by automating that testing process you can standardize it if you will and then thus introducing the 3R’s.
It means coverage of requirements so since you are decreasing the execution time you can validate a significantly the higher number of conditions and scenarios thus improving your testing process.
The other benefit is you can execute the automated test remotely and unattended, so you can schedule a series of automated tests to kick off at 9 p.m. and then actually come in the next day and perform your analysis.
Now before you automation, there are no. Of questions to ask your self. Look at your test case, your test plans and ask yourself is our environment and application supported by the automation solutions are using dot net or java, etc. , also as yourself is the application interfaces somewhat stable if there have minor changes then automated process handle those kinds of changes but if its going to be significant change such as navigation or the removal/ addition of certain objects which are necessary for the testing process now might not be the right time to automate. Another important question before you start the automation do you know the application, do you know the expected results, do you know how to navigate while you are automating the test that is not a right time to be experimenting the application.
Some good candidates for automation:
- Tests run for every build of an application
- Tests using multiple sets of data
- Tedious and prone to human errors
Good candidates for the manual:
- One time testing
- Ad-hoc testing
- Emergency testing
- Usability testing
- Tests without predictable results
Metrics for Test Automation
Annual Savings & Time and Money
QA matters in terms of bringing a real competitive advantage to your business. It is not just making sure that things work that’s a huge part of it, but if we make sure that things work as their designed day in day out then you can have a competitive advantage. We convert a lot of people from either old style scripted test/ from complete manual testing. So one of the testing we look at is what type of product of hours and annual savings can we get from moving from a manual or scripted test to a work soft automated test. It usually turns out that those numbers are quite high beyond that we started looking at things that we can do for example in terms of sap impact analysis and be saving people time to determine if they have gaps in their test library.
We also look at defect reduction as it turns out defect reduction is one of the most significant metrics that we impact on an annual basis. Because by using work soft you are just going to catch more things earlier in the development or the rollout process so, you are going to save a lot of money. Every year concerning reworking those defects because we just don’t let it get into production.
Improve productivity with automation
You can replace a lot of human labor with digital labor through work soft. By doing that we have free people to pursue other things within their companies. In terms of you know other projects that need to do additional modules, that need to be rolled out more quickly put all those things together. You get a massive time to market advantage because in addition to reducing the number of hours and money spent on manual or scripted testing you know an aggregate effect on what we call project compression.
Automated Software Testing: The Glue Between Dev and Ops
What are the tools used in DevOps lifecycle
What are the stages in DevOps
How do the tools operate in each of the DevOps stages?
How to use DevOps tools in a real-life scenario?
There are different stages in, and how they are connected we are discussing below:
1. Developers are writing the code, and after there commit, it goes to version control/SCM. It is the first stage in DevOps. In this stage, you can manage the code written by your developers.
2. Next phase is Continuous integration. Continuous means rather than waiting for all the developers to commit their code and then building and testing it, in continuous integration approach when any developer commit the code, that code is merged into the main branch. Then you can use Jenkins/Git to create an build, and after that, you can test your code in a visualized environment, It saves a lot of time in moving the code from dev environment to test environment. We need not wait for all the developers to commit. Even though you do continuous integration, it is of no use if you are not testing continuously
So the next stage is continuous testing, at this stage, you can automate your test by writing any scripts like selenium, katalon studio and then run it through Jenkins/Git in a visualized environment.
3. If the test fails then stops here and send back the issues to the developers if the test passes, we move ahead of the continuous deployment.
In continuous deployment stage we have two parts:
Consider they are five servers are up and running, and they have the same libraries, tools on them. There would be many of them are from an older background where in the past used to have shell scripts to do this job. Now if I need to have exact configuration on 100 of servers, using shell scripts, it will be pretty much complicated. So we can need a tool which can automate the job for me. Puppet is the most popular tool for configuration management.
The second part of Continuous deployment is containerization. In this stage as mentioned before you can create a test environment where you can run your application and tested. It uses for testing purpose; it is giving a visualized environment where I can run my script like selenium scripts to test my application.90% of organizations are using containerization only in the testing stage, only 10% of the organization is using it on production. In 2015 docker 1.9 came which was stable to run on production as well. For example, if you need to run some application on prod for the shorter duration, you can quickly create a container on prod and run those applications in that container, and when you don’t need those applications anymore, you can destroy the container. Now you have deployed your application; you need to monitor it to see the application is performing as desired, the environment is also stable.
It is where continuous monitoring comes into the picture. By monitoring the application, you can quickly determine when a service is unavailable and understand the underlying causes, and then you can report back the issues to the developers. Nagios is one of the most popular tools used in continuous monitoring. So this is how all stages are connected.
Why Automated Testing is Essential for CI/CD
Continuous Integration is necessary to keep everyone from working on a team in the sink. It is about having a collaborative event between all the people who are involving build something, keep them in the sink. Why that happens is through automation testing and validation and feedback.
CI is a development practice required developers to integrate code into a shared repository several times today. Each check-in is verified by the automated build, and then feedback team detects problems relate. Practicing CI requires building automated especially on testing itself. In result CI practices, gain more confidence in the code at least in a development environment. It is essential to know for continuous delivery.
The reason for being automation is because deploying software is not a simple process of taking a set of files from one place and copying and then moving to another place from one place. Especially, for software of large enterprises, there are some files like binary files need to be deployed, maybe some configuration files need to be moved over, you may have app server need to be restart if configurations are changing, you have to wait for app server restart.
Meanwhile, that’s going on maybe you can change some changes on database schema then you need to restart the database service. To make sure that all the services are talking to each other then you restart the new binary files, copy them over maybe you have multiple nodes of your application server. You need to make sure they have the same version of the software and have same configurations and all need to be done in a coordinated manner with the right events and steps happening at the right time so that there is some time involved.
So delivery automation tools help you do that automation and help to codify the time so that you can deploy to any environment. But when you are deploying, you are not deploying just software because you are also deploying the environment. The physical environment may be already provision or the virtual environment or environment in the cloud. But may it is not in the provision, I need to provision it before you can deploy the software or it is already provided may be you need to do some configuration changes because configuration settings in dev will very different with the configuration setting in QA, will suddenly different from configuration setting in prod.
The actual environment, middleware stack might be the same but configuration will be different. So automated deployment is the ability to get software deployed to any particular environment in a given time. Continuous delivery is the capability to deploy the software to any specific environment at any given time and we again including the binaries, including the configuration changes and including changes to the environment.
How to Develop an Automated Testing Strategy:
Continuous testing based on you having fixed agile process. Excellent communication and collaboration is the definition of DevOps. The second mandatory thing should good for sprint automation. It means you are not continuously playing catch up with automation suites. Strategy means the resources to achieve the desired objective effectively. Test strategy implies the plan of how to test target will be met adequately. You may have a test strategy at the organization level, at the program level, and at a project level. It is a management policy. At the project level, the test strategy is part of the test plan. Further, depending on the project nature the test strategy defined at the project level may or may not satisfy the test strategy outline data higher level before.
The advantage of test strategy :
Mitigates the risk that objective testing
It helps to focus on aspect system under test by using distinct test phases such as unit testing, integration testing, system testing and so on.
Provide clarity on the people, tools, and infrastructures.
If tester wants to define a powerful test strategy for your organization you should consider the following steps:
Before establishing any test, strategy tester should research the clients and end users with respective needs and expectations needed for the applications.
How test strategies aim is to satisfy just the test objectives.
Illuminate to create two versions of test strategy depending on the situation, one for communicate to all and add a detail/dealer test strategy for an implement with the key stakeholder.
You have the number of options of having a focus on different test phases but do not go overboard. Define only those which you think really required.
Test strategy should be totally customized according to your situation. What is work for other companies, what is work for your company may not apply to your current situation?
You should have the required test environment while designing your test strategy. You should have at least one test environment that is the same as a replica of the production environment in which the system is going to work. It is the central part while developing the test strategy.
Your test strategy should define the testing tools for testing, defect management system or automated testing. It is based if you evaluated the test tools your self. If not based on your decision on choosing the testing tools reliable tools and the popular user reports.
You should look at any define test processes example suspension criteria, resumption criteria, exit criteria for the test. The process to execute the test cases and the process to reporter a defect etc. to examine their feasibility in your situation. Ideally, the required test processes for you that will be reused, modified or created from scratch.
Identify the data that will be recorded, measured analyzed, reported to show the progress of your testing.
Challenge all assumptions while you design your test strategy. Provides safeguards if any of the prior assumptions is proved incorrectly.
Importance of Continuous Testing & Automation Testing in Agile
Early Defects, Less Cost
Test automation for continuous agile delivery helps in initial authorization & determination of software defects/bugs. As early bugs will fix; the cost to the company will get reduce. A suitable testing process in an initial stage of production allows faster recovery in case of unexpected defects /bugs in the product.
A Continuous delivery model requires Continuous testing, and this is possible only through test automation. Thus, it helps coordinate Quality Assurance efforts to balance the speed of DevOps, assisting developers to deliver unique software features quicker.
Reduce Testing Efforts
Automation tools are helping to reduce testing efforts. With the focus on continuous development, automation tools will serve the testing team and overall product development.
Challenges for Continuous Testing and Automation Testing:
Increase in Speed and Performance
Speed and performance easily accomplished with the use of automation; While it is enticing to automate every mode of testing, manual testing required in regression & exploratory testing at UI level.
Building test environments & configuring an automation framework needs a lot of expert person & efforts. The most significant work in gaining test automation coverage include time and cost associated with setting up a useful automation framework. Searching the skilled automation expert is a challenge and so due to this many organizations do confront.
Robust Planning & Execution
By automating workflows, companies can reduce cost & time in testing. This planned test management utilizes the power of automation; product owners can determine, track fixes to address critical bugs in their application quickly. Developers can concentrate on bringing new variations to achieve good user experience.
Understanding Effective Test Coverage
Test coverage lets us assign a score to a collection of test cases and so let’s be a little bit more rigorous about it. So test coverage is a measure of the proportion of a program exercised during testing. So, for example, we have just talked about the number of functions out of the total number of functions or exercised by some test that we had.
What’s right about test coverage is it gives us a score, gives us something objective that we can use to try to figure out how well we’re doing.
Additionally, when coverage is less than 100 %, that is to say, as in our example, where we had failed to execute all of the functions in the software under test, we know we need to do to get full coverage. We know what the functions are that we need to execute. Now, we need to construct test cases and execute those test functions. So, these are the good things about test coverage.
On the other hand, there some disadvantages.
Because it is a white box metric that derived from the source code for our system is not good at helping us find bugs of omission, that is to say, bugs were merely left out something that we should have implemented.
It could be tough to know what a test coverage scoreless 100% means and in safety-critical software development would sometimes be done is requiring 100% test coverage of a specific coverage metrics and that sort of removes this problem. It means we don’t have to interpret scores of less than 100% because we are not allowed to ship our product until we get 100% test coverage. For a larger, more complex software systems where the standards are correct and not as high as they are for safety-critical systems, that’s often the case, but it’s difficult or impossible to achieve 100% test coverage. Leaving us with this problem we are trying to figure out what that means about the software.
Even 100% coverage does not mean that all bugs are found, and you can see that sort of easily by thinking about the example why we measure our coverage by looking at the number of functions we executed. Just we executed some function, of course, does not mean that we found all the bugs in that function. We may not have executed very much of it or may not have somehow found very many of the new behaviors inside that function.
In which Scenarios Automation Testing is Preferred
- Load Testing: Automated testing is suitable for load testing where the tester can change the load frequently to check the point where the application will break.
- Regression Testing: Here, Automated testing is suitable when tester wants to do regression testing. Because a number of times code is changing, so it has the ability to run the regressions in a timely manner. You may love to read: Quick Guide to Regression Testing Tools and Techniques
- Repeated Execution: Testing which requires the repeated execution of a task is better to be automated.
- Performance Testing: Similarly, testing which requires the simulation of thousands of concurrent users better to be automated.
- GUI Testing: For testing, GUI displays automation testing is performed. There are many tools used for recording user actions and then replay them any number of times. This is helpful for comparing actual and expected results.
How Can XenonStack Help You?
XenonStack offers DevOps Consulting and Big Data Testing Solutions for Enterprises & Startups
How useful was this post?