Introduction to DevOps and Agile
This post will cover how to successfully implement DevOps with Agile.
DevOps is about the development and operations team making together to meet a common goal of faster delivery and quicker time to market.
Coined by Patrick Debosis in 2009, it has gained immense traction in recent times. DevOps emphasizes on closes collaboration amongst teams and better utilization of automation tools leading to enhanced efficiency in software delivery, faster time to market and higher client satisfaction as it ensures quality software being deployed.
Many research reports endorse the popularity of DevOps becoming the mainstream of software development and the changing business paradigm needing software applications to be enhanced at a rapid pace.
DevOps brings together the key aspects of software delivery which are continuous integration and continuous delivery. These fundamental components of DevOps together help in increasing deployment velocity, time to market and reduction in risk as well as improving release cycles.
However, it has to be realized that DevOps is not just about to build and deployment through a set of automated tools, it needs significant mindset shift and cultural change among team members while bringing alignment between people and process.
There are three key aspects of adoption of DevOps namely-
Different models can be adopted for the above aspects in terms of how the team is structured, what are the process to be followed and the types of tools to be used for automation.
Challenges in Agile
While the Agile approach to software development has helped cut-down the application development approaches, it is not enough to ensure rapid delivery of the application into production.
Delayed time to market
with rapid and incremental development methodologies, a potentially shippable product is available in every sprint and iteration. However, the real business value remains undelivered unless the functionality developed is also rapidly deployed into production without compromising on quality, reliability and stability of the application.
Typically, the development and operational team are separate teams working in silos, having different goals and measures. While the development team may intend rapid delivery with frequent releases, the operations team may not be equipped to handle frequent releases as they need to plan for release cheeks, adequate knowledge transfer and support in production.
So traditional handover methods for frequent releases and support induces delays, thus the agility gained in software development is not realized in production systems.
Frequent releases pose a risk of reliability and stability to the production environment and the operations team need to ensure that these are not compromised. The handover of application from the development team to the operations team happens towards the end of the development and testing phases. These may be too late to resolve any impact due to changes discovered in the handover process to the operations.
The operations team needs to spend a considerable amount of time ensuring that these releases are adequately tested for performance and security. So that they do not compromise the reliability and stability of the system.
Prolonged Timelines for Service Assurance
The frequent application releases are expected to be adequately supported in production so that the agreed service levels are met. For these, the operation team needs to acquire sufficient knowledge of the changes from the development team during the handover process.
This is mainly due to the delayed engagement of the operations team in change deployment. Here again, the traditional handover process may create delays and issues in frequent deployments.
All of the above challenges have led to the adaption of DevOps practices to ensure that the benefits of agility are realized.
Concepts of DevOps
DevOps helps achieve and to end agility in software delivery through seamless collaboration between the development and operations team through structured communication and streamlined processes. DevOps provides enhanced capability to deliver high-quality software orchestrated by a set of integrated tools for a given technology stack.
DevOps addresses challenges listed below-
- Automation through Continuous Integration and Continuous Delivery.
- Collaboration amongst development and operations teams for seamless delivery.
- Streamlined processes across the delivery life cycle from development to operations.
DevOps and Agile DevOps Adaption
Automation through Continuous Integration and Continuous Delivery
Concept of Continous Integration
Continuous integration with automation test execution has seen wide adoption in recent years. The concept of Continuous Integration has transformed how companies think about and implement Build and configuration management, release management, Deploy automation and Test orchestration.
Continuous Integration is a software development practice of continuously integrating source code into a central repository wherein each instance of integration is verified by the automated build process to ensure the stability of the project (Working Software) at all times. All through the essential concepts existed earlier, Continuous Integration normally refers to the Extreme Programming (XP) practice.
These process takes place throughout the day and ideally should trigger the automated build process after every check-in, although the frequency of the Automated Build can be decided by the team based on the project need. (e.g. every hour, once in a day).
The advantages of Automated Build and Continuous Integration are-
- Automated builds integrated the tools for code review, unit tests, and code coverage and thus, ensures the quality at source.
- Continuous integration eliminates last minutes integration issues and thus ensures cohesive build throughout the development life cycle.
- Provides the ability to completely rebuilt and test application frequently.
- Large projects often require a longer build and test cycle. Continuous Integration helps reduce effort and time as well as ensure quality.
Concept of Continous Delivery
Continuous Delivery (CD) can be considered as a logical extension to continuous integration. In this process, the software which is created in short cycles in feature or enhancement or bug fix is rapidly and reliably made available to end-users.
The advantage of continuous delivery are:-
- Lesser deployment risk as smaller changes are deployed.
- Visible process of value as changes are available in production.
- Early and quicker feedback.
Continuous delivery ensures that code can be rapidly and safely deployed to production by delivery changes to a production-like environment. Since every change is delivered to a staging environment using complete automation. It is easy to deploy the changes to production with a push of a button when the business is ready.
Continuous Deployment is the next step in Continuous delivery. Every change that passes the automated tests is deployed to production automatically.
Continuous Delivery vs Continuous Deploy
Therefore the teams which adopt Continuous Development will ensure that every change goes through the pipeline and automatically gets into production. However, in Continuous Delivery, though, the team is ready to do frequent deployments, the decision will be business-driven. Continuous Delivery is a prerequisite for Continuous Deployment.
Stags in Continuous Integration and Continuous Delivery
- Build gets initiated when developers commit their individual work/code to the version control repository system. The CI server which is installed as part of the project environment, Continually polls the repository (e.g. frequency could be every few hours based on the need of the projects for detecting any changes made by the development team.
- The server then detects changes, if any, in the Version Control repository and initiates the process of retrieving the latest copy of source code from the repository and execute the build script using build automation tools to perform integration of the software.
- Once a successful build is generated, the CI server triggers the process of validation of test cases/ scenarios and code quality inspection. The process of validation and inspection is automated using the automation tools for unit and integration testing and inspection.
- Though these steps are automated, there would be some manual intervention required in these stages. Post this activity, feedback is generated by the CI server and is communicated to the team members through various mechanisms such as to entail and text messages.
Stage for System/Functional/Non-functional Requirement Testing:-
In an automated development and delivery pipeline, the integration will be done frequently as we in the previous section. Hence test cases need to be run frequently. The gate for developed software meeting the functionality is the test cases. This is possible if the tests are automated.
In the automated development and delivery pipeline, the following test sand their management are automated mandatorily. These tests are invoked in an automated fashion by the orchestrator i.e. the Continuous integration tool.
Benefits of Continuous Testing
Increases the depth and scope of tests coverage which helps in improving quality ensure repeatability of running tests whenever required and helps in continuous integration of valuable software.
testing in DevOps
Different categories of tests can run from the CI system. This includes unit, Component, System load or performances, Security and various other types of tests.
- Component test- These tests verify portions of a system such as databases, file system network endpoints.
- System test- These tests exercise a complete software system including external interfaces.
- Functional Tests- These tests are executed to test the functionality of the application or product or product from the client perspective.
- Performance Testing- Ensures Software application will perform well under their expected workload. Provides information about the application’s speed.
Automated performance testing framework enables repeatedly building new level test., agendas without recording a new script, eliminating the additional work that may have been caused by changes to the UI.
- Security Testing- Automated security testing can be applied to tests that are done to check for known vulnerabilities. Other tests may still be manual.
- Service Virtualization- These tools help in designing and executing automated unit, functions, regression, load and performance test for distributed application architectures which leverage SOA, BPH architectures. They help emulate the behaviour of specific Components in heterogeneous Component-based application such as API- driven applications, cloud-based application and service-oriented architecture.
Component tests take a bit more time to run when compared with Unit testing because of multiple dependencies whereas the system and Functional tests take the longest time to run.
The CI System would be configured to automate and run all these types of tests. These tests can be categorized into distinct buckets and run the slower tests (Component, System) at different intervals than the faster running tests (Unit).
The CI System Configuration would run each of these test categories with different staged builds. The slower running tests (System and functional) can be automated through the interfaces of the tools with the CI Server.
The stages for Continous Delivery
Environment provisioning and Configuration
Continuous delivery is an extension to continuous integration for promoting builds through higher environments and performing incremental validation to accrue production readiness.
Various steps like environment provisioning, testing, deployments etc need on infrastructure to be in place for implementation. This infrastructure layer manages all these and the environment management tools help in managing all these environments.
The infrastructure can be managed by tools like chef, which help the spinning of virtual machines mimic servers including the complete operating system, drivers, binaries etc. They run on top of a hypervisor system which in turn runs on top of another operating system.
In a continuous effort to streamline infrastructure, optimize servers and ensure the stability of applications, cloud and virtualization are being used in DevOps. Containerization is complementary to virtualization and is playing a great role in shaping the cloud computing space.
They provide dedicated environments for application to run on and they can be deployed and run anywhere without creating an entire virtual machine set up for each application. Containerization virtualizes the operating system so that the application can be distributed across a single host.
The application is given access to a single operating system kennel. Once the application is built and QA tested, the deployment can be done through containers.
Infrastructure management (virtual/cloud) can be managed (allocation/deallocation) through automated scripts and environment management systems. These can be used to configure, allocate and deallocate test environments also. Thus they provide on-demand infrastructure and just enough infrastructure.
Continuous delivery allows a constant flow of changes into production through an automated software production pipeline called CD pipeline. The pipelines provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new features.
Continuous Delivery enables deliver working, deployable software at any point in time. By running a fully automated build including compilation, testing, inspections, packaging and deployment, the team.
Acquires the capability to release working software at any time and in any known environment.
Following points should be taken into account for automated continuous delivery –
- Reusable multitier application component models and deployment steps.
- Reusable application deployment templates for various application types.
- Orchestration of deployment workflow using such templates, components and associated deploy process.
- Ability to specify environment targets for each application component.
- Environment specific externalised app configuration setup like pre-prod.
- Ability to auto roll back to the defined stable version.
- Role-based security for controlling who can deploy where cross-team dashboard providing the status of deployments along with details of deployed application versions across environments.
Software Release Management ensures all releases through different life cycles of software development is planned, scheduled and controlled. In the context of DevOps, release management plays a critical role and can be called as a catalyst for successful DevOps implementation.
Automation is quite critical in release management to get benefits of DevOps. Few factors can impact the release management such as the unclear definition of roles and responsibilities, unavailability of test environments, improper rollback plan and lack of collaboration among stakeholders for a unified release plan.
Below are some of the key practices which contribute to the release management –
- Communication about the release plan by publishing the release calendar well in advance to all the stakeholders.
- Planning the dependency between releases of multiple applications.
- Definition of workflow and approvals for releases through environments.
- More of Automation to ensure the process is seamless.
- Dry run so that there are no last-minute issues/failures.