DevOps in workspace

In this article, you will learn about how DevOps provides teams with the components they need for success, from product conception through to delivery. You will examine the practices that make up DevOps and its impact on workplace operations.

Introduction to key DevOps concepts

In the end, you should be able to:

• List the Pillars of DevOps and Its Core Practices
• Describe the Benefits of DevOps and How Its Adoption Is Measured

Introduction

Successful business today is all about providing customers with the right products and services, in a timely manner and to a high standard. It is an environment of continuous improvement. If you are talking about software products or services, this standard may relate to things such as availability (24 x 7), appropriate security, ease of use, reliability, ongoing customer support and service, and that they are delivered when promised.

The historical approach to delivering software products followed the waterfall development methodology. Requirements were identified upfront, and work was undertaken in a sequenced, linear manner. Each team worked in isolation on their part of the product and was dependent upon the deliverables from the previous stage. The further along in a project change was introduced, the more costly it was to implement and the greater impact it had on the project’s delivery.

You may recognize waterfall development as sound practice. The required outcome is clear: what you plan is what you get. Timescales are kept, testing is easier, and any potential issues can be dealt with in the design phase.

The waterfall approach has several drawbacks in software development. It values consistency and predictability over innovation, agility, and optimization. While practices and standards associated with IT service management to standardize software development processes have been around for years, a more automated and collaborative approach was needed.

DevOps merges the traditionally siloed departments of Development and Operations into a multidisciplinary team with a shared objective.

“DevOps is the union of people, processes, and products across an organization to speed the delivery of higher quality software to its customers.”

For most organizations, DevOps encompasses more than just development and operations. DevOps is a holistic approach to work. It spans the entire software delivery lifecycle, from product conception to delivery. It also encompasses all business functions involved in the software delivery process, from development through to finance, HR, leadership, security, and operations.

This has led to the title evolving, with terms such as BizOps, CloudOps, DevSecOps, FinOps, ITOps, NetworkOps, NoOps, and others, growing increasingly common. For the purposes of simplicity, this course will focus on DevOps with the understanding that it encompasses more functions than simply development and operations in most workplaces.

There is no standard definition of DevOps. It is a methodology and mindset that vary according to the workplace and the evolution of the concept itself. At its core, it consists of a number of pillars and practices. You will look at some of these pillars and practices in this course. Your workplace may have more or fewer pillars specific to their interpretation of the philosophy.

The three pillars of DevOps you will look at are people, processes, and products.

• People: People are an essential component of DevOps. Without the right people working in harmony with an attitude of shared responsibility, an organization will be unable to realize the benefit of DevOps. The primary characteristics of a DevOps culture are increased collaboration and an environment that fosters growth, continuous learning, and leadership.
• Processes: DevOps is about improving processes. Workplace processes need to enable the efficient flow of work, support business practices, and deliver value to your customers. Anything other than this impedes work and blocks innovation. Automation is a cornerstone of this pillar, freeing people up to concentrate on value-adding activities.
• Products: DevOps products are aids that make adopting DevOps practices easier. They allow processes to be completed more quickly, with greater reliability. These products can include collaboration tools, application management tools, package management tools, and automation tools, just to name a few.

Three practices at the core of the DevOps methodology are continuous integration, continuous delivery, and continuous improvement. Continuous integration and continuous delivery are steps in their own right in the software development cycle. Continuous improvement, meanwhile, is the ongoing effort to improve all other processes over time.

Although the terminology may differ in your workplace, the objective in adopting DevOps remains the same: to release better software faster, more reliably, and to unite the teams involved in software development and deployment.

Continuous Integration

The Agile software development methodology Extreme Programming (XP) first introduced the practice of continuous integration. Integrating work frequently allows tests to be conducted against the most current code base and enables bugs to be detected earlier than they otherwise would be.

In environments with multiple teams working on separate components that rely on other applications, services, modules, and third-party offerings, this is essential. Without early integration, incompatibilities and defects may be missed until late in the development process.

When multiple teams are working on separate build components, with their own interdependencies and third-party dependencies, merging the work of all the teams into one integrated build can be a time-consuming and complex activity.

A performance issue may not be identified until the build is on a production system. A component that worked on the test site may behave unexpectedly in another environment. An unstable build may be rolled out to meet a marketing deadline.

Regular integration of work exposes risks early on and reduces the complexity of merging component builds.

Continuous Delivery

Continuous delivery builds on the practice of continuous integration. It aims to release bug fixes, new features, and changes swiftly by developing software that is deployable throughout its lifecycle.

It is important to note that continuous delivery does not solely mean delivery to a production environment or even the deployment of all changes. The practice refers to the ability to deploy to any environment in the delivery pipeline at any time, as continuous integration produces new builds. This may be delivery to a test, integration, production, or another environment.

Continuous Improvement

DevOps is an ongoing process of improvement. In the real world, no system will be perfect.

Using monitoring and metrics, your organization can analyze its applications, environments, and processes to identify areas for improvement in future iterations.

Metrics to use and how to monitor them will be shaped by your organization’s business drivers and what they are trying to achieve. What is essential is not just gathering the baseline data but running analytics on it, feeding it back into the delivery pipeline, and measuring the impact of any changes. In other words, striving to continuously improve.

So far, you have looked at continuous integration, continuous delivery, and continuous improvement. Many other practices exist which support DevOps. Some of these are listed here, although the list is by no means exhaustive.

Supporting DevOps Practices

• Continuous collaboration
• Continuous feedback
• Continuous monitoring
• Continuous operations
• Continuous planning
• Continuous quality
• Continuous security
• Continuous testing

Adopting DevOps and Measuring Its Implementation

Organizations that adopt DevOps can sustain more frequent deployments, faster cycle times, greater levels of feedback and innovation, faster delivery, more secure and stable releases, and involve stakeholders across the value stream at every stage of delivery.

According to the Puppet 2020 State of DevOps report, 45% of companies that have integrated security teams into the software development lifecycle can remediate critical vulnerabilities within a day. They are 2,604 times faster at recovering from incidents than their competitors.

DevOps shows it is possible to quickly release product improvements to market without sacrificing quality, security, or stability.

Adopting DevOps for a business is a long-term project and requires organizational transformation. The holistic view DevOps takes continuously evaluates an organization’s people, processes, and products, to identify where further improvements can be made.

There are many frameworks to identify where process improvements can be made, including Kanban, the Scaled Agile Framework (SAFe), and Deming’s Plan-Do-Study-Act (PDSA) cycle.

Plan: The planning step identifies the process change or modification to be tested. This involves clearly identifying the objective of the work, the expected outcomes, and the data collection that will be necessary to determine if the outcome is met.

Do: In the do step, you carry out your plans, performing the test or implementing the change. At this stage, you should begin data collection.

Study: In the study phase, data collection is concluded. Compare the data collected to your predictions and reflect on the impact of the change. Was the desired result achieved?

Act: In the act phase, you review the overall cycle. Did the implementation work, or should you do it differently in the next cycle? Do you need to complete the PDSA cycle again with a new change, or are you ready to roll out the change to the wider business?

In such an environment of continual transformation, how is success measured?

An organization needs to know what their business goals and drivers are to assess DevOps success. Development and delivery activities are aligned with the strategy and evaluated to determine to what extent the goals are being met. Value stream mapping is one method to perform this evaluation which you will look at further.

Can your organization do the following:
• Deliver new, innovative applications?
• Leverage modem architectures?
• Easily modernize existing applications?

How do you measure the following:
• Are you delivering value to your customer in your workplace?
• Is it changing? For the better or worse?
• How you can improve upon it?

You may be familiar with value stream mapping already, or you may be invited to participate in a value stream mapping workshop in the future. Value stream mapping visualizes the flow of work, from request through to delivery, to identify bottlenecks and pain points. The outcome is a flow chart of the development lifecycle.

It is important to accurately depict how the current process works and all the steps involved so that true performance metrics can be captured.

Value stream mapping exercises are an opportunity to identify pain points in work processes. They show the entire flow of work through an organization and what may be introducing delays or defects in that process. Many delays can be found in handoffs (wait time) rather than within the work itself.

It is not uncommon for people to be surprised at how many steps are involved in the value stream. Normally, they would not have visibility over the activities of all the departments involved across the organization or knowledge of the complexities of the work they undertake.

Lean identifies eight types of waste in value stream mapping, using the acronym DOWNTIME.

Defects:
Products or services that fail quality assurance (QA) due to mistakes or bugs and require rework, review, or retesting.

Overproduction:
Producing more than is necessary, like extra features.

Waiting:
Idle time due to delays, bottlenecks, system downtime, slow response time, or waiting on approvals.

Non-utilized talent:
Not fully utilizing all employees and their experience, skills, and knowledge. For example, not listening to a team member’s improvement suggestion.

Transport:
Excessive or unnecessary handoffs from developers, to testers, to deployment.

Inventory excess:
Partially done work building up.

Motion excess:
Individual task switching or repetitive activities, code, and development.

Extra processing:
Reworking, such as reformatting, continuing with unproductive processes and techniques because that is the way they have always be done.

Each step in the value stream should be creating value for the customer. Where waste is being created in the value stream, there may be the opportunity for improvements, such as through increased automation; introducing continuous integration, continuous release, continuous deployment, or monitoring; more collaborative environments to reduce handoff times; and more. Any recommended changes to culture, processes, or product are prioritized against businesses’ goals to determine what will add the most business value.

Through this activity, the adoption of DevOps in the workplace not only continues to evolve, but the business also has a tool through the success of the adoption and its growth can be measured.

Target future state:
• Increased visibility for stakeholders
• Faster environment provisioning
• Smaller, more frequent releases
• Automated deployment
• Automated testing and handover

As has already been established, DevOps is the convergence of people, processes, and product for a common business goal. Within a small team, implementing DevOps is easy. However, scaling DevOps to an organizational level, from the C-suite down to new hires, across organizations of all sizes and maturity levels, requires the right environment for such an evolution.

The well-respected annual State of DevOps Report identifies five stages of DevOps evolution. You may recognize your own workplace’s practices in the model and where your organization is in its DevOps journey.

Normalization:
Teams adopt version control, and operating systems are standardized to provide a standard technology base on which to build. Redundant technology is reduced. Trust between teams is built as they acquire greater autonomy.

Standardization:
Technical variation is reduced further. Teams deploy on a standard operating system and set of technologies, reducing the overhead in managing new applications, services, and technologies and increasing the time for work and innovation. Collaboration is expanded beyond the functional boundaries of “Dev” and “Ops”.

Expansion:
Individuals can work without manual approval. Bottlenecks are identified, and bureaucracy is reduced to increase productivity and empower decision-making. Reusable deployment patterns are developed, and the ability to test infrastructure changes before production is provided. Automation increases beyond problem-solving.

Automated Infrastructure Delivery:
System configuration and provisioning are automated. System configurations, infrastructure, and application configurations are all under version control. Security policy configurations are automated.

Self-Service:
Incident responses are automated, and resources are available via self-service. Security is involved in technology design and deployment. Automation is deployed to eliminate waste across multiple delivery streams.

This topic looked at the benefits DevOps practices offer in improving development time, increasing team flexibility and agility, and creating value for the customer.

Adopting DevOps is not a project with a defined start, middle, and end but a journey of continuous improvement and evolution. You also looked at the stages of DevOps maturity for organizations implementing DevOps in the workplace.

Developing a Culture for Success

In this, you will look more closely at one of the pillars of DevOps: people. The cultural characteristics required to make DevOps adoption in the workplace a success will be discussed.

At the end, you should be able to:

• Describe the Characteristics of a DevOps Workplace
• Describe the Characteristics That Contribute to a People-Centric Culture
• Describe the Structure of a DevOps Team and a DevOps Mindset

Previously, we introduced the three pillars of DevOps: people, processes, and products. It is common for one or another of these pillars to be held up as the key component to DevOps success. And while it is true that a DevOps implementation without any one of these pillars will fail, it is critical to remember that all are equally important.

In this, you will look at people and how they contribute to the foundation of DevOps success.

When people are discussed in regard to DevOps, this is not just referring to the individuals who make up a team and their skillsets. People in DevOps speaks of their mindset, attitude, and environment. You might also know this as culture.

An organization can adopt the most efficient processes or select the best products, but these investments will be worthless without the people who execute those processes and use the products. DevOps requires the right people working in harmony with an attitude of shared responsibility. These are the learned behaviors that give your organization its character and personality.

A workplace’s culture determines:
• The unspoken and unwritten rules that bond colleagues together
• How people interact with each other
• The frequency with which people communicate
• The people who are hired, fired, and promoted
• The personalities, beliefs, and behavior of people

DevOps requires continuous, transformational change, a process that organizations hold an inherent resistance to. In large organizations, with many thousands of employees and years of perfected processes and procedures, the culture DevOps calls for can be challenging to adopt.

Breaking the “this is the way things have always been done” mindset can be difficult, yet people can excel when empowered with the confidence and tools found in dynamic, flexible environments.

Fostering a workplace culture that provides employees with:
• Confidence
• Business faith
• Empowerment
• Development
• Support

Let's people excel and leads to employees who are:
• Passionate
• Committed
• Supportive of others
• Dedicated
• Innovative

So, what are the characteristics of a DevOps culture?

DevOps requires a culture of collaboration, continuous improvement, responsibility, and transparency.

Collaboration:
For DevOps teams to function successfully, members must all work cohesively and in harmony. Good communication helps to create mutual respect and reduces mistakes.

Continuous improvement:
Continuous improvement of people in DevOps focuses on the professional and personal development of team members, to develop more resilient, skilled, and responsive teams.

Responsibility:
DevOps team members should operate under an umbrella of shared ownership and responsibility. This helps the team support each other and work together as a cohesive unit.

Transparency:
Transparency underpins the success of the other characteristics of a DevOps people culture. Enabling cross-functional visibility of work activities encourages collaboration, permits multiple viewers for improved reliability, and creates shared responsibility in the activities being undertaken.

DevOps brings together individuals from different business areas into unilateral teams with a common objective. This variety enhances project success. However, breaking down the functional silos that can exist between team members and the barriers between the departments they traditionally work in can be a new way of working for many.

Collaboration is key. With many employees increasingly working remotely and possibly even based in geographically distant locations, building a culture of collaboration can require more effort. Making closer connections to team members in other areas of the business may be the key to your team achieving its mission.

Good lines of communication are established through the following ways:
• Daily standups to discuss successes and failures
• Using online platforms to foster daily interaction
• Circulating newsletters
• Carrying out surveys and requesting feedback
• Being open to new ways of working

Ensuring everyone understands the vision, aims, and objectives of the team and their role in it.

The pursuit of continuous improvement is an important goal in DevOps, across not just the processes an organization adopts and the automation products it pursues, but in the continuous improvement of the DevOps team too.

Developing new features, rapidly evaluating customer behaviors, and implementing new requirements within short timeframes are demanding tasks that require the right skills and confidence to perform.

Fostering a culture of growth and empowering people to do great work can only result in great things.

Initiatives to support continuous improvement in a DevOps team:

1. Regularly review budgets to fund innovative ways of working
2. Fostering positive, people-centered workplaces
3. Welcoming and adopting new company initiatives
4. Information sharing on business targets and their role in achieving them
5. Encouraging continuous suggestion sharing from all
6. Open and approachable leadership committed to change
7. Nurturing growth and an appetite for lifelong learning

As cross-functional DevOps teams collaborate on projects, they obtain a greater insight into and respect for the activities of their colleagues. This understanding leads to team members working together to deliver a final product to the business’s objectives, rather than working in isolation on the sole task they have been allocated.

Organizations whose DevOps teams approach the development lifecycle with a collective attitude to shared responsibility are able to iterate faster and produce more successful and reliable results.

Siloed teams:

DevOps teams:

Everyone has access to a single source of truth for incident data and deployment history.

Underpinning the characteristics of collaboration, continuous improvement, and shared responsibility of a DevOps people culture is transparency. Whether a requirement originates from a customer request, bug-fix, new feature, or business need, the audience for the development activity is not limited to a single role or business area.

Instilling transparency across incident and development workflows enables everyone in the organization to have access to a centralized, real-time source of truth informing them on the progress of change. Transparency encourages greater collaboration, drives continuous improvement, and nurtures shared responsibility in the work undertaken.

A DevOps culture in the workplace unifies focus on the customer. It emphasizes shared responsibility for work being created and maintained, fostering collaboration between all departments and roles.

So far, you looked at one of the pillars of DevOps: people. DevOps brings together team members with different areas of expertise and breaks down silos between departments. You looked at how workplace culture and its people contribute to DevOps success.

DevOps Mindsets

Realizing the transformational change required to adopt a DevOps mindset takes time and commitment. Individuals bring with them their own history and biases to the workplace. An organization aiming to work together collaboratively without functional silos or departmental boundaries requires the highest levels of the business championing this change.

Now, you will look at the structure of a DevOps team as well as approaches to developing a DevOps mindset.

Siloed departments:

Person 1: The development team does not care about how long it takes us to provision an environment.

Person 2: The deployment problems are not our responsibility. We completed our handover already.

Person 3: Let’s complete our work quicker to show we are better than development.

DevOps workplace:

Person 1: The deployment team might have some good ideas to introduce into our smoke tests.

Person 2: If we work on this together, we will get the work completed in half the time.

Person 3: Let’s work with QA to reduce the number of staging failures.

DevOps team members will each have their own area of expertise. Some may be specialist software engineers, while there will also be operations and security experts. A good DevOps team brings together its diverse team members, breaking down the barriers between their departments and creating an understanding of each other’s role in the development lifecycle. It adds the communication skills, adaptability, and mindset necessary to help them constructively work together toward common goals.

The size of a DevOps team will be different in every organization; however, the roles shown here are generally accepted as being necessary for success.

DevOps evangelist:
An organizational leader who champions DevOps. The DevOps evangelist ensures the necessary teams adopt the DevOps methodology, authorizes and encourages skills training, empowers employees, and identifies key roles. They promote the benefits of DevOps to the business and ensure DevOps strategy is not just discussed but adopted.

Cloud architect:
Cloud architects are technical experts. They identify the architecture and design necessary to support the technical requirements of a project’s business requirements and, once fulfilled, look for opportunities for upgrades and innovation.

Software developer/tester:
DevOps software developers and testers have a broad range of responsibilities, undertaking development as well as conducting unit testing, deployment, and ongoing monitoring.

Experience Assurance (XA) professional:
Experience assurance professionals evaluate and monitor feature releases with a focus on the user experience.

Security engineer:
DevOps security engineers work side-by-side with developers to ensure security considerations are addressed in product development from day one.

Automation architect:
Automation architects analyze DevOps tools and processes to design and implement strategies for the automatic management of applications and infrastructure between non-production and production environments. They focus on how development processes can be made faster, more reliable, and more efficient.

Release manager:
Release managers oversee the technical management and coordination of product development. They focus on end-to-end delivery, including integration, the flow of development, testing, and deployment to support continuous delivery.

Irrespective of the role you hold on a DevOps team, everyone needs to work together to contribute to the team’s success. DevOps workplaces require a company culture that is collaborative and experimental. The DevOps culture should extend beyond the team and encompass the entire organization. Everyone needs to share the same values and be passionate about delivering value.

Staff development activities to cultivate talent within the workplace focus on engineering behaviors that reflect the desired cultural state. You are likely to be familiar with many such development activities already, from your own workplace. Think about how they were conducted. What characteristics are your workplace reinforcing?

Measures to develop a DevOps people culture:
• Empower staff
• Cross-train team members
• Promote wider collaboration
• Align staff incentives to the desired culture
• Change reporting lines
• Hold competitions to incentivize improvement
• Encourage different voices to speak up
• Build a lifelong learning culture

DevOps processes also support the development of a DevOps culture by enabling the efficient flow of work, providing software tools for collaboration, and encouraging automation. These processes free people up to concentrate on value-adding activities.

Confidence: The increase in efficiency that automation of tasks provides builds employee confidence in the robustness of the change management process. Variability is reduced, success rates increase, and employees take more decisive decisions with self-assurance in the choices they are making.

Collaboration: Collaboration is nurtured by making the performance and quality impact of automated tests visible to everyone with every iteration, By providing immediate insight into the impact of their work, team members are compelled to work together to develop solutions.

Development: Automation enables measurement, metrics, and monitoring. This data can be utilized by management to make strategic decisions regarding employee development, such as investing in cross-training, sending the employee on a course, or running a team-building exercise. All these activities foster a culture of continuous learning and development.

A workplace following DevOps practices can be a dynamic and fast-moving environment. Chasing innovation, change, and continuous improvement can sound like a recipe for disaster for some: introducing defects, uncontrolled scope creep, and increasing rework. However, countering these risks with rigid standardization stifles experimentation and makes it difficult to realize the benefits of a DevOps approach.

Clearly, a level of standardization and good governance is necessary. Centralized standards can be specified for required tasks, such as penetration testing or meeting regulatory obligations. In other areas, recommended tools the organization favors can be promoted to encourage common approaches, but employees are not bound to these. This approach provides teams the tools to adhere to standards where necessary and the flexibility to innovate where better approaches may exist.

Building and standardizing better operational models, workflows, and
tools on the back of innovation enables enterprises to grow faster,

Good governance for a DevOps team also means team members taking responsibility for their actions and having an awareness of their impact. Each individual is the first line of defense in managing risk. The smallest of code changes can have unintended consequences, and continuous integration can quickly turn into continuous mistakes.

While your workplace will utilize technological solutions to manage risk, the key barrier to achieving good governance is workplace culture. Governance improves quality and efficiency, and when team members help each other out and work collaboratively, everyone benefits.

The way people are measured will lead how they behave. Like tools, processes, and procedures, organizational measurement techniques also need to be standardized. Although teams may be working on different technology stacks, a common middle ground to measure their value to the organization is required. Usage, velocity, and live site health thus become more constructive benchmarks over traditional measures of lines of code and team burndown. The goal is to measure outcomes, not activity.

Example DevOps Team Metrics
Lead time: Time from development to production deployment.
Mean time to recovery/repair: Time to resolve an issue from the time it is reported.
Deployment success: Percentage of successful deployments.
Project count: Number of projects completed per period.

So far in the article we discuss hypothetically how DevOps could be implemented, but what about in the real world?

State Farm is a large American insurance company. Historically, heavy industry regulation leveled competition in their market. However, recent deregulation has opened the market to newer and more agile digital competitors, putting pressure on State Farm to innovate faster and raising customer expectations.

To be their customers’ first and best choice in insurance products, State Farm has pursued DevOps across the enterprise, innovating on all platforms.

Continuous Improvement: Many tasks that were previously manual and dependent on human effort at State Farm are now automated. The introduction of automation saves time and effort for developers, who can focus more on innovation, creating a positive cycle of continuous improvement.

Testing: As soon as code leaves developers’ fingertips, State Farm can provide security tips and rapid feedback about potential issues. This is supported by development tools which include an integrated debugger, enabling developers to immediately check and refine their code in context.

Standardization and Collaboration : The addition of tools such as Git, and Jenkins has helped State Farm extend DevOps practices consistently throughout the organization. Collaboration has become an expected part of State Farm’s culture and has been built into their practices. The company is agnostic about development tools, and employees are not forced to use one single solution. They are provided with support to achieve greater efficiency and speed and also to feel comfortable, so everyone can work together across all platforms to deliver rapid innovation.

Productivity Improvements: The availability of flexible and modern tools across all platforms enables product teams to keep pace with newer applications, cutting time-to-market. Even people who were initially skeptical are very supportive of the change, which is translating into shorter development cycles.

A DevOps culture is not something that can be implemented out of a box. So far, we demonstrated that although achieving a DevOps mindset takes time and commitment, the results can be rewarding.

As mentioned earlier, people are only one pillar of DevOps. Next, you will look at some of the processes and products which underpin the DevOps methodology.

A culture of learning provides a natural mechanism for people to grow, pass on information and evolve attitudes, styles, and processes. It does not happen automatically. It takes effort, awareness, and most importantly, collaboration.

Core Practices

This part introduces the practices at the core of the DevOps methodology, continuous integration and continuous delivery, and the tools and processes that enable them to occur. These practices are key to optimizing the delivery pipeline and delivering projects in a timely manner that is reliable and secure.

At the end, you should be able to:

• Outline the Benefits of Continuous Integration and the Tools Utilized to Achieve It
• Distinguish between Continuous Delivery and Continuous Deployment
• Identify Best Practice Processes for Adopting Continuous Delivery and Continuous Deployment

As you know already, DevOps is a holistic approach to work that spans a wide range of functions in the software development lifecycle. At the core of the DevOps methodology are the practices of continuous integration and continuous delivery. Without these essential elements, DevOps does not exist.

Continuous integration (CI) refers to the practice of merging code changes into a central repository. Through frequent code commits, automated builds and tests are run, allowing bugs to be identified and rectified earlier.

Goals of continuous integration (Cl):
• Reduce time to find and address bugs
• Improve software quality and stability
• Reduce time to validate updates
• Reduce time between software releases

In traditional software development, teams of developers worked on separate functions, tasks, or the interaction of the software with external applications or services. Bringing together the products of these different teams into one stable release was complex and time-consuming.

By integrating early and frequently, bugs can be identified and addressed early in the build process, and unknown dependencies or risks are made apparent.

By integrating work regularly and conducting tests, integration incompatibilities and defects between build components can be swiftly discovered and fixed.

For organizations working on enterprise systems, this process may span multiple teams integrating code with complex technology and deployment dependencies. Breaking work down into small chunks in this environment helps to identify risks and isolate them earlier.

To keep track of application code, and application and system configurations, they need to be under version control. Small code changes are committed frequently during the practice of continuous integration using version and source control tools. Implementing version control allows you to do the following:

• Collaborate on code and track changes over time.
• Identify issues, and then roll back to known states, often at the push of a button.
• Get changes into the production environment efficiently.
• Replicate any environment reliably and quickly.
• Quickly compare and identify differences between builds and between environments.

Version control tools fall into two categories: centralized and distributed.

Centralized version control tools work with a single, central repository. Developers work with their own local copy and pull any changes made by other people, from a central repository. They complete their work and, when ready, commit their changes to the central repository for all to see. Once a commit is made, other developers can pull the change to update their local working copy.

Centralized version control tools:
• Azure DevOps Server with TFVC
• ClearCase
• IBM Rational Team Concert
• Perforce
• Subversion (SVN)

When using a distributed version control tool, every developer has a copy of the entire source repository on their machine. This allows them to see the full project, including all branch, metadata, and historical information.

As every developer has their own clone of the project repository, changes can be shared among individuals or small teams for collaboration and review before being committed to the authoritative repository.

Distributed Version Control Tools:
• Azure DevOps Server with Git
• Bazaar
• Bitbucket
• Git and GitHub
• Mercurial
• Monotone

A number of best practices should be observed when adopting continuous integration into your work activities.

Commit early and frequently: A fast pipeline enables the quick turnaround of product feedback. Bugs can be swiftly identified, and you gain user feedback on new features and ideas. Committing changes regularly also avoids merge conflicts.

Testing before commiting: Test thoroughly before committing changes. In addition to checking your code syntax is error-free, check the code behaves as expected and does not produce any side effects.

Do not commit incomplete work: Incomplete work, debugging code, log statements, or code of poor quality should never be committed. If incomplete code has been committed to the repository, there is a chance that other developers, who are pulling the latest builds and committing updates, may build functionality on top of the incomplete code. Only commit finished work.

Provide meaningful commit descriptions: Each commit should have a single purpose. Provide a clear, concise description to accompany the commit explaining why it is being made. If available, the commit message should also reference any related issue or requirement ID.

Use code reviews: Pull requests will notify your colleagues of changes ready for integration pending a code review, or you can request their input. Participating in pull requests/code reviews provides the opportunity to gain insight into other developers’ work, new code, and functionality, It allows for communication and knowledge sharing outside of your team, feedback on your own work, and is a powerful tool in the continuous integration process.

Maintain a separate configuration repository: Configuration repositories are used to store environmental settings, assets, configurations, and their dependencies. This provides a single point of reference for the assets necessary to deploy the application or one of its components.

Use branching: A branch is a copy of the codebase that is worked on independently from the source repository. It allows developers to work in parallel and experiment in isolation while still following source control best practices. Stable branches are merged back into the main codeline. Ensuring branches are merged frequently minimizes conflicts between branches and the main repository.

With code changes continuously being submitted to the authoritative repository, everyone in the software development lifecycle becomes responsible for the security, quality, and timeliness of deliverables.

This spans basic checks, such as that your code syntax is error-free prior to making a commit, to other considerations, such as security risks your code or development activities may be introducing. Continuous integration exposes an organization’s most valuable assets, and securing the pipeline must be taken into consideration in your day-to-day work.

Information stored in a version control tool:
• Intellectual property (IP)
• Source code for applications used by you or your customers
• Product designs
• Videos, graphics, or images
• Business documents

Organizational measures to protect its pipeline:
• Back up and encrypt data at rest and in transit
• Utilize access control
• Partition branches according
to intent of change
• Audit change activity
• Monitor for threats

Quality assurance (QA) practices improve security, performance, and stability.

QA team members will define the most suitable tools and frameworks for early testing so that the builds that reach staging and production are only those that are ready to be delivered to users. By configuring test cases early into the DevOps chain during the practice of continuous integration, quality assurance aids developers in keeping their code as clean and error-free as possible and helps automate large sections of the development pipeline.

QA automation
• Automating test cases
• Automating regression testing on critical areas
• Aiming for 100% code coverage
• Standardizing test environments
• Automating deployment of QA boxes
• Automating pre-testing, cleanups, and post-testing

QA priorities
• Acceptance tests
• Integration tests
• Performance tests
• System tests
• Unit tests
• White box security tests

So far, we examined the practice of continuous integration and its use to merge code changes early and often. This permits QA to run tests against the most up-to-date codebase. Continuous integration cannot be completed without the use of version control software, tools, and services to automate the build process and tests.

These efforts help the DevOps team to identify and rectify errors and integration incompatibilities as they occur.

Fun fact: Highly evolved DevOps workplaces deploy code multiple times a day, 973 times more frequently than their counterparts.

Continuous delivery (CD) builds on continuous integration by automating the release process in the software development lifecycle pipeline. It provides the means to release changes that pass initial testing to subsequent staging and production environments.

The term continuous deployment (CD) is used when changes are released to a client-accessible production environment automatically, without explicit manual approval.

Goals of continuous delivery/deployment (CD):
• Reduce release risk
• Reduce deployment downtime
• Decrease time to market
• Improve software quality and stability
• Increase business agility

Continuous delivery is the capability to deploy to any environment in the development pipeline as needed, in a repeatable, reliable, scalable manner. It does not mean the deployment of every change to production. Additional testing may be required, and there may be further business and technical approvals before an application is ready to release.

What can continuous delivery deploy?
• A full application
• An application component
• Application content
• Configuration changes
• Middleware configuration changes
• An environment
• A combination of any of the above

Multiple environments will be in use in the workplace: parallel staging environments, testing environments, and production environments. Rolling out builds to these environments enables modifications to be verified and testing to be automated across multiple dimensions, including user interface testing, integration testing, and load testing.

Continuous delivery removes handoff bottlenecks between application developers and operations by permitting the operations team to deploy from the version control repository to any environment at will.

Continuous deployment increases the efficiencies gained from continuous delivery by automating the release of changes that pass checks onwards to production and client-facing environments without manual intervention.

Deploying software involves a large degree of risk. Despite all the checks and quality gates it has passed through, there may be unforeseen problems in production. As platforms grow, infrastructure expands, and projects become more complex, balancing the risks of a deployment against the needs of your customers and the benefits promised by pending updates can be challenging.

To minimize and mitigate the impact of deployment on customers, several release management measures are available. Moving forward, you will look at the commonly encountered techniques.

In DevOps, the same team handles development and operations, and deployment happens continuously. A release occurs when the product, app, service, or feature is made available to end-users.

Release management considerations include the following:
• Are your primary users innovative early adopters?
• What is your application topology?
• What is the cost to convert your current infrastructure?

Ring-based deployment is a form of release management. It reduces the risk of releasing an update by limiting the audience to a smaller segment — or ring — which progressively increases in size as the release is validated.

Feedback from the initial rings can be used to inform the decision of whether to continue with the release to further exposure groups or not. Continuous deployment can be used to automate the deployment from one ring to the next, or manual intervention may be required.

With a blue/green deployment, there are two almost identical environments. In the example here, the green environment is currently a live production environment that all of your users are directed to. The blue environment is a staged version containing the next release of production-level software. Until you are ready, only your developers and testers have access to it.

When it is ready to be deployed as the production/current version, increasing numbers of users are directed to the new blue release through load balancing techniques, or they are switched over completely. If the release is successful, all traffic is directed to the blue release. The process is repeated with every release, switching between blue and green.

The blue deployment may be on different hardware, different virtual machines, or a single operating environment partitioned into separate zones with separate IP addresses.

Advantages:
Blue/green deployments provide an opportunity to test your disaster recovery procedures with every release and your ability to set up a hot standby.

Disadvantages:
Blue/green deployments require organizations to maintain two identical hosting environments, doubling infrastructure costs.

Your workplace will utilize multiple release practices. Exposure to experimental, high-risk feature changes may be limited to a small number of early-adopter customers. Whereas, a low-risk performance update may swiftly move through the blue/green deployment route.

Feature Flags: Feature flags are utilized when there is a requirement to deploy a functional update with limited customer exposure. The flag controls which customers gain access to the feature. For those that have the flag turned off, the code does not execute, and the normal functioning of the software is not impacted.

User opt-in: User opt-in limits exposure to updates based on the user’s risk tolerance level. The new code or feature is not executed unless a customer chooses to activate it.

Canary deployment: A canary deployment releases an application or service incrementally to a small group of users more tolerant of bugs. For example, an update may be released to employees initially. The subset of users exposed are the canaries. Canary deployment differs from blue/green deployments as it does not use two production environments, so is a cheaper deployment technique.

Here, you discovered how the use of automation to facilitate continuous delivery means that delivering code updates to new user environments is painless. A stakeholder request to deploy the current version of software to a new environment can be fulfilled at a moment’s notice.

Release management practices apply structure and governance around continuous delivery activities to minimize the risk associated with them.

Tools for Measurement and Monitoring

In this, you will look at the practice of continuous improvement and how it underpins the evolution of all other practices to further the success of DevOps in the workplace.

At the end, you should be able to:

• List the Types of Monitoring Activities Necessary to Identify Where Improvements Can Be Made
• Describe the Types of Telemetry and Metrics Monitoring Activities Return, and How They Can Be Used

One of the core practices at the heart of the DevOps methodology is the goal of continuous improvement. Implementing DevOps in the workplace is not a project with a defined start, middle, and end. It is an ongoing journey.

Achieving continuous improvement requires recognizing where efficiencies can be realized, identifying what enhancements can be made, and learning from the work just delivered. The steps necessary to achieve these aims are identified through the practice of continuous monitoring.

Continuous monitoring refers to the activities required to monitor the health, performance, and reliability of your application, infrastructure, and processes, as work moves from development to production.

It seeks to apply the same level of rigor across all areas of possible improvement as an organization pursues continuous improvement.

• Is the application functioning as expected?
• Could a handover have been completed quicker?
• Did the work result in a measurable increase in customer activity?
• Are the environment’s performance service-level agreements (SLAs) being met?

Organizations achieve true efficiency and scalability by utilizing continuous monitoring to provide real-time data for analysis. Prompt incident identification via automated alert systems refers work to the right people as soon as a problem is encountered, minimizing the impact on uptime or revenue.

Benefits of continuous monitoring:
• Network visibility and transparency
• Rapid incident response
• Minimization of system downtime
• Increased decision-making capability
• Improved business performance

The practices of continuous integration, continuous delivery, and continuous deployment increase the pace of organizational change. There may be many builds in various stages of production, running thousands of workloads, utilizing microservices and micro front-ends across hundreds of cloud environments. The processes producing work in this development pipeline are complex. Implementing mechanisms to help track the impact of updates, provide visibility, and enable teams to detect and respond to incidents is critical.

Moving forward each of these will be discussed in detail.

Application monitoring

Application monitoring provides insight into the performance of your applications and web services. This includes end-to-end transactions, connections, and the overall stability of the back and front-end.

Data on an application’s uptime, transaction time, transaction volume, system responses, and API responses allow the performance of the application to be monitored for faults or identification of areas where improvements could be made.

Availability: Availability monitoring checks that the application, site, or app is not compromised or experiencing an outage and is available 24 hours a day and 365 days a year.

Browser speed: Browser speed monitors a site’s ability to respond to end-users. More advanced performance monitoring will take into consideration variables such as file size, number of files, system architecture, user’s geographical location, device type, and connection speed.

End-user transactions: Transaction monitoring measures conversion and bounce rates, from the initiating browser click, tap, or swipe, through to the back-end application.

Error rate: Error rate monitoring tracks the percentage of HTTP response codes or other errors that are triggered in using your application, site, or app.

SLA status: Service-level agreement (SLA) monitoring provides documentary evidence to support that SLAs are being met and allows the business to ensure appropriate levels and quality of service are being delivered.

Third-party resource usage: Monitoring third-party resources ensures that if they have an outage, you know before your customers. This ensures you can keep your commitments to your customers.

Throughput: Throughput is a measure of the requests per minute through an application. It allows you to see performance trends at a glance.

Response time: Monitoring response times can help you identify which components, queries, or requests may be causing delays that the user is experiencing.

Delivery pipeline monitoring

Delivery pipeline monitoring allows visibility of the entire end-to-end delivery pipeline. It provides insights into how DevOps teams are working, exposes bottlenecks, and provides information on the speed and quality of the products being delivered.

Delivery pipeline monitoring activities include the monitoring of builds, commits, deployments, development milestones, and quality.

Infrastructure monitoring

Infrastructure monitoring provides full observability across the entire technology stack, providing continuous insight into its health, performance, and availability. Monitoring its IT infrastructure is necessary for any organization that depends on it to deliver its products and services.

The IT infrastructure in the modern workplace spans hundreds of connected devices, containers, hardware, servers, software, storage, virtual machines, and more. Monitoring these large and complex environments cannot be achieved without infrastructure monitoring tools that allow automated responses to be configured.

CPU and disk usage: CPU, disk usage, memory utilization, system temperature, and other hardware monitors provide information on the health of hardware assets and warn of server performance degradations or outages.

Database health: Database monitoring tracks performance to maintain infrastructure availability and prevent outages or slowdowns. Database monitors include query details, session details, scheduled jobs, replication details, and database performance.

Network switches: Network switch monitoring ensures networks remain healthy and provides prompt alerts when a switch or switch port goes down.

Process level usage: Process monitoring ensures that business-critical applications, files, or sites remain functional. If a service fails, corrective action can be swiftly taken.

Security: Security monitoring includes monitoring firewall rules and policies, compliance status, suspicious network activity, intrusions, and triggering alarms to alert operators.

Server availability: Servers are the heart of IT infrastructure. Monitoring their performance in large enterprises, where services may span multiple operating systems, as well as dynamic and virtualized instances, requires monitoring tools to swiftly identify problems when they occur.

User permissions: User permission monitoring detects privilege escalation where a malicious user may be exploiting a bug, design flaw, or configuration error to gain elevated access to privileged resources.

Network monitoring

Network monitoring monitors and tracks activity on the networks your infrastructure is operating on. This includes the status and activity of components like firewalls, routers, servers, switches, and virtual machines.

The primary goal of network monitoring is to prevent downtime, failures, and crashes.

User sentiment monitoring

User sentiment monitoring measures the user experience. Customer and employee experience is critical to the success of an organization. It analyzes whether the user’s experience is positive or negative and whether the organization’s work efforts are delivering business value.

User sentiment can be actively sought: through interviews, surveys with the customer, or online feedback forms. It can also be submitted by the customer: to the organization directly or expressed in the public domain on social media.

User sentiment monitoring sources include the following:
• Email
• Interviews
• Reviews
• Social media
• Surveys
• User-submitted reports

Monitoring tools

Popular infrastructure monitoring tools include Nagios, ManageEngine OpManager, Prometheus, Paessler PRTG Network Monitor, Sematext, Solarwinds, and Zabbix. If you are accessing Infrastructure as a Service (IaaS), then the vendor is likely to make available their own tools, for example, Amazon uses AWS CloudWatch.

Network monitoring tools include Cacti, ntop, nmap, Spiceworks, Traceroute, and Wireshark.

Many monitoring tools combine application and server monitoring, providing insight into the performance of applications and the services that support them. High-end enterprise tools such as BMC Helix Operations Management, BigPanda, and PagerDuty analyze events and alerts from existing monitors to determine any correlation between them and provide DevOps users with a consolidated list of issues.

So far, you looked at the goals of continuous monitoring, the need to monitor the entire application stack, and what this encompasses.

With increasingly intricate applications and deployment environments, sophisticated monitoring is both vital and complex to the modern organization.

Outages and performance degradations have a massive impact on productivity and consumer satisfaction. Identifying the source of any fault and its underlying cause so remedial action can be taken requires visibility and monitoring across the entire infrastructure.

Continuous monitoring identifies where efficiencies can be realized, enhancements made, or errors fixed. In this section, you will look at the metrics continuous monitoring produces to start the development lifecycle again.

The primary output of continuous monitoring activities is data. Combining the data continuous monitoring activities return and presenting it in a meaningful way allows an organization to gain insight into its true performance, measure quality, and predict the capacity of its processes. A well-monitored development pipeline will allow any anomalies to be spotted immediately.

The types of measurements an organization utilizes to conduct application, delivery pipeline, infrastructure, network, and user sentiment monitoring are:

Synthetic monitoring: Synthetic monitoring uses a consistent set of transactions to assess performance and availability. The quantitative data captured in or near real-time allows changes over time to be identified.

User monitoring: User monitoring measures the user’s experience from their laptop, desktop, or mobile device. Conditions such as mobile networks, internet routing, and caching will impact the metrics retuned.

Logs: Logs are records of events describing changes taking
place and activities occurring in or on a system.

Traces: Traces are records describing every step or operation performed by a system component. Traces can be enabled on-demand or by inserting code into the component to generate them. Due to the high computational overheads, they are generally only used for specific tasks over short periods of time.

Alerts: Alerts are issued by tools or systems if an underlying problem is identified or an alert threshold is triggered. They prompt manual action or alert the operator that an automated response to the alert is taking place.

Notifications: Notifications inform the operator about standard or expected events that have occurred or been undertaken. For example, when a build passes testing, or a password is reset.

Dashboards collate the output of an organization’s continuous monitoring activities into a centralized location that allows DevOps teams access to the same telemetry and tools. Everyone is provided with visibility of the software development pipeline and may have access to additional dashboards tailored to their role.

Moving forward, you will look at the most common DevOps metrics that allow the pipeline to be analyzed and automated actions initiated in response.

Lead time for change and cycle time are two important, related metrics.

Lead time for change is the time it takes from when a code commit is done to when it is in a deployable state. This is a measurement of the time taken for the code to pass all pre-release tests. Cycle time is the time it takes from work inception to delivery. The target is to make the development, test, and feedback cycle as short as possible so improvements can be rapidly implemented without sacrificing quality.

Sub-cycles found within the delivery cycle:
• Acquisition cycle time
• Approval cycle time
• Change management cycle time
• Datacenter latency cycle time
• Dev-test cycle time
• Financial approval cycle time
• Management approval cycle time
• Ops deployment cycle time

Change failure rate is the percentage of changes that fail in production. It is a measure of stability and evaluates the efficiency of the deployment processes.

26% of organizations have a change failure rate of less than 15%.

As you have seen earlier, an ideal continuous deployment process will be automated, resulting in increased measures of consistency and reliability. Manual deployments may see a higher change failure rate as they require a lot of small, repetitive changes with every deployment that carry the added potential for human error.

Failed builds are a normal part of the development process for employees of all skill levels. The key is identifying errors early in testing and preventing them from reaching deployment.

Mean time to recovery (MTTR) is a measure of the time it takes to implement a fix to a problem. This could be the recovery time from a service outage or the failure of a deployment.

Antifragile systems accept that incidents occur and focus on how to optimize the processes to restore service. Identifying and removing bottlenecks in the resolution workflow can reduce the mean time to recovery and help the DevOps team minimize the impact of incidents.

26% of organizations recover in less than an hour.

Deployment frequency is a measure of the number of times new code is deployed to production.

High-performing, successful DevOps teams deploy smaller, more similar components more frequently. Smaller changes are easier to test, easier to deploy, and easier to roll back in the case of failure. Each deployment also provides the opportunity for earlier customer feedback, so work effort can focus on items the customer actually wants.

Ref: Smith, D, Villalba, D, Irvine, M, Stanke, D, Harvey, N, (2021), Accelerate State of DevOps 2021.

Another important metric to monitor is outcome effectiveness. Work can be validated by tracking usage and comparing product versions to measure the impact of change. If the deployment of a code change is supported by the expected uptick in user activity, the DevOps team can continue their work. If usage does not reflect the expected behavior, and user sentiment indicates a negative response to the change, the team may need to pivot in their work activities.

The team can utilize usage information to measure the impact of their changes and make data-informed business decisions.

Incidents within the software development lifecycle are to be expected, from hardware and network failures to misconfiguration, resource exhaustion, data inconsistencies, and software bugs.

How you respond to incidents and react to events at scale before they affect users determines an organization’s ability to provide stable and reliable products. Organizations employ multiple tools to help manage this process. While you may not be involved in defining what constitutes an incident, in DevOps teams, everyone is involved in incident response.

Use workflows and playbooks: Pre-defined workflows and playbooks define a predictable set of operational steps to be taken when an alert is triggered. This allows automation to be leveraged for ticket allocation and asset management, and ensures the right people get the right information needed in a timely manner to commence working on a resolution.

Maximize automation: Automation helps organizations respond to incidents faster. Automation may be as simple as the initial on-call notification to a more complex fix for a common recurring issue. Alert routing and escalations can trigger additional automated processes, from notifications to resource allocation.

Encourage collaboration: Build on the culture of collaboration that DevOps encourages by integrating incident response into the tools used elsewhere in the development lifecycle. For example, real-time chat tools can integrate with service incident platforms so anyone can get up to speed at any time on what is being done and where the incident response is at.

Learn from each incident: Postmortems allow team members to share their experience on how the incident was handled, what could have been done differently, and help everyone do their jobs better in the future.

Another type of incident the DevOps team may need to respond to is security incidents. DevOps security is a continuous undertaking that requires everyone to participate. Like the other types of incidents examined on the previous page, every team should have a response plan for how to deal with security incidents.

Many organizations use war games so teams can practice detection and their mean time to recovery. Should a breach happen, exposure and recovery time are minimized due to the team’s prior preparation.

Every team should have practices in place for preventing breaches.
• How will you detect an attack?
• How will you respond if there is an attack or penetration?
• How will you recover from an attack?

Organizations that integrate security into their software delivery process can remediate critical vulnerabilities within a day.

DevOps metrics help determine whether the feedback and actions being taken are saving time and increasing productivity. In this, you looked at how continuously capturing and responding to feedback is key to delivering resilient, reliable, quality software to end-users. The same metrics also provide the data necessary for frameworks like Deming’s Plan-Do-Study-Act (PDSA) — introduced earlier — which allow organizations to evaluate the overall impact of DevOps and identify where further improvements can be made.

Everyone in the DevOps team is involved in striving for continuous improvement and activities to remediate incidents when they are encountered.

Conclusion

I know it was a long one. I hope you’ve enjoyed it and have learned something new. I’m always open to suggestions and discussions on LinkedIn. Hit me up with direct messages.

If you’ve enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorse me for relevant skills on LinkedIn.

Till the next one, happy coding!