Boosting Developer Velocity by Optimizing Feedback Loops

It’s no lie that containerization is a massive boon to the development process. It allows developers to create consistent environments across different stages of development, from local machines to production servers. Containerization involves encapsulating or packaging software code along with all its dependencies so that it can run uniformly and consistently on any infrastructure. This is achieved through containers, which are lightweight, executable units that contain everything needed to run the software, including the system tools, libraries, runtime, and settings.

The End of "It Works on My Machine"

One of the most significant advantages of containerization is its ability to solve the "it works on my machine" syndrome. This common phrase has plagued developers for decades, where an application runs perfectly on one developer's local machine but encounters issues in another environment due to slight differences in OS, underlying libraries, or configurations. Containers ensure that the application runs in the same environment, regardless of where it is deployed, be it a developer's laptop, a test server, or a production server. This consistency eliminates any discrepancies and bugs that arise from environment-specific configurations.

Streamlining Development Workflows

As an added bonus, containerization simplifies the development process by creating a common framework for all team members to work within. Developers can focus on writing code without worrying about the underlying infrastructure. Containers can be easily shared among team members, ensuring that everyone is working in an identical development environment. This notability reduces setup time and minimizes the "works on my machine" issues, leading to faster development cycles and fewer production deployment issues

And as we all know, streamlining = good. So what’s the problem?

The Downside of Containerization

However, containerization also introduces new problems, such as container builds and registry uploads, which are pure downtime for engineers and a tax on their time. During this process, nothing else can happen. This is not to say that engineers are slacking and having (probably now virtual) sword fights, but they have to wait for the build to happen before they can either commit the code or continue working on it.

This wasn’t always so. Without containers, traditional development loops were quicker, allowing higher velocity and more iteration. So, while they offer a lot of benefits—we still have to figure out how we can

get back to that speed without sacrificing the value that containers bring.

Speed Begins with Internal and External Development Cycles

The Internal Development Cycle

The problem here is with the "inner dev loop." An inner dev loop or internal dev cycle refers to the processes and practices that developers engage in on a daily basis to write, test, and debug code before it is shared with others or integrated into the main codebase. This side of the loop includes things like:

• Coding: This is the core activity where developers write new code or modify existing code to add features, fix bugs, or enhance performance.
• Building: Once the code is written, it needs to be compiled or built into an executable form that can be run on a computer. This step might involve compiling source code into binary code, interpreting code, or executing an automated build script.
• Unit Testing: After building the code, developers often run unit tests to ensure that individual parts of the application function correctly. Unit tests are written to test specific functions or methods for correctness, and they are run frequently during the development process.’
• Debugging: If the code has issues, as often happens, debugging is necessary. This involves identifying, isolating, and fixing bugs. Developers use debugging tools and techniques to examine the running code and correct the flaws.
• Integration: This involves integrating the newly written code with the existing codebase. This step ensures that changes made by different developers do not conflict with each other.’Review: Before the changes are committed to the main codebase, the code may go through a review process. Code reviews are conducted by other developers on the team to ensure the code meets quality standards and adheres to the project’s coding guidelines.
• Version Control: Developers use version control systems to manage changes to the codebase. This includes committing code to a shared repository, merging code, and resolving conflicts. Version control is essential for coordinating among multiple developers and for maintaining the history of changes.
• Continuous Integration (CI): While technically part of a slightly broader loop, CI can overlap with the development feedback loop. CI involves automatically building and testing the codebase whenever new commits are made to ensure that new changes integrate smoothly with the existing code.

This cycle is repeated throughout the day, and its efficiency greatly impacts a developer's productivity. The faster and smoother this loop, the more iterations a developer can make, leading to quicker problem-solving and feature development.

But Don’t Forget The External Development Cycle

On the other hand, the outer development loop encompasses the broader aspects of the software development lifecycle that extend beyond the immediate write-test-debug cycle of the inner loop. This loop focuses on delivering the software to end-users and gathering feedback to inform future development. Here are the key parts of the outer development loop:

• Continuous Delivery (CD): is the process of automating the delivery of applications to selected infrastructure environments. It builds on the concept of Continuous Integration by deploying all code changes to a testing, staging, or production environment after the build stage. This ensures that software can be released to customers at any time with the push of a button.
• Deployment: This involves deploying the software into a production environment where users can access it. Deployment might be done manually or automatically, and it often involves additional steps to ensure that the application is configured correctly in the live environment.
• Provisioning and Configuration: This step involves setting up the server and network infrastructure needed to host and run the application. It also includes configuring the application according to the specific requirements of the environment it is being deployed in.
• Monitoring and Logging: Once the application is deployed, it is important to monitor its performance and log any issues that occur. Monitoring tools can provide real-time data on the health of the application, while logging tools can help capture detailed information about errors and other runtime events.
• Performance Optimization: Based on the data gathered from monitoring and logging, developers may need to optimize the application to improve performance, scalability, or resource utilization. This could involve code changes, infrastructure changes, or both.
• User Feedback: Gathering user feedback is crucial for understanding how well the application meets users' needs. Feedback can be collected through various channels, such as user surveys, interviews, support tickets, and usage analytics.
• Issue Tracking and Management: As users begin to use the application, issues might arise that need to be tracked and managed. Issue tracking systems allow developers to keep track of reported bugs, requested features, and other tasks that need to be addressed.
• Compliance and Security: Ensuring that the application complies with relevant laws, regulations, and security standards is an ongoing part of the development process. This includes regular security audits, compliance checks, and updates to security protocols.

The external development cycle is critical for ensuring that the software functions correctly, meets users' evolving needs, and adheres to operational standards. This feedback helps teams deliver high-quality software that is robust, secure, and aligned with user expectations.

Where does containerization come in?

The benefits of containerization have accrued to the external dev cycle by ensuring consistency across environments and simplifying deployments. But it has introduced friction into the internal development cycle (inner dev loop), forcing it to slow down. The time spent building containers and waiting for them to start can slow down the rapid iteration developers need for efficient coding.

Before containerization, the inner dev loop might have looked like this:

So, in the traditional inner dev loop, we have just over five minutes per development iteration, with just 10 seconds of “tax” downtime. Looking back at the containerized version, this is extended to over nine minutes, with almost half of that “tax.”

If a developer codes for six hours per day, we move from 70 to 40 iterations by moving to containers. Throughout a two-week sprint, this is 300 missing cycles.

Therefore, we have to focus on the inner dev loop in a containerized environment in order to maintain high developer velocity. How do we do that? Let me introduce you to Telepresence.

Speeding Up Development: Local-to-Remote with Telepresence

One approach gaining traction is local-to-remote development. This method allows developers to run their code locally while seamlessly connecting to the remote Kubernetes cluster. Tools like Ambassador’s Telepresence enable developers to code as if their local machine were part of the remote cluster.

The idea is simple yet powerful: instead of building and deploying containers for every code change, developers can run their service locally and have it interact with other services in the remote cluster in real-time. This approach offers several advantages:

• Local Development with Remote Services: Telepresence allows developers to run their code locally while connecting to a remote Kubernetes cluster. This means that you can develop and test your code on your local machine but still interact with the full set of services and dependencies running in the remote cluster. This hybrid approach removes the need to deploy your changes to the cluster for every code iteration, significantly speeding up the development cycle.
• Immediate Feedback Loop: By running your local code in the context of the remote cluster, you can get immediate feedback on how your changes affect the entire system. This reduces the time spent waiting for builds and deployments in the cluster, enabling a faster debug and test cycle.
• Simplified Testing and Debugging: Telepresence makes it easier to debug services by allowing you to intercept network traffic between your local machine and the remote cluster. This way, you can use local debugging tools and breakpoints on your code running within the context of the full Kubernetes environment. It simplifies diagnosing issues that only manifest in the cluster without needing to use remote debugging tools.
• Environment Parity: Maintaining environment parity is crucial for catching bugs that only appear in production-like environments. Telepresence helps maintain this parity by ensuring your local code runs in the same network environment as your Kubernetes services, reducing the “it works on my machine” problem that I touched on in the beginning.
• Reduced Deployment Overhead: With Telepresence, you can avoid the overhead associated with container builds, pushes, and deployments. Instead of creating a new Docker image and deploying it to the cluster, you can run the code locally and see the changes reflected in real-time within the cluster context.
• BONUS- Collaborative Development: Telepresence can enhance collaborative development by allowing multiple developers to work on different parts of the system simultaneously, each running their code locally but interacting with the same remote cluster. This can streamline integration testing and collaborative debugging efforts.

Ultimately, by speeding up the feedback loop, you can fully unleash its potential

Look, the goal isn't to abandon containers–their benefits for the outer dev loop and production environments are too valuable. Instead, we're looking to create a hybrid approach that combines the speed of local development with the consistency and reliability of containerized environments.

By focusing on optimizing the inner dev loop, we can help developers regain their lost velocity, leading to more iterations, faster feature development, and, ultimately, better software. The key is to find the right balance between local development speed and the benefits of containerization–and with the right tools (ahem—Telepresence!) and practices, that balance is 100% achievable.