Use Your Own Tools

Ambassador Labs's VP of Engineering, Katie Wilde, is in the hot seat in the second in a series of articles about how employees at Ambassador Labs use our own tools.

As part of leading the charge to "drink our own champagne", Katie shared her passion for Ambassador products, enabling developers with better tools and cross-functional collaboration and supporting the development of game-changing software from the inside-out.

When Katie Wilde took up the role of VP of Engineering at Ambassador Labs, she was not new to Ambassador tools, such as Telepresence. In fact, as the VP of Engineering at SaaS social media management platform, Buffer, she came across Telepresence and experienced a "wow" moment. As a leader, Katie has always aimed to enable developers and be an "engineer unblocker", and identifying (or building) tools that do that are a part of the path to productivity.

Katie described Buffer as an early adopter of Kubernetes, thanks to specific use cases where it made a lot of sense. For example, many users wanted to upload video with Buffer, but it was not practical to scale the entire PHP monolith, so the team created a video upload service. This led to an increasingly services-oriented architecture, which, according to Katie, helped reliability and cost management. But it came at the expense of the development team running into a lot of issues.

"As we started adding more microservices and became more advanced in our cloud-native journey, local development became increasingly difficult, with people losing hours or even days. These issues had to do with different services running in different places with no insight into them, not being able to run enough containers, for example, Docker for Mac is famously limited, and we could not run all these services no matter what we did, so hardware started to become more expensive.

Eventually it hit a point where the expense was growing, and development so painful, that it was not worth it. Even though it was great for reliability, some aspects of cost management and the production infrastructure were amazing, developers could not be productive and could not make changes because they could not test them. It became absurd, but I also thought it had to be solvable."

It was at this point that Katie discovered Telepresence, a tool that let her team develop one service at a time and connect to an existing cluster easily and safely. "It was amazing for developers – they could finally develop and debug services running on a remote cluster locally."

Making the leap from Buffer to Ambassador, Katie officially became an ambassador for Telepresence and other Ambassador products, armed with real-world use. Since joining, Katie's enthusiasm for the product and working to improve it, in her words, "by leaps and bounds", hasn't waned. Katie saw productivity and developer experience transform using tools like Telepresence in her previous role and wanted to ensure that development teams at Ambassador were getting the most from their own work.

"Like most engineering orgs, we need to share and review a lot of our work. We also do a lot of cross-functional work at Ambassador, and there's nothing like Telepresence to be able to help colleagues share their work and get fast feedback without friction," Katie explained.

"With an intercept into a cluster you don’t have to use mock data or rebuild containers and deploy them just to test on a cluster. It's easy and allows for faster development. Telepresence also enables a preview URL feature. A preview URL shows you exactly what you are working on, making it easy to review code and share with others for further review.

Code review friction is really reduced. You don’t have to try to understand what these changes look like locally. You just click the link in the PR and go straight to the preview. You essentially see that code running in a local environment. Telepresence provides a huge boost for knowledge sharing and getting code reviews done quickly. Part of my goal is to ensure that we use this to full effect internally and make it the best it can be with every iteration we launch externally."

All aboard: Easing the onboarding process

Having worked for a long time with Kubernetes and using Telepresence, Katie did not experience onboarding challenges the way developers new to the new cloud-native paradigm do. And that, being a developer enabler, was a prompt to jump into action for Katie as she joined Ambassador.

"I realized it could be tricky for people to install Telepresence and get it running," Katie explained. If you are onboarding an app developer into this environment, battling with an install and complicated instructions, this isn't a productive use of a developer's time. This is straight-up toil – and not the developer's core job. We have radically simplified installation, eliminating the need to fight with YAML and the need to know what is going on under the hood.

A developer does not need to know the mechanics of getting something installed, just as we don't break out a soldering iron to work on our laptops. Having that simple, easy-install command reduces a huge amount of friction, especially if you are starting from zero. The clock really starts when you sit down at a blank machine with nothing on it, and we say 'go'."

Metrics for quantitative and qualitative success

Making internal tools successful internally and externally demands measurements and benchmarks to understand where things started, where they are going and what improvements have been made along the way. Most of all, how does this support the day-to-day work of developers and the teams that support getting products out the door?

Some of these metrics are more quantitative while others are anecdotal. Some key quantifiable results include a drastically reduced time to approve and time to merge PRs, which improves productivity for everyone and is something that can be influenced from day one by being able to assign PRs from a developer's first day on the job. Katie immediately expanded, "From day one, we have seen onboarding times dramatically decrease.

Quantifiably, this means we are able to get a PR in a developer's hands on day two – and we aim for day one. It used to take up to two weeks. There were so many complex things about the product that had to be understood before a developer could get their hands dirty. Now, we have made improvements to the product, so devs can use intercepts we already have, for example. We can create classic onboarding exercises, reduce friction for new developers and almost immediately achieve increased productivity. The real test for how this works is measuring how long it takes to get a brand-new person on board."

Anecdotally, fast and effective onboarding and familiarity with internal tools makes everyone's lives easier. Katie cited a lowered level of cross-team tension as well as major positive improvements, "We've certainly seen faster feedback loops throughout the Ambassador organization. For the developer, knowing that what they are doing won't break the machine, for the overall development teams, and then across teams, Telepresence enables easy sharing and feedback with everyone – product managers, developer relations, design, and so on. The developer team is more productive all the time, and certainly more than when microservices development was new."

Using, expanding and improving tools: Internal to external developer empowerment

While Telepresence is a tool that hooked Katie, and more broadly, the developer community, other popular Ambassador tools are also a focus for internal use and improvement. One such tool, critical for onboarding but otherwise always useful in production, is the service catalog. Katie recalled, "Frankly I am so used to having a service catalog that I cannot imagine not having a place where I can go and see all the running services.

I don't know how people do it otherwise. Maybe you don't need to see every service in the entire company, but you do want to see the services running in your own universe. Especially if you are onboarding, you want to start small with, 'Here's your world and what you need to know to do your job" and work outward from there."

Recognizing the value of the service catalog, Katie explained how she would like to improve on the concept, making visualizations more user friendly, potentially grouping services in ways that could be more relevant for specific teams, "I want the service catalog to be more usable at that vast scale."

Another improvement to the product functionality that Katie was excited about is being able to get insight without having to use Edge Stack as your API gateway. "Of course, we make and love AES and think it's the best, and that you should use it. But if you don't, that shouldn't stop you from getting insight from Telepresence," Katie explained. "Right now we rely on two different teams, and as a developer with Telepresence, you need Edge Stack installed with the agent running on the cluster. From this we can tell you as you are developing,

'Here's what's happening, here is an issue,' and so on. But we don't want anyone to have to be using our API gateway because it is a separate product.

Having this agent as a standalone that can run on a cluster and be used in this multipurpose way is a game changer in terms of debugging, logging, tracing – everything you need. Based on our own internal experience combined with external feedback, we are taking Telepresence from being this helpful collaboration and development tool to also being a debugging tool in vast distributed systems environments. This is particularly valuable and asked for by our customers in larger environments where they have 5,000+ services."

Conclusion: Sip by sip - The tools and culture of your own champagne

A combination of tools and culture contributes to the potent mix that makes up the champagne of one's organization. In effect, there are no better tools to give to developers or champagne to drink if there aren't step by step, or sip by sip, improvements in developer experience. Unlocking developer productivity is all about unblocking developer friction, and, as Katie put it, "being a force multiplier beyond" her own team. These enhancements start with a carefully cultivated harvest, or a culture of growth and improvement, and manifest in a great vintage. Much like an actual champagne vintage, it can be hit or miss – some experiments land better than others. But a culture of experimentation and measurement yields results from its learning either way.

'Drinking our own champagne' is a toast to developer experience, which we can do by championing, using and improving tools internally and proving their value in boosting productivity," Katie continued. "We want every second to count for developers from day one."