When You Love Champagne, You Share It

Enthusiastic about with others. A new song that we can’t get out of our heads, somewhere we traveled and want to go back to as soon as possible, and for developers, our favorite tools.

Our job is to solve problems, and when we find a great way of doing that, we love sharing it with other developers.

We release software under open-source licenses, talk at conferences, write blog posts, join Discords and Slacks, and chat at meetups for our favorite languages and frameworks.

My journey with Emissary-Ingress, Edge Stack API Gateway, Telepresence, and Ambassador Cloud has been an ultimate expression of that. As a Senior DevOps Engineer at GoSpotCheck, I was responsible for creating our Kubernetes environment and tooling. I first deployed Emissary-Ingress at v0.31.0 in (I looked it up) mid-April 2018. We later became one of the first paying customers for Edge Stack and a Telepresence early adopter.

I think one of the highest praises you can give any product is to say you kept using it when you had a chance to re-evaluate. When I took the role of CTO at UPchieve, a non-profit that provides free online tutoring for low-income high school students, I knew we would be on Kubernetes, and I had a greenfield chance to pick tools again. I knew I would use EdgeStack and Telepresence again without even thinking about it.

Now I’m a DevRel Director here at Ambassador Labs and get to share the champagne I’ve been drinking for four and a half years!

I was hired by GoSpotCheck in February 2018 to migrate their applications off Heroku, which the company had outgrown both in scale and in operational needs. It was clear (at least to me) by then that Kubernetes was going to be the dominant platform technology in the coming years, and the rest of the ops and platform team members bought in. Great. We would be using Kubernetes. Now what?

Early 2018 was an exciting but rough time to be getting into the Kubernetes ecosystem. What tooling existed was in an alpha state and constantly churning. I was inspired to give a presentation advising companies to wait, if possible, before implementing any major tools because they were likely to see massive improvements in just a few months. We were trying to create a new platform that could:

• Host the existing applications we were migrating from Heroku
• Accommodate the new microservices architecture we were adopting
• Support multiple independent development teams

We made a table of requirements, from networking to security to deployment. An early need identified was an ingress controller and/or API gateway. We had done some proof-of-concept deployments into Kubernetes that were fronted by load balancer (LB), and it became clear that putting an LB in front of every service was not going to be sustainable. Every LB used a public IP address and other resources that incurred charges and ate up quotas with our cloud provider. At that point, we weren’t sure if we would be using AWS or Google Cloud, so we couldn’t default to their API gateway products; it had to be native to Kubernetes and cloud-vendor neutral.

The requirements kept coming. We needed to be able to route traffic in our development, staging, and production clusters. Some of the services needed routing based on the domain name, and others needed routing based on the path. We had multiple teams both working and adopting the new platform at different speeds, so using the features had to be autonomous. It had to play well with Fastly, our WAF. We were building Helm charts for deploying applications, so the gateway would have to allow configuration through code. We needed observability of the gateway itself and the traffic coming through it, in Prometheus metric format.

Making the Selection: Why We Chose Emissary-ingress

After a thorough search (which, at the time, was quite short) we found Emissary-Ingress (née Ambassador). It met all of the needs listed above. Not only that, the engineers were incredibly responsive on the Ambassador community Slack, so as we ran into issues we could give and receive feedback, and get unblocked, quickly.

In those early days, we went through lots of tools. Do we build the clusters ourselves using kubeadm or use EKS? Or GKE? Do we deploy from Github or buy a continuous deployment SaaS? How do we store secrets? Emissary-Ingress was the constant. I think it was the only tool that we installed and all agreed “yep, this one.” It evolved and there were breaking changes, but Ambassador Labs’s focus on the end-user kept them from being too painful. And the sheer flexibility of

Emissary-Ingress meant that every time we ran into a new use case or requirement, it was something that Emissary already handled; we never had to second-guess our choice or start over.

As someone who had written an Envoy v1 implementation and got close to finishing right as the v2 API came out, and then finished that right as the Istio project spun up, it was a very refreshing feeling to have a tool that was ahead of me in a good and helpful way.

Ordering a Second Round: Emissary-ingress at UPchieve

In January 2021 I joined UPchieve as the sixth team member, in the role of CTO. I was hired for my experience scaling systems and teams. UPchieve had gone from facilitating 30 tutoring sessions in February 2020 to 1500 in September 2020. The whole app ran on a single Digital Ocean droplet and was falling over pretty regularly. There was a staging environment, but no great processes around keeping it in sync with production. Demos happened in production and were affected by issues there.

Initially, the team included a junior engineer and me. It eventually grew to four engineers. We needed a way to deploy quickly but in a controlled fashion, using GitOps.

The main story was keeping a tight budget. In my first year at UPchieve, the engineering budget was $10,000. Total. For all SaaS vendors, infrastructure, etc. We took advantage of every non-profit and open-source deal we could, but it was still tight. We couldn’t afford any slack in our infrastructure.

UPchieve’s system is made up of an API server using Node/Express, a Vue.js SPA, and a React Native mobile app. The server and clients use both Websockets and Socket.io, and traditional REST endpoints.

The Sniff Test: Exploring the Flexibility of Ambassador Edge Stack

This was a very different use case from GoSpotCheck, but because of its flexibility, Edge Stack API Gateway was still an ideal solution. We ran four environments in the single cluster:

• Production
• Staging
• Demo
• Hackers (used for hackers from our VDP to run tests without affecting other environments)

Now I was taking full advantage of Edge Stack’s ability to do routing based on hostname, path, and even parameters! UPchieve is an open-source organization, and you can see how we set up Edge Stack using Pulumi Infrastructure-as-Code, as well as how we set up our mappings for the API server and the SPA using Helm.

We couldn’t afford individual dev environments for each engineer, which is where Telepresence saved the day by allowing us to do Kubernetes local development. Each developer could use Preview URLs to test their work against our cloud environment to make sure there weren’t issues caused by differences between their local environment and the eventual production one.

Finally, we used Ambassador Cloud’s ArgoCD integration to facilitate continuous deployment using GitOps. We were a four-person team with the ability to do canary deployments, track differences in performance between the old and new versions, and roll back if necessary!

Sharing the Champagne

When the time came for me to move on from UPchieve, I knew I wanted to move from the end-user side of things to the tool-creation side. I think we’re all more fulfilled when we work on something we believe in. Ambassador Labs was the first place I looked for opportunities because they make my favorite tools! I’m excited to be here now in the role of DevRel Director, where I get to share the champagne full-time with people trying to solve the same problems I’ve been trying to fix since I first got into the world of DevOps, cloud operations, and platform building in 2017.