Developer Control Planes: A Platform Builder's Point of View

Daniel Bryant (00:02): Hello, and welcome to the Ambassador Labs Podcast, where we explore all things cloud native platforms, developer control planes, and developer experience. I'm your host, Daniel Bryant, head of DevRel here at Ambassador Labs, and today I have the pleasure of sitting down with Alan Barr, internal developer and security platform product owner at Veterans United Home Loans. Join us for a great conversation on technology adoption in the context of legacy systems. We also explore building the business case for investing in a developer platform, and how to define the developer experience across all those contexts. And remember, if you want to dive deeper into the motivations for and benefits of a cloud native developer control plane, or are new to Kubernetes and want to learn more on our free Kubernetes developer learning center, please visit getambassador.io to learn more. So, welcome, Alan, to the Ambassador Labs Podcast. Many thanks for joining us today. Could you briefly introduce yourself and tell the listeners a little bit about your background, please?

Alan Barr (00:51): Yeah, sure, Daniel. I'm really excited to be here, talk with you. I know we've run into each other a few times before. I am Alan Barr. I work at Veterans United Home Loans. I am a platform product owner, and I'm working on building a platform with my team that serves about 130-plus dot .Net developers, along with Python and PHP developers as well. I'm really excited to talk with you today about our journey, where we've come from and where we're going, and I think a lot of people will learn about working the enterprise and traditional companies and what change will look like.

Daniel Bryant (01:25): Perfectly set up, Alan. As you and I were talking off mic around the kind of companies we see at KubeCon. Sometimes they're the hipsters, right, the innovators, greenfield. It's a very different story for any technology adoption when you have legacy or vintage systems, the money making systems. The systems actually do the thing. And I wonder, could you set up the background a little bit of where your company's coming from? I know at one talk the company wasn't super comfortable with adopting cloud, even, as in public cloud. I thought that's super interesting. You still manage to build that platform on top. So if you get a little bit of context about the company you work for, that'd be awesome.

Alan Barr (01:58): Yeah, for sure. So the history of the company comes from about maybe 18 years of go. So it predates cloud a little bit. We've had a big focus on our own data centers and we're in the mortgage business, so a lot of the mortgage business is attempting to automate as much as possible and then also control costs. So that's where our play is. We don't get a ton of feature requests for using cloud in particular so far. It's still valuable, and I definitely recommend if you're a new business, use it as much as possible. For us, it hasn't made a ton of sense, though we are slowly moving that direction. And about three years ago, around 2017, that might be more than three years ago now, we were having a lot of conversations around where are we going to go as we're growing as a business? As you might be aware, the housing market right now is quite intense in a lot of different places.

Where we shine is that we're really focused on our niche, which is the veterans of the armed services. There is a government program called the VA benefit that allows our veterans to put zero down on a house. So the problem with the program is it's a little complicated, so we help them go through that process. And buying a house is just really stressful as it is, so we're trying to do our best to automate as much as possible and put a human touch on the whole home buying experience. And around 2017, we were talking and talking, talking. Three years later, we were able to get our executives aligned that they saw the need that a developer platform would help unlock the efficiency and effectiveness of our developers. We have a lot of virtual machines. We have a lot of windows servers and .Net framework. So this is the precursor to do .Net Core, .Net 6.

The problem with that is it's not cloud native. We spend every year about three to four months migrating from one datacenter to the next, because for various reasons, we are moving from our original choices. One data center was a closet in an original office, and then the next data center is a mall, essentially, that we're moving away from. And that's put a lot of stress and effort on our developers to maintain inventories and change settings and go into our older CICD systems that put a lot of emphasis on duplicating a build in a release and your code and the variables and all those systems. The benefit of our new system is that it's one pipeline, it's vault, it's all these different tools that make it much more manageable and takes a lot of that load off of our developers. So that's where we're seeing the benefits so far.

Daniel Bryant (04:50): Awesome. I'd love to dive into the technology, but just before we do that, watching the other presentation you gave, I really appreciated the focus on the business goals. Because I think when I was a consultant, often acting with middle management, stakeholder management was key to a lot of the projects I was involved in. You can have all the flashy technology you like in the world, but if you can't convince the right stakeholders to buy in and fund and so forth appropriately, it's really hard. So have you got any advice to folks about how they might go about building a business case or building a business context for investing in something like a developer platform?

Alan Barr (05:22): Yeah, definitely. I'm not perfect at this by any means. Imagine that you're going to take a lot of time to communicate to a lot of different people in many different ways. Your executives are probably the most important to communicate to. They will likely not have time to read a large document. That's why there's that executive summary, but preparing those and showing that you've thought through all the complications that you're going to run into. One thing we started with when we were given the okay to move forward with the developer platform is... I don't know if you've seen the book Technology Strategy Patterns by Evan Hewit, but what he recommends in that book is a ask deck. So we started with that.

So essentially if you think of the venture capital slide deck that you might come up with, what's the problem, how are we going to solve it, this is what we need. That's basically what we started with to our executives, explaining what's the problem, what we're looking from them to authorize, and then getting the permission to move forward. So that was really successful during the first few phases of the project where we asked to do the research. And then once we did the research, we asked to do the proof of concept, and then we delivered the result of the UI that they could visually see and understand, "Okay, this is different than what we're used to," and it gave us a lot of freedom to move forward and experiment and try building the platform without attempting to do it behind the scenes or not to their alignment and run into challenges.

Daniel Bryant (06:59): Love it, love it. So before we dive into some of the technology, you mentioned about the UI there, and I think a lot of my talks are focusing on developer experience, user experience really I think is so critical to all the things we do. And the irony is often the tools we as developers use day in, day out get the least UX attention. All our products get the UX attention, but the actual dev tools don't. So what's your end to end experience look like, Alan, and did you bring in external help to think about the UX at all?

Alan Barr (07:29): We didn't bring in any help with the UX itself. I think that's a really difficult area to improve and put focus on. What my goal from the beginning, and in my talk, I really cover this a lot, is creating the big vision that inspires people and make them want to do their best and see where their role is in the whole initiative. If you don't do that, I fear you run into issues where you're just trading with the each other. Like I'll do this and you'll do that. And I'll get a little bit more... And I don't think you're building a really great product that way, from my background where I come from, I really enjoy platforms and developer developer tools and using them from such a long time, I definitely empathize with the fact that they're usually very hard or there isn't any good documentation. And we don't put a lot of effort into them.

The vision of the platform was white glove. So I took a lot of inspiration around the hospitality message that I've heard at conference talks about, like Twilio, where a big part of their platform is the hospitality focus, removing all the cruft and all the challenges that is in front of the developer to make them want to adopt it. I don't know that we've completely hit the mark on that with everything that we're wanting to do. What we're running into is there's some things that we can white glove and make a very hospitality focused experience, and then there's a lot of things that we're calling dirty glove. Essentially, we expect that the engineer will need to go in, they will need to configure, they will need to understand. And we're constantly running into where is the balance with what our engineers need to know and what are the things that they don't need to know?

That's a big challenge for us. We have a lot of architecture decision boards that we do. That's a very democratic process. However, it's very difficult because a lot of the tools that we're used to using, the developers either are not trained or they don't work with them day to day. So to give you an example, we have the older load balancing of F5. We have a dedicated team that does that. And the developers are trying to figure out, "Okay, if we want to do health checks, what is the best practice?" And we're all trying to figure that out as an organization. So the lessons learned from that, we're trying to pay into the new platform. What are the things that you need to know? You may need to know how a pod is working, that there are containers in the pod. The liveness probes, the readiness probes. And then there's other things maybe you don't need to know. So we're still trying to figure that out.

Daniel Bryant (10:18): Well said. And my experience with platforms, one of the hardest things is choosing the abstractions. Every abstraction is somewhat leaky to some degrees, but... Some degree, but yeah, do we mention to folks running in containers? Do we give them access to kubectl? There's a whole bunch of more Kubernetes container focused questions. Would love to dive some of those as well in a minute. I'm just wondering what's the end to end experience like... As a developer, do I log into a portal, create a new project, check out stuff, do my work, commit it, get some metrics. Is it that white glove refined or is there still bits where I'm having to knit things together? I'm curious what the end to end flow's like as a developer.

Alan Barr (10:57):

Yeah, I think you've nailed it pretty much. The couple things that are a little bit more, you need to get your hands dirty is when you want to enable a database, when you want to work with a event publisher, an event consumer. So we have one Helm chart. What we do is the developer goes in the portal, they create a new app. It loads the summary page. They can pull the code down to their local machine. They can commit. We force them to use conventional commits, so that will cause semantic versioning. Once they commit their code, it goes through the pipeline automatically. The pipeline takes some time, so it's much faster than before, maybe 10 minutes. What we offer is Okteto, it's a very similar tool to Telepresence, for them to remotely work and skip the build pipeline.

Once that's all there, if they want to turn on the database connection or the eventing capability, that's the part where it gets a little harder. They need to go into a config folder, edit the values.yaml Dot Yamo that Helm is going to be reading, and then they can enable the different side cars that we are enabling. And we have constant discussions around is there a way we can make this easier, is it worth spending the time, or is this something that, if it's not going to be the golden path all the time, then maybe it's not worth us really changing.

Daniel Bryant (12:20): No, I like it a lot. And you've already hinted there that your choice of abstraction, at least some developers are going to be exposed to Kubernetes concepts like Helm, for example. Even if it's only values.yaml, they've got to have that... I often talk about mechanical sympathy, comes from Martin Thompson and some other folks I've learned a lot from over the years. And you need to know just enough under the covers to do your work. And his analogy was always to be a best racing car driver, you haven't got to build an engine, but you've got to understand about torque bands and power and that kind of stuff. I'm guessing now actually there are stretches that the most effective developers in your org would at least understand Helm and a bit of kubectl.

Alan Barr (13:02): I think so, to some fashion. We are not really offering that. If they really need to get under the hood, they can jump into Rancher to diagnose an issue. We don't make that the front and center experience and we are trying to figure out where do they need to go? How can we make it visual for them to understand how the systems... Because we ran into that quite a bit when we were moving the developer from the older Windows, VM, IIS to this new system. So they're asking questions that are, "Well, how do I reboot IIS?" And then we were like, "Well, you don't do that anymore. It's okay. So we haven't really mastered that yet. We're not giving them kubectl. We're really trying to keep it very secure and locked down, that we're managing their Docker file for them. However, at some point the... Maybe it could be because of our environment, the focus for us is APIs, commodity APIs. Most of the traffic that these different systems are receiving are very low volume.

However, they're using the loan application document that's being passed around in various ways. So we don't have any circumstances like others would, where for us, this seems like a great focus, but for them it might be like, "This is the wrong thing to automate. So we're still trying to figure that out. Where can we create that mechanical sympathy? Where can we help them visualize where they need to think about and be concerned with? Very few apps, one in particular had more bandwidth requirements, it had more users. We had to have more pods automatically for every single new version of that. We were able to instruct the team what they needed to do. For the majority of our other apps they've not really had to go past more than three default containers. So we're still trying to figure out where are they going to need to know to hand jam the system, whether they need custom configuration, and we haven't really figured that out yet.

Daniel Bryant (15:08): That is super interesting, Alan, because again, you and I were talking off mic and I mentioned my Cloud Foundry experience where some developers naturally just push back and were like, "The abstractions are too strong. I want to do custom stuff." And sometimes they had valid points and sometimes they didn't. It was an interesting learning experience for me. I'm kind of curious, because I saw your slide in your presentation. There's a lot of tech under the hood. You mentioned Rancher. I saw a Linkerd, I saw Grafana. There's a lot of cloud natives, some CNCF technologies, and you must have worked really hard to knit all that together and get the abstraction levels correct. I'm curious, then, would you mind just briefly talking about some of the key technologies and how you did knit them together and expose them to the developers?

Alan Barr (15:51): Yeah, for sure. The challenge at the beginning, if you've seen the cloud native landscape, there's so many choices. The one strange benefit for us is we were told at the time, "No cloud. You're not allowed to do that. We don't want to focus on that." And oddly enough, it became a benefit to us because it limited the options. So we focused on the core fundamentals of what we needed to do, and we didn't want to spend two weeks for every single tool, vetting them and making sure... We really wanted to target the core things we needed, which would be the source control, the ability to log, the ability to get better APIs and talk about them. Once we knitted all those together, most of the focus was how do we get them to communicate in a standardized way so that we could keep the name spaces, the source control, all the different systems, because they're all going to have their own idiosyncrasy around a name that's not going to totally map, and we're going to keep a centralized database of where these all are connecting to.

I have a great team that are working on this and thinking about it constantly. I have to put them on the pedestal there for figuring that out. I'm not really sure how they did it, but I think that the vision and the focus of what we were attempting to achieve... The challenge during the journey, for us, and I don't know if you saw in KubeCon, Razorpay, they did theirs in three months. So I don't know how you can... You can definitely go fast by yourself. With a team, though, we took a lot longer, we thought through a lot of different options and there were different points in time that we weren't really sure what we were building. Some of us thought, "Well, maybe this is containers as a service. Or maybe this is a platform as a service. Or why don't we go to functions as a service?"

It took a lot of discussion and communication and meetings and talking to the various stakeholders, such as our architects, around what they were wanting to get out of the system and trying to balance the very vocal stakeholders that were speaking about how they wanted to use kubectl and Helm directly, and they didn't need all this flashy stuff. Where what we saw with what our internal developers were asking for was not really needing or wanting to learn or do those things, which might be different in other people's environments. So I'm just very lucky that the team settled on this type of abstraction that they did. It seems to be working really well for us so far. However, there's still challenges and we're attempt to lean on our trainers to educate and update the documentation in such a way that it's valuable and a good reference for people to not require synchronous coordination around understanding and learning how to use the platform.

Daniel Bryant (18:45): Ah, super interesting. I've definitely had you say a few times the vision was super important. Set the direction you want go, set the goals, and then smart folks working with you will align around that. I think that's really powerful. I did hear you mention about standards as well, and I've chatted to someone, again, we mentioned off mic, Dave Sidia and the Go Spot team. He was talking a lot about this saying adopting appropriate standards gave him that key integration point where even if things did change from a tech point of view, the standard should give you some encapsulation... Sorry, encapsulation there. Is that something you've bumped into as well, picking the right standards being really useful.

Alan Barr (19:24): Yeah, standards have been very important at this company for a long time, and the reason they're important is we have a lot of, simply, cruft that's built up because we have people that want to do creative work, and sometimes they apply their creativity to these very low level, common solve problems. And that's okay. However, what we're missing when we don't standardize those lower level things is we need to train. When we move someone from one team to another, they need to ramp up. We can run into those situations where we really need Bob or Sally that knows how to update the system or knows how it works. We really don't want that as much as we used to. Where we settled on enabling a default standard is leveraging the architecture decision record community we have. If you've seen that Google has a API improvement proposal, it's a very similar kind of concept where you have these records that indicate what are the standards that, for some reason, this community has decided how to do a certain thing.

What we've done is in our focus for API development, we have a template generator via the Swagger Codegen. It's very similar to the open API code generation tool. What that's enabling us in our templates is when you create your new app, you're receiving the community standards automatically. We've wired up as much as we can for you that doesn't require configuration. There's still some things that we can't configure for you without some specific information. That's enabling us to offer the default. And then our Helm charts, we're able to add and layer on new versions of those.

The existing apps, we haven't really settled on what is the best way for those to receive the updates? Those could be merge requests. It could be various bots and tools. Or it could be running code generation again on the tool. Haven't really decided how we want to achieve that. But what we're trying to strive for is keeping these applications up to date with the architectural standards over time, as people move on and move to different teams, we want them to focus much more higher level on the business problem versus the lower level infrastructure and how things connect and do the basic things that we've solved already.

Daniel Bryant (21:49): Yeah, totally makes sense. Werner Vogel, Amazon's CTO, talks about the undifferentiated heavy lifting. Bit of a mouthful, but soon as I heard that phrase many years ago, I was like, "It totally makes sense. I've been there. I've been doing that stuff." I think Google called it toil, for example, in the SRE book, toil, right. But you want to eliminate those things on a platform. Touching on something mention there around documentation I'm hearing is really important and training. And how do you go about that? I mean, docs, I guess is self-service. Is the training live, instructor led, self-service? I'm curious about how you bring everyone along on the journey.

Alan Barr (22:22): Yeah, that's a great question. What we settle on for this system are three courses. They're all asynchronous. None of it is live training. I was lucky to have a trainer that developed custom curriculum for the platform. He focused on translating Twelve-Factor into a workshop format. So that is the one synchronous part. Where we asked the development teams to walk us through a example of a historical app that they use on dynamic framework and why it didn't apply to Twelve-Factor. And then we talked about what the Twelve-Factor principles mean. Not all of them completely. Mapped to what we're working on. It's a good framework and language. What we saw is that various teams and people put more or less effort into the exercise. Overall the community is level setting on a common knowledge base that we can communicate and talk through about these concepts. And that was the benefit there.

The rest of the material, I put a lot of effort into recording videos and tutorials. Those are wonderful to achieve a goal, but they don't explain why we're doing the platform or the reasons why we built it. That's where the curriculum shined, because it progressively revealed all the details around why we're doing what we're doing. And I was not able to hit the mark with the tutorials and documentation site that we've created. So I think that's a great lesson learned, that if you have people that are great at those abilities and creating custom curriculum, telling that story, definitely leverage them as much as possible.

Daniel Bryant (23:55): Yeah, something I do talk about there as well was telling the story, and in the presentation I watched where the user group, you talk a lot about even marketing, and I see you're rocking, the dev lab hat as we're talking, which is great. They've got the branding on point, proud marketer, which is awesome. So is the storytelling a key part of it? I guess it's explaining the why as well as the what, right?

Alan Barr (24:19): For sure. It's very important, especially where we're from. We have a lot of the classic organizational concept of architect and developer. Maybe sometimes we have a manager that acts as agile coach. What we can lose in this big system as we're growing is we lose the why around some initiative and it frustrates developers, it frustrates a lot of us when we don't receive that why are we doing a certain thing? Sometimes we're just dictated, told this is the way to do it. And what we want to move away from is that to these are the reasons why, here's why you're part of the journey, here's how you're going to contribute to the journey, that sort of thing. It's been a challenge because when you're building the platform, you're so in it, you're thinking about it constantly, you don't think what are the people that haven't seen it thinking about...

What I did is I gave a presentation one time, however, that wasn't enough to really cement the idea in people's heads about why we're doing it. So there was still a lot of people that were asking via different interviews I did, what is it, what is the purpose? How come you're not doing Angular? Lots of all over the place kinds of questions. And over time, I think it's just been part of this 18 to 24 months journey is that it's going to take time for people to embrace the concept, to see the value, to understand the background. We've tried a lot of different things to make it cement in their minds. I think naming it in the marketing was really key to make it a real thing. There were also other things I tried like blogs that were maybe a little too early, a little too rough, too focused on cloud native. I think if I went back in time, I would focus more around the why and how it's going to improve the experience for developers, but also caution not to oversell the idea.

Early on there was some feedback from product people that thought that this was going to change how fast developer might complete a project, and that might be out of scope of what the platform can really do because they're still solving hard problems, we're just removing the cruft. So there's lots of lessons learned. I think communication is just a really high leverage activity. Half the battle is building the product and then the other half is communicating the value and getting people to use it and making sure that they're not... They're feeling heard. I think that was a big lesson learned is that the people, the stakeholders, architect, developer, they felt heard about their ideas. Whether we decided to act on them or not, we were listening and taking in their feedback. I would constantly ask for testimonials as people came onto the platform and share that out so that they're seeing that their peers are seeing success and they understand the value.

Daniel Bryant (27:16): Super interesting. So it's doing stuff we might do externally.If we are selling a platform, you are doing it very much internal, because I've heard you also talk about inner source as well. Versus open source. I bumped into inner source in the PayPal folks, I think it was, many years ago. I thought it's such a interesting idea that when you get to a certain size of maturity of organization, all the stuff that works so well external can totally be applied to you internally. So is that the lesson you took with the approach as you mentioned?

Alan Barr (27:44): Yeah, definitely. The reason I went that direction with the vision and the tools is because it helps clarify our thinking and focus on not taking shortcuts and building a better product. That was the big reason, because we have these big goals around... We want to be multi-tenant, we want to build a very secure platform. It would be very tempting if we say, "Well, this is only going to be internal, so we're never going to sell it. We're never going to do anything." So we can take some shortcuts. We can do some things that will compromise the integrity and the value of the product. And we actually really don't want to do that. What we want to do and what I've really internalized is that create a really big vision, make it even daunting, and that will inspire people to want to do what's right for the product. While I didn't see a lot of success so far with inner sourcing, I'm still very optimistic that it will happen.

The benefit, what we've been able to do by professing that this is a platform that you can contribute to if you're very interested is we have some stakeholders, developers that are really passionate about different affordances they have in the older platform. Example would be blue green deployments. I just want to make sure that I can get my code out there, test that it's running good in prod, and then I can swap it with the live traffic and we're good. We don't really have that level of granularity with our deploy pipeline just yet. We might in the future. What I've offered to them is... We've talked with them many, many times, we offered, "Hey, if you are passionate about this, you can join the team for a short consultation, join us for a few days. You can see how we work. And if you're really interested, you can contribute this to the platform if this is something you really want to do."

They haven't taken me up on that just yet. However, I'm continuing to make sure that this is front and center. If there's something about these tools you are not enjoying... Obviously code generation is not something that is perfect. However, there is a big open source community that's just waiting for you to contribute. There is some internal templates that we can customize and modify to improve. However, as we're growing, scaling in more or more people, it's going to be really important for us to work together, find ways that we can coordinate without being completely synchronous, and open up the value that we can all benefit from versus focusing solely on our own apps that we own.

Daniel Bryant (30:16): Love it, love it. And as you were talking there, by inner source, I was just thinking back to the Netflix days, would you ever open source any of your platform? Because I know Netflix made a big deal of it in terms of altruistically sharing with the world, but also it's great for hiring, drives the general state of the industry's at, in terms of what we're all looking for with this platform. I'm kind of curious... But clearly it's extra work. Extra work in maintaining it, all the legal issues and all the stuff I guess the CNCF could potentially help with as well. But I'm kind of curious, Alan, has it been on the radar to open source it.

Alan Barr (30:50): I think so. I think really anything's possible with this platform. What we would love to see with any type of platform is if you're only constraining it to your own business case, you're very unlikely to get extra value that others could contribute. That it's going to be tough to maintain in the future if you only have a core set of people that can work on it. The value there I could see is that you could... If it's not a differentiating tool for your business, which many of these platforms are not, what would be the harm of open sourcing it? I think there would be value there. Obviously for us, we have a big focus on work life balance, on making sure that we're working on the things that are important and we make time to have relationships and work with our peers. And we have these things called small groups that are like book clubs that we do cool to really focus on the relational aspect of our company.

And I think that as long as we're striving to do something that will maintain our values and not burden us with lots of extra work or the burnout that we saw at the last KubeCon, there was a big focus there around making sure that we're not just focusing only on the tech things, that we make space to rest and relax. I think that's going to be key and important. So it's definitely not off the table for sure. I think with this platform, I'm really excited that we have a big vision around what we could do with it. If that means an Amazon style first and best customer, maybe this is an AWS style thing, or maybe it's more of a, "Let's open source it, let's see what the community says about it, and maybe there's other companies that can benefit." I think anything's possible.

Daniel Bryant (32:30): Excellent. No, that's great. And where should folks follow along if they want to see more, learn more about the journey or hear updates? Is it a company blog, hit you up on Twitter personally? Where's the best place?

Alan Barr (32:40): Yeah. My website and Twitter are probably the best places to follow me, get updates about what I'm working on and doing. I have a wide variety of interests, not only focusing on platforms. So that might be a challenge for some people, but for the most part, if you're interested, I'm putting out content every once in a while on where we're going, what things are changing.

Daniel Bryant (32:59): Brilliant. I'll definitely link in some of the videos you shared with me as well, because they're fantastic talks. I encourage everyone listening to check them out for the deeper dive into some of the stuff that's hard to cover fully in the podcast. The slides worked really well on that. But before we wrap up, Alan, is there anything I've not talked about you'd like to cover at all?

Alan Barr (33:16): So my perspective is that there are many companies playing in the internal developer platform, developer controlled plane space. Ambassador in particular is going that direction. And I think they're all going to be successful. In like five to ten years all these enterprises are going to be migrating to cloud native platforms. I'm just really curious where you think it's going to go? I'm trying to constantly keep tabs on all the different products that are out there and where they differentiate, whether that's this is Kubernetes a service, or this is your CICD Kubernetes as a service that you can wire up or... What do you think about where it's going? I'm really curious.

Daniel Bryant (34:01): Yeah, I have a lot of conversations around this space. The way, Ambassador Labs, we look at it internally is the code, ship, and the run. So we think everyone has got to code the apps, and then you ship them to production. And then when you're in production, you're running them. And there's clear products in each of those spaces, like you mentioned Okteto, Telepresence, Scaffold, bunch of stuff in the code space for example. In ship, you've got Argo and Flux, probably, as CNCF standards. And then when you go towards run, you've got like the edge stack, Emissary, Ingress, Contour, Kong. There's a bunch of spaces there. And there's also these projects that knit everything together, somewhat platform as a service features, GitHub, GitLab, bunch of folks in that space knitting these things together. I'm really interested to see the uptake on how opinionated people want the stack.

So in the moment at Ambassador Labs, we're very much focusing on standardization, those clear handoff points between the code, ship, the run, these kind of things. Maybe you do want to mix and match Argo, Flux. Who knows, in terms of actually how folks want to use it. But some folks will just want that Heroku, that Cloud Foundry, that OpenShift. Pick another name in the space. And I'm really curious, as in, some enterprises got a bit burned with things like OpenStack, for example, going a bit all in on something, and I'm wondering what's their level of comfort with knitting together all of the things? So we're looking at helping folks choose best of breed, particularly CNCF technologies, but best of breed cloud native technologies, and almost providing that pluggable platform, framework, call it what you will, of clearly here you're coding, clearly here you're shipping, clearly here you're running. Hopefully that points in the direction. I'd love to hear your thoughts on that. I mean, do you think about the code, the ship, the run, or do you think more holistically or...

Alan Barr (35:54): Yeah, when I think in the terms of enterprise, I imagine the traditional companies that are struggling to retain and find talent to run these systems, where I think that they could benefit is anything that minimizes the amount of talent they're going to need to run this platform. So in the presentation, I give the example that we have this team of around seven, we have a QA, we have two Kubernetes administration people, and then we have the rest traditional software developers that have built this amazing platform. And that's really great for a department of 130 plus people. I think that's a really wonderful investment and it's something that is sustainable for us. For a lot of companies, maybe they don't have the option to have that in-house Kubernetes administrator. So for them, maybe the cloud is a great option because it can manage some portion of that task, though there is going to be still some configuration and management maintenance that that will need to happen there.

So I'm really curious as well. Where I'm trying to go with this product is much more opinionated. Where we're pushing into is better API development, better focus of how to do software development work. We don't have the luxury of pursuing the top tier Silicon Valley talent. What we're looking for is we take these people that are fresh from university or from our local community, and we enable them to work in our environment, learn the skills that they can learn, and then some of them stick around, some of them don't for the most part, and where we're trying to add values is we have training, we have a really great community, we have great values that make them want to stick around. What we want to do is make them more effective and efficient. And what I would love is that they could, during their time with us, take some really good solid principles around how to be a better software engineer, how to be a better developer of APIs. I think that's an incredibly untapped market so far.

We have lots of tools that are focused on the we, "Well, it's in production, we better reconstruct the open API specification from that," or, "Here's what we could do in the pipeline to warn us, but it's very customized, you need somebody that has a really clear vision around what that needs to be." And I think that a lot of businesses will, in the long run, benefit from that. But many other companies, I imagine, are more focused on the machinery, getting the platform running, here's your common technical use case. And then also my assumption as a platform engineer is that you're going to be like me, and you're going to want to run the platform directly knowing all the tools.

And I don't know if that's going to play in every single business. I think that there's going to be room for all these different areas where they will care about the different components of build, code, run. However, I don't know how much that's going to get us at some point. I think at some point this is going to be very commoditized. We've really figured out cloud native, now what's next? What should we put our brain power to?

Daniel Bryant (39:04): Yeah, I like it a lot. I was doing some Rails work back, I guess 10 years ago now, and Heroku just fitted the goals there. We were working product market fit, start up, big code base, we could go super fast, and we were like, "We don't want to do all this op stuff. How can we not?" Heroku, boom, there we go. And similar kind of things when I was working on some Spring apps in the Java space, our Cloud Foundry popped up and it's like, "For this use case it's going to get us out of the door faster, get us that product market fit, that funding, that kind of stuff." And I think I've heard you say several times in the chat today, context is key. Really understanding your goals and your skills and where you want to get to.

And I think I've definitely worked on a few platform projects as a consultant where it was like... The goal was just build a platform, and it was like you can... Great as an engineer. Have at it. Play with all the cool tech. But the lack of constraints was actually a real... Looking back, was actually a real downfall, where I was like, "What are we going to choose here?" Makesource was a thing at the time, there was a bunch of other products out there. Those lack of constraints were really quite challenging, right?

Alan Barr (40:07): Yeah, constraints really help you fill in the details in a better way, it helps you get creative. You're really able to push yourself to come up with a solution that could even be better because you're lacking the options that you would otherwise be potentially tempted to pursue if you have all the money in the world and all the time, and unfortunately we don't usually have that or-

Daniel Bryant (40:29): Yeah, it's a luxury when it pops up.

Alan Barr (40:31): Yeah, for sure. It's a big luxury. Something that we did is we went to general availability in 10 months instead of the anniversary of the project of one year, and that was really perceived as a great thing from our executives because we compressed the risk of the schedule. There was no reason for us to go the full one year and we were really focused on shipping a good quality product. Some other lessons learned, and you can find this on my website and podcasts and whatnot, but I really had hoped that I could have gotten a business use case earlier and onto the platform. But as you know, these platforms are quite big. They're not really minimal viable products. You need to do a lot of things to make people successful, and stability is key. For us making sure that people have a good perception of our stability has been a very big focus for us.

Daniel Bryant (41:26): Awesome stuff, Alan. Awesome. There's so many... We can talk for days here, I reckon. There's so many things to unpack. It's been fantastic. I'm totally conscious of time. Really appreciate all the ideas, all the energy, all the knowledge sharing today. I'll definitely share folks where they can contact you in the show notes, but really appreciate your time today, Alan. Thanks very much.

Alan Barr (41:43): Yeah. Thank you, Daniel. Really appreciate the talk.