Dana Lawson on Kubernetes, Local Development Loops, and Constant Learning

Daniel (00:03):

Hello everyone. I'm Daniel Bryant, and I'd like to welcome you to the Ambassador, Living on the Edge Podcast, a show that focuses on all things related to cloud native platforms, creating effective developer workflows, and building modern APIs. Today I'm joined by Dana Lawson, VP of engineering at GitHub. I bumped into Dana's work several times over the last year, and I was particularly interested in both her career resume. She's worked across interesting companies like New Relic, Heptio, and Envision. And I was also interested in recent interviews I'd seen of her where she talks about how platforms impact continuous delivery. I was keen to dive into this topic in much more detail.

Daniel (00:35):

If you like what you hear today, I definitely encourage you to pop over to our website. That's getambassador.io, where we have a range of articles, white papers, and videos that provide more information for engineers working in the Kubernetes and cloud space. You can also find links there to our latest releases, such as the ambassador edge track, also our open source ambassador API gateway, and our CNCF hosted telepresence tools too. Hello, Dana, and welcome to the podcast.

Dana (00:57):

Thanks for inviting me, Daniel.

Daniel (00:59):

Could you briefly introduce yourself and share a recent career highlight please?

Dana (01:04):

I am Dana Lawson, VP of engineering at GitHub. And a recent career highlight would be I've joined the dark side of product engineering after a very long, awesome career in platform and systems. So I think looking at career highlight, it was just being at GitHub and shipping support facing products. We just released free private repos for teams, and we've been planning this forever. And all this stuff going down in the world and the timing was weird, but it's just really awesome because I truly believe that the world runs on software. It's just exciting because that's why I do it. I do it for the developer community. I do it for the next generation, and it's awesome. It's just awesome, so that's a career highlight. A very recent one is my team did that, and I'm like, "Aw."

Daniel (01:51):

Totally cool. I'll definitely dive into a bit more of that in a minute in terms of leadership. But first off, I wanted to talk about something like developer loop, developer experience. Yeah. So when I talk about that, I'm kind of meaning that capability of having an idea, either yourself, your team, code in testing, deploying, releasing, verifying, all that good stuff. Now I looked at your LinkedIn and I see you've got super interesting career highlights, New Relic, Ambition, Heptio, GitHub, big-hitting names there. Without naming any of those names, I'm guessing, could you describe your worst developer experience?

Dana (02:19):

Oh my gosh. Okay, worst developer experience without naming names. Well, I'm going to just say this. I have had the joy, pleasure, and pain of getting to work on three monoliths. Everywhere I go, I seem to attract a monolith problem and a mono repo problem. And when you have a monolith, it really introduces challenges. There's no one way, right way to have your architecture, right? There's this hot news of like, "Oh, everything has to be microservices." And before that, guess what? Everything was service-oriented. And guess what? Before that, it doesn't matter. You've got to pick the right architecture. But with that, it does change how you approach your dev cycle and loop.

Dana (03:03):

And so you can see I'm bouncing around because I do not want to answer this question. I would just have to say in my long experience, the similarities that all companies I've been with are pretty close to the same, especially as you're trying to transform your environment while you're transforming how your team works because if you're making architectural changes, it's going to change how you approach your deployment systems. And so I would say the worst would have to be just the manual tooling.

Dana (03:29):

There was this one place in history where we built our own homegrown tool for change management because we had PCI requirements and FedRAMP requirements. Well, it was pre-FedRAMP, but we had Sox compliance. And so when you have compliance, you have to have this two-flight system checklist to ensure their code is safe. We had the most manual process you could ever imagine. I think we were using Subversion, so I can tell you that ... Subversion and Tortoise. And we built this little change order website, and literally, I'd have to go in there every day because, as a startup, there's no release managers, and go and say, "Did somebody go and like make thechange?" And it was all manually driven. And I think we would deploy like twice a day. And that was pretty fast for back in the day.

Daniel (04:16):

That's not bad.

Dana (04:18):

It's not bad. It's not bad, especially when you're talking about monoliths, but it still was completely just so much administrative overload. People were just cutting corners because it just took so long to get your code in production. We were building our own pipelines to the side when that they shouldn't. The worst is ops changes when people are like, "Oh, this is going to take too long to do a NoOp infrastructure change. I'm just going to SSH in the box. I don't know. They've all been kind of janky, but I'm really excited about our modern tool set.

Dana (04:47):

I will say in the last five, even I'll give them eight years, I won't go a full decade. The growth in just developer tooling and the attention in having pragmatic tool sets to do repeatable things has been amazing. And just all of the new technology is stitching it together for you. The ease of stitching it together, where your developer cycle is really cool, merge to master, let the stuff go. But there's still a ton of friction on the local environment. There's still so much friction on getting your local environment set up, and I think that's the next area where people continue to put more energy into it. It's like, "Cool. We got Kubernetes. We can orchestrate. We can deploy. We have GitHub actions and other CICD providers that can automate that workflow. But what about your local? What about local dev? It's still an area that I think needs some love and care.

Daniel (05:39):

Yeah, totally, Dana. I actually would love to pick your brains on that a bit deeper. As you were talking there, it made me think around some of the tooling in my career has progressed. Almost the actual tooling itself, the CD tooling has progressed from monolith to microservices, to abuse an analogy there. Do you know what I mean? In terms of being able to compose things through actions, GitHub actions, that kind of stuff. Do you think that's a trend we're going to continue seeing now, as in being able to compose your workflows?

Dana (06:03):

Oh, definitely. I mean, everybody has their own small little tweak to their workflow that maybe ... I mean, we want to say that there's a nirvana out there, nirvana tool set that every engineer in every company can utilize to get their product at scale. Right? It was Docker, it was containers, and now it's Kubernetes and server lists. I mean, it's been everything, right? A nirvana tool set that can allow you to have that, and the reality of it is you have to have tools that are extensible.

Dana (06:33):

And so I think when you build these pipelines and the way that all of us are approaching it is that extensibility so that you can have that piece that goes out, but in a predictable way. Right? So I believe that it's going to continue to happen. I mean, you also see companies like Zapier who are doing it right with not even the deployment side, but they came out and they were like, "Oh cool. We can stitch together all the other pieces by doing these apps."

Dana (06:56):

A simple one is like, "Oh, I need to order certificates." Guess what? You still have to go and order site certificates. Right? And so how can you automate that certificate creation? How can we do all these things that increase that kind of dev workflow from local environment to production? There's so many things we take for granted and don't think about in our workflow. And we're trying to get to a nirvana of everything is automated. Well, as most as everything could be automated, and that includes doc writing, release notes, everything.

Daniel (07:25):

Yeah, totally hear you. Let's take a step back to the local dev experience you mentioned there because yeah, I totally struggle with this. I've actually bought like a 64-gig laptop just because I run so many containers, yeah. It makes my life easier. But what's your thoughts about the current local dev experience, and where do you think it's going to go with things like Kubernetes and so forth?

Dana (07:42):

When Kubernetes was introduced, a lot of product engineers were like, "How am I actually going to know how my code is running because I don't have a local environment that mimics what production actually does and the orchestration behind it? How do I get this container in a pod localized?" Right? That's is very challenging, and so I really think we have to be smart about it. We've come and we've cracked the nut on how we can automatically have high availability and capacity using orchestration and different types of rule sets, whether it's on your edge or internal, on how you scale, de-scale, and provision. Right? We've got that solved.

Dana (08:20):

I think now it's like taking that local environment and giving it more of that production feel is key, right? Because we can do as much as we can do locally, but you're not having real load. You're not having the network layer, especially when you add in Kubernetes and now you have the software driven network layer. It brings in another nuance that you don't account for because we all know it's not real until that code is out in the wild. And so I think trying to get local environments closer to production, I mean, smart about it, and bringing the dev to that experience so they can think about reliability, and they can think about accessibility because that's our jobs. Right?

Dana (08:54):

There's so many products out there. There's so many companies out there. You can have an amazing product, but if it's not fast and speedy, and the user experience is just ... not even, it can be slightly, slightly created. They're going to say, "Whatever. I'm going to go to this company that's always up. Maybe I don't have your complete feature set."

Dana (09:14):

And so long story longer, I think it's incredibly important to have that local experience as mimicked as close to production. But the reality is, the way that we're doing it, unless we have these mega machines ... I am a sinner that I now just have MacBook Airs and iPads because yeah, I know I'm an executive and that's all we do, right, is spreadsheet ops. But anytime I do try to write any sense of code, I'm just like, "Why do I have this book?" My computing program comes on. I'm just like I almost want to rage quit because you just don't write code anymore because you're a little MacBook Air is not built to do that.

Dana (09:50):

And I think the reality of it is we need to have a way to, especially given what's going on in the world, a way that we aren't always ... I know mostly everybody is at home now, but we can go where our environments are. Right? And I think it's also important now. I maybe want to go sit on the couch and program on my MacBook Air. I want a tool that allows me to do that. And I think it's happening. I mean, we're talking right now, Datawire definitely has jumped into that. Early on, I was a guinea pig about their teleport environments. And I swear Daniel did not I asked me to ask this, it's what I think about. It's what I think about. So I think that's our next step, really. Kubernetes, we solve for that. How can we get that local environment really to feel like production in a safe way?

Daniel (10:36):

Totally makes sense, yeah. Something that you said there is something I sort of struggle with. So I bumped in Telepresence, like exactly same kind of deal about putting my laptop in the class there. I've seen a bunch of folks. I think it was the AWS folks I was chatting to. They're pitching more about we should be doing our development in the cloud. So it's almost like we've got dumb terminals. And the MacBook Air, perfect example. Chromebook, perfect example, yeah. What's your thoughts on the kind of dumb terminal approach where we're coding on a VM or something in the cloud?

Dana (11:00):

I think I'm ready for it. Let's do it. I want to be able to code on my iPad. I know! It sounds cool. I do! But honestly, and because of just having the experience of working with hosting, it sounds awesome, but shit's expensive. It is expensive!

Daniel (11:17):

Yeah.

Dana (11:18):

A part of me is like, "Oh, that's great. Cool!" Because think about it, local environment is actually in the cloud. You're all ready to go. Your changes are hopefully within your protected server cluster, whether it be-

Dana (11:29):

remote or on-prem. This was like, "Okay, cool. I'm going to promote to the next few environments." Just boom, like a click of a button and not hold on, hold on, let me do this pull request. Let me go merge it in, and let me do this and that. It's all there. I think it's going to happen, but I think the thing that people have to realize is the cost behind it. Cloud computing is expensive. It's expensive, and rightly so because you're paying for that toolset that is around it. You're paying for the actual physical space. There's pieces that go within it that we pay for.

Dana (11:59):

So that's the one place that makes me nervous. I'm like, "It sounds great, but is it fiscally responsible?" I don't know. Somebody is going to solve it though. Somebody is going to solve it. They're solving it right now. But let's be real, right? Not all projects are the same. There are going to be some that you're just like, "No way, this does not work for me." And that's totally fine. Or the opposite, you're doing something that doesn't need that kind of environment. Maybe you're just running a Jekyll page or a stat. Do you actually need anything? What is your IDE? What is your developer space? You're not going to need that many toolsets if that's what you're doing.

Dana (12:35):

So I think there's this desire sometimes for us to go get the hottest tools. I would say ask yourself do you really need that? Come on, do you need it? I still will open up Emacs and just throw some stuff in there depending on what doing. It's the fastest way. So I think always being open to what's working for you at that moment is always going to reign supreme no matter how you develop. It's like don't pace the hotness, do the thing that works for you.

Daniel (13:03):

Yeah. I hear a similar argument actually with things like paths as well. A bunch of folks like spitting up. Kubernetes like a Jekyll site or something. Yeah. And it's just like, you don't need answer to that. Heroku is fine. Yeah. What's your thoughts on that?

Dana (13:14):

A hundred percent. I think in one sense, I was a very early Kubernetes adopter. And the barrier of entry to get it up was kind of challenging back in the day. It was. I still lived through the horrors of going from a Virgin 0.8 to 11. And you had to do ... All of y'all Kubernetes nerds know what I'm talking about. Okay. Because you couldn't just do a live upgrade, it was painful.

Dana (13:40):

Anyway. It's easier to use now. It's available. Most hosting providers have built in for Kubernetes, vanilla Kubernetes, the community has done amazing. That Kubernetes team, oh my gosh, all the randos in the world, that is one of my favorite projects, and it always will be, being a Heptio alumni.

Daniel (14:03):

Oh, Of course, yeah.

Dana (14:04):

Right? But I think just because it's out there and it's accessible, you've still got to ask yourself do you need that because you're still adding on complexity. And when you add on complexity, you add the opportunity for error, whether it be system failure, human error, or other. So you've got to ask yourself, this page or this product, does it need this complexity? Because I want to be able to remediate and respond as quickly as possible. Some failure does happen. And if you add all this stuff you don't need, you're just going to make it harder. And then what do you do? The customers get sad, and they quit on you and leave.

Daniel (14:38):

Yes. Yeah, totally. It's something I've struggled with in my career. DACA was one of the shining examples, and I totally went DACA mad, yeah, back in five years, whenever it was. Have you got any advice now in the leadership position? Say I was at one of your engineers and I was going DACA, DACA, DACA, have you got any advice for me to try and pull me back from the shiny technology?

Dana (14:55):

I'm so excited you are excited and curious about new techs. I think we should always watch trends. I don't think we should always go and invest in them, but I think it's important to watch the trends. And I would ask you, I would be like, "Why?" I know I'm just like, Dana, Why? You tell me why because you know what? Developers are smart humans. I typically believe we all are informed. I ask a question why, I expect an informed answer. You'll be like, "Oh, because we can do, I don't know, X, Y, and Z." If the answer doesn't meet the need, then I'll say, "That's really cool. Maybe this is something we need to investigate for the future." Because your hope they're at a startup, right, our hope within startup is that it makes it.

Dana (15:34):

And so it's about timing too, and if you're looking through that startup lens, I'm like, "Ask yourself, how can we grow into this?" And if your answer was really awesome, I think then it's like, "Cool. Let's do an experiment." If you're running an agile shop, you have capacities for experiments. I always say if it's an informed decision and answer because I definitely am from the book of knowledge of don't ask permission, just come with me in an informed way. And so I would be like, "Hey, why don't you do a time-boxed, measurable experiment to see if this meets our needs because you shouldn't do anything without measurement." You absolutely shouldn't. You know?

Dana (16:09):

And so that's one of the ways that I would come back to a dev because I don't want to want hamper curiosity. I also want to do the right thing for the business. That's my job. And two, I love the fact that we're thinking strategically that our company, our business is going to grow to what we need for these different types of technology. So it's finding that balance of promoting curiosity, doing it in a safe way that promotes the business. And then like I said, watching the trends and thinking about what you will be in three, six, 12, one year.

Dana (16:43):

And so I don't know, I love it when engineers hit me up with that stuff because that shows me, they're thinking about the future. Now, it may not be the right time, but at least they're thinking about it. Unless they think about it right now, and then I'm like, "Okay, you just were on Hacker News. I was on Hacker News. I saw the same article you did."

Daniel (17:01):

Good call. Yeah, watching the Hacker News, actually, that is a good tip for leadership, yeah. Check out what's currently trending.

Dana (17:08):

Oh my God, check it out. And then when you get a random Slack message of like, "Do you see this?" You're like, "Hold on. I know this is probably trending on Hacker News." I call it Hacker News development just like Twitter-driven development. There's two types of problems there.

Daniel (17:21):

Yeah. Nice. Very nice. Yeah, I like that. It's a good tip. So we've touched a few times on this, and then about the business aspect of stuff. And I've totally got this as my career has progressed. I wish I'd learned some of this stuff earlier, I guess. But how important do you think it is for all engineers, really, for the frontend, backend ops, whatever, how important is it for them to understand the business context they're working in?

Dana (17:41):

I think that's incredibly important. Do I think they need to know all of the customer use cases? Do I think that there is a need to dive as far as a product manager would know? No. I mean that's a separate job. However, I do think we all have a responsibility to understand what we're building, why we're building it, and what is the next steps in evolution, to have that kind of product acumen, especially when you're in a startup because you're going to wear multiple, multiple hats. There's going to be cases where you are the PM designer, support person, office manager, you are everything, in addition to being the number one lead developer, or just another developer on the team. There's going to be a time that you're doing all of this.

Dana (18:20):

If you don't have product sense, how do you know what you're building? How can you go and partner, and collaborate, and cooperate, and make the good decisions? As engineers, it is our responsibility to go and ask good questions because I keep coming back to it, we all want this pristine, awesome experience, but it takes time to get there. And sometimes we have to be real about the time. And to be able to make those tradeoffs, you have to know what the customer experience is. What is the end-user experience? Because honestly, that's all that really matters at the end of the day is how the human interacts with whatever you're building, and how they feel about it.

Dana (18:57):

And if an engineer doesn't have any kind of concept on that, on what they're building, they're going to build something shallow. They're not going to ask the good questions. I mean, it just squashes creativity. And also, it pushes agency and ownership to somebody else. I like to build teams that have a sense of ownership, right, that have a complete line that says, "I am invested in this. I am motivated, and I believe in delighting my customers." You can't do that if you're just giving a list of features to develop and not asking questions.

Dana (19:31):

And so I think it's important, but I think there's a limitation. And honestly, it depends upon your experience and skill level, right? As a new engineer, you should have awareness, and you should be hungry for it. But you should be focusing on delivering the task, right? You're more than likely going to be having a senior engineer partnering with you saying, "Hey, let's drive through this because you're just honing your skills." A senior engineer, staff engineer, especially as we get in those higher ranks of technical ability, you're working solo. You're staff principal engineer, you're typically tagged to go do an MVP to run an experiment to try a proof of concept. And how can you do that when you have no idea of the end product or the user experience?

Dana (20:16):

And so it's incredibly important. We should start honing those skills in the beginning of your career, but don't over worry about it. I come back to being curious is probably the most important thing. And then also, it's going to allow you to build better. So when your product manager or team are like, "Oh, we're going to do this." And maybe there's a technical impediment, you're still thinking about that user experience. Well, it's like, "Okay, why did we want to do ... We wanted this user experience, but we have this engineering limitation or impediment." Well, how can we still work towards that experience?

Dana (20:49):

And I think understanding the end experience, why you work through the impediments, is so important. In a part of my career, I had the pleasure of working at Envision, which is really design-oriented. And I, for the most part of my career, have been a platform engineer, which isn't necessarily externally customer-facing. But I do believe in the principles of design thinking. And I believe every engineer should go through some of those exercises because even if you're building an internal API or you're building a toolset to support your production infrastructure, there is still a person that will be operating that at some point. There is a person that will be interacting.

Dana (21:24):

So thinking about that first, even if you're on an internally facing team, is going to delight your people because that's what you're doing, trying to enable developer productivity. You're trying to take away some of the cognitive load on the administrative or system overhead so that they can push those features in a reliable, predictable way.

Daniel (21:43):

Well, I wish I'd had that advice like 10 years ago. Probably I've been here like 15 years ago, and the beginning of my career. That's like, yeah, golden, golden. On that note, so you and I were talking off mic in the beginning about how do you balance up all the requirements of skills as an engineer these days. You have got this business focus. I'm totally down with that too. You've clearly got to be good at whatever you do, be it development, QA, ops. But then I've noticed, in particular, around development, a push to be even more ops capable. Have you got any advice of how, as an engineer ... So I'm sort of a mid level engineer, I'm getting the business ideas. My coding is good, but I'm now being asked to understand more about the operational aspects of the platform. That can be a lot to some folks.

Dana (22:20):

I mean, it is a lot. In the teams that I've been able to support and had the privilege and benefit of working with, these are the transformations that a lot of companies are making because, once again, we're getting smarter. And the barrier of entry for doing the op side is lowering. And we also are thinking about how we can have more ownership in that meantime to recovery and meantime to availability, really be closer to home so that we can fix it faster. Right? You think about why are we pulling people in closer on all these primitives? And it is, at the end of the day, to have a really highly reliable site with quality. Right? Which is quality and care. And so it can be a lot.

Dana (22:58):

And I think really we've been talking about devops and building developer tools for operations for quite a while now, and it's really starting to come across the board in the industry. I'm seeing enterprise companies take this leap into these toolsets.

Daniel (23:11):

Agreed.

Dana (23:12):

They may have not before. And it's a cultural shift. And so coming back to your question, I think it's about one, everybody learns in a different way, and they have different backgrounds. I'm a mid-level engineer. Maybe I didn't work at a place that had me do this particular thing. Maybe I was at a large, large company where I had a very focused job because you have the startup where it's like, "Oh wow. My mental cognitive load is overloaded because I'm doing everything. And I don't know if I can take this operational type of expertise any further." And then you have the other version, right? "I'm an enterprise engineer that's very focused. We have four or five different teams that handle that." That's going away. Toolsets are doing that. It's becoming clickable.

Dana (23:52):

So what I think this is about creating a culture. And then also, that's where it starts. Is this important to your culture? Why is it important to your culture? Understanding why somebody should learn this because you can find the space if you believe in the reason. Right? We always say this, "I'm too busy." I say this all the time, which, if it is a priority in your life, you're probably not too busy, and everybody has 20 minutes they can carve away. And so I think it's about putting the pressure and talking about with your leadership and your manager about your personal capacity, and then saying, "Hey, what's more important? To go deeper on knowing business knowledge because my career wants to go in this way, or is it important for me?" Because it comes down to a personal decision, but I think it starts with the culture.

Dana (24:38):

First and foremost, why is it important to your team? Is this something you need to double down in? Are you a startup that has Jack and Jill's of all trades that have to wear multiple hats? Or maybe you've got an enterprise company and it's not that important, but it's good because your next career move may be to a startup where you're asked about this. I think it comes down to your personal desires. And then also, what's the kind of company that you're in? And then question. Say, "Why is this important to us?" And if so, build a culture, ensure that you have the right skills and the curation of skills. I guess the skills in the sense of the support system to build those skills, right? If this is new core production engineers, get them help. Hire SREs and devops to help train and learn them, not make it their problem, but to help them understand how to do this because it's new.

Dana (25:27):

So I don't have a one size fits all for that because it really comes down to who you're working for and what you're doing, but it can be a lot. It just can be a lot. And I think this is where we sit and say, "What's most important and where are we at?" And then I think personally, you've got to ask yourself, what are you driving for? Are you going into management? Are you going into product management? Because at a startup, you may start as an engineer, and you may be in a totally different field. You may be in a totally different field. Shoot, I've been a PM. I've been in support. In 21 years, I have probably done every role except I've never been an actual doc writer, even though I started to write the docs. I think that's the one title I've never had. But you've got to ask those questions.

Daniel (26:14):

I like it. I like it a lot. So wrapping up with a couple of final questions. I wanted to get your thoughts on what do you think the future holds? Where's the biggest innovation coming down the pipeline? And I'm talking about maybe languages, architectures, platforms. You can definitely spin it from a GitHub slather. I don't mind that. But what's the most interesting thing you're thinking about in the next few years?

Dana (26:30):

In next few years, well, I mean, that's what we started talking about right now, right? How can we complete the software development loop and life cycle in a real automated way from local to production? For me, that's the place where I'm seeing it. But I'm going to throw one out there for my developers. You called it, people like Rust. I've been thing on it for a while, not because it's not offered. I love you, Mozilla. But I was like, "Wow! React, and now we've got Rust." I think Rust is going to continue to pick up some momentum. We'll see. I don't know. We'll see. I'm watching that one, but honestly, because of my interest, and I'm going to just tell you my personal nerd out, it's going to be that local environment, that dev environment, that complete cycle and that circle. What can we do to get into production faster? Right?

Dana (27:20):

We all want to be agile. I think really using dark data and using production load beyond the local environment, using feature flags and incrementally, we're already doing that. We're already doing that on the industry. I think it's going to be even more so. It's like, "Cool. I really want to see how it works." I've gone from my virtual local, and it's gone right into production in a Kubernetes pod that has only got 2% of traffic. And I can sit and watch it, and I put more traffic in it. And maybe it'll double right, so I can see the load. These interesting things that help us determine what our future scale is and what it actually does within the systems once those transactions are written. That's what's interesting. And I think that's the next steps for us.

Dana (28:01):

And the continuation of how we store data and how we present data, that's always so cool and interesting just with the enhancements, and GPUs, and graphics, and animation, and all of those things. And now I'm getting buzzwordsy. It's machine learning. Come on, you-

Daniel (28:18):

Blockchain.

Dana (28:20):

Blockchain, you know what? Let's just say the future is blockchain. I think that's fair, Daniel. That's the future. You heard it here.

Daniel (28:28):

Nice. Very nice. Well, yeah, thanks for your time today. I really appreciate it. It's been great chatting to you. And thanks so much.

Dana (28:33):

Thanks. It's been great to be on the show. I appreciate it so much.