Developer Control Planes: A Community Practice Engineer's Point of View
About
Episode Guests
Apostolis Apostolidis (better known as Toli) joined Daniel Bryant in the latest Ambassador Labs podcast. Toli discussed the culture of software engineering, how best to engineer platforms to help teams achieve their goals, and how to set up and benefit from communities of practice.
Here are some of the key takeaways from their conversation:
- Consider how to improve the craft of software engineering
Knowledge loss is a downside to the way many development teams are organized. Discussion on the possible need for internal developer relations functions led Toli to conclude, "One of the biggest problems we have, in Team Topologies speak, is that we have a lot of stream-aligned teams, but minimal platform teams. This leads to overindexing on streams, and each team focuses on their individual domains, and the different disciplines are not talking to each other. You lose critical knowledge-sharing. The way teams are often set up, people are not building cross-functional relationships with each other, but we want people to learn together and from each other. In questioning, "How do we continuously improve our craft?”, we need to consider how teams work together (and ensure that they do) and look to informal communities of practice."
- Create and communication around communities of practice
Communities of practice might be a new concept for many, although Toli adopted the idea and introduced it to his organizations several years ago. As defined by the Harvard Business Review, communities of practice are "groups of people informally bound together by shared expertise and passion for a joint enterprise." Bringing different people from different roles together to discuss shared interests in an informal but regular way to focus on practice yields benefits.
"With time, your community of practice matures. You go from generating interest, getting people to join, meeting at a regular cadence to beginning to see cross-functional benefits. You aren't just talking about things once your mature curve accelerates. Instead you may be doing a project together. For example, one project might be about improving CI/CD across the org. How can we look at the backend template and recommend, as a community, tangible changes that will improve the CI/CD mechanism? And further, how can we improve and share knowledge around this? This involves not just the community but extending it to communicate to a wider audience, for example in GitHub Discussions or a Confluence page," Toli explains. "Facilitation is a particularly useful skillset. It enables communities, makes them inclusive, helps to move things along, encourages people to participate. Facilitation techniques are a massive bonus for use in fostering communities of practice."
- Foster cross-org relationships and learning in a fast-scaling environment
Once the cinch team recognized that the silos they faced were in part due to its rapid scale-up process, it became more critical than ever to enable learning and technical knowledge sharing at the organizational level. This involved both team structure and technology and tooling choices.
Toli explains how, at least for some time, teams were designed to ensure that each team had a dedicated person who wore several hats, for example, a software engineer who had an extra focus on infrastructure, CI/CD, SRE, and so on. Having this dedicated resource with more of a full life cycle perspective helped the teams build software faster and better. "The idea," Toli continues, "is to introduce enablers, or many coaches, who help teams scale."
Technology, too, is a consideration for fast-scaling organizations. As Toli describes, the pace of growth will throw enough of a learning curve into the mix without adding the burden of too many unfamiliar technologies. In his example, cinch already had a website with a backend hosted in a Kubernetes cluster. Despite taking time to learn Kubernetes and Helm charts and concepts and tooling, the team realized that if they wanted to build a team around Kubernetes, they had to understand it completely and make it stable. They made the decision to move away from Kubernetes to serverless in order to reduce the learning curve and cognitive load.
- Reduce developer cognitive load and get closer to customers
Most organizations grapple with the problem of saddling developers with too much cognitive load. At cinch, they found the cognitive load of adopting and using Kubernetes to be too high. Moving to serverless, the team was able to shift focus from infrastructure to observability and monitoring.
"By leaving containers in the past, we gained cognitive space that we could redirect to observability, which had the bonus of letting us get closer to the customer. A year and a half later, rather than becoming a Kubernetes expert, I became immersed in observability and monitoring using DataDog, and this helped us understand business metrics and understand how customers were using the website rather than metrics around Kubernetes clusters."
Transcript
Daniel Bryant (00:02):
Hello and welcome to the Ambassador Labs podcast, where we explore all things about cloud native platforms, developer control planes, and developer experience. I'm your host, Daniel Bryant, head of DevRel here at Ambassador Labs, and today I had the pleasure of sitting down with Apostolis Apostolidis, or Toli to his friends, a principal practice engineer at UK-based car purchasing platform, Cinch. Join us for a fantastic, wide-ranging discussion covering topics such as how to improve the craft of software engineering, creating communities of practice, and how to reduce developers' cognitive load when dealing with Kubernetes and serverless.
Daniel Bryant (00:31):
And remember, if you want to dive deeper into the motivation four and the benefits of platform engineering and cloud native developer control planes or are new to Kubernetes and want to learn how to get started, you can do so in our free Kubernetes learning center. Please visit getambassador.io to learn more. Well, welcome, Toli. Many things for joining us today. Could you briefly introduce yourself and give us a little bit about your background as well, please?
Toli Apostolidis (00:53):
Well, thanks for having me. My name's Toli. I'm a principal practice engineer at Cinch. It's a secondhand car seller in the UK. My role sounds a bit weird. It's almost like a practice lead. I specialize in DevOps and observability, SRE, anything in and around software engineering. I found myself in this role about two and a half years ago, when Cinch started as an idea. I was the first software engineer to be hired. And I went through the journey with Cinch. As Cinch grew, I grew as well, and went through a few roles.
Toli Apostolidis (01:27):
But the thread between it has always been DevOps and observability. My background before that is I studied maths, like mathematical physics, a master's, and then I literally found myself doing software development because we're looking for mathematicians. And I found it really, really hard, very hard. A lot of terminology. It was about the time when software was very much a 13-year-old who's learned to code and is really, really an expert because they've done it for a long time, and it doesn't have the socio-technical aspect that we talk about today.
Toli Apostolidis (02:05):
And so it was really hard. But then it took me a few years to even call myself a Dev. And I stayed for the same company for about seven years, working on optimization algorithms, which was a good kind of way into software. And then moved onto one more company, and now I'm at Cinch, but in terms of tech stack, I'm very much a .NET / C# developer. I've worked Microsoft products my whole life, but I've touched all stacks now, infrastructure, the backend, front end, but I would probably say backend is my most...
Toli Apostolidis (02:40):
Nowadays, I'm very, very interested in software engineering as a culture, as a mindset, and how to help teams. I'd almost say software engineering coaching would be something-
Daniel Bryant (02:51):
Ooh, interesting.
Toli Apostolidis (02:51):
... that I could currently do. Yeah. I guess I was a basketball coach as well-
Daniel Bryant (02:57):
Nice.
Toli Apostolidis (02:57):
... so there's some synergy there, because there's definitely things that are shared between the two disciplines.
Daniel Bryant (03:06):
Oh yeah, I can definitely relate to coaching because I moved into more management roles, coaching is a super valuable tool, right? But I think even when I was a consultant, coaching at a technical level is valuable because often, I used to joke I was a software psychologist. Because often I'd go in as a consultant and people would know their problems. I'd just need to ask the right questions, right, an idea would come.
Toli Apostolidis (03:24):
Yeah.
Daniel Bryant (03:27):
I love it, Toli, I love it. We can definitely double pick on that, go to that a bit more. But you and I initially connected based on a tweet I sent out about internal DevRel, right? It generated quite a bit of buzz, and people were like, "Oh yeah, I'm kind of doing this. I'm not sure if I'm liking the terminology internal DevRel." But you and I have talked off mic around what are the communities of practice, things like that, and then you mentioned about coaching. I'd love to get your thoughts on, when you replied to my tweet, what did you think? Were you like, "This is interesting. Haven't we seen this been talked about?" Or I was curious what were your thinking around that.
Toli Apostolidis (03:56):
Yeah, I thought you were in my head. I was like, "How'd this guy get in my head?" It's something I've been thinking about a lot and we've been talking about a lot in terms of how we go about sharing practice and sharing knowledge between teams. One of the biggest problems that you have, we have very much a cross-functional team orientated topology, if you like. So from the team topology speak where we've got a bunch of stream aligned teams and a bare minimum number of platform teams. And the problem with that is you over index on stream and the team's focusing on their domain. And disciplines between them not talking to each other.
Toli Apostolidis (04:35):
And what happens there is you lose the sharing modes, you lose the okay, so how do we improve our craft? That's where my role came in and we have got colleagues who focus on other aspects, back end, front end. We've got head of engineering practice. And the reason for that is because we want people to share things. We want people to learn together. And we've had learning there from the start. But they haven't built their relationships with each other. As a matter of scaling, we're getting more people than we started about two years ago, two and a half years ago, with 10, 15 people in Manchester. And now we're over 250 people-
Daniel Bryant (05:12):
Oh, wow.
Toli Apostolidis (05:12):
... as a company. So all of this does contribute to quite a bit of siloing. So I was very interested in your tweet, because I thought how do you organizationally enable learning and technical enablement and just advancing our understanding of the craft itself?
Daniel Bryant (05:33):
You've got a bunch of notifications rocking in, Toli. Nice, nice. I'll have you pause there.... Yeah, fantastic. And before perhaps we dive into some of the socio aspect of the socio-technical, Toli, just to give folks a bit of background, because I think this is fascinating background, you and I were chatting off mic about since you don't actually use Kubernetes, because you know I live and breathe Kubernetes at the moment, but I'm fully aware that other things do exist. And I'm serverless. I've done a bunch of stuff with Knative. Obviously with AWS Lambda back in the day. I'd love to get your take on why your organization chose serverless. And you even hinted at maybe chose it over Kubernetes, which I thought was super interesting.
Toli Apostolidis (06:12):
Yeah. It was my first question when you asked me about a podcast. Are you okay if I don't talk about Kubernetes?
Daniel Bryant (06:16):
Yeah.
Toli Apostolidis (06:18):
Yeah, so it's interesting, right? Because when I started, I was hired as a DevOps SME, whatever that means. But the platform that the initial version of the website was built on was AKS, so Kubernetes and Azure.
Daniel Bryant (06:31):
Oh, yeah.
Toli Apostolidis (06:32):
And my background was I was hired in the role because the recruiter heard that I used Azure DevOps, and the company was looking for a DevOps SME. So making the two together-
Daniel Bryant (06:42):
Check.
Toli Apostolidis (06:42):
... completely makes sense.
Daniel Bryant (06:43):
Yeah.
Toli Apostolidis (06:45):
But as it happened in my previous role, I got very interested into the DevOps movement and DevOps culture because we had a lot of DevOps anti-patterns. We had club meetings. We had ops and dev. We had things going on. We had very, very slow cadence of releases. And all sorts of things, all the things that you would... So I got very interested in it. And when I joined, since we already had a website that was built by an agency and the backend was all hosted in Kubernetes cluster, so naturally me and a couple of other engineers, we got into the habit of... We started learning Kubernetes and all the Helm charts and all the concepts and terminology and tooling around that.
Toli Apostolidis (07:28):
And we started to realize we have to build a team around this. One of the first things we need to do is build a team around this and make this stable, and then build out teams to develop software for the website that we wanted. It didn't help that one day when we upgraded Kubernetes, we had to take the website down and we had to handle that. Everyone was really nice about it and I remember the technology director called me to ask me if I wanted any breakfast.
Daniel Bryant (08:00):
Oh, that's leadership, right?
Toli Apostolidis (08:02):
Yeah, it was really, really nice. But that was the beginning of the end for Kubernetes. We were like, we don't understand it enough. We need to hire in people. So we made the brave decision to move to serverless. And the natural evolution from there would be Azure functions. But we didn't think Azure functions were mature enough at the time, and although we were .NETs heavy and we'd hired a few .NET people in, we decided to go for AWS...
Daniel Bryant (08:28):
Oh, interesting. Lambda?
Toli Apostolidis (08:30):
Yeah. It's worked for Lambda and anything that didn't mean that we had to deal with hosting.
Daniel Bryant (08:37):
Nice. So really, I've talked off mic about the fantastic work the team topologies crew have done, and I look back on my dev career and cognitive load... Didn't call it that back then, but cognitive load was a key concept when I look back. When I was most confused, my cognitive load was high. Is that a fair role up there, Toli? Do you think the cognitive load was super high for Kubernetes?
Toli Apostolidis (09:00):
Yeah, that's fantastic, yeah. I termed it as the cognitive load shift from infrastructure to observability and monitoring. So what serverless enabled us is that we didn't have to care even about containers. We don't have any containers. The only thing we needed to care about is a little bit of infrastructure configuration, our GitHub actions that we had that we also moved to from Azure DevOps interestingly, and then we went strong on observability and we thought, well, what can we gain? So we have a certain cognitive load. We gained a bit of cognitive load from not having to deal with containers.
Daniel Bryant (09:41):
Yeah, yeah.
Toli Apostolidis (09:41):
And plus we redirected that load towards observability and monitoring.
Daniel Bryant (09:49):
Oh, nice.
Toli Apostolidis (09:50):
And that helped us become closer to the customer ultimately. So one half year later, rather than me becoming an expert in Kubernetes and containers, I started knowing a lot more about observability and monitoring. And our platform that we use, which was data, which helped massively because we became close to the customer. We started talking about business metrics or metrics that help us understand how customers are using a website rather than metrics related to Kubernetes clusters and-
Daniel Bryant (10:20):
Yeah, interesting, interesting.
Toli Apostolidis (10:21):
And nothing against them. But that was the reality of what we experienced.
Daniel Bryant (10:24):
I like it, because I often talk about business KPIs and then things like SLIs, service level indicators, right? And I do think they're almost two concepts. Full stack engineers in theory, full cycle developers, would have to know both things. But I do wonder at a certain scale of organization or certain scale of platform or applications, the cognitive load is just too high. You have to choose, right? And it sounds like you made that choice, Toli, because the customer's what you're most focusing on as a business. That's where you're adding your value and getting your revenue, right? So you made that conscious choice to focus on business KPIs over platform SLIs.
Toli Apostolidis (11:02):
Yes, yeah. We still try to use the concept of SLOs, although we're finding it quite hard. But it's all about, from day one, we started emitting metrics and telemetry data that's related to business. So you'll have things like vehicles sold or funds application approved, things like that, that relate to our business domain rather than our infrastructure platform.
Daniel Bryant (11:27):
Very nice. Very nice indeed. And was that an easy shift for the rest of the team? Because I'm thinking using... And we're going to be chatting for 30 minutes or whatever, but you do seem quite empathic. You seem to be thinking in the bigger context all the time. And I'm conscious that earlier in my career, I didn't do that, I'll be honest, right? And definitely developers I've worked with in the past are just focused on what's my CPU? What's my heap size? What's the memory load? How do you reconcile your goals with the rest of the team?
Toli Apostolidis (11:53):
Yeah, it was really hard. The first reaction from developers is where are my logs?
Daniel Bryant (11:59):
Yeah.
Toli Apostolidis (12:00):
I want my logs. Give me my logs. And then you have to show them the new concepts and you have to show them working for them to understand it. It's that extra cognitive load that they need, which they gained from not having infrastructure considerations. But it took time, so we had to really try and take them along the journey and encourage usage of the tooling. So the biggest problem to start with is what platform do we use? We started with Honeycomb, which was really good. It has all the observability principles, right? But then we moved into Datadog because it encompassed a lot more of what we might need, including front end and SLOs at the time.
Toli Apostolidis (12:46):
So I guess the challenge was how do we take engineers along the way? And going back to our initial Twitter conversation, we've deliberately designed our organization so that each team had a dedicated ops person, but they weren't really ops people. They were software engineers who wrote code, who had an extra focus on infrastructure, CI/CD, and observability and monitoring, and a little bit of SRE. So there's a lot there, a lot of software life cycle. But we hired them explicitly to say, "We'd like you to think about these things and we'd like you to pair with people in your team and we'd like to bring things in from what you've learned from outside the team to help your team build software faster and better." So that worked for a while. I was one of them.
Daniel Bryant (13:34):
Oh, nice.
Toli Apostolidis (13:34):
I worked in a team like that for a while, so I know how it feels. And a lot of it was actually, not the DevRel that much, but the enabler.
Daniel Bryant (13:41):
Yeah, totally.
Toli Apostolidis (13:41):
So you have many coaches of okay, so here's how we use observability and here's how we use monitoring. And that worked for a while. But as you scale, it becomes a lot harder to actually hire one per team as well. It's a really hard role to hire for as well.
Daniel Bryant (13:57):
I can imagine, yeah. So I guess following up on that, how do you scale it? Is it a case of... I was chatting to Crystal Herschel, and again, we were chatting off mic, and I was chatting to Crystal earlier in the week, and she was at Sneak moving a similar kind of approach, but moving more towards self-service education. So bootstrapping teams with people, but when that didn't scale, adding a bit of, say, video tutorials or adding some articles to read as part of the onboarding process. Is that something you've done as well?
Toli Apostolidis (14:27):
More or less. We're running a book club right now on the DevOps handbook internally. So one of the things mentioned in part two, which is all about where do you start with DevOps, the three models that you can use for this kind of thing, so you either embed people into teams or you have a separate central ops team as the traditional way. Or you have the DevOps advisories or the ops advisories, whatever you want to call them. And I think we went with the first model initially, but growing towards the latter model, which is around... You have maybe one SME, if you like, for two or three teams that people can advise.
Toli Apostolidis (15:11):
And that's naturally happening already. And on top of that, we were thinking about how to become better knowledge stewards and we've been reading a lot about community practice. And we've been reading a lot about how you can better improve your knowledge stewardship through community practice, because we believe... Well, the community practice manager believes that the people who do the work know the work better, the best.
Daniel Bryant (15:38):
Yep, makes sense.
Toli Apostolidis (15:39):
So I think we agree with that. And we want to help people build communities and help people participate in communities so that we can share our knowledge, so we can learn together, so people can build relationships with each other.
Daniel Bryant (15:56):
Nice.
Toli Apostolidis (15:57):
And across teams, because cross-functional teams or cross-discipline teams create silos.
Daniel Bryant (16:03):
Yeah, yeah, yeah.
Toli Apostolidis (16:03):
And we've over indexed on cross-discipline teams, so now we need to bring the disciplines together or bring the people who have common interests together. And we're hoping that out of that, we will have better knowledge stewardship. We will have better learning. And we're already seeing that quite a bit. We're seeing people picking up things like GitHub discussions. That's a new feature in GitHub, which is like stack overflow. You can use stack overflow just to have an idea. And just get better at async communication. That's something we're working on a lot.
Daniel Bryant (16:38):
To scale, right?
Toli Apostolidis (16:39):
But the onboarding thing that you mentioned, so I don't forget to mention that, it's hard. I think as we've grown, we've found the new engineer managers as well, and they're helping quite a bit. We built out the engineer progression framework that describes a bit what the rarest roles we have are and what skills are required. And making things visible quite a bit I think is the biggest thing. So we're on a journey, if I had to summarize it, would be communities. It would be engineer managers and just better tooling, I would say, around basic collaboration.
Daniel Bryant (17:18):
Super interesting. Definitely something I'd be keen to dive into a bit more there, Toli, is based on our Twitter conversation. For me, the mention of the communities of practice is very close to that internal DevRel thing, right? For me, DevRel is a lot about communication, but you've got to get that feedback loop, too. And it's much like continuous delivery, right? Everyone's like, "Ship the thing." And I'm like, yeah, shipping is great. Got to get feedback so we get better.
Daniel Bryant (17:40):
And I think it's the same with communities of practice, right? It's that communication of things and it's that feedback. So could you share with the listeners how does that work in that framework? Because I'm guessing for a lot of folks listening, communities of practice is probably somewhat of a new concept.
Toli Apostolidis (17:56):
Yeah. So, to start with, none of these concepts are mine. I've read Emily Webber's book on communities of practice, and the book that she recommends as well in the book, which is a book called Cultivating Community. It's a practice, I think. And it was written 20 years ago.
Daniel Bryant (18:13):
Oh, wow. I've not read that one. That's on my list. Yeah, that sounds great.
Toli Apostolidis (18:17):
It's a bit of a big book, but it's a very interesting one, because it doesn't feel like it was written 20 years ago. A lot of the concepts are very much, you can relate to them still. So one of the big things about communities, you might have a community around, for example, security. So people want to learn more about security. We have a few roles that are quite obscure, called security engineers. A security engineer might start a community and what that means is you start a group of people that meets every so often, and they have a little bit of an agenda. And they go through the normal cadence of forming, norming, all that stuff that we talk about-
Daniel Bryant (18:57):
Gotcha, teamwork, yeah.
Toli Apostolidis (18:59):
... yeah, to get into a mature community. And the things that you need to do around that is first, you need to get the interest. Then you get people in and you meet regularly. I think that's the biggest thing. And that's where you start modeling that over index that I was talking about earlier around cross-functional teams, whereby before you were siloed and you just met with your team. Now once a week maybe you're meeting for an hour with people who have a similar interest across their organization that they talk about things and that you have an agenda. And as you mature, it becomes less about talking about things but more about maybe having a project that you do.
Daniel Bryant (19:41):
Interesting.
Toli Apostolidis (19:41):
And an example could be that these enablers that we have, we have a community of practice around that. They have a project around okay, so how can we improve our CI/CD across the organization? We have a backend template that we use, for example. How can we look at the backend template and recommend some tangible changes that will improve the CI/CD mechanism that is almost recommended by the core backend template, if you like? Or they will do things like, well, how can we improve the knowledge around this? How can we not answer the same question all the time? And this is where the GitHub discussion things comes in.
Toli Apostolidis (20:21):
And okay, well, we can have a good confluent space. We can have discussions and mark things as answered and better curate knowledge. I don't know if that makes it a bit more clear. It's not that clear for us right now. We're on the journey and-
Daniel Bryant (20:39):
Well, that's great though. It's definitely made it clear in my mind, Toli, and I always like to dive into these things even if they're emerging or nascent, right? Because I think other folks, as we've seen on our Twitter conversations, other folks going, "Me too." And sometimes people will literally have no idea. And I've been there, no idea where to start sometimes. And even if it's not where you ultimately end up, the beginning of the journey can be really critical sometimes. And I do think to your point, particularly in this post-COVID or maybe not being post-COVID, but COVID era, the natural human interaction has been removed to some degree.
Daniel Bryant (21:15):
And I think you have to work extra hard across time zones, locations, these kind of things. Async is not natural for a lot of folks and how you foster collaboration under that condition is even more challenging. So having some of the frameworks you're talking about, I think they're a really good bootstrapping mechanism. Would you agree?
Toli Apostolidis (21:35):
Yeah, yeah, I can totally relate to that. The scale that I was talking about earlier at Cinch, we did it all remotely, because we started building up teams in January 2019. 2019? 2020?
Daniel Bryant (21:50):
2020 is when COVID all kicked off, pretty much.
Toli Apostolidis (21:53):
Yeah. That was it. 2020, and then March 2020, we all went into lockdown.
Daniel Bryant (21:57):
That's it.
Toli Apostolidis (21:58):
So we had about four teams at the time and now we have 20-odd.
Daniel Bryant (22:01):
Oh, wow.
Toli Apostolidis (22:01):
So we scaled all remotely. And I have to say, the communities, one of the communities that I lead, that for a while was really hard to do remotely, especially for people who don't know each other. The moment we were back in the office, magic started happening.
Daniel Bryant (22:17):
I know.
Toli Apostolidis (22:18):
It's crazy. I don't know what it is about it.
Daniel Bryant (22:20):
I'm with you.
Toli Apostolidis (22:21):
It was incredible. I think the big thing for me as well is that with communities, I'm finding that there's a particular skillset that's interesting and useful is facilitation.
Daniel Bryant (22:31):
Connect that to coaching a little bit as well, like we were talking earlier, right? That's a great topic. Yeah.
Toli Apostolidis (22:35):
Yeah. So there's a whole community out there of facilitators who you go into a room, they facilitate something, and you're like, how did they do that magic? And it's learning that magic that I've found the most fascinating in my journey so far. And it's crazy how much it enables to make things inclusive, to help moving things along, to encourage people to participate. All these things and what kind of techniques you can use at different stages of the community is massive. One of the other things that we did quite recently is we started having an internal blogging platform-
Daniel Bryant (23:19):
Oh, interesting.
Toli Apostolidis (23:21):
Yeah, I'm hoping a lot of people out there have got similar experiences, but for recent communication, we had confluence amongst our teams.
Daniel Bryant (23:31):
Yeah. Yeah.
Toli Apostolidis (23:32):
And a lot of people, when you'll ask them, "Why don't you write this up or why don't you write a blog post? That sounds very interesting. Go for it," they're like, "Where shall I write it? I don't know where, where, where, is the biggest problem. So we launched an internal blogging platform. By launched, I mean we've paid for a blogging platform and asked people to join. And they joined, and people started writing blog posts for work, for outside of work. A lot of-
Daniel Bryant (24:01):
Is this public or just internal?
Toli Apostolidis (24:03):
It's not public right now. I think that's actually a big learning for communities and general, and it relates to blogging as well, is that there's a tension between psychological safety and working in the open.
Toli Apostolidis (24:16):
So what I've found with communities is that you have to start small and let people bond with each other and get to know each other. Even with a community that may be eight to 10 people, it's only when they break down into activities of two, for example, that they get to ask the questions. Okay, so where are you from? What do you do? All this stuff, and get to know each other. Same with blogging. We do want to blog externally but they didn't have a blogging culture internally. So it didn't really translate. So we thought the first step is we start an internal blogging platform and get-
Daniel Bryant (24:52):
That makes a lot of sense.
Toli Apostolidis (24:53):
... that blogging culture, and then naturally we're hoping that we'll blog externally as well.
Daniel Bryant (24:58):
Absolutely fantastic. I know we're kind of at time here, but I'd like to wrap up the podcast, because there's so many fantastic things discussed today. I've got a bunch of notes and I always like a book recommendation. You've given me a couple today, which is like, mission accomplished, right? But if you were to chat to someone that is in a similar position as you were a couple of years ago, knowing all the mistakes you've made, successes you've had... I love to share this kind of stuff. What advice would you give them starting on the journey of how to build a platform, how to share the knowledge, and how to ultimately meet the customers' needs, right?
Toli Apostolidis (25:33):
Very good question. Very good question. I would say I think my biggest learning is that if you believe in the platform, if you believe in your approach, don't despair. Because I've had a lot of those moments. It does work in the end. I had a lot of those moments. To give specific examples, around observability, it was a really, really hard concept to introduce to engineers. They were not used to it. It's not something that engineers put on their CVs.
Daniel Bryant (26:03):
Interesting. Interesting.
Toli Apostolidis (26:03):
You won't see them mention data in their CVs. But I'm hoping that they will now because it's a capability that's quite crucial in 2022. So I would say persevere. I was lucky to have line managers and people who really influenced me to support me in that process, and when I went on a call and said, "It's not working. No one's getting it. I don't think it's the right approach," they encouraged me that it is the right approach. So I'd say perseverance is one. The second thing actually is don't go against your tooling.
Toli Apostolidis (26:35):
Really do understand your tooling. If you're going to use Kubernetes, for example, understand what that provides. Don't try and retrofit something if you're using serverless. Likewise, if you're using... A big example for me was Datadog versus Honeycomb. I would advocate for a completely different practice if I was using Honeycomb compared to what using Datadog. And that's a good example because I was very much influenced by Charity Majors and all the stuff that she says.
Toli Apostolidis (27:04):
Some of those things just don't work in Datadog and you would probably best off having a slightly different practice. Maybe some of the principles are the same but don't go against your tooling. Use your tooling to your advantage.
Daniel Bryant (27:19):
Fantastic advice, Toli. Yeah, the tooling thing, I feel like I'm bumping into that a lot at the moment, because one thing I say in the cloud native space is we're still at the divergent phase. There's more tools coming and I love KubeCon. I'm looking forward to going to events here in a few weeks, months' time. But I know there's going to be more tools there. And I'm guilty as charged because I work for a vendor. We create tools. But at Ambassador Labs, we're really trying to say, "Hey, let's converge." And we talk a lot about developer control planes as a way to centralize some of these things, the code, ship, and run. But do you see a slowing down on the tooling that's going out there?
Toli Apostolidis (27:58):
I don't know. I think there's tooling... In our case, for example, again, take Honeycomb was a great tool and really, really pushes you toward the right principles, and they're principles I think that do work. But it's all about what you're trying to achieve. The answer's always it depends. And actually gives you options. I don't see the existence of the two, Honeycomb and Datadog, being contradictory. I think different organizations will want a different thing. I guess having options is good.
Toli Apostolidis (28:34):
One actually good advice I just thought about is that be careful about the tooling that you choose. It's not that we weren't careful. It's just that you will not easily migrate away from it. We migrated from Azure to AWS and we migrated from Azure DevOps to GitHub actions. But it's not every day that we're going to do that, especially to scale. So I'd say know what you want from the tool before you go into a tool.
Daniel Bryant (29:01):
That's great advice, Toli. I'll double down on that as well. In my time as an individual contributor and as a consultant, too many times we did not think about the requirements, the constraints, and the skillsets of our team, because this hot new thing looks great but when you start plugging it in, you realize there's a mismatch or whatever. So I'll double down on that comment there. Fantastic. Well, wrapping up, Toli, thank you so much. Really appreciate your time. It's been fantastic. Hopefully we'll meet in person and we'll continue the conversation in the future. We can even podcast again.
Daniel Bryant (29:30):
But in the meantime, if folks want to get involved in the conversation, want to reach out to you, where's best? Twitter? LinkedIn?
Toli Apostolidis (29:36):
LinkedIn, Twitter. Twitter, I'm @Apostolis09 and LinkedIn, Apostolis Apostolidis, you'll find me there, or my email address if you want, dev@toli.io, which is a lot simpler. But yeah, thanks for having me. This has been a really, really good conversation. You mentioned conferences. I went to a conference last week and it's just such a good feeling to go in person again. So yeah, hope I meet you at a conference at some point, and hope you enjoy your... Is it KubeCon you're going to?
Daniel Bryant (30:05):
Yeah, KubeCon next week, and then KubeCon EU not too far away. We've even got KubeCon NA, in North America. I think it's in Detroit this year, in October time. I know you're probably not in the Kubernetes space, but as in yeah, I hope we do get to meet at a conference. It'd be awesome, right?
Toli Apostolidis (30:17):
Yeah.
Daniel Bryant (30:17):
Brilliant. Thanks, Toli.
Toli Apostolidis (30:17):
Thanks very much.
