communities of practice & Improving the Software Engineering Craft and Developer Control Planes

When Toli began his career in software, the socio technical aspects of software development were not a big consideration. Today, both within the industry and for Toli himself, software engineering as a culture drives both business goals and the developer experience. Taking culture to heart, and the mindset of change, and reconciling goals with developer experience is a big part of helping developer teams achieve the goals.

As Toli describes, an example of this was a cultural shift and a big decision to move away from Kubernetes and containers as technologies because they exacted too much cognitive load from developers and diverted their focus from the primary goals at cinch. Moving away from infrastructural issues that became everyone's problem, the team redirected their efforts to customer-facing metrics rather than cluster-based metrics, which helped the team move closer to customers and serve them better.

However, simply making these choices is never enough to get developer teams on board. As Toli explains, at first developers were demanding, "Where are my logs?" when in fact they needed to learn new concepts in practice, and that required a journey and encouraging the use of specific tooling (their journey started with Honeycomb and landed on Datadog). How could cinch introduce these changes while understanding and respecting the existing culture and bringing developers along on the journey?

"We deliberately designed the organization to give each team a dedicated person to function as something of an ops person, not really ops but more like software engineers who had an extra focus on infrastructure, CI/CD and observability and monitoring, and a little bit of SRE. Someone for each team who thought about the software life cycle and relieved the rest of the team from that heavy lifting. This helped solidify the new processes while slowly evolving the culture. We found that this approach helped teams build software, faster and better. It introduced the idea of many enablers, or coaches, which was great for a while but hard to scale."

Scale up to coach software engineering teams: "Put me in, Coach; I'm ready to play"

While the idea of being a "software engineering coach" is a role Toli has claimed for himself since early on in his career, and it still rings true in how he thinks about developers needing to learn from each other, it is harder to put coaching into practice at the level he initially introduced at cinch – with a "coach" on each team.

Scaling up to coach more developers at more "industrial" levels, as companies like cinch grow, is a tall order, but not impossible. Instead, it relies less on individuals who can wear all hats, and shifts the responsibility in a few directions: helping make developers more productive, reducing cognitive load with developer control planes and sharing knowledge more efficiently and effectively.

Cloud luminary Kelsey Hightower recently explained in a DockerCon talk that you, "...don't want a 10x developer... what you want is someone that can come in and make 10 other developers more productive", and this is the kind of thinking that typifies Toli's approach to scaling up developer coaching. Codifying and sharing knowledge (and automating workflows) bridges many of the gaps in developer knowledge, and makes it possible to scale up when one-to-one or small-group coaching becomes unrealistic.

Reduce developer cognitive load: Adopt internal developer control planes

Reducing cognitive load is a major concern in building productive developer teams, and the industry appears to be converging on the idea that a centralized developer platform is the key. A developer needs a control plane from which to ship and run the code they have written and to trust that once they write the code, package it correctly and ensure the presence of its necessary dependencies, everything from that point on should be in place without their intervention. The code should just run.

For Toli's cinch team, reducing cognitive load required peeling back existing technology and starting from scratch. Their site was built on Kubernetes, and when they realized that they would have to build their entire team around this, ensure its stability and then develop the software for the site they wanted, they realized that it was not the right path for them. Despite the risk, Toli's team opted to move to serverless, and this immediately reduced the cognitive load. Serverless helped the team to worry only about their core focus areas, not about containers and infrastructure.

"We gained cognitive space by not having to deal with containers – we redirected that load to observability, and helped us get closer to the customer. Almost two years later, we refocused that energy to being fluent in Datadog, which has helped massively because we became closer to the customer and started talking about business metrics. Now we can understand how customers are using the website rather than gathering metrics related to Kubernetes clusters. That was the reality of what we experienced."

With this reality and stability, it became clearer how to proceed in building a developer platform that would serve developers working on key business goals. By extension, other forms of learning became easier to put into place, and discussion opened about the need for internal developer relations and knowledge sharing.

Toli explains, "One of the biggest problems we have, in Team Topologies speak, is that we have a lot of stream-aligned teams, but minimal platform teams. This leads to over indexing on streams, and each team focuses on their individual domains, and the different disciplines are not talking to each other. You lose critical knowledge-sharing. The way teams are often set up, people are not building cross-functional relationships with each other, but we want people to learn together and from each other. In questioning, 'How do we continuously improve our craft?', we need to consider how teams work together (and ensure that they do) and look to informal communities of practice."

Embrace communities of practice: Share knowledge to improve the software engineering craft

"How do we go about sharing practice and sharing knowledge between teams?" Toli has asked himself this question in relation to his team, knowing that people across development teams rarely had the opportunity to build relationships with and learn from each other. "We want people to learn together. How can we organizationally enable learning and advance our understanding of the software engineering craft itself?"

The approach Toli's team took mirrors the approach Crystal Hirschorn took with her engineering teams at Snyk – bootstrapping teams using and different modes of self-service education. Practically, for Toli's teams, this involves bringing like-minded people together, fostering discussion and giving them tools to more easily communicate their knowledge with each other. This can move from smaller groups with shared interests to more organization-wide knowledge sharing and hands-on projects, which can be anything from launching an internal blog (as cinch has done) to curating better materials on Confluence pages to an actual project to, for example, improve CI/CD across the organization based on what the community of practice has discovered or learned.

Toli describes how the cinch team thinks about becoming better knowledge stewards, reading about and implementing communities of practice, getting participation in these communities and figuring out how to share knowledge. Why? According to Toli, "The people actually doing the work know the work best."

Conclusion: Persevere on the platform - the sociotechnical is as important as the technical in developer experience

Improving the software building craft is about a lot more than technical know-how. Even though technical expertise and knowledge needs to be shared, the underlying culture and the sociotechnical environment in which the developer experience is shaped is just as important. Toli shares, "Bringing together people with common interests helps to forge the sharing and building of knowledge and stewardship we need to gain a true understanding of how people and technology interact."

Teams thus need to be built to knit together business, technical and human understanding to deliver on both external business goals and on internal needs. Toli again echoes similar sentiments of those shared by Crystal Hirshorn, about the need for internal developer control planes, or "paved paths", in enabling and supporting the developer experience in both the short and long term. Toli states, just as Crystal did, "Platform engineering – and the journey to building your developer platform – requires perseverance and the ability to foresee what tooling you need and then to stick with it.

Be careful upfront because it will be difficult to migrate away from it, so be quite clear about what you want from a tool before you adopt it. But overall, if you believe in your platform and your approach, don't despair if it does not seem to work at first. It will work in the end if you've made careful decisions in the beginning."