Daniel Mangum on Crossplane, Building a PaaS, and Multi-Cluster Kubernetes

Daniel Bryant (00:03):

Hello everyone, I'm Daniel Bryant and I'd like to welcome you to the Ambassador Livin' on the Edge Podcast. The show that focuses on all things related to cloud native platforms, creating effective developer workflows and building modern APIs. Today I'm joined by Daniel Mangum, software engineer at Upbound and fellow podcaster in the cloud space.

At Ambassador Labs we've been following Upbound's work on the Crossplane project for quite some time. Crossplane is an open source Kubernetes add-on that augments your clusters and enables you to provision and manage infrastructure services and applications from kubectl or using well-established continuous delivery best practices like GitOps.

Daniel Bryant (00:36):

A lot of folks I chat to in my day job are looking to create some kind of internal platform as a service experience. Something like an internal Heroku or some other PaaS, and Crossplane is aimed at helping teams provide, manage, and configure the infrastructure and application bundles associated with this.

If you like what you hear today, I would definitely encourage you to pop over to our website. That's www.getambassodor.io, where we have a range of articles, white papers and videos that provide more information for engineers working in the Kubernetes and cloud space. You can also find links there to our latest releases, such as the Edge Stack, which focuses on the edge and API gateway use cases in Kubernetes and allows advanced routing and security config. You can also find our latest open source releases at the Edge Stack API gateway and our CNCF-hosted Telepresence tool too

Daniel Mangum (01:19):

Thanks, I'm happy to be here. It's quite the honor to come on today.

Daniel Bryant (01:22):

Super. So before introducing yourself, I know you wanted to say a few words about the current happenings in the world.

Daniel Mangum (01:27):

Yeah, for sure. I appreciate you giving me a moment. I'm here to talk. And as you said before we get started, I did want to say a few words about everything that's going on in the United States and many other countries as well, as folks are taking a stand against systemic racism and police brutality. First off as a white person, this is a time where I mostly need to close my mouth and open my ears in most settings. We need to hear our brothers and sisters who have had enough with centuries of oppression. Because of this, as I mentioned to Daniel before we got started, I was pretty hesitant to record this podcast today. I've had the opportunity to be a guest on this podcast in large part because of privileges I've been afforded due to the color of my own skin.

Daniel Mangum (02:05):

That being said, one of the most important things I can do is use my voice when given a platform to speak, and we all can do when given a platform to speak. Today we're going to talk about software, Cloud, Kubernetes, and plenty of other topics that I care really deeply about. But if you take anything away from this episode, I hope it is this, each and every one of us has a sphere of influence. We have people in our lives who listen when we speak, you may have a large social media following, you may have a podcast, or you may just have your family and friends, whatever your sphere of influence is, but you have the opportunity and the responsibility not just today, but every day for the rest of your life to fight injustice in that sphere. So fight it with your words, with your actions and with your money, do it at the national level, the local level, and within your own home.

Daniel Mangum (02:51):

There is no easy fix and if you're like me, you have a lot of learning and listening to do every day to even begin to bring change. But don't wait until you know everything. You'll never know everything, and you're always going to make mistakes. Just wake up every day and consider what your role is in bringing change to that day and then go out and do it. If you don't exactly hit the mark, except criticism, have humility, listen to correction and do better next time. Inaction holds back change, start taking these steps today, and don't forget their importance a year from now when the new cycle has moved on to something different. Fighting injustice is a constant daily battle. Do not let yourself get comfortable.

Daniel Mangum (03:30):

I know Daniel has penned some tweets on his account with some good learning materials, but please feel free to also reach out to me directly on Twitter, or any other medium where you'd find me. And I will not give you advice, but I will try to point you to some resources that have been helpful to me, and people I trust who have more experience in fighting oppression that have recommended to me, I'll try to point you to those.

Daniel Mangum (03:53):

So yeah, I just wanted to start off with saying that because that's way more important than anything we're going to talk about today. And the reason why I think we decided to go ahead and record this is because we do have an opportunity where people are going to be listening to us, and we need to make the most of that. So, I just wanted to give that little spiel before we get started. And please, please feel free to reach out if you have any questions or thoughts on that or anything like that.

Daniel Bryant (04:17):

Well said Daniel, well said. One of the key takeaways I've had myself the last few days is that need to learn more. That tweet I've pinned on my account is a small thing to do, but it's somewhere where I want to start, I want to educate myself. And like you, I want to then share my learnings and so forth. And difficult times for everyone, but I think this community we have, the Cloud community, is fantastic at generating awareness, fantastic at sharing ideas, listening to each other. So I think we need to not only focus on technology, we need to focus on important social issues too. So yes, very well said

Daniel Mangum (04:45):

Thanks.

Daniel Bryant (04:47):

So, could you introduce yourself Daniel and share a recent career highlight for us please?

Daniel Mangum (04:51):

Yeah, for sure. So my name is Daniel Mangum. I work at Upbound, which is a company based in Seattle, Washington, but we're remote-first. So, I actually live in St. Louis, Missouri in the US. And Upbound's main focus is a project called Crossplane, which is an open source, basically Cloud control plane that is standardized on the Kubernetes API. I personally work on what we call the core Crossplane team within our company, which means I exclusively do open source stuff. So, that means I'm spending most of my day working on Crossplane things. I also do some upstream Kubernetes stuff. Daniel mentioned this Cloud native community, and the Kubernetes community is a subset within that. So I've had the privilege of being a part of that specifically with the release team and some other SIGs within the community. So that's something that I definitely spend a bit of my day every day, and then also some of my hours on the weekends and nights doing as well.

Daniel Mangum (05:47):

As far as a recent career highlight, I definitely say, and this is very fresh, it'll be a few weeks old by the time this podcast comes out, but we recently announced that we're donating Crossplane to the CNCF. And that is a really awesome thing from my perspective, being involved with other CNCF projects and kind of looking up to some of those communities in my own career and journey in software.

Daniel Mangum (06:09):

And as part of that, we revised some of our governance where I was able to officially become a maintainer of the project, which is just a name change honestly, but it was a big thing for me that I felt really honored mostly because the other folks who work on the project who are maintainers right now, which would be some of the founders of Upbound and Crossplane are people I really look up to. So, considering myself a maintainer alongside them is a tremendous privilege and honor I'm in my book. So, that's definitely my most recent career highlight I'd say.

Daniel Bryant (06:41):

Super stuff. I actually saw when I was looking through Bassam's presentation from a couple of weeks ago about the donation to the CNCF. Funny enough, we've also donated Ambassador, our opensource API gateway to CNCF recently, or in the process of doing that. So when I saw the news at Crossplane, I was super excited to read that. We'll definitely dive more into that later Daniel, I'm totally keen to cover Crossplane and go into more depth on that. Before though, I've got to ask the traditional Ambassador Living on The Edge Podcast question. Everyone that comes on the podcast has got to share their worst devloop. And when I talk about that, I mean that ability to come up with an idea, a code, test, deploy, release, and verify without naming names, unless you're allowed to. Could you share your devloop.

Daniel Mangum (07:23):

So, I've listened to, I think every episode that you've put out on the Life on the Edge Podcast. So, I am familiar with this question, and I did come prepared. I will not be mentioning the name of anyone I worked with or the company, but my first real paid developer job that wasn't just a one off contract thing, I was writing visual basic scripts for Excel spreadsheets, which was an incredibly painful experience. And I'll frame the setting in which I was doing this. So, this was the summer after my freshman year of college. And I worked out in I'm originally from Nashville, Tennessee in the US, and I worked about an hour outside the city at this big basically data center for a really large US company. And I worked in a cubicle where I didn't speak to anyone all day.

Daniel Mangum (08:13):

And I basically just sat at this desktop machine and wrote these scripts, but my supervisor was in a different state. So my devloop was basically write these scripts. And if you're familiar with visual basic, there's not really a testing loop for that, except for you just provide your inputs. Or maybe there is a really great visual basic testing framework that I am not familiar with, which could totally be the case. But what I was doing was writing these scripts, then running massive amounts of accounting data through these Excel spreadsheets and seeing if it basically checked the stuff that had been done manually before, which obviously could be wrong as well.

Daniel Mangum (08:49):

And then I would zip up the file and email it to my manager who would then look through it and review it, maybe give some feedback and then send it back. Maybe actually adding code comments and then sending back, there's obviously no version control here whatsoever. It's all XLSX, or whatever the extension is, underscore this is the real one or something that. So yeah, so that was our git back then. So, that was definitely my worst devloop, and I think it definitely made me very tolerant of some other bad developer experiences because you really, you can't do a lot worse than that, unless if maybe you're doing punch cards still.

Daniel Bryant (09:33):

Well said, well said. Oh, that's definitely a great experience to share. I can feel your pain on that one. I can relate to your pain, definitely. So you mentioned Upbound and Crossplane and I'm super interested. I've been following what's going on with Crossplane for a year or so now I guess. I think when I bumped into it in my InfoQ work. Could you give the kind of pitch maybe a little bit deeper to folks what Crossplane is please?

Daniel Mangum (09:55):

Yeah, for sure. So at the most basic level, Crossplane allows you to provision infrastructure from the Kubernetes API. So there's a lot of companies, Kubernetes has kind of become the standard across organizations where you're deploying your workloads. So, Kubernetes in its original form is a container orchestration engine. So, you have basically a way of packaging software and then running it on a node somewhere. So, that's the basic part. And then the big revolution in my mind of the extension story of Kubernetes was the introduction of third party resources, which eventually became custom resource definitions. So, basically when you create a custom resource definition in Kubernetes, you're adding what is essentially an API end point to the Kubernetes API.

Daniel Mangum (10:41):

And then you can also introduce controllers, or operators, you'll hear it called different things, that basically monitor these API end points or these API objects, and take action based on things that the user requests to the API. So what we do is just like Kubernetes has lower level primitives like a pod, and then higher level primitives like a deployment. We kind of model that same thing for infrastructure resources for cloud providers. It can actually be any infrastructure resource. You could write them for your on-prem systems as well, your custom organizational systems. And so what we do is we model those infrastructure resources as Kubernetes objects, which then allows you to, with those same commands where you're orchestrating your containers, allows you to also provision infrastructure around that, because you introduced that into the Kubernetes API.

Daniel Mangum (11:32):

We also have the opportunity to do things like inject secret information to connect to a database or something that that's provisioned on AWS, or GCP, or Azure, et cetera. We have the ability to control that workflow of getting that secret information to your applications. And then the higher level value proposition here is we have something which is relatively new, which we've iterated on for a while called composition. Where it basically allows you to combine these small building blocks, like maybe a database and a VPC, some sub-net security groups, et cetera, and provision them as a single abstract unit, if you will. So within your organization, you can define these higher level primitives that may be developers interact with, and then the platform team defines it.

Daniel Mangum (12:18):

So you can think of it, and this is kind of a tagline you'll hear often used with the project as kind of like a build your own Heroku, right? So Heroku is a really easy to use cloud provider, I guess you could say, that allows you to get software up and running. And it has some integrations, like some databases and that sort of thing. But essentially what we're allowing you to do is say, I'm not constrained by these abstractions that Heroku makes, I'm going to be able to define my own abstractions. And then when I need to tweak those, I can go in as a platform team, change that around, make some new abstractions and then provide those to my developers. So that's kind of the full story of the most granular things we do up to the most I guess abstract or higher level things that Crossplane does.

Daniel Bryant (13:01):

No, that's a superb introduction. And that's definitely why I was keen to chat to you today because we're all at building platforms and it's quite a specialist thing. I've got a bunch of mentoring folks as you know on the podcast, but no one's really talked that building the platform. I think that's where your expertise is really interesting. I'm guessing a lot of folks in this podcast, like I started out with are coming from say Chef, Puppet and the likes, and then you go to Terraform and things like that. How would I guess the workflow, maybe the mental model even, how would it differ? If I'm coming from Puppet and Chef to Crossplane, what's the jump I have to make?

Daniel Mangum (13:34):

Yeah, yeah. We definitely hear that quite a bit. And especially Terraform, which I think is maybe a level in-between maybe Chef and Puppet before you'd get something like Crossplane. So, I guess the big advantage of using something like Crossplane, that's a different developer loop is with Chef, or Puppet, or Terraform, that's kind of sequestered from what you're doing in Kubernetes, right? So you have your Kubernetes manifests, and you have to figure out a way to wire up the different things that you're provisioning with Chef, or Puppet, or Terraform to those Kubernetes objects. And generally either something you provision on Kubernetes is either connecting to something you're provisioning with an infrastructure as code tool, which I'll use broadly to describe those different products there. There it's either running on those or it's connecting to them, right?

Daniel Mangum (14:17):

So what we allow you to do is both in Crossplane provision things, and then schedule workloads to them, as well as consume them. And you can do that all on a single bundle, right? So let's imagine you have a git repository or something like that that has a Kubernetes deployment manifest. It has your website running that has a database backing it. And then you also have an RDS instance YAML, and it's one command. And it could be way more resources than that as well, of course, but it's one command to control, create, apply, et cetera, all of those resources. And then Crossplane kind of takes care of wiring all that up for you. And then you have all your infrastructure and your application together.

Daniel Mangum (14:54):

So, that's kind of bringing it to a senior workflow. And also the part that I haven't touched on there, which is pretty distinctly different from those projects is that Kubernetes, its basic kind of way it does anything is in a control loop. So, you're watching things continuously, right? So if you're familiar with Terraform, you usually do a TF plan and then a TF apply, right? And then it kind of does this static operation of going and creating these resources and I'll make sure they come up successfully and inform you of that. But then you're done until you want to make some change to it. Crossplane on the other hand, is going to go and provision those resources, make sure they get to a steady state. I mean that's going to continue to watch those, and it's going to continue to see if you make any changes or if changes happen on the other end.

Daniel Mangum (15:38):

So, one of the demos we like to show off, if we're just showing the basic control loop functionality there is to let's say, create an RDS instance with Crossplane running in a Kubernetes cluster and then go into the AWS console and change the size of the database. So maybe we provision it with 20 gigs and we bump it to 30 gigs, and then we'll see Crossplane and automatically say, "Hey, that's not what you said you wanted here in this source of truth. So I'm going to go ahead and go correct that for you." And so obviously with changing a database size, that's kind of a trivial example, but with things like a resource just getting into a bad state, you could see your infrastructure automatically healing itself

Daniel Bryant (16:16):

That's very cool. As you were talking now, I was definitely thinking of GitOps. Yeah, so we're big fans of GitOps, Ambassador, and Datawire, and we know the work team quite well. I learned a lot from Alexis over the years, the CEO there, I'm guessing this is pretty much, you mentioned single source of truth, declarative, this is totally built for GitOps style workflows, yeah?

Daniel Mangum (16:34):

Absolutely. And you mentioned Alexis there and stuff like that, Argo, and Flux, and some of those projects, which are now coming together to kind of be this Kubernetes GitOps engine there, those are things that we are super interested in because you get that GitOps CICD workflow now with your infrastructure, right? So those different projects allow you to get GitOps workflows with your deployments and containers and that sort of thing. But now you can also have your infrastructure in that as well. So, definitely one of the biggest demos, and the one that we had running on our screen at the last cube con was one with Argo where we'd show making a commit to git repository and having Argo, apply that in your Kubernetes cluster, which then Crossplane sees and goes and spins up a database, or spins up a Kubernetes cluster or something like that. And so there's some different complex workflows we have there, but definitely very interested in GitOps and think that is kind of the way forward here.

Daniel Bryant (17:27):

Very nice. I'm going to come back to some of the dev stuff and how this all relates to workflows in a minute. Something I bumped into when I was having a look at the Crossplane website is the Upbound registry, sort of like a free community library for all things Crossplane was the pitch. And it reminded me a bit of the Rancher folks who've got sort of a community library, kind of like Helm, all that kind of stuff. Basically it's a catalog for things, yeah. The Upbound registry is pretty much that in the Crossplane world. Is that how I understand it?

Daniel Mangum (17:52):

Yeah, yeah. I think that's the way to look at it. I think that we are still figuring out exactly what that's going to look like because we don't want it to be restricted to any type of Kubernetes workload. So we have different ways of packaging things within Crossplane that make it really easy to do some of that stuff I was talking about, about provisioning infrastructure and workloads, and then wiring them up together. However, there's a lot of existing projects and apps and things like that that are already packaged using helm, or customized, or something like that. And we definitely don't want to be restrictive to that. We want you to be able to basically be able to install anything in a Kubernetes cluster from a registry. And there's some other projects that are working on this as well. There's Operator Hub, and I think the CNCF just started.

Daniel Mangum (18:35):

I can't remember, I think they changed the name after they announced it, but there's essentially kind of like a hub for helm charts and that sort of thing I believe you can download from. And so we're definitely looking at those communities as well, and just trying to find something that allows you to basically get this infrastructure alongside with things you're provisioning. So exactly how that packaging looks, I think there's still some sort of convergence to happen in the community about how we want to package things, or if we want this disparate packaging thing. And if so, is there some sort of interface we can standardize on to make that easy to install on a uniform way

Daniel Bryant (19:09):

As you were talking about earlier on the packaging of things, it reminded me of... so bump into Pulumi quite a bit, and they have this notion of modules. But even taking it a bit further is what I hear you talking about. Could you almost imagine say having a WordPress package in Crossplane that say literally deployed my LAMP stack, my database, and the app? Would that be a thing in Crossplane?

Daniel Mangum (19:29):

Yeah, absolutely. And it's funny that you bring up WordPress, that's kind of like our core demo right now. I promise I did not pay Daniel to bring this up before. But in Upbound, if you go and watch some of the demo videos or Upbound Cloud is now available, so you can go create an account and do it yourself. But WordPress is kind of the thing where in the WordPress scenario we have, we will create a VPC and the subnets, and then we'll create the Kubernetes cluster, and the database and have them be able to securely connect to each other. So, no public IPs or anything that. So we're setting up the subnets and security groups behind the scenes there as well. And then, so we have WordPress running the database they're connected up, and then it basically just gives you back a URL and says, here's your full WordPress stack running, configure it as it needs to be.

Daniel Mangum (20:17):

And we show that also across four different clouds. So we have within the Crossplane or anyone can write a provider for a Crossplane, but within Crossplane right now we support GCP, AWS, Azure and Alibaba cloud. So we show that running across different ones. We also have a few other apps that you can try out as well there, but yes, that's definitely the vision of one click infrastructure app, everything, your stack is ready to go. And the thing that I like to think about, which is obviously down the line with more adoption and that sort of thing, is basically companies being able to public open source their platforms.

Daniel Mangum (20:52):

So you can think about if you're a startup saying maybe I'm running an eCommerce site, and let's say Nike or something has this platform where they've defined their abstractions and apps and those sorts of things. And then they're saying we're going to open source this platform, and I can basically as a new startup go grab pieces of that platform and kind of take building blocks and build my own Heroku, if you will, or maybe something a little more complex than that from that. And it's also portable potentially across regions, geographies, clouds, et cetera.

Daniel Bryant (21:23):

That's a super interesting abstraction, isn't it? Yeah. We'll perhaps come back to that a little bit more. Something you mentioned that caught my ear there was security, which are the classic IAM rules, and security groups, and ACLs and so forth, and Amazon. I've spent many an hour figuring that stuff out sometimes, but it's super important. I think I did bump into somewhere, the mention of open policy agent. Because I had a chat with Michael Hausenblas a couple of weeks ago on the podcast. Michael was really interested in that OPA, he has got some fantastic demos he was talking about. Is OPA maybe a way to do some fine grain control within things like Crossplane, kind of guardrails I guess?

Daniel Mangum (21:59):

Definitely. And there's a couple of different integration points there. So first of all, OPA is a awesome project. It's super cool, it can be used in a lot of different ways. It truly is a general purpose policy engine therefore that you can use across a lot of different scenarios. And actually, so we had our community day last week where we had a bunch of different speakers, which I was emceeing and definitely felt a bit out of place with that caliber of speakers we had there. So definitely we're going to be kind of chopping up that video and publishing the separate sessions, and you can see those on YouTube in that. And then also on TBS, which is our biweekly live stream show, we had OPA present both those. So on TBS Torin Sandall, who's one of the cofounders of the project. And then at the community day, Tim Hendricks, who is the other co founder of the project. And they both worked for Styra, which is kind of the core company behind that.

Daniel Mangum (22:51):

They both showed demos of using Crossplane with OPA. So in those demos, OPA integrates really well with the Kubernetes API, as well as other things. And there's lots of other projects as well. So we've already mentioned GitOps and that sort of thing. One of the advantages of standardizing on the Kubernetes API is that we can basically talk the same language as all these different projects, right? So OPA is frequently used in Kubernetes to restrict maybe security contexts on a deployment or something like that. But it also allows you to write policies for any custom resources as well. So, once you begin to define all of your infrastructure as Kubernetes custom resources, then you have a general purpose policy agent that goes across them.

Daniel Mangum (23:34):

So in the demo that was shown at community day, for instance, once again going back to this database size example, because it's just an easy one to go to. Maybe you don't want your developers to be able to spin up a database larger than 20 gigs or something like that. So you can write a policy specifically with gatekeepers, what we're using in this example, which is OPA's Kubernetes controller, but you can write a policy there that basically says any time an RDS instance is created, check and make sure that it's not over 20 gigs. And if it is over 20 gigs, that it uses an admission web hook to basically say you're not allowed to create that. And since that object doesn't actually get created, then then Crossplane isn't going to go provision that.

Daniel Mangum (24:15):

So by being part of this Kubernetes API ecosystem, we get a lot of those things out of the box. So, definitely policy is a big part of provisioning infrastructure, particularly because cloud spend is such a massive part of any company's bill. So, this is definitely something we are very interested in as well as the OPA folks. So, it's cool to see the integration there for sure.

Daniel Bryant (24:36):

Super interesting. Something you said there Daniel a couple times, it reminds me I chatted to the Buoyant folks a while back, Oliver from Buoyant. And Linkerd is obviously their service mesh, he was all in on Kubernetes. He said, "We're betting the Kubernetes is the abstraction of the cloud." And that's why, at the time I was chatting to Oliver, which was a while back, yet they were not looking at supporting other abstractions with the service mesh. I'm guessing what I'm hearing, and I know it's personal opinion mixed in with some company policy here as well, but it sounds like you are all in on Kubernetes as the abstraction of the sort of future data plane if you like of the cloud?

Daniel Mangum (25:09):

Yeah, for sure. I definitely think we are obviously, Crossplane pitches itself as essentially an add on to Kubernetes to allow you to do infrastructure provisioning. Some of the finer points around that, as you said, can be a little complicated. I will say, just to kind of give a behind the scenes look here. So, Kubernetes has different components that basically make it be able to do all the things it does. So, some of the primary components are you generally have this control plane and then you have the nodes which run your containers for you. So in Crossplane, besides the controllers that are actually running, you actually don't need any nodes if you're not provisioning workloads on that cluster.

Daniel Mangum (25:46):

So we generally talk about this. If you're familiar with cluster API, there's kind of a similar concept. So, what cluster API does is it uses a Kubernetes cluster to spin up other Kubernetes clusters. And they'll generally call that first one a bootstrap cluster. And so you can use kind or something like that, which allows you to basically spin up your new Kubernetes cluster and then kill the bootstrap cluster. You don't really care about it. But in Crossplane, we might call that a control cluster, and it's going to be a more of a long-lived thing. So, Crossplane not only allows you to provision external infrastructure, it can also allow you to provision other Kubernetes clusters and then schedule workloads to those remote clusters.

Daniel Mangum (26:23):

So, you can think in the control cluster, which this is probably a little bit hard to explain just over words, so maybe I'll follow up in a tweet or something with a diagram of this. But in the control cluster, imagine that is your bootstrap cluster, but it's going to live in perpetuity essentially. So in that cluster, you're not doing anything but creating other infrastructure and scheduling workloads to it. So, you don't actually need the ability to run container workloads in that control cluster. So, that actually allows you to cut down a lot of the components in Kubernetes. So, you really only need etcd, which is the key value store, which basically holds all the data for Kubernetes, and these different objects that you're creating are stored there, as well as some other metadata.

Daniel Mangum (27:03):

And then the API server, which basically serves up those end points we were talking about and interacts with etcd on your behalf. So really all you need to actually get Crossplane working is the API server and etcd. So you can imagine that you actually don't need a "full Kubernetes cluster" to be able to run Crossplane. You could just have a control plane with those components, and then be able to schedule workloads to essentially other remote full Kubernetes clusters.

Daniel Mangum (27:32):

So, we allow you in Crossplane to run what's called hosted mode, where you can run multiple Crossplane clusters within a single Kubernetes cluster. So there's some documentation on that as well, but essentially what we're doing there is spinning up an API server and an etcd instance, or connecting you back to the main etcd instance with a different namespace that you're writing to, and just having Crossplane working there. So basically you only need a place to register these custom resources, and then Crossplane can kind of watch and interact with those.

Daniel Bryant (28:03):

That has been a super interesting dive into behind the scenes. I'd not thought of it in that way. Yeah, that's really using Kubernetes and abstraction of the cloud. Because I chatted to Katie Gamanji a while back and she's doing a lot of work in the cluster API space. I learned a lot from Katie then, and I'm thinking the combination of what you've just described there, cluster API, and then even that multi-cluster, that multi-cluster seems like a big thing in the moment in service mesh space. You can almost imagine what you just described there with Crossplane as it being the overarching control plane for multiple clusters. And then maybe you have API gateway or service mesh during the traffic management, but Crossplane is more like the infrastructure and the glue at the infrastructure level I guess.

Daniel Mangum (28:45):

Right. That's a terrific way to look at it, and not to keep plugging our live stream, but it seems like every person you bring up has either been on our live stream, or their project or company has been. So we've had Linkerd on there, and we've also had the cluster API. So we're definitely not in direct competition with cluster API because what cluster API does is it basically spins up self managed Kubernetes. So actually provisions easy two instances or something like that, and then uses Kubeadm to bootstrap a Kubernetes cluster there. Crossplane actually allows you to bring a cluster API provision cluster and schedule workloads to that, or what the kind of clusters that we would allow you to create would be like GKE, or EKS, or some hosted offering like that, a managed service which you can also schedule too. Or you can have your Kubernetes running on your Raspberry Pi, and as long as you have a valid cube config, you can schedule workloads as well there.

Daniel Mangum (29:39):

And what you mentioned about Linkerd, Linkerd has recently been working on this. I think they're calling it service mirroring, which basically allows you to do this multi-cluster data plane. We do not work on the data plane whatsoever, except for providing you basically credentials to talk to something. So exactly what you said about Kubernetes being a control plane. We had one of the engineers from Linkerd on the live stream, and this was when they were first starting on the service mirroring stuff. And we were talking about how Kubernetes could serve as that control plane. And if you have this service mirroring set up with Linkerd in these remote clusters, then your workloads can talk to each other. And then you can get into a whole nother realm of basically doing dynamic scheduling to Kubernetes clusters based on certain attributes.

Daniel Mangum (30:25):

So maybe geography, maybe fail over, latency, cost, all those sorts of things. And if you have this generic way to communicate between these clusters, then that makes that a lot more powerful. As you mentioned, Linkerd and other folks, there's some upstream Kubernetes work happening with multi-cluster stuff. And we're basically saying on that data plane side, you all own that. we think you have a much, much better insight there, and we don't want to get outside our bounds there. So we're going to allow you to basically put the configuration in that cluster for what you want to happen there. But in terms of connecting them up, we'll defer to other projects and allow people to bring their service meshes or their upstream Kubernetes to handle that problem.

Daniel Bryant (31:05):

I like this a lot. As I've moved from my career, sort of having that notion of single responsibility principle, I think is a really, really core separation of concerns, like do one thing and do it well classical Unix philosophy. But then also having the integration points is really important. So the picture for me Daniel was there's clear value in each piece of the puzzle here, but we've got the integration points set up well.

Daniel Mangum (31:25):

Yep, absolutely

Daniel Bryant (31:26):

Nice. Very nice. Now it's all coming together. I'm super privileged I get to chat with interesting people on the podcast, and I'm gradually joining the dots on where the future perhaps is going. Yeah, we're all making the future as we do this now, but I think this is a super interesting vision. So that was yeah, really enjoyed that. That was a great insight. Before we sort of wrap up, I did want to briefly dive into your project. I see you're working doc.crds.dev. Could you briefly give the pitch for what you've done there? It looks super interesting on Twitter, I was kind of curious.

Daniel Mangum (31:54):

Yeah. Well, I appreciate you calling that out. That is a project that I've kind of taken on as a side project to some of the Crossplane and Kubernetes stuff I do. So, I'll give you the original motivation here. We have lots of different CRDs, custom resource definitions, in Crossplane. But you can imagine if every cloud managed service on AWS, GCP, Azure, et cetera, has its own CRD, then you're going to have quite a few. And then we also have some that allow us to do some other things, like wiring those together. So, you're talking hundreds potentially getting to thousands of CRDs in our project, which we are a little bit unique in that regard, but there are a lot of projects that bring along their own CRDs and that's kind of becoming the way that you talk in Kubernetes now.

Daniel Mangum (32:39):

And so before we were starting, you mentioned Javadoc and Godoc, and that sort of thing. So I have definitely looked at those projects with admiration, mostly because they are extremely simple, but they provide a ton of value, and people use them all day every day. So what we were doing in Crossplane is we basically ... Nick, who is Nick Cope, who is the lead of the core Crossplane team, who fantastic engineer, had this tool that was basically allowing us whenever we made a change to any of the different providers, which would be, AWS, GCP, et cetera, added new resources, changed them. We would run this tool, and we had a centralized repository for the documentation. I mean, it would generate new docs for the CRDs, basically tell you the fields, tell you what was required, et cetera

Daniel Mangum (33:25):

And that was great, but when you get more and more providers and more and more CRDs, and you have this centralized documentation, that was getting really hard to maintain. And we'd often cut a release and not have run the doc generation, as soon as we needed to, or something snuck in at the last minute. It was also someone had to download a binary. So if a new contributor comes in and says, "I want to modify this resource or add a new one," they may not to be familiar with that they need to go run this tool. So basically, it was a very manual process. It was error prone, it was arduous. So, I was trying to find a better way to do this. And basically what I did was exactly copied those projects.

Daniel Mangum (34:04):

So, there's not a lot of innovation happening here, but what doc.crds.dev does is it basically goes and looks at repos on GitHub, and it parses them for YAML files that are custom resource definitions, and does some validation there, and then serves them up in a web app. Right now it does that basically by me feeding it a few different repositories. So, obviously the Crossplane ones, it is now the standard place for all of our documentation, but I also have some, other repos that are popular in the community there like jet stack cert-manager, cluster API, OPA some of those. So those are manually provided right now, but if you're familiar with those documentation engines, they basically index all of GitHub and all of get repositories and that sort of thing.

Daniel Mangum (34:51):

So, that's definitely the mission there. And I'm slowly moving towards that by adding more and more projects and seeing what breaks, manipulate to make it more general purpose. And the idea there is that we will do ... there's not really a standard for having one CRD that is the source of truth in your repo. So we're adding the ability to have configuration that basically says, this is the source of truth for this kind of CRD, and I want you to serve that on the website. And so we can allow people to get some better performance there. So that the end goal is it serves up any repository anywhere, shows you the CRD in a nice format, shows you what are the required fields. Also, maybe generates example valid CRDs for you.

Daniel Mangum (35:34):

So, these are all things that are in the works. It's been really cool to get to do that. The infrastructure behind that is sponsored by Upbound, so I really appreciate them sponsoring me there. And also fun fact, it runs on Crossplane. So, the reddest behind the scenes is running on GCP, which is provision by a crosswind cluster where also the web app is running, as well as the gate repository indexing. So it's kind of a dog food opportunity and scratch your own itch. So, if folks are interested in that, I've definitely seen some interest from some projects that want their CRDs index then definitely open an issue and I'll get support there. And then eventually we'll have kind of this general purpose place to go look for CRDs, and know what is required to create instances of them.

Daniel Bryant (36:21):

All right, great Daniel. Documentation, like throughout my dev career, and even my ops career are super, super important, right? Build that mental model, figuring out what to do, how to fix tasks, awesome stuff. I've definitely got the link in the show notes, no worries on that one. As a wrapping up question, what are you excited about over the next few years in the dev space?

Daniel Mangum (36:40):

Yeah, that's a great question. I would say, and this is not specifically related toCrossplane, but I think it fits in. I am excited about moving away from local machines. So this is kind of already happening through projects that have been around for a while, like Repl.it, or something like that if you're familiar, where you basically have your editor in a browser. I know GitHub just started code sandboxes. So that's kind of a new direction, but I think we're going to move more and more towards that. I know there's actually a startup in San Francisco somewhere, of course, that basically streams Chrome to your laptop as opposed to you actually running Chrome locally. And the idea is that as network bandwidth becomes more and more robust, that it's going to actually be more efficient for you to stream, you're basically doing a very advanced screen share to stream applications to you, as opposed to actually running them on your local machine.

Daniel Mangum (37:36):

Obviously this isn't going to work for every scenario, but as someone who gets frustrated when my computer restarts itself or something that, even though I have a pretty nice machine here, I'm really excited to make that developer workflow more efficient. And I think when we have that ability, then you get all sorts of more robust opportunity for testing and having quick development loops. So for instance, with testing Kubernetes things, you kind of have to have at least some components of a Kubernetes cluster running. But you can imagine if you had some sort of code sandbox, then whoever was offering that, whatever company or organization, they could basically provide the Kubernetes API there if they wanted. And you could connect directly to it, and it would sort of be basically like compiling your code. You can kind of do whatever you want there. So, I think that as this continues to develop that we may start to see some interfaces and ways to write generic plugins, to provide different development things in these kinds of hosted IDEs.

Daniel Mangum (38:33):

So I'm excited for that, for sure. And I selfishly think that some of the things that we can provide there are maybe it's some managed services that Crossplane could provision for you, and allow you connect to in your dev loop. I will say one company that's working on this is Octedo, which I don't know if you had them on the show, but Ramiro is their CEO. Yeah, he's their CEO and just super nice person, just one of the kindest people I've ever met. So, definitely would advise having him on. And they're working on the Kubernetes devloop there, and we've done some demos with them about showing your devloop of provisioning an application, but also provisioning your dev database and then swapping it out for production database, and using Crossplane with that as well. So, that's definitely an area that I'm excited to see how that continues to mature, and potentially how the work I do can fit into that as well.

Daniel Bryant (39:21):

Super, great answer, and on brand as well, what more can you ask for? You got the Crossplane mention in there. Octeto, funnily enough, they've cropped several times. They've cropped up on the Datawire radar, we talk about them in our Slack, in all good terms. And just this morning I was checking out their website again because I was like this is super interesting tech. So, cheers to that recommendation. I should definitely reach out to the Okteto folks. It's been a genuine pleasure. So many amazing things we've covered here today, I'll be sure to write up show notes, and then link to various things we've talked about. But yeah, really appreciate you coming on the podcast. Thanks again.

Daniel Mangum (39:48):

Yeah, absolutely. Well, like I said at the beginning, it is an honor and privilege to be here. And I really appreciate the work you do, and will continue to promote it. And excited to see some of the guests that you kind of gave me a behind-the-scenes look that are going to be coming up. There's some really, really great stuff that's already going to be published, and people have listened to by the time they hear this episode. So, excited to hear that.