What if learning the Kubernetes API is all you need to provision any infrastructure? And we’re not only talking about AWS, Azure & Google – but also IONOS, DigitalOcean and even vSphere. Let’s have a look at Crossplane and how we can create an S3 Bucket on AWS step by step.
No code required, it’s all declarative!
Crossplane claims to be the “The cloud native control plane framework”. It introduces a new way for managing any cloud resource (be it Kubernetes-native or not). It’s an alternative Infrastructure-as-Code tooling to Terraform , AWS CDK /Bicep or Pulumi and introduces a different level of abstraction, based on Kubernetes CRDs . One could name it the cloud native way to do GitOps.
Why Crossplane is different
Crossplane can be also compared to tools which enable the management of cloud resources through the Kubernetes API, like AWS Controllers for Kubernetes (ACK) , Azure Service Operator for Kubernetes (ASO) or Google Config Connector . Crossplane providers are even generated from ACK and ASO . But the CNCF incubating project Crossplane adds on top of these:
The Crossplane community believes that the typical developer using Kubernetes to deploy their application shouldn’t have to deal with low level infrastructure APIs.
Crossplane uses the Kubernetes API and extends it with a set of Custom Resource Definitions (CRDs) to abstract from actual cloud provider APIs. Additionally, these CRDs are a great foundation to build an Internal Developer Platform (IDP) . Crossplane promotes self-service by introducing building blocks like Composite Resource Claims (XRCs) that are a great API for the typical application developer. On the other hand, Composite Resources (XR) and Composites are a magnitude more powerful and ideal for platform operators.
Drawing on our experiences as platform builders, SREs, and application developers, we’ve designed Crossplane as a toolkit to build your own custom resources on top of any API – often those of the cloud providers. We think this approach is critical to enable usable self-service infrastructure in Kubernetes.
Crossplane basic concepts
The crossplane concept of a Composite Resource is composed out of several building blocks :
- Composite Resources (XR) : they compose Managed Resources into higher level infrastructure units (especially interesting for platform teams) and an optional CompositeResourceClaim (XRC) (which is also referred to as ‘Claim’ and can be seen as an abstraction of the XR for the application team to consume)
CompositeResourceDefinition(XRD) which defines an OpenAPI schema the Composition needs to be conform to (think of Kubernetes CRDs)
Compositionthat describes the actual infrastructure primitives aka
Managed Resourcesused to build the Composite Resource. One XRD could have multiple Compositions – e.g. one for every environment like development, stating and production
Composite Resources can also be nested together, which allows for higher level abstractions and better separation of concerns. Think of an AWS EKS cluster where you need lot’s of network/subnetting setup (which can form one XR) and the actual EKS cluster creation with NodeGroups etc (which would be the XR using the networking XR). More in-depth details on nested XRs can be found here .
As an optional step, you can package these Composite Resource artifacts using a Configuration into an OCI container image. Now your custom Configuration package can be installed easily in any other Crossplane clusters.
Composite Resources are composed of infrastructure building blocks called Managed Resources (MR) that are bundled by a Provider:
- Managed Resourced (MR) : a Kubernetes custom resources (CRDs) that represent infrastructure primitives (mostly in cloud providers). All Crossplane Managed Resources could be found via https://doc.crds.dev
- Providers : are Packages that bundle a set of Managed Resources and a Controller to provision infrastructure resources – all providers can be found on GitHub, e.g. provider-aws or on docs.crds.dev . A list of all available Providers can also be found on GitHub.
You may also stumble upon Packages . They were formerly named
Stacks and are simply OCI container images. Packages handle the distribution, version updates, dependency management and permissions for
Getting started with Crossplane: fire up a k8s cluster with kind
In order to use Crossplane, we’ll need any kind of Kubernetes cluster to let it operate in. This management cluster with Crossplane installed will then provision the defined infrastructure. Using any managed or local Kubernetes cluster like EKS, AKS, Minikube or k3d is suitable. In my example project (which is available on GitHub ), I used kind to host the management cluster.
Using kind is pretty easy. Let’s get our hands dirty! Just be sure to have kind, the package manager Helm and kubectl installed. On a Mac you can use
brew like this (have a look at the docs for other systems ):
1brew install kind helm kubectl 2
We should also install the crossplane CLI:
1curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh 2sudo mv kubectl-crossplane /usr/local/bin 3
kubectl crossplane --help command should be ready to use.
Now spin up a local kind cluster with:
1kind create cluster --image kindest/node:v1.23.0 --wait 5m 2
Install Crossplane with Helm
The Crossplane docs tell us to use Helm for installation :
1kubectl create namespace crossplane-system 2 3helm repo add crossplane-stable https://charts.crossplane.io/stable 4helm repo update 5 6helm upgrade --install crossplane --namespace crossplane-system crossplane-stable/crossplane 7
As a Renovate -powered alternative we can create our own simple
Chart.yaml to enable automatic updates of our installation if new crossplane versions get released:
To install Crossplane using our own Chart.yaml simply run:
1helm dependency update crossplane-config/install 2helm upgrade --install crossplane --namespace crossplane-system crossplane-config/install 3
Be sure to exclude
/charts directories and
Chart.lock files via
.gitignore. Now Renovate will have an eye on our Crossplane versions:
Before we can actually apply a Provider, we have to make sure that Crossplane is actually healthy and running. Therefore we can use the
kubectl wait command like this:
1kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s 2
Otherwise we will run into errors like this when applying a
1error: resource mapping not found for name: "provider-aws" namespace: "" from "provider-aws.yaml": no matches for kind "Provider" in version "pkg.crossplane.io/v1" 2ensure CRDs are installed first 3
Finally check the Crossplane status with
kubectl get all -n crossplane-system:
1$ kubectl get all -n crossplane-system 2NAME READY STATUS RESTARTS AGE 3pod/crossplane-7c88c45998-d26wl 1/1 Running 0 69s 4pod/crossplane-rbac-manager-8466dfb7fc-db9rb 1/1 Running 0 69s 5 6NAME READY UP-TO-DATE AVAILABLE AGE 7deployment.apps/crossplane 1/1 1 1 69s 8deployment.apps/crossplane-rbac-manager 1/1 1 1 69s 9 10NAME DESIRED CURRENT READY AGE 11replicaset.apps/crossplane-7c88c45998 1 1 1 69s 12replicaset.apps/crossplane-rbac-manager-8466dfb7fc 1 1 1 69s 13
Configure Crossplane to access AWS: create aws-creds.conf
1echo "[default] 2aws_access_key_id = $(aws configure get aws_access_key_id) 3aws_secret_access_key = $(aws configure get aws_secret_access_key) 4" > aws-creds.conf 5
Don’t ever check this file into source control – it holds your AWS credentials! In the example project I added
*-creds.confto the .gitignore file.
If you’re using a CI system like GitHub Actions (as this repository is also based on), you need to have both
AWS_SECRET_ACCESS_KEY configured as repository secrets:
Also make sure to have your
default region configured locally, or as a
env: variable in your CI system. All three needed variables in GitHub Actions for example look like this:
Create AWS Provider secret
Now we need to use the
aws-creds.conf file to create the Crossplane AWS Provider secret:
1kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf 2
If everything went well there should be a new
aws-creds secret ready:
Install the Crossplane AWS Provider
To be able to provision infrastructure on a cloud provider like AWS, we need to install the corresponding Crossplane Provider first. Remember: the Provider packages all needed Managed Resources (MRs) and their respective controllers. As a Provider is a Crossplane Package, we can install it using the Crossplane CLI like this:
1kubectl crossplane install provider crossplane/provider-aws:v0.22.0 2
Alternatively we can create our own provider-aws.yaml file like this:
apiVersion: pkg.crossplane.io/v1is completely different from the
kind: Providerwhich we want to consume. The latter uses
Don’t forget to install the AWS provider using
1kubectl apply -f crossplane-config/provider-aws.yaml 2
package version in combination with the
packagePullPolicy configuration here is crucial, since we can configure an update strategy for the Provider here. A full table of all possible fields can be found in the docs . We can also let Crossplane manage the upgrade to new versions for us. If you installed multiple package versions, you’ll see them prefixed with
providerrevision.pkg.x when running
kubectl get providerrevision:
1$ kubectl get providerrevision 2NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE 3providerrevision.pkg.crossplane.io/provider-aws-2189bc61e0bd True 1 crossplane/provider-aws:v0.22.0 Inactive 6d22h 4providerrevision.pkg.crossplane.io/provider-aws-d87796863f95 True 2 crossplane/provider-aws:v0.28.1 Active 43h 5
Now our first Crossplane Provider has been installed. You may check it with
kubectl get provider:
1$ kubectl get provider 2NAME INSTALLED HEALTHY PACKAGE AGE 3provider-aws True Unknown crossplane/provider-aws:v0.22.0 13s 4
Before we can actually apply a
ProviderConfig to our AWS provider we have to make sure that it’s actually healthy and running. Therefore we can use the
kubectl wait command again like this:
1kubectl wait --for=condition=healthy --timeout=120s provider/provider-aws 2
Otherwise we may run into errors when applying the
ProviderConfig right after the Provider.
Create ProviderConfig to consume the Secret containing AWS credentials
Now we need to create a
ProviderConfig object that will tell the AWS Provider where to find it’s AWS credentials . Therefore we create a provider-config-aws.yaml :
Crossplane resources use the
defaultif no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.
secretRef.key have to match the fields of the already created Secret. Apply the ProviderConfig with:
1kubectl apply -f crossplane-config/provider-config-aws.yaml 2
Provision a S3 Bucket with Crossplane
The Crossplane core controller and the Provider AWS controller should now be ready to provision any infrastructure component in AWS! So let’s start with a simple S3 Bucket.
The first step towards using Composite Resources is configuring Crossplane so that it knows what XRs you’d like to exist, and what to do when someone creates one of those XRs. This is done using a
CompositeResourceDefinition(XRD) resource and one or more
So in order to provision a S3 Bucket in AWS using Crossplane we have to craft three building blocks:
1. Defining a CompositeResourceDefinition (XRD) for our S3 Bucket
All possible fields a XRD may consist of are documented here . The field
spec.versions.schema must contain a OpenAPI schema , which is similar to the ones used by any Kubernetes CRDs. They determine what fields the XR (and Claim) will have. The full CRD documentation and a guide on how to write OpenAPI schemas could be found in the Kubernetes docs .
Note that Crossplane will automatically extend this section. These includes the following fields, which will be ignored if they’re found in the schema:
The file uses lots of comments in the attempt to give some assistance for crafting your own XRD. A CompositeResourceDefinition can be roughly divided into the top area featuring Crossplane specific configuration (with
spec.claimNames and a compositionRef using
spec.defaultCompositionRef) and the bottom section leveraging the OpenAPI schema defining the parameters of our resources. In our case in which we want to provision an S3 Bucket, there are only two parameters:
We need to install the XRD into our cluster with:
1kubectl apply -f aws/s3/definition.yaml 2
Optionally one can check the CRDs beeing created with
kubectl get crds and filter them using
grep to our group name
1$ kubectl get crds | grep crossplane.jonashackt.io 2objectstorages.crossplane.jonashackt.io 2022-06-27T09:54:18Z 3xobjectstorages.crossplane.jonashackt.io 2022-06-27T09:54:18Z 4
2. Craft a Composition to provision a S3 Bucket for static website hosting
The main work in Crossplane has to be done crafting the Compositions. This is because they interact with the infrastructure primitives the cloud provider APIs provide.
There are detailed docs to many of the possible manifest configurations . A
Composition to manage an S3 Bucket in AWS with public access for static website hosting could for example look like the example project’s composition.yaml :
Again this example uses lots of comments in the attempt to give some assistance for crafting your own Composition. After defining some
spec.writeConnectionSecretsToNamespace the actual resource configuration happens in
spec.resources. Since we want to provision a S3 Bucket in AWS we definitely should have a look into the Crossplane AWS provider API docs . It also helps to have the Terraform docs opened up to clarify some questions or to skim the internet for some example code. The latter wasn’t that comprehensive when writing this post. I hope to see more Crossplane example implementations around in the near future.
The Composition also needs to be installed with
kubectl apply -f aws/s3/composition.yaml.
3. Craft a Composite Resource (XR) or Claim (XRC)
Only the platform team itself typically has the permissions to create XRs directly. Everyone else uses a lightweight proxy resource called
CompositeResourceClaim (XRC or simply “Claim”) to create them with Crossplane. If you’re familiar with Terrafrom you can think of an XRD as similar to
variable blocks of a Terrafrom module. The
Composition could then be seen as the rest of the HCL code describing how to instrument those variables to create actual resources.
Regardless of the role you only have to write a Composite Resource (XR) or Claim (XRC) ! You don’t need to craft both, since the XR will be automatically generated from the XRC by Crossplane. If you step into the role of an platform engineer, you can start crafting the XR directly and no Claim will be generated.
Since we want to create a S3 Bucket, the example project hosts a claim.yaml :
As you can see the Claim is one of the simpler building blocks in Crossplane. You need to look at the correct
apiVersion and name the
kind exactly to what you defined in the CompositeResourceDefinition (XRD). Also a
namespace is needed to define a valid Claim (which isn’t true for XRs, since they’re cluster scoped). Also a
compositionRef or alternatively
compositionSelector need to be defined to reference the Composition the Claim should use. Finally both our parameters
region must be defined.
Now kubectl applying our Claim will finally trigger the provisioning of our S3 Bucket in AWS! If you went through all the steps, you can now test-drive your setup with
kubectl apply -f aws/s3/claim.yaml.
The CLI validation will have your back, if you held the YAML ruler incorrectly like me:
1$ kubectl apply -f aws/s3/claim.yaml 2error: error validating "claim.yaml": error validating data: [ValidationError(S3Bucket.metadata): unknown field "crossplane.io/external-name" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta_v2, ValidationError(S3Bucket.spec): unknown field "parameters" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec.writeConnectionSecretToRef): missing required field "namespace" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec.writeConnectionSecretToRef, ValidationError(S3Bucket.spec): missing required field "bucketName" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec): missing required field "region" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec]; if you choose to ignore these errors, turn validation off with --validate=false 3
This will also help you debug your configuration – it hints for the actual problems that are still present.
Waiting for resources to become ready
There are some possible things to check while your resources get deployed after running a
kubectl apply -f aws/s3/claim.yaml. The best overview gives a
kubectl get crossplane which will simply list all the crossplane resources:
1$ kubectl get crossplane 2NAME ESTABLISHED OFFERED AGE 3compositeresourcedefinition.apiextensions.crossplane.io/xs3buckets.crossplane.jonashackt.io True True 23m 4 5NAME AGE 6composition.apiextensions.crossplane.io/s3bucket 2d17h 7 8NAME INSTALLED HEALTHY PACKAGE AGE 9provider.pkg.crossplane.io/provider-aws True True crossplane/provider-aws:v0.22.0 4d21h 10 11NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE 12providerrevision.pkg.crossplane.io/provider-aws-2189bc61e0bd True 1 crossplane/provider-aws:v0.22.0 Active 4d21h 13 14NAME AGE TYPE DEFAULT-SCOPE 15storeconfig.secrets.crossplane.io/default 5d23h Kubernetes crossplane-system 16
There are also some other useful commands you should know:
kubectl get claim: get all resources of all Claim kinds, like PostgreSQLInstance.
kubectl get composite: get all resources that are of kind Composite (aka XR), like XPostgreSQLInstance.
kubectl get composition: don’t confuse with
composite! Get’s you all Compositions.
kubectl get managed: get all resources that represent a unit of external infrastructure.
kubectl get name-of-provider: get all resources related to the Provider.
Troubleshooting your Crossplane configuration
There’s a great explanation in the docs on how to efficiently track down errors in Crossplane configuration:
Per Kubernetes convention, Crossplane keeps errors close to the place they happen. This means that if your Claim is not becoming ready due to an issue with your Composition or with a composed resource you’ll need to “follow the references” to find out why. Your Claim will only tell you that the XR is not yet ready.
The docs also tell us what they mean by “follow the references”:
- Find your XR by running
kubectl describe claim-kind claim-metadata.nameand look for its “Resource Ref” (aka
kubectl describeon your XR. This is where you’ll find out about issues with the Composition you’re using, if any.
- If there are no issues but your XR doesn’t seem to be becoming ready, take a look for the “Resource Refs” (or
spec.resourceRefs) to find your composed resources.
kubectl describeon each referenced composed resource to determine whether it is ready and what issues, if any, it is encountering.
Inspect the S3 Bucket and deploy a static website
Now let’s check our Claim with
kubectl get claim-kind claim-metadata.name like this:
1$ kubectl get ObjectStorage managed-s3 2NAME READY CONNECTION-SECRET AGE 3managed-s3 managed-s3-connection-details 5s 4
To watch the provisioned resources become ready, we can run
kubectl get crossplane -l crossplane.io/claim-name=claim-metadata.name:
1kubectl get crossplane -l crossplane.io/claim-name=managed-s3 2
We can also check if the S3 Bucket has been created successfully via aws CLI with
aws s3 ls.
1$ aws s3 ls 22022-06-27 11:56:26 microservice-ui-nuxt-js-static-bucket 3... 4
Our bucket should be provisioned by now! It will then also be visible in the AWS console:
To really “proof” it’s working, let’s deploy a website (the example project holds a simple index.html ) to our S3 Bucket using the aws CLI like this:
1aws s3 sync static s3://microservice-ui-nuxt-js-static-bucket --acl public-read 2
Now we can open up microservice-ui-nuxt-js-static-bucket.s3-website.eu-central-1.amazonaws.com in our Browser and should see our website already deployed:
To delete the S3 Bucket again, we simply need to remove the Claim. But before deleting the Claim, we should remove our
index.html. Otherwise we’ll run into errors like
BucketNotEmpty: The bucket you tried to delete is not empty:
1aws s3 rm s3://microservice-ui-nuxt-js-static-bucket/index.html 2
kubectl delete claim we can remove our S3 Bucket again:
1kubectl delete claim managed-s3 2
A full cycle of all the described steps and commands in this post can be found in the example repositories GitHub Actions workflow provision.yml . It’s always a good practice to have everything automatically executable and fully comprehensible (!) inside a CI/CD pipeline :).
Learn the Kubernetes API – and you can rule the (infrastructure) world with Crossplane!
It’s really as simple as that: Crossplane extends the Kubernetes API in a way that you don’t need to leave your YAML manifests for long anymore. Some work still remains though. You’ll need a management cluster to be set up before you can actually use Crossplane. A combination of GitHub Actions CI/CD workflows with kind or k3d suffices.
The Crossplane Providers already provide a wide coverage of so many infrastructure providers . And since Crossplane introduced the Terraform-to-Crossplane CRD generator Terrajet a while ago, everything Terraform can provision now Crossplane can do to.
But the real work is hiding inside the Compositions! Since skilled platform engineers are needed to craft these using the Managed Resources as building blocks, they really need to know what they’re doing. Just think about more complex infrastructure like an AWS EKS cluster. There are some blog posts and guides around that show how to create “a production ready EKS cluster with Crossplane”. But I wouldn’t name them like that.
My first impression looking at Crossplane was that there’s maybe something missing: A curated library of higher level abstractions as we’re used to from the Pulumi Crosswalk collection for example. I only saw some initiatives from cloud vendors to create things like AWS Blueprints for Crossplane . But digging a bit deeper you’ll find out about Upbound , the company behind Crossplane. They have some really great folks on board (I can fully recommend Nate Reid’s blog who works as Staff Solutions Engineer at Upbound) and they also build the so called Upbound Platform Reference Architectures . These look really promising and may show a kind of Composite Resource library I was missing in the first place. I hope to find the time to dig into them deeper in another post!
Dein Job bei codecentric?
More articles in this subject area\n
Discover exciting further topics and let the codecentric world inspire you.