//

Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3

3.7.2022 | 17 minutes of reading time

What if learning the Kubernetes API is all you need to provision any infrastructure? And we’re not only talking about AWS, Azure & Google – but also IONOS, DigitalOcean and even vSphere. Let’s have a look at Crossplane and how we can create an S3 Bucket on AWS step by step.

No code required, it’s all declarative!

Crossplane claims to be the “The cloud native control plane framework”. It introduces a new way for managing any cloud resource (be it Kubernetes-native or not). It’s an alternative Infrastructure-as-Code tooling to Terraform , AWS CDK /Bicep or Pulumi and introduces a different level of abstraction, based on Kubernetes CRDs . One could name it the cloud native way to do GitOps.

Why Crossplane is different

Crossplane can be also compared to tools which enable the management of cloud resources through the Kubernetes API, like AWS Controllers for Kubernetes (ACK) , Azure Service Operator for Kubernetes (ASO) or Google Config Connector . Crossplane providers are even generated from ACK and ASO . But the CNCF incubating project Crossplane adds on top of these:

The Crossplane community believes that the typical developer using Kubernetes to deploy their application shouldn’t have to deal with low level infrastructure APIs.

Crossplane uses the Kubernetes API and extends it with a set of Custom Resource Definitions (CRDs) to abstract from actual cloud provider APIs. Additionally, these CRDs are a great foundation to build an Internal Developer Platform (IDP) . Crossplane promotes self-service by introducing building blocks like Composite Resource Claims (XRCs) that are a great API for the typical application developer. On the other hand, Composite Resources (XR) and Composites are a magnitude more powerful and ideal for platform operators.

Drawing on our experiences as platform builders, SREs, and application developers, we’ve designed Crossplane as a toolkit to build your own custom resources on top of any API – often those of the cloud providers. We think this approach is critical to enable usable self-service infrastructure in Kubernetes.

Finally, Crossplane also enables effective multi-cloud interoperability . But don’t worry, you can even order Pizza using Crossplane :).

Crossplane basic concepts

The crossplane concept of a Composite Resource is composed out of several building blocks :

  • Composite Resources (XR) : they compose Managed Resources into higher level infrastructure units (especially interesting for platform teams) and an optional CompositeResourceClaim (XRC) (which is also referred to as ‘Claim’ and can be seen as an abstraction of the XR for the application team to consume)
  • a CompositeResourceDefinition (XRD) which defines an OpenAPI schema the Composition needs to be conform to (think of Kubernetes CRDs)
  • a Composition that describes the actual infrastructure primitives aka Managed Resources used to build the Composite Resource. One XRD could have multiple Compositions – e.g. one for every environment like development, stating and production

Composite Resources can also be nested together, which allows for higher level abstractions and better separation of concerns. Think of an AWS EKS cluster where you need lot’s of network/subnetting setup (which can form one XR) and the actual EKS cluster creation with NodeGroups etc (which would be the XR using the networking XR). More in-depth details on nested XRs can be found here .

As an optional step, you can package these Composite Resource artifacts using a Configuration into an OCI container image. Now your custom Configuration package can be installed easily in any other Crossplane clusters.

Composite Resources are composed of infrastructure building blocks called Managed Resources (MR) that are bundled by a Provider:

You may also stumble upon Packages . They were formerly named Stacks and are simply OCI container images. Packages handle the distribution, version updates, dependency management and permissions for Providers and Configurations.

Getting started with Crossplane: fire up a k8s cluster with kind

In order to use Crossplane, we’ll need any kind of Kubernetes cluster to let it operate in. This management cluster with Crossplane installed will then provision the defined infrastructure. Using any managed or local Kubernetes cluster like EKS, AKS, Minikube or k3d is suitable. In my example project (which is available on GitHub ), I used kind to host the management cluster.

Using kind is pretty easy. Let’s get our hands dirty! Just be sure to have kind, the package manager Helm and kubectl installed. On a Mac you can use brew like this (have a look at the docs for other systems ):

1brew install kind helm kubectl
2

We should also install the crossplane CLI:

1curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh
2sudo mv kubectl-crossplane /usr/local/bin
3

The kubectl crossplane --help command should be ready to use.

Now spin up a local kind cluster with:

1kind create cluster --image kindest/node:v1.23.0 --wait 5m
2

Install Crossplane with Helm

The Crossplane docs tell us to use Helm for installation :

1kubectl create namespace crossplane-system
2 
3helm repo add crossplane-stable https://charts.crossplane.io/stable
4helm repo update
5 
6helm upgrade --install crossplane --namespace crossplane-system crossplane-stable/crossplane
7

As a Renovate -powered alternative we can create our own simple Chart.yaml to enable automatic updates of our installation if new crossplane versions get released:

To install Crossplane using our own Chart.yaml simply run:

1helm dependency update crossplane-config/install
2helm upgrade --install crossplane --namespace crossplane-system crossplane-config/install
3

Be sure to exclude /charts directories and Chart.lock files via .gitignore. Now Renovate will have an eye on our Crossplane versions:

Before we can actually apply a Provider, we have to make sure that Crossplane is actually healthy and running. Therefore we can use the kubectl wait command like this:

1kubectl wait --for=condition=ready pod -l app=crossplane --namespace crossplane-system --timeout=120s
2

Otherwise we will run into errors like this when applying a Provider:

1error: resource mapping not found for name: "provider-aws" namespace: "" from "provider-aws.yaml": no matches for kind "Provider" in version "pkg.crossplane.io/v1"
2ensure CRDs are installed first
3

Finally check the Crossplane status with kubectl get all -n crossplane-system:

1$ kubectl get all -n crossplane-system
2NAME                                           READY   STATUS    RESTARTS   AGE
3pod/crossplane-7c88c45998-d26wl                1/1     Running   0          69s
4pod/crossplane-rbac-manager-8466dfb7fc-db9rb   1/1     Running   0          69s
5 
6NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
7deployment.apps/crossplane                1/1     1            1           69s
8deployment.apps/crossplane-rbac-manager   1/1     1            1           69s
9 
10NAME                                                 DESIRED   CURRENT   READY   AGE
11replicaset.apps/crossplane-7c88c45998                1         1         1       69s
12replicaset.apps/crossplane-rbac-manager-8466dfb7fc   1         1         1       69s
13

Configure Crossplane to access AWS: create aws-creds.conf

I assume that you have aws CLI installed and configured . The command aws configure should work on your system. With this prepared we can create an aws-creds.conf file as described in the docs :

1echo "[default]
2aws_access_key_id = $(aws configure get aws_access_key_id)
3aws_secret_access_key = $(aws configure get aws_secret_access_key)
4" > aws-creds.conf
5

Don’t ever check this file into source control – it holds your AWS credentials! In the example project I added *-creds.conf to the .gitignore file.

If you’re using a CI system like GitHub Actions (as this repository is also based on), you need to have both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured as repository secrets:

Also make sure to have your default region configured locally, or as a env: variable in your CI system. All three needed variables in GitHub Actions for example look like this:

Create AWS Provider secret

Now we need to use the aws-creds.conf file to create the Crossplane AWS Provider secret:

1kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf
2

If everything went well there should be a new aws-creds secret ready:

Install the Crossplane AWS Provider

To be able to provision infrastructure on a cloud provider like AWS, we need to install the corresponding Crossplane Provider first. Remember: the Provider packages all needed Managed Resources (MRs) and their respective controllers. As a Provider is a Crossplane Package, we can install it using the Crossplane CLI like this:

1kubectl crossplane install provider crossplane/provider-aws:v0.22.0
2

Alternatively we can create our own provider-aws.yaml file like this:

This kind: Provider with apiVersion: pkg.crossplane.io/v1 is completely different from the kind: Provider which we want to consume. The latter uses apiVersion: meta.pkg.crossplane.io/v1.

Don’t forget to install the AWS provider using kubectl:

1kubectl apply -f crossplane-config/provider-aws.yaml
2

The package version in combination with the packagePullPolicy configuration here is crucial, since we can configure an update strategy for the Provider here. A full table of all possible fields can be found in the docs . We can also let Crossplane manage the upgrade to new versions for us. If you installed multiple package versions, you’ll see them prefixed with providerrevision.pkg.x when running kubectl get providerrevision:

1$ kubectl get providerrevision
2NAME                                                           HEALTHY   REVISION   IMAGE                             STATE      DEP-FOUND   DEP-INSTALLED   AGE
3providerrevision.pkg.crossplane.io/provider-aws-2189bc61e0bd   True      1          crossplane/provider-aws:v0.22.0   Inactive                               6d22h
4providerrevision.pkg.crossplane.io/provider-aws-d87796863f95   True      2          crossplane/provider-aws:v0.28.1   Active                                 43h
5

Now our first Crossplane Provider has been installed. You may check it with kubectl get provider:

1$ kubectl get provider
2NAME           INSTALLED   HEALTHY   PACKAGE                           AGE
3provider-aws   True        Unknown   crossplane/provider-aws:v0.22.0   13s
4

Before we can actually apply a ProviderConfig to our AWS provider we have to make sure that it’s actually healthy and running. Therefore we can use the kubectl wait command again like this:

1kubectl wait --for=condition=healthy --timeout=120s provider/provider-aws
2

Otherwise we may run into errors when applying the ProviderConfig right after the Provider.

Create ProviderConfig to consume the Secret containing AWS credentials

Now we need to create a ProviderConfig object that will tell the AWS Provider where to find it’s AWS credentials . Therefore we create a provider-config-aws.yaml :

Crossplane resources use the ProviderConfig named default if no specific ProviderConfig is specified, so this ProviderConfig will be the default for all AWS resources.

The secretRef.name and secretRef.key have to match the fields of the already created Secret. Apply the ProviderConfig with:

1kubectl apply -f crossplane-config/provider-config-aws.yaml
2

Provision a S3 Bucket with Crossplane

The Crossplane core controller and the Provider AWS controller should now be ready to provision any infrastructure component in AWS! So let’s start with a simple S3 Bucket.

The first step towards using Composite Resources is configuring Crossplane so that it knows what XRs you’d like to exist, and what to do when someone creates one of those XRs. This is done using a CompositeResourceDefinition (XRD) resource and one or more Composition resources.

So in order to provision a S3 Bucket in AWS using Crossplane we have to craft three building blocks:

1. CompositeResourceDefinition (XRD) (reference docs )
2. Composition (reference docs )
3. Composite Resource (XR) or Claim (XRC) (reference docs )

1. Defining a CompositeResourceDefinition (XRD) for our S3 Bucket

All possible fields a XRD may consist of are documented here . The field spec.versions.schema must contain a OpenAPI schema , which is similar to the ones used by any Kubernetes CRDs. They determine what fields the XR (and Claim) will have. The full CRD documentation and a guide on how to write OpenAPI schemas could be found in the Kubernetes docs .

Note that Crossplane will automatically extend this section. These includes the following fields, which will be ignored if they’re found in the schema:

  • spec.resourceRef
  • spec.resourceRefs
  • spec.claimRef
  • spec.writeConnectionSecretToRef
  • status.conditions
  • status.connectionDetails

The example project on GitHub hosts an a Composite Resource Definition (XRD) for an S3 Bucket. The definition.yaml could look like this:

The file uses lots of comments in the attempt to give some assistance for crafting your own XRD. A CompositeResourceDefinition can be roughly divided into the top area featuring Crossplane specific configuration (with spec.group, spec.names, spec.claimNames and a compositionRef using spec.defaultCompositionRef) and the bottom section leveraging the OpenAPI schema defining the parameters of our resources. In our case in which we want to provision an S3 Bucket, there are only two parameters: bucketName and region.

We need to install the XRD into our cluster with:

1kubectl apply -f aws/s3/definition.yaml
2

Optionally one can check the CRDs beeing created with kubectl get crds and filter them using grep to our group name crossplane.jonashackt.io:

1$ kubectl get crds | grep crossplane.jonashackt.io
2objectstorages.crossplane.jonashackt.io                         2022-06-27T09:54:18Z
3xobjectstorages.crossplane.jonashackt.io                        2022-06-27T09:54:18Z
4

2. Craft a Composition to provision a S3 Bucket for static website hosting

The main work in Crossplane has to be done crafting the Compositions. This is because they interact with the infrastructure primitives the cloud provider APIs provide.

There are detailed docs to many of the possible manifest configurations . A Composition to manage an S3 Bucket in AWS with public access for static website hosting could for example look like the example project’s composition.yaml :

Again this example uses lots of comments in the attempt to give some assistance for crafting your own Composition. After defining some metadata labels, spec.compositeTypeRef and spec.writeConnectionSecretsToNamespace the actual resource configuration happens in spec.resources. Since we want to provision a S3 Bucket in AWS we definitely should have a look into the Crossplane AWS provider API docs . It also helps to have the Terraform docs opened up to clarify some questions or to skim the internet for some example code. The latter wasn’t that comprehensive when writing this post. I hope to see more Crossplane example implementations around in the near future.

The Composition also needs to be installed with kubectl apply -f aws/s3/composition.yaml.

3. Craft a Composite Resource (XR) or Claim (XRC)

Only the platform team itself typically has the permissions to create XRs directly. Everyone else uses a lightweight proxy resource called CompositeResourceClaim (XRC or simply “Claim”) to create them with Crossplane. If you’re familiar with Terrafrom you can think of an XRD as similar to variable blocks of a Terrafrom module. The Composition could then be seen as the rest of the HCL code describing how to instrument those variables to create actual resources.

Regardless of the role you only have to write a Composite Resource (XR) or Claim (XRC) ! You don’t need to craft both, since the XR will be automatically generated from the XRC by Crossplane. If you step into the role of an platform engineer, you can start crafting the XR directly and no Claim will be generated.

Since we want to create a S3 Bucket, the example project hosts a claim.yaml :

As you can see the Claim is one of the simpler building blocks in Crossplane. You need to look at the correct apiVersion and name the kind exactly to what you defined in the CompositeResourceDefinition (XRD). Also a namespace is needed to define a valid Claim (which isn’t true for XRs, since they’re cluster scoped). Also a compositionRef or alternatively compositionSelector need to be defined to reference the Composition the Claim should use. Finally both our parameters bucketName and region must be defined.

Now kubectl applying our Claim will finally trigger the provisioning of our S3 Bucket in AWS! If you went through all the steps, you can now test-drive your setup with kubectl apply -f aws/s3/claim.yaml.

The CLI validation will have your back, if you held the YAML ruler incorrectly like me:

1$ kubectl apply -f aws/s3/claim.yaml
2error: error validating "claim.yaml": error validating data: [ValidationError(S3Bucket.metadata): unknown field "crossplane.io/external-name" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta_v2, ValidationError(S3Bucket.spec): unknown field "parameters" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec.writeConnectionSecretToRef): missing required field "namespace" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec.writeConnectionSecretToRef, ValidationError(S3Bucket.spec): missing required field "bucketName" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec, ValidationError(S3Bucket.spec): missing required field "region" in io.jonashackt.crossplane.v1alpha1.S3Bucket.spec]; if you choose to ignore these errors, turn validation off with --validate=false
3

This will also help you debug your configuration – it hints for the actual problems that are still present.

Waiting for resources to become ready

There are some possible things to check while your resources get deployed after running a kubectl apply -f aws/s3/claim.yaml. The best overview gives a kubectl get crossplane which will simply list all the crossplane resources:

1$ kubectl get crossplane
2NAME                                                                                          ESTABLISHED   OFFERED   AGE
3compositeresourcedefinition.apiextensions.crossplane.io/xs3buckets.crossplane.jonashackt.io   True          True      23m
4 
5NAME                                               AGE
6composition.apiextensions.crossplane.io/s3bucket   2d17h
7 
8NAME                                      INSTALLED   HEALTHY   PACKAGE                           AGE
9provider.pkg.crossplane.io/provider-aws   True        True      crossplane/provider-aws:v0.22.0   4d21h
10 
11NAME                                                           HEALTHY   REVISION   IMAGE                             STATE    DEP-FOUND   DEP-INSTALLED   AGE
12providerrevision.pkg.crossplane.io/provider-aws-2189bc61e0bd   True      1          crossplane/provider-aws:v0.22.0   Active                               4d21h
13 
14NAME                                        AGE     TYPE         DEFAULT-SCOPE
15storeconfig.secrets.crossplane.io/default   5d23h   Kubernetes   crossplane-system
16

There are also some other useful commands you should know:

  • kubectl get claim: get all resources of all Claim kinds, like PostgreSQLInstance.
  • kubectl get composite: get all resources that are of kind Composite (aka XR), like XPostgreSQLInstance.
  • kubectl get composition: don’t confuse with composite! Get’s you all Compositions.
  • kubectl get managed: get all resources that represent a unit of external infrastructure.
  • kubectl get name-of-provider: get all resources related to the Provider.

Troubleshooting your Crossplane configuration

There’s a great explanation in the docs on how to efficiently track down errors in Crossplane configuration:

Per Kubernetes convention, Crossplane keeps errors close to the place they happen. This means that if your Claim is not becoming ready due to an issue with your Composition or with a composed resource you’ll need to “follow the references” to find out why. Your Claim will only tell you that the XR is not yet ready.

The docs also tell us what they mean by “follow the references”:

  • Find your XR by running kubectl describe claim-kind claim-metadata.name and look for its “Resource Ref” (aka spec.resourceRef).
  • Run kubectl describe on your XR. This is where you’ll find out about issues with the Composition you’re using, if any.
  • If there are no issues but your XR doesn’t seem to be becoming ready, take a look for the “Resource Refs” (or spec.resourceRefs) to find your composed resources.
  • Run kubectl describe on each referenced composed resource to determine whether it is ready and what issues, if any, it is encountering.

Inspect the S3 Bucket and deploy a static website

Now let’s check our Claim with kubectl get claim-kind claim-metadata.name like this:

1$ kubectl get ObjectStorage managed-s3
2NAME         READY   CONNECTION-SECRET               AGE
3managed-s3           managed-s3-connection-details   5s
4

To watch the provisioned resources become ready, we can run kubectl get crossplane -l crossplane.io/claim-name=claim-metadata.name:

1kubectl get crossplane -l crossplane.io/claim-name=managed-s3
2

We can also check if the S3 Bucket has been created successfully via aws CLI with aws s3 ls.

1$ aws s3 ls
22022-06-27 11:56:26 microservice-ui-nuxt-js-static-bucket
3...
4

Our bucket should be provisioned by now! It will then also be visible in the AWS console:

To really “proof” it’s working, let’s deploy a website (the example project holds a simple index.html ) to our S3 Bucket using the aws CLI like this:

1aws s3 sync static s3://microservice-ui-nuxt-js-static-bucket --acl public-read
2

Now we can open up microservice-ui-nuxt-js-static-bucket.s3-website.eu-central-1.amazonaws.com in our Browser and should see our website already deployed:

To delete the S3 Bucket again, we simply need to remove the Claim. But before deleting the Claim, we should remove our index.html. Otherwise we’ll run into errors like BucketNotEmpty: The bucket you tried to delete is not empty:

1aws s3 rm s3://microservice-ui-nuxt-js-static-bucket/index.html
2

Using kubectl delete claim we can remove our S3 Bucket again:

1kubectl delete claim managed-s3
2

A full cycle of all the described steps and commands in this post can be found in the example repositories GitHub Actions workflow provision.yml . It’s always a good practice to have everything automatically executable and fully comprehensible (!) inside a CI/CD pipeline :).

Learn the Kubernetes API – and you can rule the (infrastructure) world with Crossplane!

It’s really as simple as that: Crossplane extends the Kubernetes API in a way that you don’t need to leave your YAML manifests for long anymore. Some work still remains though. You’ll need a management cluster to be set up before you can actually use Crossplane. A combination of GitHub Actions CI/CD workflows with kind or k3d suffices.

The Crossplane Providers already provide a wide coverage of so many infrastructure providers . And since Crossplane introduced the Terraform-to-Crossplane CRD generator Terrajet a while ago, everything Terraform can provision now Crossplane can do to.

But the real work is hiding inside the Compositions! Since skilled platform engineers are needed to craft these using the Managed Resources as building blocks, they really need to know what they’re doing. Just think about more complex infrastructure like an AWS EKS cluster. There are some blog posts and guides around that show how to create “a production ready EKS cluster with Crossplane”. But I wouldn’t name them like that.

My first impression looking at Crossplane was that there’s maybe something missing: A curated library of higher level abstractions as we’re used to from the Pulumi Crosswalk collection for example. I only saw some initiatives from cloud vendors to create things like AWS Blueprints for Crossplane . But digging a bit deeper you’ll find out about Upbound , the company behind Crossplane. They have some really great folks on board (I can fully recommend Nate Reid’s blog who works as Staff Solutions Engineer at Upbound) and they also build the so called Upbound Platform Reference Architectures . These look really promising and may show a kind of Composite Resource library I was missing in the first place. I hope to find the time to dig into them deeper in another post!

I can only recommend to keep an eye on Crossplane e.g. by watching the roadmap and visiting the Crossplane and Upbound blogs regularly.

share post

Likes

2

//

More articles in this subject area\n

Discover exciting further topics and let the codecentric world inspire you.

//

Gemeinsam bessere Projekte umsetzen

Wir helfen Deinem Unternehmen

Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.

Hilf uns, noch besser zu werden.

Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.