Widening the API With a CRD

Goal

You write a CustomResourceDefinition yourself to extend the Kubernetes API, confirm that schema validation actually rejects requests, and grant permissions on the new resource with an Aggregated ClusterRole.

Why it matters

CRDs are in the CKA scope not to make you build an Operator. They ask whether you can operate a cluster that has Operators installed. Production clusters already have dozens of CRDs installed.

There is a structure here you need to understand. Creating a CRD adds a new endpoint to the apiserver, and that endpoint provides storage, validation, and watch. But that is all. What actually does something is the controller that watches that CR, and that is separate software. If you create a CR and nothing happens, usually the controller is missing or dead.

The schema is in the same vein. In apiextensions.k8s.io/v1, the schema is not optional but required. The schema is the contract of that API, and the apiserver, not the controller, is the first to block wrong values.

Steps

1. Create the CRD widgets.labhub.io. Group labhub.io, scope Namespaced, kind Widget, plural widgets, singular widget, wg in shortNames, and a single version v1alpha1 with both served and storage true.
2. Create the namespace cka-crd and create a Widget demo in it. spec.replicas is 3 and spec.tier is small.
3. Edit the CRD's v1alpha1 schema to add validation rules. spec.replicas is type integer with minimum 1 and maximum 10. spec.tier is type string with enum [small, large]. The required list of the spec object is [replicas, tier].
4. Try to create a Widget whose spec.replicas is 99, named too-big, and save the rejection error output to /root/cka-crd/reject.txt. too-big must not be created.
5. Add two additionalPrinterColumns to v1alpha1. Name REPLICAS (type integer, jsonPath .spec.replicas) and name TIER (type string, jsonPath .spec.tier).
6. Create the CRD clusterwidgets.labhub.io. Scope Cluster, kind ClusterWidget, plural clusterwidgets, group labhub.io, version v1alpha1. Then create a ClusterWidget global.
7. Create the ClusterRole cka-widget-viewer. Label rbac.labhub.io/aggregate-to-widget=true, with a rule for apiGroups labhub.io, resource widgets, and verbs get, list, and watch. Then create the ClusterRole cka-widget-aggregate so that its aggregationRule selects that label.

Reference

• The minimal form of a schema is openAPIV3Schema: {type: object, properties: {spec: {type: object, x-kubernetes-preserve-unknown-fields: true}}}. In step 3 you make this spec concrete.
• It is quick to extract the current schema with kubectl get crd widgets.labhub.io -o yaml, edit it, and apply it again.
• Common mistake 1: writing the CRD name in the singular, like widget.labhub.io. You must join the plural and the group.
• Common mistake 2: writing rules together with an aggregationRule. The controller overwrites rules, so hand-written rules disappear.

Create a CustomResourceDefinition

Create the CRD widgets.labhub.io. Group labhub.io, scope Namespaced, kind Widget, plural widgets, singular widget, wg in shortNames, and a single version v1alpha1 with both served and storage true.

A CRD's metadata.name must be in the form 'plural.group'. In apiextensions.k8s.io/v1, each entry of the versions array requires a schema.

Create a custom resource

Create the namespace cka-crd and create a Widget demo in it. spec.replicas is 3 and spec.tier is small.

A CR's apiVersion is 'group/version'. If the schema is loose, arbitrary fields are accepted, so it is convenient to first allow unknown fields under spec.

Add validation rules to the schema

Edit the CRD's v1alpha1 schema to add validation rules. spec.replicas is type integer with minimum 1 and maximum 10. spec.tier is type string with enum [small, large]. The required list of the spec object is [replicas, tier].

Under properties.spec.properties inside openAPIV3Schema, add the type and minimum/maximum/enum for each field. required is an array at the level of that object, not a value.

See the moment validation rejects a request

Try to create a Widget whose spec.replicas is 99, named too-big, and save the rejection error output to /root/cka-crd/reject.txt. too-big must not be created.

The error goes to standard error, not standard output. Do not forget 2>&1 when you redirect. It is normal for the rejected resource not to be created.

Add columns to kubectl get output

Add two additionalPrinterColumns to v1alpha1. Name REPLICAS (type integer, jsonPath .spec.replicas) and name TIER (type string, jsonPath .spec.tier).

additionalPrinterColumns goes inside each version in the versions array. It needs three fields: name, type, and jsonPath, and the jsonPath starts with a dot.

Create a cluster-scoped CRD

Create the CRD clusterwidgets.labhub.io. Scope Cluster, kind ClusterWidget, plural clusterwidgets, group labhub.io, version v1alpha1. Then create a ClusterWidget global.

You cannot change the scope after the CRD is created. A cluster-scoped resource does not accept the -n option.

Widen permissions with an Aggregated ClusterRole

Create the ClusterRole cka-widget-viewer. Label rbac.labhub.io/aggregate-to-widget=true, with a rule for apiGroups labhub.io, resource widgets, and verbs get, list, and watch. Then create the ClusterRole cka-widget-aggregate so that its aggregationRule selects that label.

You do not write the rules of a ClusterRole that has an aggregationRule yourself. The controller finds the other ClusterRoles that carry the label and merges them in.