Setting Up Signature Verification Policy and Credentials

Goal

You write Kyverno policies that verify cosign signatures and attestations, and put onto a real cluster the credential resources that Kyverno and Pods each need to access a private registry.

Why it matters

The operating conditions of a signature verification policy are harder than its syntax. Whether a signature exists, whether that signature can be looked up, and whether the signature found belongs to a subject I trust are each a different failure mode, and the symptom is the same for all of them: the Pod does not come up. In particular, if you leave out attestors.count, all of entries must pass, so the moment you add a key everything is blocked, and if you turn on mutateDigest, a pipeline that used to push a tag and recreate Pods is quietly neutralized. This lab lets you step on those traps in advance on manifests.

Steps

Create the /root/kca-verify/ directory and write kind ClusterPolicy, metadata.name kca-verify-images, and spec.rules[0].name verify-signature in verify-keys.yaml. Put 10.0.0.202/platform/* in verifyImages[0].imageReferences, and put a PEM public key that starts with BEGIN PUBLIC KEY in attestors[0].entries[0].keys.publicKeys.
In the same attestors[0], set count to 1 and make entries two. The second entry is keyless, with url https://fulcio.sigstore.dev, rekor.url https://rekor.sigstore.dev, issuer https://token.actions.githubusercontent.com, and for subject write a workflow path that contains .github/workflows/.
In the same verifyImages[0], set required, verifyDigest, and mutateDigest all to true and put 10.0.0.202/platform/legacy-* in skipImageReferences. And set spec.webhookConfiguration.timeoutSeconds to a value from 20 to 30, but do not use the deprecated spec.webhookTimeoutSeconds.
In /root/kca-verify/verify-attestations.yaml, write kind ClusterPolicy and put two entries in verifyImages[0].attestations. One has type https://slsa.dev/provenance/v1 and checks the builder identity with conditions[0].all[0].key, and the other has type https://cyclonedx.org/bom/v1.4 and checks the version of the log4j-core component with conditions[0].all[0].key.
In /root/kca-verify/allowed-registries.yaml, write kind ClusterPolicy and metadata.name kca-allowed-registries. spec.rules[0].validate.foreach[0].list is request.object.spec.[initContainers, containers][], and do not wrap it in curly braces. deny.conditions.all[0].operator is AnyNotIn, put 10.0.0.202/platform/* in the value list, and set validate.failureAction to Enforce.
Create the namespace kca-verify in the cluster and actually create the Secret harbor-cred in it. The type is kubernetes.io/dockerconfigjson and the server is 10.0.0.202.
In the same namespace, create the ServiceAccount kca-deployer and attach harbor-cred to its imagePullSecrets, and actually create the Deployment kca-payments with serviceAccountName set to kca-deployer and the container image written as 10.0.0.202/platform/payments@sha256: followed by 64 hexadecimal characters.

Notes

If you use kubectl create secret docker-registry harbor-cred --docker-server=10.0.0.202 --docker-username=robot --docker-password=... -n kca-verify, the type and keys come out right on their own.
For the 64-character digest you may make up any value with something like printf 'kca' | sha256sum.
Common mistake 1: only adding entries to attestors and leaving out count. It becomes a state where all must pass.
Common mistake 2: wrapping the foreach list in curly braces. The list takes a JMESPath expression itself.

verifyImages with a static key

Create the /root/kca-verify/ directory and write kind ClusterPolicy, metadata.name kca-verify-images, and spec.rules[0].name verify-signature in verify-keys.yaml. Put 10.0.0.202/platform/* in verifyImages[0].imageReferences, and put a PEM public key that starts with BEGIN PUBLIC KEY in attestors[0].entries[0].keys.publicKeys.

verifyImages is an array inside the rule, and each item has imageReferences and attestors. You put the public key in as a multi-line PEM string.

Adding keyless and the meaning of count

In the same attestors[0], set count to 1 and make entries two. The second entry is keyless, with url https://fulcio.sigstore.dev, rekor.url https://rekor.sigstore.dev, issuer https://token.actions.githubusercontent.com, and for subject write a workflow path that contains .github/workflows/.

If you only increase entries, all items must pass. To get through a key rotation period without downtime, you have to state explicitly how many need to pass.

Digest pinning, exceptions, and the timeout

In the same verifyImages[0], set required, verifyDigest, and mutateDigest all to true and put 10.0.0.202/platform/legacy-* in skipImageReferences. And set spec.webhookConfiguration.timeoutSeconds to a value from 20 to 30, but do not use the deprecated spec.webhookTimeoutSeconds.

The three booleans mean, respectively, enforce verification, enforce the use of digests, and replace tags with digests. Write exceptions in the dedicated field rather than duplicating the policy. A policy that involves a registry round-trip is not served well by the default webhook timeout.

provenance and SBOM conditions

In /root/kca-verify/verify-attestations.yaml, write kind ClusterPolicy and put two entries in verifyImages[0].attestations. One has type https://slsa.dev/provenance/v1 and checks the builder identity with conditions[0].all[0].key, and the other has type https://cyclonedx.org/bom/v1.4 and checks the version of the log4j-core component with conditions[0].all[0].key.

attestations selects which statement it is by type and checks its contents with conditions. If you look only at the signature, you have not checked who built it.

Checking allowed registries

In /root/kca-verify/allowed-registries.yaml, write kind ClusterPolicy and metadata.name kca-allowed-registries. spec.rules[0].validate.foreach[0].list is request.object.spec.[initContainers, containers][], and do not wrap it in curly braces. deny.conditions.all[0].operator is AnyNotIn, put 10.0.0.202/platform/* in the value list, and set validate.failureAction to Enforce.

The list of foreach is not wrapped in curly braces. You have to iterate over init containers as well, and pick the operator that rejects when something is outside the allowlist.

The registry credential Secret

Create the namespace kca-verify in the cluster and actually create the Secret harbor-cred in it. The type is kubernetes.io/dockerconfigjson and the server is 10.0.0.202.

From here on it is the real cluster. There is a dedicated Secret type for registries, and kubectl has a subcommand that creates it.

A ServiceAccount and a digest-pinned deployment

In the same namespace, create the ServiceAccount kca-deployer and attach harbor-cred to its imagePullSecrets, and actually create the Deployment kca-payments with serviceAccountName set to kca-deployer and the container image written as 10.0.0.202/platform/payments@sha256: followed by 64 hexadecimal characters.

If you attach imagePullSecrets to a service account, Pods that come up with that account inherit it. The image must be pinned by digest, not by tag, and a sha256 is 64 characters.