TT Lab
Get started
Learn Learning paths Courses

CGOA — GitOps Certified Associate

Doing GitOps Without Putting Secrets in Git — and What GitOps Cannot Protect

Continue in TT Lab

In one line

GitOps demands "everything in Git," but secrets alone cannot go there. There are four approaches, and each has a different trade-off. And no matter how well you do GitOps, it is useless if the layer beneath it is open — that is what the 4C model says.

Why this was needed

You first need to know exactly what happens when you commit a plaintext secret. A credential pushed to a public repository is collected by automated scanners within seconds to minutes. So the correct order when you discover one is revoke → investigate the impact → clean up the history. Starting with a history rewrite is the wrong order — because values already copied into forked repositories, local copies that were already cloned, PR references the platform retains, code-search caches, and CI caches cannot be taken back. A history rewrite is not a measure that undoes the leak; it is hygiene that reduces recurrence.

There is another misunderstanding you must address. The base64 in a Kubernetes Secret is not encryption. There is no key, and one command turns it back. Moreover, with the default settings, Secrets are stored in etcd in plaintext. Anyone who obtains an etcd backup file, a snapshot, or a disk image reads every Secret. You must turn on encryption at rest (EncryptionConfiguration), and in that case the identity provider must be placed last in the list. If it comes first, you revert to plaintext storage.

How it works — four approaches

Method What goes into Git Who decrypts Strengths Trade-offs
Sealed Secrets A SealedSecret encrypted with a public key The in-cluster controller (private key) The value is inside Git, so it is self-contained The key differs per cluster, everything must be re-encrypted if the key is lost, replacing a value requires a commit
External Secrets Operator Only a reference to an external store (ExternalSecret) The operator fetches from the external manager and creates the Secret The value is not in Git at all, rotation is the store's job Depends on the external manager, the operator's credential becomes a new weak point
SOPS YAML with only the values encrypted (keys in plaintext) CI or the GitOps tool (age/KMS/PGP) Diffs are readable and can be reviewed Decryption-key distribution problem, Argo CD needs a plugin (Flux is native)
Argo CD Vault Plugin A manifest containing placeholders A plugin in the repo-server substitutes at render time Manifests stay clean Plaintext appears in the render result, the burden of running a CMP sidecar

Let us make clear the point that is most often misunderstood here. An ExternalSecret manifest is safe to commit — because it has no value. But the operator materializes that value into a real Kubernetes Secret, so the RBAC problem and the encryption-at-rest problem remain as they were. Using a secret manager does not exempt you from hygiene on the cluster side. Anyone with get secrets permission in a namespace reads every Secret in that namespace. It is very common for edit permission given to developers as a convenience to effectively be permission to view production credentials.

Signed commits and signatureKeys

If Git is the source of truth, then "who wrote that truth" is itself the security boundary. If you register a GPG key ID in an AppProject's signatureKeys, the Applications in that project sync only from commits (or tags) whose signature has been verified. Even if the repository credentials are stolen and an attacker pushes in a commit, if it is not signed with a registered key, the agent refuses to apply it. If branch protection is a server-side control, signatureKeys is a cluster-side control, and the two do not replace each other.

The trap of argocd cluster add

This is the command used when registering an external cluster.

argocd cluster add my-cluster-context --name production

This one line creates an argocd-manager ServiceAccount on the target cluster and binds the cluster-admin ClusterRole to it. It is a default for convenience and the documentation says so, but in practice it means "if Argo CD is breached, every cluster is breached." Because Argo CD is a system that gathers the keys to many clusters in one place, it is one of the components with the largest blast radius when compromised.

There are two responses. First, replace the target cluster's ClusterRole with a least-privilege one — keep read broad (get/list/watch), but narrow writes to the resource kinds you actually deploy. Second, if the target namespaces are fixed, give only namespace-scoped permissions instead of cluster scope, and tighten further with the AppProject's destinations.

Where GitOps sits within 4C

The 4C of cloud native security, from the outside in, are Cloud → Cluster → Container → Code. The core proposition is a single one.

If an outer layer is vulnerable, an inner layer cannot save it.

However well you harden a container, it is useless if the cluster API is open to the internet, and even if the code has no vulnerabilities, it is useless if the cloud account's IAM is loose.

On this map, the layer GitOps handles is mainly Cluster. It pins down by declaration what goes into the cluster, narrows the change path to Git alone, and reverts drift. As a side effect it also contributes to the Code layer — since every change goes through review and signing, the last stretch of the supply chain is audited. On the other hand, there are clearly things GitOps does not do. It does not find vulnerabilities inside images (the Container layer, the job of a scanner), it does not design the permissions of a cloud account for you (the Cloud layer), and it does not detect intrusions that happen at runtime. If the exam asks something like "Does adopting GitOps solve container image vulnerabilities?", the answer is no.

What it looks like in the field

The author's homelab puts Harbor at 10.0.0.202, Gitea at 10.0.0.200, and Argo CD at 10.0.0.201, exposed through the MetalLB L2 pool 10.0.0.200-215. The network is handled by Cilium 1.20 eBPF without kube-proxy, and even hostFirewall and WireGuard encryption between nodes are turned on. Here the layered sense of 4C actually shows — the fact that the Argo CD UI is open only inside the private network (the Cluster/network layer) is preventing a good part of mistakes in Argo CD's own RBAC configuration (inside the Cluster layer). Conversely, if that network boundary disappeared, you would have to hold up on the inner settings alone.

One more thing. This cluster has an etcd quorum with 3 control planes, but because controlPlaneEndpoint is the physical IP of the first node, if that node dies, the data is alive yet nobody can connect to the API. Just as availability and accessibility are separate, having encrypted secrets and few people being able to read those secrets are also separate. Encryption stops an attacker who obtains the storage medium, and RBAC stops people who pass through the API. You need both.

What to check in the next quiz

This module ends with a quiz. The lab environment has no Sealed Secrets controller, no Vault, and no internet, so it is more accurate to learn the selection criteria and trade-offs of secret tools through questions. If you add signatureKeys and a narrow destinations to the AppProject file you made in the earlier module, the content of this reading connects to something real.