Tenant Boundaries and an App Factory

Goal

Declare a tenant boundary with an AppProject, write manifests that stamp out Applications with two kinds of ApplicationSet, and then put the real namespaces and ResourceQuotas that correspond to that boundary into the cluster.

Why it matters

Multi-tenancy consists of two layers. The upper layer is "who can declare what," and the lower layer is "how many resources that declaration can use." An AppProject is responsible for the upper layer only. Even if you narrow the deployment targets with destinations, it cannot stop that namespace from eating all the nodes, and that is the job of ResourceQuota. Conversely, with only ResourceQuota, you cannot stop a tenant from hooking up someone else's repository as a source or creating a ClusterRoleBinding. Only by standing up both layers together does each tool's reach stay with you.

Steps

Create the /root/capa-project/ directory and, in appproject.yaml, write kind AppProject, metadata.name capa-tenants, and metadata.namespace argocd. In spec.sourceRepos put https://gitea.homelab.internal/platform/*, and in spec.destinations[0] put server https://kubernetes.default.svc and namespace capa-team-*.
In the same file, put group "" and kind Namespace in spec.clusterResourceWhitelist, and put the two kinds ResourceQuota and LimitRange of group "" in spec.namespaceResourceBlacklist.
In spec.roles[0], set name deployer, include p, proj:capa-tenants:deployer, applications, sync, capa-tenants/*, allow in policies, and put capa-platform in groups.
In spec.syncWindows, create one item with kind deny, schedule 0 0 * * 0,6, duration 24h, and applications *.
In /root/capa-project/appset-list.yaml, write kind ApplicationSet and metadata.name capa-tenants-list, and turn spec.goTemplate on with true. In spec.generators[0].list.elements, put 3 items whose tenant key values are alpha, bravo, and charlie, and set spec.template.spec.project to capa-tenants.
In /root/capa-project/appset-matrix.yaml, write kind ApplicationSet and metadata.name capa-tenants-matrix, and turn spec.goTemplate on with true. In spec.generators[0].matrix.generators, put exactly two: one is a git generator whose repoURL is https://gitea.homelab.internal/platform/tenants.git and whose directories[0].path is apps/*, and the other is a list generator with at least one element.
Actually create the three namespaces capa-team-alpha, capa-team-bravo, and capa-team-charlie in the cluster, and attach the label capa.io/tenant with the values alpha, bravo, and charlie respectively.
In each of the three namespaces, actually create a ResourceQuota named tenant-quota. spec.hard.pods is 10 and spec.hard["requests.cpu"] is 2.

Notes

You can create it quickly with kubectl create quota tenant-quota --hard=pods=10,requests.cpu=2 -n <네임스페이스> (the placeholder is the namespace).
To verify the manifests, extract just the fields, as in yq '.spec.clusterResourceWhitelist' /root/capa-project/appproject.yaml.
Common mistake 1: putting the same kind in both the allowlist and the denylist. The two fields have different scopes.
Common mistake 2: putting only one generator under matrix. A Cartesian product holds only with two.

The AppProject's allowed sources and deployment targets

Create the /root/capa-project/ directory and, in appproject.yaml, write kind AppProject, metadata.name capa-tenants, and metadata.namespace argocd. In spec.sourceRepos put https://gitea.homelab.internal/platform/*, and in spec.destinations[0] put server https://kubernetes.default.svc and namespace capa-team-*.

sourceRepos and destinations are both allowlists. You can use a wildcard in the namespace field of destinations.

Allowlist and denylist

In the same file, put group "" and kind Namespace in spec.clusterResourceWhitelist, and put the two kinds ResourceQuota and LimitRange of group "" in spec.namespaceResourceBlacklist.

For the cluster scope, the default is to forbid everything, so you write what needs to be opened, and for the namespace scope, the default is to allow everything, so you write what needs to be closed. You only need to grasp that the directions are opposite.

Project roles and group mapping

In spec.roles[0], set name deployer, include p, proj:capa-tenants:deployer, applications, sync, capa-tenants/*, allow in policies, and put capa-platform in groups.

A policy string has six slots: p, SUBJECT, RESOURCE, ACTION, OBJECT, EFFECT. The subject of a project role takes the form proj:PROJECT-NAME:ROLE-NAME.

Block sync on weekends

In spec.syncWindows, create one item with kind deny, schedule 0 0 * * 0,6, duration 24h, and applications *.

A syncWindows item consists of kind, schedule, duration, and applications. schedule is a cron expression pointing to the moment the window opens, and duration is the length of the window.

The list generator and goTemplate

In /root/capa-project/appset-list.yaml, write kind ApplicationSet and metadata.name capa-tenants-list, and turn spec.goTemplate on with true. In spec.generators[0].list.elements, put 3 items whose tenant key values are alpha, bravo, and charlie, and set spec.template.spec.project to capa-tenants.

The elements of a list generator are an array of maps with arbitrary keys. To use conditions or loops, you have to turn on one boolean directly under spec that switches the template engine.

Build a Cartesian product with the matrix generator

In /root/capa-project/appset-matrix.yaml, write kind ApplicationSet and metadata.name capa-tenants-matrix, and turn spec.goTemplate on with true. In spec.generators[0].matrix.generators, put exactly two: one is a git generator whose repoURL is https://gitea.homelab.internal/platform/tenants.git and whose directories[0].path is apps/*, and the other is a list generator with at least one element.

The generators under matrix must be exactly two to make a Cartesian product. A git generator scans directories with directories and produces one set of parameters per directory.

Create 3 tenant namespaces

Actually create the three namespaces capa-team-alpha, capa-team-bravo, and capa-team-charlie in the cluster, and attach the label capa.io/tenant with the values alpha, bravo, and charlie respectively.

From here on it is a real cluster. Make the names and the label values correspond to each other. It is repetitive work, so using a shell loop is fine.

A ResourceQuota that sets up the resource boundary

In each of the three namespaces, actually create a ResourceQuota named tenant-quota. spec.hard.pods is 10 and spec.hard["requests.cpu"] is 2.

spec.hard of a ResourceQuota is a map and its keys contain dots. If you recall that what the AppProject drew is only a syntactic boundary, the meaning of this step becomes clear.