Only one of three services changed, yet all were rebuilt

Goal

You expand jobs to match the combinations with parallel:matrix, handle exception combinations and waiting for a specific combination, and then run a monorepo with gitlab-ci-local, split into per-service child pipelines and a dynamic pipeline generated from the repository structure.

Why it matters

The way configuration grows is usually copy and paste. Each time one Python version is added you copy the job, and each time one service is added you paste a block into the parent file, until one file accumulates every team's rules and every change rebuilds every service. A matrix reduces the combinations to one block, a child pipeline sends a service's configuration back to that service's directory, and changes and dynamic generation make only what changed run. In exchange, you have to know how job names, artifacts, and variables flow in order to apply needs and rules correctly.

Steps

Make /root/glci-mono a git repository (with .gitlab-ci-local/ in .gitignore), and in .gitlab-ci.yml put stages [build, package, trigger, generate, dynamic] and a job build (stage build). With parallel: matrix, make the combinations of PY as "3.11" and "3.12" and OS as linux and alpine; the script writes to dist/py$PY-$OS.txt the line py=<PY> os=<OS> and uploads dist/ as artifacts. Commit and run it. Four jobs should appear, with names in the form build: [3.11,linux].
Add rules to build so that only the combination where OS is alpine and PY is "3.11" gets when: never, and the rest get when: on_success. When you commit and run, only three jobs should run.
Add a job package-linux (stage package) and, with needs, point to only the one combination of build with PY: "3.12" and OS: linux (needs:parallel:matrix), with the script ls dist. When you commit and run, the package-linux log should show only one file, py3.12-linux.txt.
In /root/glci-mono/services/api/ci.yml, put a job api-unit (echo "api unit svc=$SVC"), and add a job trigger-api (stage trigger) to the parent with the variable SVC: api and trigger: include: services/api/ci.yml. When you commit and run, the api-unit log of the child pipeline should be api unit svc=api.
In services/web/ci.yml, put a job web-unit (echo "web unit svc=$SVC") and add trigger-web (SVC web) to the parent. On the two trigger jobs, put rules: changes: with services/api/**/* and services/web/**/* respectively. Make a bare repository at /root/glci-mono-origin.git and register it as origin, push the committed main, and then run git remote set-head origin main. The grader makes a branch in a copy in which only web was changed and checks that only trigger-web is in the list.
Make /root/glci-mono/scripts/generate.sh print to standard output YAML that contains, for each directory under services/, a job lint-<이름> (echo "lint <이름>") (the placeholder stands for the directory name). In the parent, a job generate (stage generate) saves that output as generated.yml and uploads it as artifacts, and a job run-generated (stage dynamic, needs: [generate]) runs it as a child pipeline with trigger: include: - artifact: generated.yml, job: generate. When you commit, push, and run, lint-api and lint-web should run.
In /root/glci-mono/services/billing/, instead of ci.yml, make only a README.md (any content), then commit and push. Do not touch the parent .gitlab-ci.yml. When you run it, lint-billing should newly appear and run in the dynamic child pipeline. The grader also checks by creating one more service directory in a copy.

Notes

This VM has no GitLab server or runner, and gitlab-ci-local 4.75.1 interprets .gitlab-ci.yml by the same rules as GitLab and runs jobs with the shell. If you write image:, it tries to run with Docker, so do not use it. Protected variables, masking, CI_JOB_TOKEN, runner tags, and merge request pipeline creation are server features and are not reproduced here.
Run: at the repository root, gitlab-ci-local --shell-isolation --no-artifacts-to-source (a separate working directory per job, artifacts not written back to the repository), job list: gitlab-ci-local --list-csv-all, interpreted configuration: gitlab-ci-local --preview. gitlab-ci-local passes only files tracked by git to jobs, so git add a file after creating it. The grader copies the repository, commits all files, and runs it again with the same tool.
Differences in gitlab-ci-local (measured): child pipelines are an experimental feature and do not follow trigger:strategy, so even if a child fails, the trigger job ends in success. --variable is not passed to the child, and only the variables of the trigger job are passed.
The logs of child pipeline jobs are also left in .gitlab-ci-local/output/<잡이름>.log (the placeholder stands for the job name).
parallel:matrix (Job control) · Downstream pipelines · needs · Specify when jobs run with rules

Two Pythons times two OSes, four jobs in one block

Make /root/glci-mono a git repository (with .gitlab-ci-local/ in .gitignore), and in .gitlab-ci.yml put stages [build, package, trigger, generate, dynamic] and a job build (stage build). With parallel: matrix, make the combinations of PY as "3.11" and "3.12" and OS as linux and alpine; the script writes to dist/py$PY-$OS.txt the line py=<PY> os=<OS> and uploads dist/ as artifacts. Commit and run it. Four jobs should appear, with names in the form build: [3.11,linux].

If you write several variables inside one entry of the matrix, every combination becomes a job. Wrap version values in quotes so that 3.10 never turns into 3.1. You see the list with gitlab-ci-local --list-csv-all.

Leave out just one unsupported combination

Add rules to build so that only the combination where OS is alpine and PY is "3.11" gets when: never, and the rest get when: on_success. When you commit and run, only three jobs should run.

Matrix variables can usually be used in rules:if like CI/CD variables. If you split the matrix into two entries to leave out a combination, you have to recalculate the list each time it grows, but if you leave it out with a rule, you only write the exception.

Wait for and receive only one combination's artifacts

Add a job package-linux (stage package) and, with needs, point to only the one combination of build with PY: "3.12" and OS: linux (needs:parallel:matrix), with the script ls dist. When you commit and run, the package-linux log should show only one file, py3.12-linux.txt.

Instead of writing a job created by a matrix in needs by its name as it is (build: [3.12,linux]), you point to it by writing the variable values with parallel:matrix. You receive only the artifacts of the combination you pointed to.

Keep a service's configuration in the service's directory

In /root/glci-mono/services/api/ci.yml, put a job api-unit (echo "api unit svc=$SVC"), and add a job trigger-api (stage trigger) to the parent with the variable SVC: api and trigger: include: services/api/ci.yml. When you commit and run, the api-unit log of the child pipeline should be api unit svc=api.

A trigger job creates another pipeline instead of a script. The file pointed to by include becomes the entire configuration of the child pipeline, and the variables of the trigger job are passed to the child. It is a structure that lets a service team edit only the files in its own directory.

Turn on only the changed service's pipeline

In services/web/ci.yml, put a job web-unit (echo "web unit svc=$SVC") and add trigger-web (SVC web) to the parent. On the two trigger jobs, put rules: changes: with services/api/**/* and services/web/**/* respectively. Make a bare repository at /root/glci-mono-origin.git and register it as origin, push the committed main, and then run git remote set-head origin main. The grader makes a branch in a copy in which only web was changed and checks that only trigger-web is in the list.

If you build every service every time in a monorepo, the pipeline time grows by the number of services. changes looks at the changes compared with the remote default branch (as in gitlab-ci-local), so a remote is needed.

Read the repository structure and create a child pipeline

Make /root/glci-mono/scripts/generate.sh print to standard output YAML that contains, for each directory under services/, a job lint-<이름> (echo "lint <이름>") (the placeholder stands for the directory name). In the parent, a job generate (stage generate) saves that output as generated.yml and uploads it as artifacts, and a job run-generated (stage dynamic, needs: [generate]) runs it as a child pipeline with trigger: include: - artifact: generated.yml, job: generate. When you commit, push, and run, lint-api and lint-web should run.

A job can create the configuration of a child pipeline during execution. Instead of editing the parent configuration each time a service is added, you put the rule (directory = job) in a script. The official documentation has a constraint that CI/CD variables cannot be used in an include inside generated configuration.

Even if you add one more service, the parent configuration stays as it is

In /root/glci-mono/services/billing/, instead of ci.yml, make only a README.md (any content), then commit and push. Do not touch the parent .gitlab-ci.yml. When you run it, lint-billing should newly appear and run in the dynamic child pipeline. The grader also checks by creating one more service directory in a copy.

The rule of the dynamic pipeline is in the repository structure, so a job appears just because a directory appears. Conversely, it means that if you put a non-service directory under services, that becomes a job too, so you must leave the rule written down as documentation.