Kubernetes Distributions — Build Them Yourself

A k3s with the certification logo failed a conformance test

Goal

On a k3s with its version pinned, you actually pick and run a few of the official Kubernetes conformance tests, and read the JUnit results to tell passed, failed, and timed out apart. You will be able to explain, from the results, what conformance tests check and what they do not.

Why it matters

When choosing a distribution, the "Certified Kubernetes" logo looks like a strong signal. But that certification is the result of checking with one test suite that the required APIs and behavior that are GA are the same as upstream, and the tests have premises such as two or more nodes. Performance, availability, security settings, and optional features are out of scope. To read the logo correctly, you have to see for yourself what the tests consist of and what shape a failure is recorded in. Also, if the versions of the test binary and the server diverge, the results themselves become meaningless, so you begin by matching the versions. Finally, when you deliberately break DNS and see the same test fail and then pass again after recovery, you learn that conformance tests can also be used as a "recovery verification tool."

Steps

Write to /root/conformance/versions.json server (the API server gitVersion), server_minor (in the 1.36 form), e2e_test (the output of e2e.test --version), ginkgo (only the version number of ginkgo version, for example 2.0.0), and same_minor (whether the minor versions of the server and e2e.test are the same, a boolean).
Read /usr/local/conformance/conformance.yaml (the list at the v1.36.4 tag) and write to /root/conformance/catalog.json total (the number of entries), sig_network (the number of entries whose codename starts with [sig-network]), dns_tests (a sorted array of the codenames that start with [sig-network] DNS), and dns_cluster_release (the release value of [sig-network] DNS should provide DNS for the cluster [Conformance]).
Run e2e.test twice with --ginkgo.dry-run and write to /root/conformance/dryrun.json conformance_will_run (the number of specs that will run with the focus \[Conformance\]), total_specs (the total number of specs), dns_will_run (the number that will run with the focus \[sig-network\] DNS.*\[Conformance\]), and matches_catalog (whether conformance_will_run equals the total from step 2, a boolean).
Run with only [sig-network] DNS should provide DNS for the cluster [Conformance] as the focus, leaving the JUnit result in /root/conformance/dns-pass/junit_01.xml (--report-dir) and the standard output and error in /root/conformance/dns-pass/e2e.log. It must pass.
Run [sig-architecture] Conformance Tests should have at least two untainted nodes [Conformance] and leave the results in /root/conformance/two-nodes/junit_01.xml and /root/conformance/two-nodes/e2e.log, and write to /root/conformance/two-nodes.json status (the status of that testcase in the JUnit), reason (among the [FAILED] lines in e2e.log, the sentence of the line that states the reason rather than the source location (in [It] - ...)), and schedulable_nodes (the number of Ready nodes without taints now, a number).
Reduce the coredns Deployment in kube-system to 0 and confirm the Pods are gone, then run the same DNS test as in step 4 with --ginkgo.timeout=45s and leave the results in /root/conformance/dns-broken/junit_01.xml and /root/conformance/dns-broken/e2e.log. Write to /root/conformance/broken.json coredns_replicas (spec.replicas right before the run), coredns_pods (the number of CoreDNS Pods right before the run), and status (the status of that testcase in the JUnit). You recover in the next step.
Return coredns to 1 and make it Available, then run the same DNS test again and leave the results in /root/conformance/dns-restored/junit_01.xml and /root/conformance/dns-restored/e2e.log. It must pass.
Write to /root/conformance/report.json server (the server gitVersion), catalog_total (the total from step 2), passed (a sorted, deduplicated array of the testcase names that passed in steps 4 and 7, without [It] ), failed_single_node (the codename of the test that failed in step 5), timed_out_when_broken (whether the result of step 6 was a timeout, a boolean), submission_files (a sorted array of the four file names that go into a CNCF certification PR), certified_focus (the E2E_FOCUS value required for a certification run, exactly as is), and skip_allowed (whether E2E_SKIP may be given in a certification run, a boolean).

Notes

In the VM there is one k3s v1.36.4+k3s1, and the e2e.test, ginkgo, and conformance.yaml of the same version are in /usr/local/conformance. The test images have been pulled in advance.
The basic form of a run: e2e.test --kubeconfig $KUBECONFIG --provider skeleton --ginkgo.no-color --report-dir <디렉터리> --ginkgo.focus='<정규식>' (the placeholders are the directory and the regular expression)
Common mistake: not escaping the brackets in the focus. They are interpreted as a character set of the regular expression, and hundreds of unrelated tests get picked. Check the count first with --ginkgo.dry-run.
Common mistake: running step 6 without a time limit. It waits 600 seconds for the DNS result.
The full conformance run (446 tests) is not done in this lab. The certification submission of k3s took about 2 hours 54 minutes on a two-machine configuration.
Documentation: cncf/k8s-conformance · instructions.md · Conformance Testing in Kubernetes

Match the versions of the test binary and the server

Write to /root/conformance/versions.json server (the API server gitVersion), server_minor (in the 1.36 form), e2e_test (the output of e2e.test --version), ginkgo (only the version number of ginkgo version, for example 2.0.0), and same_minor (whether the minor versions of the server and e2e.test are the same, a boolean).

The test binaries are in /usr/local/conformance. The k3s version has a suffix such as +k3s1. The conformance test list differs from version to version, so you must use tests built from the same release branch as the cluster version for the results to have meaning.

Read the conformance test list

Read /usr/local/conformance/conformance.yaml (the list at the v1.36.4 tag) and write to /root/conformance/catalog.json total (the number of entries), sig_network (the number of entries whose codename starts with [sig-network]), dns_tests (a sorted array of the codenames that start with [sig-network] DNS), and dns_cluster_release (the release value of [sig-network] DNS should provide DNS for the cluster [Conformance]).

It is YAML, so some entries have a codename folded across several lines. Counting lines with grep gives the wrong answer, so read it with Python's yaml module. Each entry has the keys testname, codename, description, release, and file. The release is the version in which that test entered conformance.

Count the scope before running

Run e2e.test twice with --ginkgo.dry-run and write to /root/conformance/dryrun.json conformance_will_run (the number of specs that will run with the focus \[Conformance\]), total_specs (the total number of specs), dns_will_run (the number that will run with the focus \[sig-network\] DNS.*\[Conformance\]), and matches_catalog (whether conformance_will_run equals the total from step 2, a boolean).

A dry-run creates nothing in the cluster and only shows which specs are picked. Read the Will run N of M specs line of the output. If you give --ginkgo.no-color, color codes are not mixed in.

Really run one conformance test

Run with only [sig-network] DNS should provide DNS for the cluster [Conformance] as the focus, leaving the JUnit result in /root/conformance/dns-pass/junit_01.xml (--report-dir) and the standard output and error in /root/conformance/dns-pass/e2e.log. It must pass.

The focus is a regular expression, so you must escape the brackets. If you write only part of the name, tests with similar names may be picked too, so first confirm with a dry-run that it is 1. The JUnit also includes all the skipped specs, as skipped.

A conformance test that a single-machine cluster fails

Run [sig-architecture] Conformance Tests should have at least two untainted nodes [Conformance] and leave the results in /root/conformance/two-nodes/junit_01.xml and /root/conformance/two-nodes/e2e.log, and write to /root/conformance/two-nodes.json status (the status of that testcase in the JUnit), reason (among the [FAILED] lines in e2e.log, the sentence of the line that states the reason rather than the source location (in [It] - ...)), and schedulable_nodes (the number of Ready nodes without taints now, a number).

It is normal for this test to fail. Compare it with the configuration in which k3s ran the tests in its conformance submission (the README in cncf/k8s-conformance). In the JUnit, look at the failure child element and the status attribute together.

Reduce CoreDNS and run the same test

Reduce the coredns Deployment in kube-system to 0 and confirm the Pods are gone, then run the same DNS test as in step 4 with --ginkgo.timeout=45s and leave the results in /root/conformance/dns-broken/junit_01.xml and /root/conformance/dns-broken/e2e.log. Write to /root/conformance/broken.json coredns_replicas (spec.replicas right before the run), coredns_pods (the number of CoreDNS Pods right before the run), and status (the status of that testcase in the JUnit). You recover in the next step.

This test waits 600 seconds for the DNS lookup result, so if you run it without a time limit it sits frozen for 10 minutes. When the suite limit ends, ginkgo records a timeout, not a failure. Look in e2e.log for which name's lookup failed.

Revert it and prove it with the same test

Return coredns to 1 and make it Available, then run the same DNS test again and leave the results in /root/conformance/dns-restored/junit_01.xml and /root/conformance/dns-restored/e2e.log. It must pass.

The surest way to confirm recovery is to make the very test that failed when you broke it pass again. Even if the namespace from the previous run remains in Terminating, the test creates a new namespace.

Interpret the results in the words of conformance

Write to /root/conformance/report.json server (the server gitVersion), catalog_total (the total from step 2), passed (a sorted, deduplicated array of the testcase names that passed in steps 4 and 7, without [It] ), failed_single_node (the codename of the test that failed in step 5), timed_out_when_broken (whether the result of step 6 was a timeout, a boolean), submission_files (a sorted array of the four file names that go into a CNCF certification PR), certified_focus (the E2E_FOCUS value required for a certification run, exactly as is), and skip_allowed (whether E2E_SKIP may be given in a certification run, a boolean).

Reread the names and statuses from the JUnit files of earlier steps. The submission files and the focus and skip rules are in instructions.md of cncf/k8s-conformance.