OTCA — OpenTelemetry Certified Associate

We masked it, yet emails still reached storage

Goal

Send the same six spans through a real Collector, changing only the processor order and pipeline layout, see how the resulting span count, the remaining emails, and the value counted by a connector change, and then complete a production-ready set of pipelines.

Why it matters

A Collector configuration can be syntactically correct and still do something different, silently, if the order is wrong. If a processor that filters on an attribute comes before the processor that creates that attribute, the filter does nothing; personal data removed by key name remains in the values of other attributes; and a receiver attached to several pipelines hands each one its own copy. The result depends on where PII masking and the cost metric sit in the pipeline, so you need both the eye to read a configuration and the habit of actually running data through it to check.

Prepared environment

python3 /opt/fixtures/otca_order_lab.py init places spans.json (6 spans), processors.yaml (a collection of processor definitions), and base.yaml (a baseline configuration with no processors) in /root/otca-order/. python3 /opt/fixtures/otca_order_lab.py run 설정.yaml (the placeholder is the configuration file) actually starts otelcol-contrib 0.116.0 from the lab-k8s image, sends the spans over OTLP/HTTP, summarizes what the file exporter wrote, copies it to /root/otca-order/out/, and shuts the Collector down. To avoid colliding with a Collector you started separately, only the receiver port and the file path are changed in a copy. The processors and their order stay the same. The grader reruns your configuration the same way.

Steps

Create the materials with python3 /opt/fixtures/otca_order_lab.py init, then run python3 /opt/fixtures/otca_order_lab.py run /root/otca-order/base.yaml. A real otelcol-contrib starts, receives the spans in spans.json, and writes them with a file exporter. From the summary, write spans=, spans_with_user_email=, and spans_with_any_email= (the number of spans that have an email in any attribute value) into /root/otca-order/01-baseline.txt.
Create /root/otca-order/filter-first.yaml. Copy the filter/health and transform/route definitions from processors.yaml into base.yaml as they are, and put the traces pipeline processors in the order [filter/health, transform/route] (the file path is out/filter-first.json). Write the run result into /root/otca-order/02-filter-first.txt as spans= and routes= (the routes value from the summary, as is).
Create /root/otca-order/transform-first.yaml from the same two definitions, but change the order to [transform/route, filter/health] (out/transform-first.json). Write spans= and routes= into /root/otca-order/03-transform-first.txt and compare with step 2.
In /root/otca-order/redact.yaml, add attributes/redact after the order from step 3 ([transform/route, filter/health, attributes/redact], out/redact.json). Write spans_with_user_email=, spans_with_any_email=, and leaked_attribute= (the attribute key where the email remained) into /root/otca-order/04-redact.txt.
In /root/otca-order/mask.yaml, add one masking processor after the three processors of redact.yaml. With the transform processor's OTTL replace_all_patterns(attributes, "value", 정규식, "***") (the placeholder is the regular expression), mask anything shaped like an email inside every attribute value. The grader checks that 4 spans remain, that no email appears anywhere, and that the url.query attribute remains with the part before notify= intact.
In /root/otca-order/fanout.yaml, put two pipelines. traces/raw exports to file/raw with no processors, and traces/clean goes through the four processors from step 5 and exports to file/clean; both take the same otlp receiver. Write raw_spans=, clean_spans=, and raw_spans_with_any_email= into /root/otca-order/06-fanout.txt.
In /root/otca-order/count.yaml, put the count connector as an exporter of the traces pipeline (processors [transform/route, filter/health]) and as a receiver of the metrics pipeline. The metrics pipeline exports to a file exporter (for example file/metrics). Write span_count_metric= (the trace.span.count value) and counted_after= (the name of the last processor in the traces pipeline) into /root/otca-order/07-count.txt.
Write /root/otca-order/final.yaml. The traces pipeline starts with memory_limiter and ends with batch, and in between it does path normalization → dropping health checks → removing user.email → masking emails inside values. Spans go to one file exporter, and the trace.span.count counted by the count connector goes through the metrics pipeline to another file exporter. The grader checks with otelcol-contrib validate and an actual run for 4 spans, 0 emails, two kinds of routes, and trace.span.count of 4.

Notes

What determines execution order is the order of the service.pipelines.<이름>.processors list (the placeholder is the pipeline name), not the position of the processor definitions (processors:).
Common mistakes: putting filtering before normalization, thinking that deleting the key is enough to finish removing personal data, and forgetting that the processors before a connector affect the value it counts.
The masking regular expression in this lab is for practice. Real personal data removal needs a separate field list and verification.
Collector configuration · Transforming telemetry

Six spans sent through with no processors

Create the materials with python3 /opt/fixtures/otca_order_lab.py init, then run python3 /opt/fixtures/otca_order_lab.py run /root/otca-order/base.yaml. A real otelcol-contrib starts, receives the spans in spans.json, and writes them with a file exporter. From the summary, write spans=, spans_with_user_email=, and spans_with_any_email= (the number of spans that have an email in any attribute value) into /root/otca-order/01-baseline.txt.

Even without a user.email key, an email can be inside the value of another attribute. Read the per-span attributes below the summary one line at a time.

If you put filtering first

Create /root/otca-order/filter-first.yaml. Copy the filter/health and transform/route definitions from processors.yaml into base.yaml as they are, and put the traces pipeline processors in the order [filter/health, transform/route] (the file path is out/filter-first.json). Write the run result into /root/otca-order/02-filter-first.txt as spans= and routes= (the routes value from the summary, as is).

filter/health looks at the http.route attribute. Who creates that attribute, and when?

If you put normalization first

Create /root/otca-order/transform-first.yaml from the same two definitions, but change the order to [transform/route, filter/health] (out/transform-first.json). Write spans= and routes= into /root/otca-order/03-transform-first.txt and compare with step 2.

A pipeline's processors list is the execution order, not the declaration order. The position of the definition block does not matter.

The key was deleted, but the email remained

In /root/otca-order/redact.yaml, add attributes/redact after the order from step 3 ([transform/route, filter/health, attributes/redact], out/redact.json). Write spans_with_user_email=, spans_with_any_email=, and leaked_attribute= (the attribute key where the email remained) into /root/otca-order/04-redact.txt.

The delete action of the attributes processor deletes by key name. It knows nothing about other keys that have an email mixed into their value.

Keep the attribute and mask only the value

In /root/otca-order/mask.yaml, add one masking processor after the three processors of redact.yaml. With the transform processor's OTTL replace_all_patterns(attributes, "value", 정규식, "***") (the placeholder is the regular expression), mask anything shaped like an email inside every attribute value. The grader checks that 4 spans remain, that no email appears anywhere, and that the url.query attribute remains with the part before notify= intact.

delete removes the whole attribute. To change only part of a value, you need a pattern replacement function. The regular expression goes inside a YAML string, so check the quotation marks.

The same receiver, different pipelines

In /root/otca-order/fanout.yaml, put two pipelines. traces/raw exports to file/raw with no processors, and traces/clean goes through the four processors from step 5 and exports to file/clean; both take the same otlp receiver. Write raw_spans=, clean_spans=, and raw_spans_with_any_email= into /root/otca-order/06-fanout.txt.

A processor changes only the data of the pipeline it belongs to. When a receiver is attached to several pipelines, each pipeline receives the same data separately.

What does the connector count?

In /root/otca-order/count.yaml, put the count connector as an exporter of the traces pipeline (processors [transform/route, filter/health]) and as a receiver of the metrics pipeline. The metrics pipeline exports to a file exporter (for example file/metrics). Write span_count_metric= (the trace.span.count value) and counted_after= (the name of the last processor in the traces pipeline) into /root/otca-order/07-count.txt.

A connector receives data at the end of one pipeline (the exporter position) and passes it to the start of another pipeline (the receiver position). The processors before it have already run.

One set of pipelines for production

Write /root/otca-order/final.yaml. The traces pipeline starts with memory_limiter and ends with batch, and in between it does path normalization → dropping health checks → removing user.email → masking emails inside values. Spans go to one file exporter, and the trace.span.count counted by the count connector goes through the metrics pipeline to another file exporter. The grader checks with otelcol-contrib validate and an actual run for 4 spans, 0 emails, two kinds of routes, and trace.span.count of 4.

memory_limiter only makes sense if it rejects data first when load piles up, and batch must bundle the result after all transformations are done. count must sit after the dropping step for the operational metric to be correct.