TT Lab
Get started
Learn Learning paths Courses

In Front of an Unfamiliar System

From 200 lines and 12 steps down to two lines and four steps

Continue in TT Lab

Goal

You shrink the incident bundle the customer sent (one processor, a 200-line feed, and a 12-step reproduction procedure) into a minimal reproducible case. First you make a machine judge "is it still the same failure?", shrink using only that judge, and go as far as proving that removing any one of what is left stops the reproduction.

Why it matters

The reproduction procedure the customer wrote is usually everything that person did that day. If you look for the cause while carrying all of that, the combinations you have to look at explode. Even so, starting with shrinking is more dangerous — you remove one line and the program dies with a different error, and since that is also an error, you mistake it for still reproducing. A case produced that way reproduces some other defect that has nothing to do with the defect we were chasing. So there is an order. You build the judge first, and shrink using only that judge. The judge looks together at the exit code and at a signature string that appears only in that failure. Values that differ from run to run, such as the execution time or temporary file names, do not go into the judgment. A machine does the shrinking. If you remove one line at a time, you call the judge as many times as there are lines, but delta debugging (ddmin), which splits the candidate into chunks, brings that count close to the logarithm of the number of lines. The grader does not trust the numbers you write down. It feeds several candidates it built into your oracle, and gives your shrinker a 256-line input with the answer hidden in a different place each time, measuring the result and the number of judge calls together.

Steps

  1. Create and run /root/mre/gen_case.py to unpack /root/mre/app/ingest.py, /root/mre/input.txt (200 lines), and /root/mre/repro/steps.txt (12 steps).
  2. Create /root/mre/oracle.py. If the candidate received through --input produces the failure we are chasing, the exit code is 0, and otherwise 1. A different error is not a reproduction.
  3. Shrink the feed with the oracle to produce /root/mre/min_input.txt. The remaining lines are lines of the original and keep the original order.
  4. In /root/mre/minimality.json, write for each remaining line whether it still reproduces when that line is removed, to prove 1-minimality.
  5. Implement delta debugging (ddmin) in /root/mre/shrink.py. It takes --input --oracle --out, and the number of judge calls must not be proportional to the number of lines.
  6. Shrink the 12-step reproduction procedure to produce /root/mre/min_steps.txt. The remaining steps are lines of the original exactly as they are and keep the order.
  7. Check whether it still reproduces when you change the values, and leave /root/mre/generalize.json and the set of variant files.
  8. Report in four sections in /root/mre/mre_report.md.

Notes

Unpack the incident bundle

Create and run /root/mre/gen_case.py to produce /root/mre/app/ingest.py, /root/mre/input.txt (200 lines), and /root/mre/repro/steps.txt (12 steps).

What you receive on site is only three things: the program, the data, and the procedure. After unpacking, run the procedure once as it is and see the failure with your own eyes — set MRE_WORK to a directory that does not yet exist and run it.

Make a machine judge whether it is the same failure

Create /root/mre/oracle.py. If the candidate received through --input produces the failure we are chasing, the exit code is 0, and otherwise 1. The oracle creates the configuration file itself, and looks together at the exit code and the signature string.

If you treat every nonzero exit code as a reproduction, you will not notice when the shrinking drifts to a completely different error. This processor ends with 2 if it cannot read the configuration, 3 if it is the defect we are chasing, and 4 if there is not a single qty line. Look at the signature string on standard error together with it.

Shrink the feed

Shrink the feed with the oracle to produce /root/mre/min_input.txt. Every remaining line must be a line of the original input.txt and must keep the original order, and removing any one line must stop the reproduction.

You may do this step by hand. The simplest method is to remove one line at a time and keep only the lines for which the reproduction holds. You end up calling the judge about 200 times, and that slowness is why you build ddmin in step 5.

Prove why every remaining line is needed

In /root/mre/minimality.json, write for each line of min_input.txt whether it still reproduces when that line is removed. If the reproduction stops for every line, minimal is true. lines is in exactly the order of min_input.txt.

1-minimal means 'removing any one of what is left stops the reproduction'. Without this proof you cannot answer the question 'why is this line needed?', and the receiving side has to simply trust you. Leave the judging to the oracle and write down only the results.

Automate the shrinking

Implement delta debugging (ddmin) in /root/mre/shrink.py. It takes --input --oracle --out, shrinks after first checking that the original reproduces, and the number of judge calls must not be proportional to the number of lines.

The remove-one-line method calls the judge as many times as there are lines. ddmin splits the candidate into n pieces and first checks whether one piece alone reproduces it, and if not, checks whether the rest with one piece removed reproduces it. If neither does, it splits more finely and repeats. The grader runs your shrinker on a 256-line input and counts how many times you called the judge.

Shrink the reproduction procedure too

Shrink the 12-step reproduction procedure to produce /root/mre/min_steps.txt. The remaining steps are lines of steps.txt exactly as they are and must keep the original order, and removing any one step must stop the reproduction.

Each time you test the procedure, take a new MRE_WORK as a directory that does not yet exist. If there is a configuration file left by the previous run, it still reproduces even when you remove the 'step that creates the configuration', and you reach the wrong conclusion that the step is unnecessary. The reproduction judgment is whether the signature string appears on standard error.

Check whether it is structure, not values

From the minimal case, make a variant with changed values and variants with changed structure, and write them in /root/mre/generalize.json. There are three variants; /root/mre/min_input_generic.txt, in which only the values are changed, must reproduce, and the other two must not reproduce. Actually leave all three files.

This is the step that separates whether the remaining two lines are due to an accidental value or due to structure. If it reproduces even when you change a number to another number, the cause is not that number. Conversely, if you show that it stops when you change the comma to a period or change the locale, that proves the cause is the combination of the two conditions.

Write down what you called a reproduction

In /root/mre/mre_report.md, write four sections: ## 무엇을 재현이라고 불렀나 ## 얼마나 줄였나 ## 남은 줄이 왜 다 필요한가 ## 남은 가정 (the Korean headings mean "What we called a reproduction", "How much we shrank it", "Why every remaining line is needed", and "The remaining assumptions"). The number of lines and steps of the original and of the minimal case, and the exit code and the signature string the oracle looks at, must all appear.

From just two input lines, the receiving side starts by asking 'why is this a problem?' First write what you counted as the same failure, then write how much you shrank it and why you cannot shrink it further. In the last section, write the fact that it reproduces even when you change the values, and what stops it when changed.