Strip comments from a polyglot repo for a public mirror

Steps#

1. Put the scripts in one directory. test-oss-strip-comments.py loads oss-strip-comments.py from its own directory, so the two must sit side by side.

2. Run the trap and control tests.

   $ python3 -I test-oss-strip-comments.py -v
   test_awk (__main__.Controls.test_awk) ... ok
   test_go (__main__.Controls.test_go) ... ok
   test_py (__main__.Controls.test_py) ... ok
   test_sh (__main__.Controls.test_sh) ... ok
   test_xml (__main__.Controls.test_xml) ... ok
   test_yaml (__main__.Controls.test_yaml) ... ok
   test_awk_including_spliced (__main__.Traps.test_awk_including_spliced) ... ok
   test_dockerfile (__main__.Traps.test_dockerfile) ... ok
   test_go (__main__.Traps.test_go) ... ok
   test_java (__main__.Traps.test_java) ... ok
   test_licence_exempt (__main__.Traps.test_licence_exempt) ... ok
   test_properties (__main__.Traps.test_properties) ... ok
   test_python (__main__.Traps.test_python) ... ok
   test_sh (__main__.Traps.test_sh) ... ok
   test_xml (__main__.Traps.test_xml) ... ok
   test_yaml (__main__.Traps.test_yaml) ... ok
   
   ----------------------------------------------------------------------
   Ran 16 tests in 1.062s
   
   OK
   

   Expected: OK, 16 tests. The Traps tests feed inputs where a comment marker is not a comment (${x#y}, $#, heredoc bodies, "http://x", Java text blocks, YAML block scalars, <![CDATA[, Go raw strings, /* in a Dockerfile rm -rf …/lists/*). Each Controls test swaps in a deliberately wrong stripper (renames echo, swaps an import, edits a quoted YAML value, empties a Go raw string, renames an XML element, changes 4 / 2 to 4 / 3 inside awk) and asserts that the oracle stops the run.

3. Prove the controls can fail. A control that passes no matter what proves nothing. blind-oracle.py replaces one oracle with one that always answers "same" and re-runs the controls:

   #!/usr/bin/env python3
   """Control of the controls: blind one oracle (always says "same") and run the
   control tests. The control for that language must now FAIL.
   Usage: blind-oracle.py <test-oss-strip-comments.py> <ext>"""
   import importlib.util, sys, unittest
   spec = importlib.util.spec_from_file_location("t", sys.argv[1])
   t = importlib.util.module_from_spec(spec); spec.loader.exec_module(t)
   ext = sys.argv[2]
   t.s.STRIPPERS[ext] = (t.s.STRIPPERS[ext][0], lambda src, new: True)
   suite = unittest.defaultTestLoader.loadTestsFromTestCase(t.Controls)
   sys.exit(not unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful())
   

   $ python3 -I blind-oracle.py test-oss-strip-comments.py .sh
   test_awk (t.Controls.test_awk) ... ok
   test_go (t.Controls.test_go) ... ok
   test_py (t.Controls.test_py) ... ok
   test_sh (t.Controls.test_sh) ... FAIL
   test_xml (t.Controls.test_xml) ... ok
   test_yaml (t.Controls.test_yaml) ... ok
   
   ======================================================================
   FAIL: test_sh (t.Controls.test_sh)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
   …
   AssertionError: SystemExit not raised
   
   ----------------------------------------------------------------------
   Ran 6 tests in 0.181s
   
   FAILED (failures=1)
   

   Expected: exit status 1; exactly the control for the blinded language fails, the other five pass. Repeat with .py, .yml, .go, .xml; each fails only its own control. test_awk stays ok when .sh is blinded because awk programs have their own oracle.

4. Export one branch into a throwaway tree and keep a pristine copy.

   $ mkdir tree
   $ git -C <SRC_REPO> archive origin/<BRANCH> | tar -x -C tree
   $ cp -a tree pristine
   $ find tree -type f | wc -l
   144
   

   Expected: a file count, no errors. The source repo is only read.

5. Strip.

   $ time python3 -I oss-strip-comments.py tree
     comments stripped from 55 file(s)
   
   real	0m0.363s
   user	0m0.313s
   sys	0m0.135s
   

   Expected: one comments stripped from N file(s) line and no WARN: or FATAL: line. A FATAL: line means an oracle or the unchanged-line net found a change in meaning. The run stops at that file; files before it are already rewritten, so export again before retrying (Failure modes, row 1).

6. Read what changed. The awk program inside lib.sh lost its column notes; only comments went:

   $ diff -u pristine/scripts/bench/lib.sh tree/scripts/bench/lib.sh
   --- pristine/scripts/bench/lib.sh	2026-10-16 18:41:24.000000000 +0300
   +++ tree/scripts/bench/lib.sh	2026-10-16 22:26:50.626119877 +0300
   …
   @@ -132,11 +108,6 @@
      log "Analyzing prometheus CSV..."
      awk -F',' '
      BEGIN {
   -    # columns: 1:ts 2:rss 3:heap_used 4:heap_committed 5:heap_max
   -    # 6:nonheap_used 7:nonheap_committed 8:buf_mem 9:buf_count
   -    # 10:gc_count 11:gc_total 12:cpu_usage 13:cpu_seconds 14:uptime
   -    # 15:threads 16:threads_daemon 17:pool_active 18:pool_idle
   -    # 19:pool_pending 20:pool_total 21:http_count 22:http_dur
      }
      NR>1 {
        n++
   …
   

   On this branch the 55 changed files were 31 .java, 13 .sh, 3 .properties, 3 .yml, 2 Dockerfiles, 1 .go, 1 .xml, 1 .js. All 31 shebangs stayed on line 1. The branch has no Go directives and no Dockerfile parser directives, so those two keep-rules are exercised only by the Step 2 trap tests.

7. Run a naive stripper through the same oracle. This is the control on real files: a plain regex stripper for shell comments, judged by the real bash --pretty-print oracle.

   #!/usr/bin/env python3
   """Control: replace one language's stripper with a naive regex and let the
   real oracle judge each file. Usage: naive.py <oss-strip-comments.py> <ext> <regex> <file>..."""
   import importlib.util, pathlib, re, sys
   spec = importlib.util.spec_from_file_location("strip", sys.argv[1])
   s = importlib.util.module_from_spec(spec); spec.loader.exec_module(s)
   ext, rx = sys.argv[2], re.compile(sys.argv[3], re.M)
   s.STRIPPERS[ext] = (lambda t: rx.sub("", t), s.STRIPPERS[ext][1])
   caught = 0
   for f in map(pathlib.Path, sys.argv[4:]):
       try:
           s.strip_text(f, f.read_text())
           print(f"passed  {f}")
       except SystemExit as e:
           caught += 1
           print(e)
   print(f"{caught} of {len(sys.argv) - 4} caught")
   

   $ python3 -I naive.py oss-strip-comments.py .sh '[ \t]#.*$|^[ \t]*#(?!!).*$' $(/usr/bin/find pristine -name '*.sh' | sort) | sed '/^passed/d'
   FATAL: pristine/scripts/bench/lib.sh changed meaning after stripping
   FATAL: pristine/scripts/prom-scraper.sh changed meaning after stripping
   FATAL: pristine/scripts/report-detailed.sh changed meaning after stripping
   FATAL: pristine/scripts/report-summary.sh changed meaning after stripping
   4 of 30 caught
   

   Expected: at least one FATAL. In report-detailed.sh and report-summary.sh the regex cut CSS colours out of an HTML template (background: #0d1117; color: #c9d1d9; } became background:). Those two are real breaks, and both still pass bash -n, so a syntax check alone would have shipped them. In lib.sh and prom-scraper.sh the regex removed only comments: # lines inside single-quoted awk programs (plus one trailing shell comment in lib.sh). The shell oracle sees the awk program as part of a shell string, so it flags any edit there; those two are cautious catches of comment-only changes, which the real stripper handles with its own gawk oracle.

8. Compile what has no oracle. Java has no equivalence oracle here; a full test-compile of the stripped tree is the check. Build the Go module too.

   $ cp -a tree build
   $ (cd build && mvn -q -o -DskipTests test-compile > ../mvn.log 2>&1; echo "mvn exit=$?")
   mvn exit=0
   $ (cd build/mock-server && go build -o /dev/null . ; echo "go exit=$?")
   go exit=0
   

   Expected: both exit=0. test-compile covers test sources too, which plain compile skips. The copy keeps target/ out of tree/.

9. Repeat for every branch you will publish. strip-all.sh exports, strips, test-compiles and builds each branch:

   #!/usr/bin/env bash
   # usage: strip-all.sh <SRC_REPO> <OUT_DIR> <STRIPPER> <BRANCH>...
   set -u
   src=$1 out=$2 stripper=$3; shift 3
   rc=0
   for b in "$@"; do
     d="$out/${b//\//_}"
     rm -rf "$d"; mkdir -p "$d"
     git -C "$src" archive "origin/$b" | tar -x -C "$d"
     s=$(python3 -I "$stripper" "$d" 2>&1) || { echo "$b STRIP FAILED: $s"; rc=1; continue; }
     ( cd "$d" && mvn -q -o -DskipTests test-compile > "$d.mvn.log" 2>&1 ) && m=OK || { m=FAIL; rc=1; }
     ( cd "$d/mock-server" && go build -o /dev/null . > "$d.go.log" 2>&1 ) && g=OK || { g=FAIL; rc=1; }
     echo "$b |${s#  comments} | mvn=$m go=$g"
   done
   exit $rc
   

   $ time bash strip-all.sh <SRC_REPO> "$PWD/all" oss-strip-comments.py $(git -C <SRC_REPO> for-each-ref --format='%(refname:lstrip=3)' 'refs/remotes/origin/*/*synth*')
   micronaut/imperative-jdbc-synth | stripped from 55 file(s) | mvn=OK go=OK
   micronaut/imperative-synth | stripped from 52 file(s) | mvn=OK go=OK
   micronaut/imperative-synth-more-io | stripped from 55 file(s) | mvn=OK go=OK
   …
   springboot/reactive-r2dbc-synth | stripped from 49 file(s) | mvn=OK go=OK
   springboot/reactive-synth | stripped from 44 file(s) | mvn=OK go=OK
   springboot/reactive-synth-more-io | stripped from 46 file(s) | mvn=OK go=OK
   
   real	1m51.490s
   user	8m36.538s
   sys	0m27.609s
   

   Expected: one line per branch, every one mvn=OK go=OK, no STRIP FAILED, exit status 0. The mock-server path and the branch pattern are specific to my repo; change them for yours.