Software / code / prosody
Annotate
tools/test_mutants.sh.lua @ 13801:a5d5fefb8b68 13.0
mod_tls: Enable Prosody's certificate checking for incoming s2s connections (fixes #1916) (thanks Damian, Zash)
Various options in Prosody allow control over the behaviour of the certificate
verification process For example, some deployments choose to allow falling
back to traditional "dialback" authentication (XEP-0220), while others verify
via DANE, hard-coded fingerprints, or other custom plugins.
Implementing this flexibility requires us to override OpenSSL's default
certificate verification, to allow Prosody to verify the certificate itself,
apply custom policies and make decisions based on the outcome.
To enable our custom logic, we have to suppress OpenSSL's default behaviour of
aborting the connection with a TLS alert message. With LuaSec, this can be
achieved by using the verifyext "lsec_continue" flag.
We also need to use the lsec_ignore_purpose flag, because XMPP s2s uses server
certificates as "client" certificates (for mutual TLS verification in outgoing
s2s connections).
Commit 99d2100d2918 moved these settings out of the defaults and into mod_s2s,
because we only really need these changes for s2s, and they should be opt-in,
rather than automatically applied to all TLS services we offer.
That commit was incomplete, because it only added the flags for incoming
direct TLS connections. StartTLS connections are handled by mod_tls, which was
not applying the lsec_* flags. It previously worked because they were already
in the defaults.
This resulted in incoming s2s connections with "invalid" certificates being
aborted early by OpenSSL, even if settings such as `s2s_secure_auth = false`
or DANE were present in the config.
Outgoing s2s connections inherit verify "none" from the defaults, which means
OpenSSL will receive the cert but will not terminate the connection when it is
deemed invalid. This means we don't need lsec_continue there, and we also
don't need lsec_ignore_purpose (because the remote peer is a "server").
Wondering why we can't just use verify "none" for incoming s2s? It's because
in that mode, OpenSSL won't request a certificate from the peer for incoming
connections. Setting verify "peer" is how you ask OpenSSL to request a
certificate from the client, but also what triggers its built-in verification.
| author | Matthew Wild <mwild1@gmail.com> |
|---|---|
| date | Tue, 01 Apr 2025 17:26:56 +0100 |
| parent | 13432:116b756ced71 |
| rev | line source |
|---|---|
|
12765
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
1 #!/bin/bash |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
2 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
3 POLYGLOT=1--[===[ |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
4 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
5 set -o pipefail |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
6 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
7 if [[ "$#" == "0" ]]; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
8 echo "Lua mutation testing tool" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
9 echo |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
10 echo "Usage:" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
11 echo " $BASH_SOURCE MODULE_NAME SPEC_FILE" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
12 echo |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
13 echo "Requires 'lua', 'ltokenp' and 'busted' in PATH" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
14 exit 1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
15 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
16 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
17 MOD_NAME="$1" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
18 MOD_FILE="$(lua "$BASH_SOURCE" resolve "$MOD_NAME")" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
19 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
20 if [[ "$MOD_FILE" == "" || ! -f "$MOD_FILE" ]]; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
21 echo "EE: Failed to locate module '$MOD_NAME' ($MOD_FILE)"; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
22 exit 1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
23 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
24 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
25 SPEC_FILE="$2" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
26 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
27 if [[ "$SPEC_FILE" == "" ]]; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
28 SPEC_FILE="spec/${MOD_NAME/./_}_spec.lua" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
29 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
30 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
31 if [[ "$SPEC_FILE" == "" || ! -f "$SPEC_FILE" ]]; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
32 echo "EE: Failed to find test spec file ($SPEC_FILE)" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
33 exit 1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
34 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
35 |
|
13432
116b756ced71
tools/test_mutants.sh: Load loader helper when running busted
Matthew Wild <mwild1@gmail.com>
parents:
12765
diff
changeset
|
36 if ! busted --helper=loader "$SPEC_FILE"; then |
|
12765
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
37 echo "EE: Tests fail on original source. Fix it"\!; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
38 exit 1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
39 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
40 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
41 export MUTANT_N=0 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
42 LIVING_MUTANTS=0 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
43 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
44 FILE_PREFIX="${MOD_FILE%.*}.mutant-" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
45 FILE_SUFFIX=".${MOD_FILE##*.}" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
46 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
47 gen_mutant () { |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
48 echo "Generating mutant $2 to $3..." |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
49 ltokenp -s "$BASH_SOURCE" "$1" > "$3" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
50 return "$?" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
51 } |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
52 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
53 # $1 = MOD_NAME, $2 = MUTANT_N, $3 = SPEC_FILE |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
54 test_mutant () { |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
55 ( |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
56 ulimit -m 131072 # 128MB |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
57 ulimit -t 16 # 16s |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
58 ulimit -f 32768 # 128MB (?) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
59 exec busted --helper="$BASH_SOURCE" -Xhelper mutate="$1":"$2" "$3" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
60 ) >/dev/null |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
61 return "$?"; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
62 } |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
63 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
64 MUTANT_FILE="${FILE_PREFIX}${MUTANT_N}${FILE_SUFFIX}" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
65 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
66 gen_mutant "$MOD_FILE" "$MUTANT_N" "$MUTANT_FILE" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
67 while [[ "$?" == "0" ]]; do |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
68 if ! test_mutant "$MOD_NAME" "$MUTANT_N" "$SPEC_FILE"; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
69 echo "Tests successfully killed mutant $MUTANT_N"; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
70 rm "$MUTANT_FILE"; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
71 else |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
72 echo "Mutant $MUTANT_N lives on"\! |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
73 LIVING_MUTANTS=$((LIVING_MUTANTS+1)) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
74 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
75 MUTANT_N=$((MUTANT_N+1)) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
76 MUTANT_FILE="${FILE_PREFIX}${MUTANT_N}${FILE_SUFFIX}" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
77 gen_mutant "$MOD_FILE" "$MUTANT_N" "$MUTANT_FILE" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
78 done |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
79 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
80 if [[ "$?" != "2" ]]; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
81 echo "Failed: $?" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
82 exit "$?"; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
83 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
84 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
85 MUTANT_SCORE="$(lua -e "print(('%0.2f'):format((1-($LIVING_MUTANTS/$MUTANT_N))*100))")" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
86 if test -f mutant-scores.txt; then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
87 echo "$MOD_NAME $MUTANT_SCORE" >> mutant-scores.txt |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
88 fi |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
89 echo "$MOD_NAME: All $MUTANT_N mutants generated, $LIVING_MUTANTS survived (score: $MUTANT_SCORE%)" |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
90 rm "$MUTANT_FILE"; # Last file is always unmodified |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
91 exit 0; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
92 ]===] |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
93 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
94 -- busted helper that runs mutations |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
95 if arg then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
96 if arg[1] == "resolve" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
97 local filename = package.searchpath(assert(arg[2], "no module name given"), package.path); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
98 if filename then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
99 print(filename); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
100 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
101 os.exit(filename and 0 or 1); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
102 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
103 local mutants = {}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
104 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
105 for i = 1, #arg do |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
106 local opt = arg[i]; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
107 print("LOAD", i, opt) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
108 local module_name, mutant_n = opt:match("^mutate=([^:]+):(%d+)"); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
109 if module_name then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
110 mutants[module_name] = tonumber(mutant_n); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
111 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
112 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
113 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
114 local orig_lua_searcher = package.searchers[2]; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
115 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
116 local function mutant_searcher(module_name) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
117 local mutant_n = mutants[module_name]; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
118 if not mutant_n then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
119 return orig_lua_searcher(module_name); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
120 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
121 local base_file, err = package.searchpath(module_name, package.path); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
122 if not base_file then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
123 return base_file, err; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
124 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
125 local mutant_file = base_file:gsub("%.lua$", (".mutant-%d.lua"):format(mutant_n)); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
126 return loadfile(mutant_file), mutant_file; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
127 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
128 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
129 if next(mutants) then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
130 table.insert(package.searchers, 1, mutant_searcher); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
131 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
132 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
133 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
134 -- filter for ltokenp to mutate scripts |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
135 do |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
136 local last_output = {}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
137 local function emit(...) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
138 last_output = {...}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
139 io.write(...) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
140 io.write(" ") |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
141 return true; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
142 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
143 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
144 local did_mutate = false; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
145 local count = -1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
146 local threshold = tonumber(os.getenv("MUTANT_N")) or 0; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
147 local function should_mutate() |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
148 count = count + 1; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
149 return count == threshold; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
150 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
151 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
152 local function mutate(name, value) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
153 if name == "if" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
154 -- Bypass conditionals |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
155 if should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
156 return emit("if true or"); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
157 elseif should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
158 return emit("if false and"); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
159 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
160 elseif name == "<integer>" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
161 -- Introduce off-by-one errors |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
162 if should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
163 return emit(("%d"):format(tonumber(value)+1)); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
164 elseif should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
165 return emit(("%d"):format(tonumber(value)-1)); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
166 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
167 elseif name == "and" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
168 if should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
169 return emit("or"); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
170 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
171 elseif name == "or" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
172 if should_mutate() then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
173 return emit("and"); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
174 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
175 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
176 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
177 |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
178 local current_line_n, current_line_input, current_line_output = 0, {}, {}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
179 function FILTER(line_n,token,name,value) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
180 if current_line_n ~= line_n then -- Finished a line, moving to the next? |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
181 if did_mutate and did_mutate.line == current_line_n then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
182 -- The line we finished was mutated. Store the original and modified outputs. |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
183 did_mutate.line_original_src = table.concat(current_line_input, " "); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
184 did_mutate.line_modified_src = table.concat(current_line_output, " "); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
185 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
186 current_line_input = {}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
187 current_line_output = {}; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
188 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
189 current_line_n = line_n; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
190 if name == "<file>" then return; end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
191 if name == "<eof>" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
192 if not did_mutate then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
193 return os.exit(2); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
194 else |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
195 emit(("\n-- Mutated line %d (changed '%s' to '%s'):\n"):format(did_mutate.line, did_mutate.original, did_mutate.modified)) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
196 emit( ("-- Original: %s\n"):format(did_mutate.line_original_src)) |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
197 emit( ("-- Modified: %s\n"):format(did_mutate.line_modified_src)); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
198 return; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
199 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
200 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
201 if name == "<string>" then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
202 value = string.format("%q",value); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
203 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
204 if mutate(name, value) then |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
205 did_mutate = { |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
206 original = value; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
207 modified = table.concat(last_output); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
208 line = line_n; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
209 }; |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
210 else |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
211 emit(value); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
212 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
213 table.insert(current_line_input, value); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
214 table.insert(current_line_output, table.concat(last_output)); |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
215 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
216 end |
|
132a3c7b25fa
tools: Add initial mutation testing script
Matthew Wild <mwild1@gmail.com>
parents:
diff
changeset
|
217 |