The Tigress Test System
To be able to test Tigress thorough across platforms, we have built a testing framework called tigress-test.
The program tigress_test2.exe
runs tests of Tigress transformations that are described in a test cases JSON file and outputs results to a results JSON file and brief results to the console.
This page aims to be a comprehensive guide of configuring & running tigress-test
on Linux and macOS, and any non platform-specific steps on Windows and Android.
tigress-test
Installing dependencies on Debian-based Linux distributions:
# to install OCaml and dune
sudo apt install -y ocaml ocaml-dune opam
Installing dependencies on MacOS using homebrew:
brew install ocaml dune opam
Note that tigress-test
requires ocaml >= 4.14
. Before you proceed , make sure to check your OCaml version with
ocaml --version
Once you have the correct OCaml version, you may proceed to initialize the opam package manager. Optionally, add -c <switch-version> to initialize opam with the ocaml version you want. As of the time of writing, the latest OCaml stable version is 5.2.0. Also, install dependencies:
opam init
opam install -y yojson atdgen re logs sha ppx_deriving ppx_enumerate
tigress-test
Change your working directory to $TIGRESS_TEST_HOME/src
, and run
dune build
Environment variables:
Add the following line to your ~/.bashrc
file, with the appropriate path to your tigress-test
directory:
export TIGRESS_TEST_HOME=/path/to/your/tigress/test/directory
Then, restart your terminal or update your current shell environment with:
source ~/.bashrc
.tigress-test-config
. It should look like so:
"tigress_home": "/usr/local/bin/tigresspkg/"
tigress-test
Once tigress-test
has been built, its executable should be placed under $TIGRESS_TEST_HOME/src/_build/default/tigress_test2/tigress_test2.exe
.
We suggest creating a link to the tigress-test
executable in your TIGRESS_TEST_HOME
directory. Cd to TIGRESS_TEST_HOME
, then run the following:
ln -s src/_build/default/tigress_test2/tigress_test2.exe .
tigress-test
supports two different testing modes: script mode and ad-hoc mode.
Script mode: Run the Tigress Test program with a command like the following:
./tigress_test2.exe run script_01.json
Replace the script file argument as necessary.
The results are output to a file named: results_
. It contains the result and a duplicate of the test case information, which would be used in a future version to reproduce the test.
Ad-hoc mode:
In this mode, tigress-test
receives transformation options directly on the command line, in a somewhat similar fashion to running Tigress manually. Some notable flags include:
tigress_test2.exe
run [options] |
rerun [results-file] |
continue [results-file] |
continue-queue [options] |
testcases-run [options] |
worker-run [options] |
adhoc [options] |
status [options] |
stop [options] |
help [topic]
run
Command
--path-to-c-compiler
|
Location of C compiler to use; Either this must be provided or the path must be provided in the configuration file found in a conventional location: ~/.tigress-test/config or ~/.tigress-test-config or .tigress-test-config |
--No-Config-File
|
Don't use a config file. |
--config-file
|
Name of the configuration file to use, overriding the default |
--parallel-processes
|
Number of parallel processes for testing on the local machine; if this is set, the program will also try to run tests in parallel on remote machines if they are configured. |
--parallel-run
|
The program will try to run tests in parallel on remote machines if they are configured and on the local machine depending on the setting of "--remote-hosts" and "--remote-only". |
--generate-only
|
Generate test cases but do not run the tests. |
--remote-only
|
Run test cases only on remote machines, not on the local machine. |
--remote-hosts
|
Run test cases for the hosts with the host names/addresses given in a comma separated list. |
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
-help
|
Display this list of options |
--help
|
Display this list of options |
rerun
Command
--test-numbers
|
A comma separated list of test numbers or test number ranges to rerun; ranges are separated with hyphens, e.g., 0 - 10. |
--failed
|
Re-run only all failed tests and any tests specified using the --Test-Numbers option |
--enqueue
|
Put test-cases to rerun in the queue, so that they can be monitored with "status" subcommand or restarted with "continue-queue" < TABLE-COL-END |
--all-remotes
|
Rerun the most recent tests on all remote hosts listed in the remote hosts file configured in the configuration file. |
--remote-hosts
|
Rerun the most recent tests on the hosts with the host names/addresses given in a comma separated list. |
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
-help
|
Display this list of options |
--help
|
Display this list of options |
continue
Command
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
-help
|
Display this list of options |
--help
|
Display this list of options |
addhoc
Command
--path-to-c-compiler
|
Location of C compiler to use; Either this must be provided or the path must be provided in the configuration file found in a conventional location: ~/.tigress-test/config or ~/.tigress-test-config or .tigress-test-config |
--no-config-file
|
Don't use a config file. |
--config-file
|
Name of the configuration file to use, overriding the default |
--transform
|
Name of a transformation to be performed in the transformation pipeline. This option may be provided more than once, and the transformations will be performed in the order in which they are provided. |
--transform-option =<name,value>
|
An option to the last transformation that was specified with a --Transform option. This may appear multiple times. |
--source-file
|
Path to a file to transform. Wildcards may be included to specify that multiple files should be transformed. |
--seeds
|
A comma separated list of seeds to pass to Tigress. |
--input
|
Input to pass to the C program. |
--
|
Arguments to pass to the C program. |
--save
|
Filename to use to save the test_cases as a script to run again later. |
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
-help
|
Display this list of options |
--help
|
Display this list of options |
testcases-run
Command
--path-to-c-compiler
|
Location of C compiler to use; Either this must be provided or the path must be provided in the configuration file found in a conventional location: ~/.tigress-test/config or ~/.tigress-test-config or .tigress-test-config |
--No-Config-File
|
Don't use a config file. |
--config-file
|
Name of the configuration file to use, overriding the default |
--parallel-processes
|
Number of parallel processes for testing on the local machine; if this is set, the program will also try to run tests in parallel on remote machines if they are configured. |
--parallel-run
|
The program will try to run tests in parallel on remote machines if they are configured and on the local machine depending on the setting of "--remote-hosts" and "--remote-only" .
|
--generate-only
|
Generate test cases but do not run the tests. |
--remote-only
|
Run test cases only on remote machines, not on the local machine. |
--remote-hosts
|
Run test cases for the hosts with the host names/addresses given in a comma separated list. |
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
--test-number-start
|
Test number to start running tests from. |
--gen-spec-index
|
Index of transformations generator specification within test_cases (generator) file to use (for running parallel test from a single test cases generator file). |
-help
|
Display this list of options |
--help
|
Display this list of options |
worker-run
Command
--test-number-start
|
Test number to start running tests from. |
--path-to-c-compiler
|
Location of C compiler to use; Either this must be provided or the path must be provided in the configuration file found in a conventional location: ~/.tigress-test/config or ~/.tigress-test-config or .tigress-test-config |
--No-Config-File
|
Don't use a config file. |
--continue-queue
|
Interpret any filenames given as files to continue from, and also continue running files found in the queue directory. |
--gen-spec-index
|
Index of transformations generator specification within test_cases (generator) file to use (for running parallel test from a single test cases generator file). |
--verbosity
|
A verbosity level between 0 and 5 for log and console output: 0 - no log or console output; 1 - output in brief; 2 - include error messages as well; 3 - include warning messages as well; 4 - include verbose information; 5 - include verbose debugging information |
-help
|
Display this list of options |
--help
|
Display this list of options |
status
Command
Coming Soon
stop
Command
Coming Soon
script-file
Coming Soon
results-file
Title: test results file
Type: object
Description: file containing information about each test run, especially the result of testing each test case
Properties:
file_type (required):
Title: file type
Type: string
Description: string naming the type of file to distinguish it from other JSON files used by the program; this field always has the value "results"
file_format_version (required):
Title: file format version
Type: array
Purpose: distinguish formats of the same file type that may be incompatible
Description: version for this file, as a list of numbers; version 1.0.1 would be [ 1, 0, 1 ]; the first number is interpretted as "major" version and the last number as "revision"
Usage: increment one of the numbers in the list to indicate a non-trivial change to the file format
Items:
Type: int
framework_build_identifier (required):
Title: framework build identifier
Type: string
Purpose: information and debugging
Description: identifies the build of the framework that produced this file, which is an automatically generated value; if I forget to update the version number, I can still determine which source version produced the executable that produced this results file
creation_time (required):
Title: creation time
Type: float
Purpose: enables searching for results by time, so as to find previous results for the same test cases
Description: time since the epoch when file was created
Usage: field may be set using "Unix.time ()"
test_cases_file (required):
Title: test cases file
Type: object
Description: information uniquely describing the test cases file containing the test cases used in the test run that produced the results in this file
Properties:
filename (required):
Title: filename
Type: string
Description: full (absolute) path to the test cases file
modified_time (required):
Title: modified time
Type: float
Description: the modification time of the file, recorded when the file was used by the test program for producing the current results
checksum (required):
Title: checksum
Type: string
Purpose: uniquely identify the test cases file
Description: SHA-256 hash of the file
gen_spec_index (optional):
Title: gen_spec_index
Type: int
Purpose: identify which element of the list of transformation specifications was used to generate the set of test cases from which these results were produced
Description: Index into the list of generator specifications for transformations to use in creating test cases that were run to produce the results file output
results (required):
Title: results
Type: array
Description: list of results of running test cases
Usage: One result object is added to this list after each test case has been run. Before running any test case, a "null" is written into this list as a placeholder when the file is initialized.
Items:
Type: object
Properties:
status (required):
Title: status
Type: string
Description: whether the test was completed (an incomplete test implies a result of "FAIL", but a completed test may be either a "PASS" or a "FAIL"
result (required):
Title: result
Type: string
Description: whether the test passed or failed
start_time (optional):
Title: start time
Type: float
Purpose: keep tract of how long a test program takes to execute
Description: time since the epoch when the obfuscated program was invoked to execute
end_time (optional):
Title: end time
Type: float
Description: time since the epoch when the obfuscated program exited or was terminated and that state was registered by the calling process
elapsed_time (optional):
Title: elapsed time for obfuscated program
Type: float
Description: time elapsed while the calling process was waiting on the obfuscated program to produce output
elapsed_time_orig (optional):
Title: elapsed time for original program
Type: float
Description: time elapsed while the calling process was waiting on the original program to produce output
elapsed_time_delta_percent (optional):
Title: elapsed time delta from previous result (percent)
Type: variant
Description: difference between the elapsed time for running the obfuscated program and that same measure made in the most recent previous run, as a percent (i.e., (new - old)/old * 100.0)
Variants:
DELTA_PERCENT of:
Type: float
MISSING_VALUE
DIVIDE_BY_ZERO
test_case (required):
Title: test case
Type: object
Description: all of the information needed to run one test case
Properties:
compiler (optional):
Title: compiler
Type: object
Description: full path to the compiler to use to compiler the source code
Properties:
filename (required):
Title: filename
Type: string
Description: path to the program
version (optional):
Title: version
Type: string
Description: version of the program
version_command (optional):
Title: version command
Type: string
Description: a commmand to run to cause the program to output its version string, which could be compared with the contents of the "version" field
compiler_options (optional):
Title: compiler options
Type: string
Description: command line options that should alwasy be passed to the compiler
tigress (optional):
Title: Tigress executable
Type: object
Description: full path to the Tigress executable to use
Properties:
filename (required):
Title: filename
Type: string
Description: path to the program
version (optional):
Title: version
Type: string
Description: version of the program
version_command (optional):
Title: version command
Type: string
Description: a commmand to run to cause the program to output its version string, which could be compared with the contents of the "version" field
tigress_options (optional):
Title: Tigress options
Type: string
Description: command line options that should always be passed to Tigrees (in addition to options for the transformations)
max_run_seconds (required):
Title: max run seconds
Type: int
Description: number of seconds after which to time out and terminate the process running the test for any given test case
test_number (optional):
Title: test number
Type: int
Description: a unique number (within this test run) identifying the test case; generally it will start from one and increment by one
seed (optional):
Title: seed for random number generator
Type: int
Description: seed to pass to Tigress
transformations (optional):
Title:
Type: array
Description:
Items:
Title: transformations
Type: object
Properties:
transform_name (required):
Title: transformation name
Type: string
Description: name of a transformation (which gets passed to Tigress in a --Transform= option)
transform_options (required):
Title: transformation options
Type: array
Items:
Type: object
Properties:
option_name (required):
Title: option name
Type: string
Description: name of the option; appears before the equals sign in a name/value pair on the Tigress command line
option_value (optional):
Title: option value
Type: string
Description: value for the option; appears after the equals sign in a name/value pair on the Tigress command line
source_file (optional):
Title: source file
Type: object
Description: description of the source file containing the code to transform for this test case
Properties:
filename (required):
Title: filename
Type: string
Description: path to the source code file
checksum (optional):
Title: checksum
Type: string
Purpose: uniquely identify the source code
Description: SHA-256 hash of the file
args (optional):
Title: args
Type: array
Description: arguments to pass to the program when executed
Items:
Type: string
input (optional):
Title: input
Type: string
Description: data to pass to the program through its standard input when it is executed
functions (optional):
Title: functions
Type: array
Description: list of all of the function in the source code or of all of the function which should be considered for transformation
Items:
Type: string
tigress_output (optional):
Title: standard output from Tigress
Type: string
Description: complete contents of the standard output from running Tigress for the test case that produced this result
tigress_error (optional):
Title: standard error from Tigress
Type: string
Description: complete contents of the standard error from running Tigress for the test case that produced this result
compiler_output (optional):
Title: standard output from compiler, compiling obfuscated program
Type: string
Description: complete contents of the standard output from running the compiler to compile the obfuscated program
compiler_error (optional):
Title: standard error from compiler, compiling obfuscated program
Type: string
Description: complete contents of the standard error from running the compiler to compile the obfuscated program
compiler_output_orig (optional):
Title: standard output from compiler, compiling original program
Type: string
Description: complete contents of the standard output from running the compiler to compile the original program
compiler_error_orig (optional):
Title: standard error from compiler, compiling original program
Type: string
Description: complete contents of the standard error from running the compiler to compile the original program
halstead_before (optional):
Title: Halstead metric before obfuscation
Type: string
Description: the output of the SoftwareMetric transformation's Halstead metric for the functions which will be obfuscated
halstead_after (optional):
Title: Halstead metric after obfuscation
Type: string
Description: the output of the SoftwareMetric transformation's Halstead metric for the functions which were obfuscated
halstead_function_length_greatest_difference (optional):
Title: greatest difference between before and after Halstead function lengths
Type: int
Purpose: detect change produced by the obfuscating transformations (e.g., make sure the transformation had an effect)
Description: for each function that was obfuscated, there is an (absolute value of the) difference between the length given by the Halstead metric before the transformation and that given after the transformation for that function; one function (or possibly more that one, if there is a tie) has the largest value for absolute value of difference in length; that value is this "greatest difference".
halstead_difference_delta_percent (optional):
Title: Halstead function length delta from previous result (percent)
Type: variant
Description: difference between the greatest difference in the Halstead function lengths and that same measure made in the most recent previous run, as a percent (i.e., (new - old)/old * 100.0)
Variants:
DELTA_PERCENT of:
Type: float
MISSING_VALUE
DIVIDE_BY_ZERO
previous_result (optional):
Title: previous result
Type: object
Description: all of the same result fields containing the data from one (most recent) previous run
Properties:
test-cases-file
Coming Soon