From c4cc6bc5b614bd7f5383f8a85c7fc81facdc4b20 Mon Sep 17 00:00:00 2001 From: Albert Chen Date: Thu, 16 Jul 2020 16:59:28 -0700 Subject: Add Expression Fuzzer (#1741) Includes: * Random generator of FIRRTL Expressions (UInt and SInt types) * JQF SBT plugin and CLI * Documentation in README.md Co-authored-by: Jack Koenig --- README.md | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) (limited to 'README.md') diff --git a/README.md b/README.md index 65df861c..70921ed5 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,73 @@ utils/bin/firrtl -i regress/rocket.fir -o regress/rocket.v -X verilog // Compile utils/bin/firrtl --help // Returns usage string ``` +##### Using the JQF Fuzzer +The `build.sbt` defines the `fuzzer/jqfFuzz` and `fuzzer/jqfRepro` tasks. These +can be used to randomly generate and run test cases and reproduce failing test +cases respectively. These tasks are Scala implementations of the [FuzzGoal and +ReproGoal](https://github.com/rohanpadhye/JQF/tree/master/maven-plugin/src/main/java/edu/berkeley/cs/jqf/plugin) +of the JQF maven plugin and should be functionally identical. + +The format for the arguments to jqfFuzz are as follows: +``` +sbt> fuzzer/jqfFuzz ... +``` + +The available options are: +``` + --classpath the classpath to instrument and load the test class from + --outputDirectory the directory to output test results + --testClassName the full class path of the test class + --testMethod the method of the test class to run + --excludes comma-separated list of FQN prefixes to exclude from coverage instrumentation + --includes comma-separated list of FQN prefixes to forcibly include, even if they match an exclude + --time the duration of time for which to run fuzzing + --blind whether to generate inputs blindly without taking into account coverage feedback + --engine the fuzzing engine, valid choices are zest|zeal + --disableCoverage disable code-coverage instrumentation + --inputDirectory the name of the input directory containing seed files + --saveAll save ALL inputs generated during fuzzing, even the ones that do not have any unique code coverage + --libFuzzerCompatOutput use libFuzzer like output instead of AFL like stats screen + --quiet avoid printing fuzzing statistics progress in the console + --exitOnCrash stop fuzzing once a crash is found. + --runTimeout the timeout for each individual trial, in milliseconds +``` + +The `fuzzer/jqfFuzz` sbt task is a thin wrapper around the `firrtl.jqf.jqfFuzz` +main method that provides the `--classpath` argument and a default +`--outputDirectory` and passes the rest of the arguments to the main method +verbatim. + +The results will be put in the `fuzzer/target/JQf/$testClassName/$testMethod` +directory. Input files in the +`fuzzer/target/JQf/$testClassName/$testMethod/corpus` and +`fuzzer/target/JQf/$testClassName/$testMethod/failures` directories can be +passed as inputs to the `fuzzer/jqfRepro` task. + + +The format for the arguments to jqfRepro are the same as `jqfFuzz` +``` +sbt> fuzzer/jqfRepro ... +``` + +The available options are: + +``` + --classpath the classpath to instrument and load the test class from + --testClassName the full class path of the test class + --testMethod the method of the test class to run + --input input file or directory to reproduce test case(s) + --logCoverage output file to dump coverage info + --excludes comma-separated list of FQN prefixes to exclude from coverage instrumentation + --includes comma-separated list of FQN prefixes to forcibly include, even if they match an exclude + --printArgs whether to print the args to each test case +``` + +Like `fuzzer/jqfFuzz`, the `fuzzer/jqfRepro` sbt task is a thin wrapper around +the `firrtl.jqf.jqfRepro` main method that provides the `--classpath` argument +and a default `--outputDirectory` and passes the rest of the arguments to the +main method verbatim. + ##### Citing Firrtl If you use Firrtl in a paper, please cite the following ICCAD paper and technical report: -- cgit v1.2.3