Add FirrtlStage, make Driver compatibility layer

This adds FirrtlStage, a reimplementation of the original FIRRTL
Driver as a Stage. This updates the original firrtl.options package to
implement FirrtlStage (namely, TransformLike is added) along with
FirrtlMain. Finally, the original FIRRTL Driver is converted to a
compatibility wrapper around FirrtlStage.

For background, Stage and Phase form the basis of the Chisel/FIRRTL
Hardware Compiler Framework (HCF). A Phase is a class that performs a
mathematical transformation on an AnnotationSeq (in effect, a
generalization of a FIRRTL transform). Curtly, a Stage is a Phase that
also provides a user interface for generating annotations. By their
construction, Phases are designed to be composed sequentially into a
transformation pipeline.

This modifies the existing options package (which provides
Stage/Phase) to build out a type hierarchy around Stage/Phase. This
adds TransformLike[A] which implements a mathematical transformation
over some type A. Additionally, and as an interface between different
TransformLikes, this adds Translator[A, B] which extends
TransformLike[A], but does an internal transformation over type B.
This is used to interface Phases with the existing FIRRTL compiler.

This adds a runTransform method to Phase that, like
Transform.runTransform, will automatically detect deleted Annotations
and generate DeletedAnnotations.

The new FirrtlStage, a reimplementation of FIRRTL's Driver, is added
as a Stage composed of the following Phases:

  1. AddDefaults - add default annotations
  2. AddImplicitEmitter - adds an implicit emitter derived from the
     compiler
  3. Checks - sanity check the AnnotationSeq
  4. AddCircuit - convert FIRRTL input files/sources to circuits
  5. AddImplicitOutputFile - add a default output file
  6. Compiler - run the FIRRTL compiler
  7. WriteEmitted - write any emitted modules/circuits to files

The Driver is converted to a compatibility layer that replicates old
Driver behavior. This is implemented by first using new toAnnotation
methods for CommonOptions and FirrtlExecutionOptions that enable
AnnotationSeq generation. Second, the generated AnnotationSeq is
preprocessed and sent to FirrtlStage. The resulting Phase order is
then:

  1. AddImplicitAnnotationFile - adds a default annotation file
  2. AddImplicitFirrtlFile - adds a default FIRRTL file using top name
  3. AddImplicitOutputFile - adds an output file from top name
  4. AddImplicitEmitter - adds a default emitter derived from a
     compiler and any split modules command line option
  5. FirrtlStage - the aforementioned new FirrtlStage

Finally, the output AnnotationSeq is then viewed as a
FirrtlExecutionResult. This compatibility layer enables uninterrupted
usage of old Driver infrastructure, e.g., FirrtlExecutionOptions and
CommonOptions can still be mutated directly and used to run the
Driver.

This results in differing behavior between the new FirrtlStage and the
old Driver, specifically:

  - FirrtlStage makes a clear delineation between a "compiler" and an
    "emitter". These are defined using separate options. A compiler is
    "-X/--compiler", while an emitter is one of "-E/--emit-circuit" or
    "-e/--emit-modules".
  - Related to the above, the "-fsm/--split-modules" has been removed
    from the FirrtlStage. This option is confusing once an implicit
    emitter is removed. It is also unclear how this should be handled
    once the user can specify multiple emitters, e.g., which emitter
    should "--split-modules" apply to?
  - WriteOutputAnnotations will, by default, not write
    DeletedAnnotations to the output file.
  - The old top name ("-tn/--top-name") option has been removed from
    FirrtlStage. This option is really a means to communicate what
    input and output files are as opposed to anything associated with
    the circuit name. This option is preserved for the Driver
    compatibility layer.

Additionally, this changes existing transform scheduling to work for
emitters (which subclass Transform). Previously, one emitter was
explicitly scheduled at the end of all transforms for a given
compiler. Additional emitters could be added, but they would be
scheduled as transforms. This fixes this to rely on transform
scheduling for all emitters. In slightly more detail:

  1. The explicit emitter is removed from Compiler.compile
  2. An explicit emitter is added to Compiler.compileAndEmit
  3. Compiler.mergeTransforms will schedule emitters as late as
     possible, i.e., all emitters will occur after transforms that
     output their input form.
  4. All AddImplicitEmitter phases (DriverCompatibility and normal)
     will add RunFirrtlTransformAnnotations to add implicit emitters

The FIRRTL fat jar utilities are changed to point at FirrtlStage and not
at the Driver. This has backwards incompatibility issues for users
that are using the utilities directly, e.g., Rocket Chip.

The Logger has been updated with methods for setting options based on
an AnnotationSeq. This migrates the Logger to use AnnotationSeq as
input parameters, e.g., for makeScope. Old-style methods are left in
place and deprecated. However, the Logger is not itself a Stage.
The options of Logger Annotations are included in the base Shell and
Stage is updated to wrap its Phases in a Logger scope.

Additionally, this changes any code that does option parsing to always
prepend an annotation as opposed to appending an annotation. This is
faster, but standardizing on this has implications for dealing with
the parallel compilation annotation ordering.

A Shell will now put the initial annotations first (in the order the
user specified) and then place all annotations generating from parsing
after that. This adds a test case to verify this behavior.

Discovered custom transforms (via `RunFirrtlTransformAnnotation`s) are
discovered by the compiler phase in a user-specified order, but are
stored in reverse order to more efficiently prepend (as opposed to
append) to a list. This now reverses the transform order before
execution to preserve backwards compatibility of custom transform
ordering.

The Compiler phase also generates one deleted annotation for each
`RunFirrtlTransformAnnotation`. These are also reversed.

Miscellaneous small changes:

  - Split main method of Stage into StageMain class
  - Only mix in HasScoptOptions into Annotation companion objects (h/t
    @jackkoenig)
  - Store Compiler in CompilerAnnotation
  - CompilerNameAnnotation -> CompilerAnnotation
  - Make Emitter abstract in outputSuffix (move out of FirrtlOptions)
  - Add DriverCompatibility.AddImplicitOutputFile that will add an
    output file annotation based on the presence of a
    TopNameAnnotation. This is important for compatibility with the
    old Driver.
  - Cleanup Scaladoc
  - Refactor CircuitOption to be abstract in "toCircuit" that converts
    the option to a FirrtlCircuitAnnotation. This allows more of the
    conversion steps to be moved out of AddCircuit and into the actual
    annotation.
  - Add WriteDeletedAnnotation to module WriteOutputAnnotations
  - A method for accessing a FirrtlExecutionResultView is exposed in
    FIRRTL's DriverCompatibilityLayer
  - Using "--top-name/-tn" or "--split-modules/-fsm" with FirrtlStage
    generates an error indicating that this option is no longer
    supported
  - Using FirrtlStage without at least one emitter will generate a
    warning
  - Use vals for emitter in Compiler subclasses (these are used to
    build RunFirrtlTransformAnnotations and the object should be
    stable for comparisons)
  - Fixes to tests that use LowTransformSpec instead of
    MiddleTransformSpec. (SimpleTransformSpec is dumb and won't
    schedule transforms correctly. If you rely on an emitter, you need
    to use the right transform spec to test your transform if you're
    relying on an emitter.)

Signed-off-by: Schuyler Eldridge <schuyler.eldridge@ibm.com>

5 files changed, 89 insertions, 46 deletions

diff --git a/src/test/scala/firrtlTests/DriverSpec.scala b/src/test/scala/firrtlTests/DriverSpec.scala
index ae1e08e7..d26aadf2 100644
--- a/src/test/scala/firrtlTests/DriverSpec.scala
+++ b/src/test/scala/firrtlTests/DriverSpec.scala
@@ -206,18 +206,35 @@ class DriverSpec extends FreeSpec with Matchers with BackendCompilationUtilities
   }
 
   // Deprecated
-  "Annotations can be read implicitly from the name of the circuit" in {
+  "Annotations can be read implicitly from the name of the circuit" - {
+    val input = """|circuit foo :
+                   |  module foo :
+                   |    input x : UInt<8>
+                   |    output y : UInt<8>
+                   |    y <= x""".stripMargin
     val top = "foo"
     val optionsManager = new ExecutionOptionsManager("test") with HasFirrtlOptions {
       commonOptions = commonOptions.copy(topName = top)
+      firrtlOptions = firrtlOptions.copy(firrtlSource = Some(input))
     }
     val annoFile = new File(optionsManager.commonOptions.targetDirName, top + ".anno")
-    copyResourceToFile("/annotations/SampleAnnotations.anno", annoFile)
-    optionsManager.firrtlOptions.annotations.length should be(0)
-    val annos = Driver.getAnnotations(optionsManager)
-    annos.length should be(12) // 9 from circuit plus 3 general purpose
-    annos.count(_.isInstanceOf[InlineAnnotation]) should be(9)
-    annoFile.delete()
+    "Using Driver.getAnnotations" in {
+      copyResourceToFile("/annotations/SampleAnnotations.anno", annoFile)
+      optionsManager.firrtlOptions.annotations.length should be(0)
+      val annos = Driver.getAnnotations(optionsManager)
+      annos.length should be(12) // 9 from circuit plus 3 general purpose
+      annos.count(_.isInstanceOf[InlineAnnotation]) should be(9)
+      annoFile.delete()
+    }
+    "Using Driver.execute" in {
+      copyResourceToFile("/annotations/SampleAnnotations.anno", annoFile)
+      Driver.execute(optionsManager) match {
+        case r: FirrtlExecutionSuccess =>
+          val annos = r.circuitState.annotations
+          annos.count(_.isInstanceOf[InlineAnnotation]) should be(9)
+      }
+      annoFile.delete()
+    }
   }
 
   // Deprecated
diff --git a/src/test/scala/firrtlTests/annotationTests/TargetDirAnnotationSpec.scala b/src/test/scala/firrtlTests/annotationTests/TargetDirAnnotationSpec.scala
index 8f157131..eb061d8f 100644
--- a/src/test/scala/firrtlTests/annotationTests/TargetDirAnnotationSpec.scala
+++ b/src/test/scala/firrtlTests/annotationTests/TargetDirAnnotationSpec.scala
@@ -5,7 +5,6 @@ package annotationTests
 
 import firrtlTests._
 import firrtl._
-import firrtl.stage.TargetDirAnnotation
 
 /** Looks for [[TargetDirAnnotation]] */
 class FindTargetDirTransform(expected: String) extends Transform {
diff --git a/src/test/scala/firrtlTests/options/ShellSpec.scala b/src/test/scala/firrtlTests/options/ShellSpec.scala
index d87a9a30..50000550 100644
--- a/src/test/scala/firrtlTests/options/ShellSpec.scala
+++ b/src/test/scala/firrtlTests/options/ShellSpec.scala
@@ -2,12 +2,24 @@
 
 package firrtlTests.options
 
-import org.scalatest._
+import org.scalatest.{FlatSpec, Matchers}
 
+import firrtl.annotations.NoTargetAnnotation
 import firrtl.options.Shell
 
 class ShellSpec extends FlatSpec with Matchers {
 
+  case object A extends NoTargetAnnotation
+  case object B extends NoTargetAnnotation
+  case object C extends NoTargetAnnotation
+  case object D extends NoTargetAnnotation
+  case object E extends NoTargetAnnotation
+
+  trait AlphabeticalCli { this: Shell =>
+    parser.opt[Unit]('c', "c-option").unbounded().action( (x, c) => C +: c )
+    parser.opt[Unit]('d', "d-option").unbounded().action( (x, c) => D +: c )
+    parser.opt[Unit]('e', "e-option").unbounded().action( (x, c) => E +: c ) }
+
   behavior of "Shell"
 
   it should "detect all registered libraries and transforms" in {
@@ -19,4 +31,10 @@ class ShellSpec extends FlatSpec with Matchers {
     info("Found BarLibrary")
     shell.registeredLibraries.map(_.getClass.getName) should contain ("firrtlTests.options.BarLibrary")
   }
+
+  it should "correctly order annotations and options" in {
+    val shell = new Shell("foo") with AlphabeticalCli
+
+    shell.parse(Array("-c", "-d", "-e"), Seq(A, B)).toSeq should be (Seq(A, B, C, D, E))
+  }
 }
diff --git a/src/test/scala/firrtlTests/transforms/GroupComponentsSpec.scala b/src/test/scala/firrtlTests/transforms/GroupComponentsSpec.scala
index c54e02e3..b4c27875 100644
--- a/src/test/scala/firrtlTests/transforms/GroupComponentsSpec.scala
+++ b/src/test/scala/firrtlTests/transforms/GroupComponentsSpec.scala
@@ -8,7 +8,7 @@ import firrtl.ir._
 
 import FirrtlCheckers._
 
-class GroupComponentsSpec extends LowTransformSpec {
+class GroupComponentsSpec extends MiddleTransformSpec {
   def transform = new GroupComponents()
   val top = "Top"
   def topComp(name: String): ComponentName = ComponentName(name, ModuleName(top, CircuitName(top)))
@@ -71,15 +71,18 @@ class GroupComponentsSpec extends LowTransformSpec {
         |    output out: UInt<16>
         |    inst inst of Child
         |    node n = UInt<16>("h0")
-        |    inst.in_IN <= in
-        |    node a = UInt<16>("h0")
-        |    node b = a
+        |    wire a : UInt<16>
+        |    wire b : UInt<16>
         |    out <= inst.w_OUT
+        |    inst.in_IN <= in
+        |    a <= UInt<16>("h0")
+        |    b <= a
         |  module Child :
-        |    input in_IN : UInt<16>
         |    output w_OUT : UInt<16>
-        |    node w = in_IN
+        |    input in_IN : UInt<16>
+        |    wire w : UInt<16>
         |    w_OUT <= w
+        |    w <= in_IN
       """.stripMargin
     execute(input, check, groups)
   }
diff --git a/src/test/scala/firrtlTests/transforms/TopWiringTest.scala b/src/test/scala/firrtlTests/transforms/TopWiringTest.scala
index 5a6b3420..d5b1aa6d 100644
--- a/src/test/scala/firrtlTests/transforms/TopWiringTest.scala
+++ b/src/test/scala/firrtlTests/transforms/TopWiringTest.scala
@@ -55,7 +55,7 @@ trait TopWiringTestsCommon extends FirrtlRunners {
 /**
  * Tests TopWiring transformation
  */
-class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
+class TopWiringTests extends MiddleTransformSpec with TopWiringTestsCommon  {
 
    "The signal x in module C" should s"be connected to Top port with topwiring prefix and outputfile in $testDirName" in {
       val input =
@@ -78,8 +78,8 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output x: UInt<1>
            |    x <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                    s"topwiring_"),
                          TopWiringOutputFilesAnnotation(testDirName, topWiringTestOutputFilesFunction))
       val check =
@@ -113,7 +113,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal x in module C inst c1 and c2" should 
+   "The signal x in module C inst c1 and c2" should
     s"be connected to Top port with topwiring prefix and outfile in $testDirName" in {
       val input =
          """circuit Top :
@@ -177,7 +177,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal x in module C" should 
+   "The signal x in module C" should
    s"be connected to Top port with topwiring prefix and outputfile in $testDirName, after name colission" in {
       val input =
          """circuit Top :
@@ -203,8 +203,8 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output x: UInt<1>
            |    x <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                                  s"topwiring_"),
                                TopWiringOutputFilesAnnotation(testDirName, topWiringTestOutputFilesFunction))
       val check =
@@ -213,14 +213,16 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output topwiring_a1_b1_c1_x_0: UInt<1>
            |    inst a1 of A
            |    inst a2 of A_
-           |    node topwiring_a1_b1_c1_x = UInt<1>("h0")
+           |    wire topwiring_a1_b1_c1_x : UInt<1>
+           |    topwiring_a1_b1_c1_x <= UInt<1>("h0")
            |    topwiring_a1_b1_c1_x_0 <= a1.topwiring_b1_c1_x_0
            |  module A :
            |    output x: UInt<1>
            |    output topwiring_b1_c1_x_0: UInt<1>
            |    inst b1 of B
-           |    node topwiring_b1_c1_x = UInt<1>("h0")
+           |    wire topwiring_b1_c1_x : UInt<1>
            |    x <= UInt(1)
+           |    topwiring_b1_c1_x <= UInt<1>("h0")
            |    topwiring_b1_c1_x_0 <= b1.topwiring_c1_x
            |  module A_ :
            |    output x: UInt<1>
@@ -240,7 +242,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal x in module C" should 
+   "The signal x in module C" should
    "be connected to Top port with topwiring prefix and no output function" in {
       val input =
          """circuit Top :
@@ -262,8 +264,8 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output x: UInt<1>
            |    x <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                    s"topwiring_"))
       val check =
          """circuit Top :
@@ -296,7 +298,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal x in module C inst c1 and c2 and signal y in module A_" should 
+   "The signal x in module C inst c1 and c2 and signal y in module A_" should
    s"be connected to Top port with topwiring prefix and outfile in $testDirName" in {
       val input =
          """circuit Top :
@@ -321,11 +323,11 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output x: UInt<1>
            |    x <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                    s"topwiring_"),
-                               TopWiringAnnotation(ComponentName(s"y", 
-                                                                 ModuleName(s"A_", CircuitName(s"Top"))), 
+                               TopWiringAnnotation(ComponentName(s"y",
+                                                                 ModuleName(s"A_", CircuitName(s"Top"))),
                                                    s"topwiring_"),
                          TopWiringOutputFilesAnnotation(testDirName, topWiringTestOutputFilesFunction))
       val check =
@@ -350,8 +352,9 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |  module A_ :
            |    output x: UInt<1>
            |    output topwiring_y: UInt<1>
-           |    node y = UInt<1>("h1")
+           |    wire y : UInt<1>
            |    x <= UInt(1)
+           |    y <= UInt<1>("h1")
            |    topwiring_y <= y
            |  module B :
            |    output x: UInt<1>
@@ -371,7 +374,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal x in module C inst c1 and c2 and signal y in module A_" should 
+   "The signal x in module C inst c1 and c2 and signal y in module A_" should
    s"be connected to Top port with topwiring and top2wiring prefix and outfile in $testDirName" in {
       val input =
          """circuit Top :
@@ -396,11 +399,11 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output x: UInt<1>
            |    x <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"x",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                    s"topwiring_"),
-                               TopWiringAnnotation(ComponentName(s"y", 
-                                                                 ModuleName(s"A_", CircuitName(s"Top"))), 
+                               TopWiringAnnotation(ComponentName(s"y",
+                                                                 ModuleName(s"A_", CircuitName(s"Top"))),
                                                    s"top2wiring_"),
                          TopWiringOutputFilesAnnotation(testDirName, topWiringTestOutputFilesFunction))
       val check =
@@ -425,8 +428,9 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |  module A_ :
            |    output x: UInt<1>
            |    output top2wiring_y: UInt<1>
-           |    node y = UInt<1>("h1")
+           |    wire y : UInt<1>
            |    x <= UInt(1)
+           |    y <= UInt<1>("h1")
            |    top2wiring_y <= y
            |  module B :
            |    output x: UInt<1>
@@ -446,7 +450,7 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
       execute(input, check, topwiringannos)
    }
 
-   "The signal fullword in module C inst c1 and c2 and signal y in module A_" should 
+   "The signal fullword in module C inst c1 and c2 and signal y in module A_" should
    s"be connected to Top port with topwiring and top2wiring prefix and outfile in $testDirName" in {
       val input =
          """circuit Top :
@@ -471,11 +475,11 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    output fullword: UInt<1>
            |    fullword <= UInt(0)
            """.stripMargin
-      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"fullword", 
-                                                                 ModuleName(s"C", CircuitName(s"Top"))), 
+      val topwiringannos = Seq(TopWiringAnnotation(ComponentName(s"fullword",
+                                                                 ModuleName(s"C", CircuitName(s"Top"))),
                                                    s"topwiring_"),
-                               TopWiringAnnotation(ComponentName(s"y", 
-                                                                 ModuleName(s"A_", CircuitName(s"Top"))), 
+                               TopWiringAnnotation(ComponentName(s"y",
+                                                                 ModuleName(s"A_", CircuitName(s"Top"))),
                                                    s"top2wiring_"),
                          TopWiringOutputFilesAnnotation(testDirName, topWiringTestOutputFilesFunction))
       val check =
@@ -500,8 +504,9 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |  module A_ :
            |    output fullword: UInt<1>
            |    output top2wiring_y: UInt<1>
-           |    node y = UInt<1>("h1")
+           |    wire y : UInt<1>
            |    fullword <= UInt(1)
+           |    y <= UInt<1>("h1")
            |    top2wiring_y <= y
            |  module B :
            |    output fullword: UInt<1>
@@ -576,8 +581,9 @@ class TopWiringTests extends LowTransformSpec with TopWiringTestsCommon  {
            |    topwiring_b1_c2_fullword <= b1.topwiring_c2_fullword
            |  module A_ :
            |    output fullword: UInt<1>
-           |    node y = UInt<1>("h1")
+           |    wire y : UInt<1>
            |    fullword <= UInt(1)
+           |    y <= UInt<1>("h1")
            |  module B :
            |    output fullword: UInt<1>
            |    output topwiring_fullword: UInt<1>