Support for verilog memory loading. (#840)

* Ability to load memories at simulation startup
* first pass
* create annotation
* create skeleton Transform

* Work in progress
Building out transform and pass now

* Support for LoadMemory annotation
* Creates chisel and firrtl LoadMemory annotations
* LoadMemoryTransform converts annotation into BlackBox InLine
* Simple test that verilog bound modules get created.

* Support for LoadMemory annotation
* Supports Bundled/multi-field memories
* more tests
* support for `$readmemh` and `$readmemb`
* warns if suffix used in file specification.

* Support for LoadMemory annotation
* Use standard chisel annotation idiom

* Support for LoadMemory annotation
* Fixes for @seldridge nits and super-nits

* Support for LoadMemory annotation
- transform now only runs if emitter is an instance of VerilogEmitter
- suffixes on memory text files are now respected
  - if suffix exists and memory is aggregate, aggregate sub-fields will now be inserted before suffix
- every bind module created gets a unique number
  - this is required when multiple loaded memories appear in a module
  - this should be generalized for other uses of binding modules

* Support for LoadMemory annotation
- remove un-needed suffix test

* Support for LoadMemory annotation
- remove instance walk, now just processes each module

* Support for LoadMemory annotation
- Move LoadMemoryTransformation into Firrtl for treadle to access it.

* Support for LoadMemory annotation
- One more bug in suffix handling has been eliminated

* Support for LoadMemory annotation
- remove unused findModule per jackkoenig
- fixed complex test, bad filename edge case

* Support for LoadMemory annotation
- changed to not use intellij style column alignment for : declarations

* Load memory from file
Fixes based on @jkoenig review
- remove unused BindPrefixFactory
- Moved code from CreateBindableMemoryLoaders into to LoadMemoryTransfrom
- Made map to find relevant memory annotations faster
- Made map to find modules referenced by annotations faster
- Made things private that should be private
- DefAnnotatedMemorys are no longer referenced, shouldn't be found here.
- println of error changed to failed

* Loading memories from files
- Many changes based on review
- move stuff into experimental
- clean up annotation manipulation
- manage tests better
- use more standard practices for transform

* Loading memories from files
- More review changes
- Move doc from annotation to the object apply method that generates the annotation
- Make scalastyle directives more specific
- Use more efficient collect to generate name to module map
- Made lines obey style length limit
- a couple of cleanups of imports in tests
- removed some commented out code
- optimized checking for lines using .exists
- use _ for unused variable in match

3 files changed, 424 insertions, 0 deletions

diff --git a/src/main/scala/chisel3/package.scala b/src/main/scala/chisel3/package.scala
index f0542d7e..7f1ad040 100644
--- a/src/main/scala/chisel3/package.scala
+++ b/src/main/scala/chisel3/package.scala
@@ -245,6 +245,8 @@ package object chisel3 {    // scalastyle:ignore package.object.name
 
   type BlackBox = chisel3.core.BlackBox
 
+  type InstanceId = chisel3.internal.InstanceId
+
   val Mem = chisel3.core.Mem
   type MemBase[T <: Data] = chisel3.core.MemBase[T]
   type Mem[T <: Data] = chisel3.core.Mem[T]
diff --git a/src/main/scala/chisel3/util/experimental/LoadMemoryTransform.scala b/src/main/scala/chisel3/util/experimental/LoadMemoryTransform.scala
new file mode 100644
index 00000000..04a01fe9
--- /dev/null
+++ b/src/main/scala/chisel3/util/experimental/LoadMemoryTransform.scala
@@ -0,0 +1,234 @@
+// See LICENSE for license details.
+
+package chisel3.util.experimental
+
+import chisel3._
+import chisel3.experimental.annotate
+// import chisel3.InstanceId
+import chisel3.experimental.{ChiselAnnotation, RunFirrtlTransform}
+import firrtl.annotations.{MemoryLoadFileType, _}
+import firrtl.ir.{Module => _, _}
+import firrtl.transforms.BlackBoxInlineAnno
+import firrtl.Mappers._
+import firrtl.{AnnotationSeq, CircuitForm, CircuitState, EmitCircuitAnnotation, LowForm, Transform, VerilogEmitter}
+
+import scala.collection.mutable
+
+/** This is the annotation created when using [[loadMemoryFromFile]], it records the memory, the load file
+  * and the format of the file.
+  * @param target        memory to load
+  * @param fileName      name of input file
+  * @param hexOrBinary   use $readmemh or $readmemb, i.e. hex or binary text input, default is hex
+  */
+case class ChiselLoadMemoryAnnotation[T <: Data](
+  target:      MemBase[T],
+  fileName:    String,
+  hexOrBinary: MemoryLoadFileType.FileType = MemoryLoadFileType.Hex
+)
+  extends ChiselAnnotation with RunFirrtlTransform {
+
+  if(fileName.isEmpty) {
+    throw new Exception(
+      s"""LoadMemory from file annotations file empty file name"""
+    )
+  }
+
+  def transformClass: Class[LoadMemoryTransform] = classOf[LoadMemoryTransform]
+
+  def toFirrtl: LoadMemoryAnnotation = {
+    LoadMemoryAnnotation(target.toNamed.asInstanceOf[ComponentName], fileName, hexOrBinary)
+  }
+}
+
+
+object loadMemoryFromFile {
+  /** Use this annotation generator to load a memory from a text file by using verilator and
+    *  verilog's $readmemh or $readmemb.
+    *  The treadle backend can also recognize this annotation and load memory at run-time.
+    *
+    * This annotation triggers the [[LoadMemoryTransform]] which will take add the verilog directive to
+      * the relevant module by using the creating separate modules that are bound to the modules containing
+    * the memories to be loaded.
+    *
+    * ==Example module==
+    *
+    * Consider a simple Module containing a memory
+      * {{{
+        * import chisel3._
+        * class UsesMem(memoryDepth: Int, memoryType: Data) extends Module {
+          *   val io = IO(new Bundle {
+            *     val address = Input(UInt(memoryType.getWidth.W))
+            *     val value   = Output(memoryType)
+            *   })
+          *   val memory = Mem(memoryDepth, memoryType)
+          *   io.value := memory(io.address)
+          * }
+        * }}}
+    *
+    * ==Above module with annotation==
+      *
+    * To load this memory from a file /workspace/workdir/mem1.hex.txt
+    * Just add an import and annotate the memory
+      * {{{
+        * import chisel3._
+        * import chisel3.util.experimental.loadMemoryFromFile   // <<-- new import here
+        * class UsesMem(memoryDepth: Int, memoryType: Data) extends Module {
+          *   val io = IO(new Bundle {
+            *     val address = Input(UInt(memoryType.getWidth.W))
+            *     val value   = Output(memoryType)
+            *   })
+          *   val memory = Mem(memoryDepth, memoryType)
+          *   io.value := memory(io.address)
+          *   loadMemoryFromFile(memory, "/workspace/workdir/mem1.hex.txt")  // <<-- Note the annotation here
+          * }
+        * }}}
+    *
+    * ==Example file format==
+      * A memory file should consist of ascii text in either hex or binary format
+    * Example (a file containing the decimal values 0, 7, 14, 21):
+      * {{{
+        *   0
+        *   7
+        *   d
+        *  15
+        * }}}
+    * Binary file is similarly constructed.
+    *
+    * ==More info==
+    * See the LoadMemoryFromFileSpec.scala in the test suite for more examples
+    * @see <a href="https://github.com/freechipsproject/chisel3/wiki/Chisel-Memories">Load Memories in the wiki</a>
+    */
+  def apply[T <: Data](
+    memory: MemBase[T],
+    fileName: String,
+    hexOrBinary: MemoryLoadFileType.FileType = MemoryLoadFileType.Hex
+  ): Unit = {
+    annotate(ChiselLoadMemoryAnnotation(memory, fileName))
+  }
+}
+
+/**
+  * This transform only is activated if verilog is being generated
+  * (determined by presence of the proper emit annotation)
+  * when activated it creates additional verilog files that contain
+  * modules bound to the modules that contain an initializable memory
+  *
+  * Currently the only non-verilog based simulation that can support loading
+  * memory from a file is treadle but it does not need this transform
+  * to do that.
+  */
+//scalastyle:off method.length
+class LoadMemoryTransform extends Transform {
+  def inputForm: CircuitForm  = LowForm
+  def outputForm: CircuitForm = LowForm
+
+  private var memoryCounter: Int = -1
+
+  private val bindModules: mutable.ArrayBuffer[BlackBoxInlineAnno] = new mutable.ArrayBuffer()
+
+  private val verilogEmitter:    VerilogEmitter = new VerilogEmitter
+
+  /**
+    * run the pass
+    * @param circuit the circuit
+    * @param annotations all the annotations
+    * @return
+    */
+  def run(circuit: Circuit, annotations: AnnotationSeq): Circuit = {
+    val groups = annotations
+      .collect{ case m: LoadMemoryAnnotation => m }
+      .groupBy(_.target.serialize)
+    val memoryAnnotations = groups.map { case (key, annos) =>
+        if (annos.size > 1) {
+          throw new Exception(
+            s"Multiple (${annos.size} found for memory $key one LoadMemoryAnnotation is allowed per memory"
+          )
+        }
+        key -> annos.head
+      }
+
+    val modulesByName = circuit.modules.collect { case module: firrtl.ir.Module =>  module.name -> module }.toMap
+
+    /**
+      * walk the module and for memories that have LoadMemory annotations
+      * generate the bindable modules for verilog emission
+      *
+      * @param myModule     module being searched for memories
+      */
+    def processModule(myModule: DefModule): DefModule = {
+
+      def makePath(componentName: String): String = {
+        circuit.main + "." + myModule.name + "." + componentName
+      }
+
+      def processMemory(name: String): Unit = {
+        val fullMemoryName = makePath(s"$name")
+
+        memoryAnnotations.get(fullMemoryName) match {
+          case Some(lma @ LoadMemoryAnnotation(ComponentName(componentName, moduleName), _, hexOrBinary, _)) =>
+            val writer = new java.io.StringWriter
+
+            modulesByName.get(moduleName.name).foreach { module =>
+                val renderer = verilogEmitter.getRenderer(module, modulesByName)(writer)
+                val loadFileName = lma.getFileName
+
+                memoryCounter += 1
+                val bindsToName = s"BindsTo_${memoryCounter}_${moduleName.name}"
+                renderer.emitVerilogBind(bindsToName,
+                  s"""
+                     |initial begin
+                     |  $$readmem$hexOrBinary("$loadFileName", ${myModule.name}.$componentName);
+                     |end
+                      """.stripMargin)
+                val inLineText = writer.toString + "\n" +
+                  s"""bind ${myModule.name} $bindsToName ${bindsToName}_Inst(.*);"""
+
+                val blackBoxInline = BlackBoxInlineAnno(
+                  moduleName,
+                  moduleName.serialize + "." + componentName + ".v",
+                  inLineText
+                )
+
+                bindModules += blackBoxInline
+              }
+
+          case _ =>
+        }
+      }
+
+      def processStatements(statement: Statement): Statement = {
+        statement match {
+          case m: DefMemory          => processMemory(m.name)
+          case s                     => s map processStatements
+        }
+        statement
+      }
+
+      myModule match {
+        case module: firrtl.ir.Module =>
+          processStatements(module.body)
+        case _ =>
+      }
+
+      myModule
+    }
+
+    circuit map processModule
+  }
+
+  def execute(state: CircuitState): CircuitState = {
+    val isVerilog = state.annotations.exists {
+      case EmitCircuitAnnotation(emitter) =>
+        emitter == classOf[VerilogEmitter]
+      case _ =>
+        false
+    }
+    if(isVerilog) {
+      run(state.circuit, state.annotations)
+      state.copy(annotations = state.annotations ++ bindModules)
+    }
+    else {
+      state
+    }
+  }
+}
diff --git a/src/test/scala/chiselTests/LoadMemoryFromFileSpec.scala b/src/test/scala/chiselTests/LoadMemoryFromFileSpec.scala
new file mode 100644
index 00000000..330cdd3e
--- /dev/null
+++ b/src/test/scala/chiselTests/LoadMemoryFromFileSpec.scala
@@ -0,0 +1,188 @@
+// See LICENSE for license details.
+
+package chiselTests
+
+import java.io.File
+
+import chisel3._
+import chisel3.util.experimental.loadMemoryFromFile
+import chisel3.util.log2Ceil
+import firrtl.FirrtlExecutionSuccess
+import firrtl.annotations.MemoryLoadFileType
+import org.scalatest.{FreeSpec, Matchers}
+
+class UsesThreeMems(memoryDepth: Int, memoryType: Data) extends Module {
+  val io = IO(new Bundle {
+    val address = Input(UInt(memoryType.getWidth.W))
+    val value1  = Output(memoryType)
+    val value2  = Output(memoryType)
+    val value3  = Output(memoryType)
+  })
+
+  val memory1 = Mem(memoryDepth, memoryType)
+  val memory2 = Mem(memoryDepth, memoryType)
+  val memory3 = Mem(memoryDepth, memoryType)
+  loadMemoryFromFile(memory1, "./mem1")
+  loadMemoryFromFile(memory2, "./mem1")
+  loadMemoryFromFile(memory3, "./mem1")
+
+  io.value1 := memory1(io.address)
+  io.value2 := memory2(io.address)
+  io.value3 := memory3(io.address)
+}
+
+class UsesMem(memoryDepth: Int, memoryType: Data) extends Module {
+  val io = IO(new Bundle {
+    val address = Input(UInt(memoryType.getWidth.W))
+    val value   = Output(memoryType)
+    val value1  = Output(memoryType)
+    val value2  = Output(memoryType)
+  })
+
+  val memory = Mem(memoryDepth, memoryType)
+  loadMemoryFromFile(memory, "./mem1")
+
+  io.value := memory(io.address)
+
+  val low1 = Module(new UsesMemLow(memoryDepth, memoryType))
+  val low2 = Module(new UsesMemLow(memoryDepth, memoryType))
+
+  low2.io.address := io.address
+  low1.io.address := io.address
+  io.value1 := low1.io.value
+  io.value2 := low2.io.value
+}
+
+class UsesMemLow(memoryDepth: Int, memoryType: Data) extends Module {
+  val io = IO(new Bundle {
+    val address = Input(UInt(memoryType.getWidth.W))
+    val value   = Output(memoryType)
+  })
+
+  val memory = Mem(memoryDepth, memoryType)
+
+  loadMemoryFromFile(memory, "./mem2")
+
+  io.value := memory(io.address)
+}
+
+class FileHasSuffix(memoryDepth: Int, memoryType: Data) extends Module {
+  val io = IO(new Bundle {
+    val address = Input(UInt(memoryType.getWidth.W))
+    val value   = Output(memoryType)
+    val value2  = Output(memoryType)
+  })
+
+  val memory = Mem(memoryDepth, memoryType)
+
+  loadMemoryFromFile(memory, "./mem1.txt")
+
+  io.value := memory(io.address)
+
+  val low = Module(new UsesMemLow(memoryDepth, memoryType))
+
+  low.io.address := io.address
+  io.value2 := low.io.value
+}
+
+class MemoryShape extends Bundle {
+  val a = UInt(8.W)
+  val b = SInt(8.W)
+  val c = Bool()
+}
+
+class HasComplexMemory(memoryDepth: Int) extends Module {
+  val io = IO(new Bundle {
+    val address = Input(UInt(log2Ceil(memoryDepth).W))
+    val value   = Output(new MemoryShape)
+  })
+
+  val memory = Mem(memoryDepth, new MemoryShape)
+
+  loadMemoryFromFile(memory, "./mem", MemoryLoadFileType.Hex)
+
+  io.value := memory(io.address)
+}
+
+
+/**
+  * The following tests are a bit incomplete and check that the output verilog is properly constructed
+  * For more complete working examples
+  * @see <a href="https://github.com/freechipsproject/chisel-testers">Chisel Testers</a> LoadMemoryFromFileSpec.scala
+  */
+class LoadMemoryFromFileSpec extends FreeSpec with Matchers {
+  def fileExistsWithMem(file: File, mem: Option[String] = None): Unit = {
+    info(s"$file exists")
+    file.exists() should be (true)
+    mem.foreach( m => {
+      info(s"Memory $m is referenced in $file")
+      val found = io.Source.fromFile(file).getLines.exists { _.contains(s"""readmemh("$m"""") }
+      found should be (true)
+    } )
+    file.delete()
+  }
+
+  "Users can specify a source file to load memory from" in {
+    val testDirName = "test_run_dir/load_memory_spec"
+
+    val result = Driver.execute(
+      args = Array("-X", "verilog", "--target-dir", testDirName),
+      dut = () => new UsesMem(memoryDepth = 8, memoryType = UInt(16.W))    )
+
+    result match {
+      case ChiselExecutionSuccess(_, _, Some(FirrtlExecutionSuccess(_, _))) =>
+        val dir = new File(testDirName)
+        fileExistsWithMem(new File(dir, "UsesMem.UsesMem.memory.v"), Some("./mem1"))
+        fileExistsWithMem(new File(dir, "UsesMem.UsesMemLow.memory.v"), Some("./mem2"))
+        fileExistsWithMem(new File(dir, "firrtl_black_box_resource_files.f"))
+      case _=>
+        throw new Exception("Failed compile")
+    }
+  }
+
+  "Calling a module that loads memories from a file more than once should work" in {
+    val testDirName = "test_run_dir/load_three_memory_spec"
+
+    val result = Driver.execute(
+      args = Array("-X", "verilog", "--target-dir", testDirName),
+      dut = () => new UsesThreeMems(memoryDepth = 8, memoryType = UInt(16.W))
+    )
+
+    result match {
+      case ChiselExecutionSuccess(_, _, Some(FirrtlExecutionSuccess(_, _))) =>
+        val dir = new File(testDirName)
+        fileExistsWithMem( new File(dir, "UsesThreeMems.UsesThreeMems.memory1.v"), Some("./mem1"))
+        fileExistsWithMem( new File(dir, "UsesThreeMems.UsesThreeMems.memory2.v"), Some("./mem1"))
+        fileExistsWithMem( new File(dir, "UsesThreeMems.UsesThreeMems.memory3.v"), Some("./mem1"))
+        fileExistsWithMem( new File(dir, "firrtl_black_box_resource_files.f"))
+      case _=>
+        throw new Exception("Failed compile")
+    }  }
+
+  "In this example the memory has a complex memory type containing a bundle" in {
+    val complexTestDirName = "test_run_dir/complex_memory_load"
+
+    val result = Driver.execute(
+      args = Array("-X", "verilog", "--target-dir", complexTestDirName),
+      dut = () => new HasComplexMemory(memoryDepth = 8)
+    )
+
+    result match {
+      case ChiselExecutionSuccess(_, _, Some(FirrtlExecutionSuccess(emitType, firrtlEmitted))) =>
+        val dir = new File(complexTestDirName)
+        val memoryElements = Seq("a", "b", "c")
+
+        memoryElements.foreach { element =>
+          val file = new File(dir, s"HasComplexMemory.HasComplexMemory.memory_$element.v")
+          file.exists() should be (true)
+          val fileText = io.Source.fromFile(file).getLines().mkString("\n")
+          fileText should include (s"""$$readmemh("./mem_$element", HasComplexMemory.memory_$element);""")
+          file.delete()
+        }
+
+      case _=>
+        fail(s"Failed compile")
+    }
+  }
+
+}