Add On HOWTO

This page has initial notes on some add-on functionality that could be added to Visual2, as well as information on how to implement this functionality.

For HLP2019 discussion and additional help view or post on Slack channels:

How to add interrupt simulation?

Hardware definition file

Write code that can parse a string (the contents of a Visual2 file buffer) as an interrupt and hardware definition file, and output a datatype representing all the needed info to set up interrupts and the hardware which will generate them when running a simulation.

The parsing process is similar to that in renderer/testbenches.fs (which implements some automated testing using test definitions in a Visual2 buffer). The actual parse function used is parseTests in #src/emulator/testlib.fs.

You should if possible use the tokenisder and base parser code from my Sample code in github because this allows clever parsing that identifies parse errors and (in group phase) can be used to make comprehensible parse error messages. Otherwise cheap and cheerful parsing as in parseTests can be used - see the Slack v2-parsing channel.

You need a definition of what is the interrupt hardware and how is it configured. You are free to develop this, either looking at ARM microcontrollers (which have built-in timer blocks) or others, or just rolling your own. I give a baseline starting point in the hardware section

Hardware definitions

(1) Timer block: functionality: one-shot (interrupt after specified time0 or repetitive (interrupt repeatedly after time - note that new interrupts are triggered by timer which is never stopped, so time of subsequent interrupts is not chnaged by delay in processing earlier interrupt.

(2) (more complex) UART Rx / Tx. Transmit and receive data on serial stream. As far as simulation goes the bytes in or out would need to be tracked as part of teh hardware simulation. The device would mormally have a buffer (can be anything from 0 to 16 bytes) transfers could be eitehr 32 bit (converted to/from 4 bytes) or 8 bit using LS 8 bits in word.) Interrupts can be configured on empty buffer / full buffer nearly full / nearly empty etc. How muh of this is configurable, and how much you choose sensible definition and if it, is up to you. You can check microcontroller UARTs (or UASRTS) for inspiration. The speed of transmission (reception) is always configurable as well, and obviously relevant in a simulation.

For complex definitions see ARM LPC2106 etc user manual general purpose Timer Block and UART Block. But note that this level of complexity would be unhelpful in an educational tool like Visual2.

These definitions are not unique, you could take later peripherals from a cortex microcontroller, or even peripherals from an AVR miccrocontroller (simpler but same functionality).

Hardware simulation

To simulate interrupt hardware you need a function that will be called (once per instruction) from the main Visual2 simulation loop in emulator/executionTop.asmStep:

IntState: all the internal state needed to define the hardware (contents of hardware registers etc)
IntStatus: value indicating is an interrupt raised and if so (perhaps) from which hardware block
state: IntState -> cycles: int64 -> IntState * IntStatus

You need a function that will be called whenever data memory is written that can change hardware state (for example switching off an interrupt, or resetting a timer) when a specific memory location is read or written. Where the registers for each hardware block sit in the Visual2 data memory is up to you - a good candidate for configuration.

Write peripheral memory location, returning new state.
peripheralWrite: intState: IntState -> cycles: int64 -> address: uint32 -> data: uint32 -> IntState

// read peripheral location, returning new state and value read
peripheralRead: intState: IntState -> cycles: int64 -> address: uint32 -> IntState * uint32

Both read and write functions can potentially change state (though read often leaves state unchanged).

Hardware Block Simulation

The type IntState contains the contents of the simulated hardware register(s) AND the time in cycles at which those contents were valid. This allows the new contents at some later time to be calculated. IntState works like DataPath in tracking the state of the system as simulation progresses. Because the hardware is time dependent it must include the time, and its value (when read) will depend on current time as well as previous value.

For example, suppose the hardware simulated is a digital 32 bit down counter, with a memory write to location 0x1000,000 (say) that loads a new value into the counter and a memory read from the same location that returns the current value. Suppose the counter counts 1 for every elapsed cycle.

IntState contains

Field	Type
Counter	uint32
Time	int64

// called when address = 0x1000000 and memory is read
let counterRead (its: IntState) (cycles: uint64) =
    let newcount = its.Counter - (cycles - its.Time)
    { Time = cycles ; Counter = newcount}

// called when address = 0x1000000 and memory is written
let counterWrite (its: IntState) (cycles: uint64) (data: uint32)=
    { Time = cycles ; Counter = data}

// called after each instruction to determine if interrupt has happened (in this simple example)
let counterHasOverflowed (its: IntState) (cycles: uint64) = 
// IntState does not change so no need to return it
    let newcount = its.Counter - (cycles - its.Time)
    newcount < 0

Hardware Block Hooks

The hardware simulation method used is strange if you have an imperative mindset - thinking about functions that change things, but straightforward if you realise that we are writing all the interrupt simulation in a purely functional manner without any mutable variables. (The mutable variables in the asmStep while loop are there for efficiency only, they carry local state that could instead be handled as extra local variables in a tail recursive function). Therefore whenever something changes in the simulated state, this must correspond to passing round an updated version of the DataPath value that represents current simulation state.

Adding simulated hardware thus corresponds to adding new state to the passed round DataPath value, and implementing (functional) updates to this state whenever the hardware state needs to change.

The interrupt hardware interfaces with the CPU via memory mapped registers. A memory mapped register is a peripheral register that can be read and written by CPU memory instructions. It occupies some fixed word address in CPU memory (usually high memory, above the default stack location). Instructions in the Memory module LDM,STM,LDR,STR can read or write these registers.
IntState must be added as a new field to Datapath record so that the hardware state is simulated as well as the CPU state. This record is passed round the simulator - whenever a CPU instruction is simulated it is passed in and out so that changes in CPU state caused by the instruction execution can be made.
The peripheralRead and peripheralWrite functions above will be integrated with the normal CPU memory read and write operations used by Memory module in simulation (see point 6.). They will update (or read) IntState in the case that the read or written address is one of those allocated to the memory mapped registers.
The addresses for this can be constant, or defined by your hardware definition file.
The neatest way to perform this hooking is to rewrite peripheralRead and peripheralWrite as functions returning an Option type. Return a None value when the address does not match a memory-mapped interrupt hardware register, and therefore the normal read/write functionality is used.
Rewrite Memory.getDataMemWord and Helpers.UpdateMemDataand possibly Helpers.UpdateMemByte (which should probably be an error on a peripheral register). These functions must incorporate peripheralRead and peripheralWrite
Add your is there an interrupt? check to the asmStep loop making it update the current simulation dataPath with the most recent hardware state based on time elapsed. Since this gets called once per instruction it should be enough to keep the hardware state cycles count correct and then this (which is available inside dataPath) can be used as a cycles input to peripheralRead and peripheralWrite providing the current time.

Interrupt Mechanism

The DataPath type from emulator needs some additional state to give mode (THREADED or HANDLER), are interrupts switched on or off, shadow registers for R13,R14 or whatever Cortex uses.

Two functions are needed (at least):

// take the interrupt (if interrupts are switched on)
// will set PC to some known interrupt handler address. 
// configure your interrupt vector as low memory: e.g. address 4, etc. Address 0 is reserved for the main code start
// which when interrupts are being used must contain a branch over the handler code to the main program
interruptStart: dataPath: Datapath -> IntStatus -> DataPath

// return from the interrupt
interruptEnd: dataPath: DataPath -> DataPath

The interruptEnd function will be called from an interrupt return instruction. The interruptStart function will be called one per instruction simulated and change state as needed if an interrupt should happen so that the PC has changed and the next instruction executed will be the first instruction in the interrupt handler.

Start and end functionality should if possible be identical to Cortex; but must correctly save and restore enough state (including flags). For the interrupt handler to be written. Cortex interrupts are neat in that they save enough state so that handlers can be identical (in code) to subroutines. A subroutine return in HANDLER mode implements a return from interrupt because LR will contain the correct return address.

Interrupt instructions

As a minimum, to implement interrupts, you need a few instructions:

RETI (return from interrupt)
special instruction to read THREADED mode SP register when in HANDLER mode
SWI AKA SVC - a software interrupt isntruction which can simulate an interrupt.

Oscilloscope-style display

Some teams will want to implement this. It could be a separate task for anyone, but integrates well with interrupts.

The idea is that after having run a simulation you may want to display, as in a scope trace, the values of registers vs times in cycles. You could also display stuff like when interrupts happen. You can have ways to zoom and pan over a long trace in the GUI, ways to set up which registers are displayed and is the display analog (signed or unsigned value as position of a point on Y axis) or digital: hex or decimal value in digits.

The scope display can be done nicely using SVG (see renderer/tooltips.fs). However you could write in individual phase code that will take information from the simulation (asmStep) and convert this into an output suitable for conversion to svg. E.g. a list of pairs of numbers could represent the vertices of a trace. A pair of numbers and string could represent text to be displayd in a given position. Have a look at the sample SVG code in renderer.tootips.fs to see what type of thing can easily be converted to svg.

You can test your code independently of the GUI (without the final "convert to SVG" stage). However to see what you generate you will need to convert to svg and display under FABLE (dotnet core emulator only code will not do this).

You can write a function that has various control inputs: what should be displayed, pan, zoom, maybe some sort of trigger, and outputs what should be on the screen from a given execution trace.

It is easy to instrument the simulator loop to output info as a trace (e.g. array or list) for you to process; e.g. list of pairs, dataPath, cycle count which would contain all info. There are perhaps performance issues on long simulations doing that: but as first attempt it would be acceptable.

In group work stage, hooking your functions up to the GUI will be interesting. (You can do the SVG generation in indiv phase work if you have time, but I don't expect this). Getting good interactive performance so you can navigate and change the display will also be interesting.

Adding New Instructions to Visual2

Here is how you do it.

Note that you only need to add to emulator code.
For an overview of the emulator structure and how to change it read the wiki page FIRST
Note that the emulator instructions currently are all under memory.fs, dp.fs, misc.fs for memory, dp, misc instructions respectively. Look at these modules for samples of how to write instructions.

Always define a new module (like DP, Memory, etc) don't add to or change existing modules.
look at existing modules to see how your instructions will be hooked in (eventually)
Write for your module two functions:
- Parse which takes a single assembly line and returns an error or an instruction value with other info in a Parse type.
- Execute which takes an instruction value and a DataPath value and returns a modified DataPath value or a run-time error.
- NB each of these functions can be found in the existing modules

Note there is some extra stuff - does your instruction take more than one cycle, etc, that you need to include in return values - the types you are given make this pretty clear - stuff you don't understand ask.

The emulator wiki page gives much more information on how the emulator and its instruction modules work.

Re-engineering the simulation

As always, write new functions in a separate module. Hook the into Visual2 via smallest possible change (typically a few function calls). Where you need to add to existing datatypes do this in a way that minimies disturbace - typically adding a new field to a record somewhere. The compielr will force you to update all teh Visual2 places that create new records of that type - but there are not many of them.

it is acceptable in individual work phase just to work on the functions - and worry about hooking them into Visual2 simulation later. However is some of you want to do the hooks - as long as you have time - that is OK too.

How does Visual2 simulation work:

parse a program, resolving labels, to generate a LoadImage record with code, symbol values, defined data.
Initialse a RunInfo record from LoadInfo
Simulate by updating the RunInfo value each step as it is chnaged bye xecuting and instruction.
Simulation can be to a given instruction step (which does not terminate). If simulation terminated it can be:
1. An instruction has given a run-time error
2. a "break condition" has happened (allows breakpoints etc to be added, though so far this has been done in only a minimal way)
3. The program has a normal end (falling off bottom of file or hitting and END label)

Most of the code here is in executionTop.

important types:

Datapath - the CPU register, memory, flag state
RunInfo - the simulation state which contains a whole load of extra info as well as the current and last Datapath value
- last DataPath value is useful because current value can be a run time error if the last instruction errored.
To run a simulation you need to generate an initial "start of simulation" runInfo record:

let lim = reloadProgram srcLines
let runinfo = getRunInfoFromImage lim

Important functions:

asmStep: top-level move to given instruction number in a simulation, this function keeps a historic record of steps (not every one) so it can reconstruct any past step in teh simulation quickly having done it once

Register Memory Change History

Some teams are planning to track this, in order to display on demand in the GUI a highly useful list of changes to registers and memory locations perhaps hyperlinked to the relevant step in the simulation or instruction that made the change.

You will need at some point to read about re-engineering the simulation (asmStep).

You need to define a type that can hold one or more changes where a Change is defined as a location or register that changes, the most recent value it changed from and the step number that caused the change. You can do this separately for registers and memory and combine the two parts, or have a unified Change type that can be either register of memory.
Given this, your spec is to reproduce reasonably quickly the last N Changes (as defined above) for any given register or memory location. You know this can be incorporated into the GUI in team stage work.
All Changes cannot be stored in the execution history currently stored - it would be an unnaceptable space increase, with at least one Change in most instructions. The current history stores a list of snapshots of execution every historyMaxGap instructions where historyMaxGap is currently 500.
However, it would be acceptable for each history record to include the (single) most recent Change for every register and memory location that has even changed. That would roughly double the size of the history. OK. If needed historyMaxGap could be increased.
The current simulation provides, for each instruction executed, current and last Datapath values. From these it is possible to obtain Changes to registers and memory locations in that step. Registers can be worked out quickly - to get the list of memory changes would be slow in the case that thousands of memory locations had been written, because they would all need to be compared to find the one or two that had changed. Even so, I'm not sure it will be a problem because the Map module is very efficient, and if it is the memory instruction module can quickly be refactored to provide as output of each instruction a list of the locations that are written - from which the ones that are changed can quickly be determined.

This set of facts gives you a clear strategy for individual coding.

Get the current step Change info you need from the two dataPath values available to you each instruction. (One simple function).
During execution ensure that the most recent Change for each register / memory location is held (in a suitable datatype that can store this info). Every register and memory location that has been written will have one Change
Store this same info (the most recent change) in the histroy for each history record
Reconstruct your needed list of changes as follows:
- Look up the most recent Change
- Follow its step number (calling asmStep to retrieve from history the relevant data) the next Change
- Repeat as needed

The above is not difficult once you get your head round it. You can write your code in a module separate from existing Visual2 as follows (each function is simple):

Work out the Change type and the type for the set of most recent Change for each register / memory location.
Write a function (called once per instruction) that computes the next step Changes from RunInfo and the previous instruction per-step changes
Write a function that generates the new (enhanced) history record from the old one
Copy existing asmStep to your module and include your functions so it records Changes and has a history that records changes
Write a recursive function that calls your new asmStep etc to generate from the info stored in your modified History a list of at most N changes (you cannot generate all changes because that could be very long indeed)

You can test your function with unit tests (Expecto) on simple programs. Also, difficult but fun, property-based checks using (for example) the complex demo code. There are lots of properties that Changes satisfy if you compute them for the same program at different points in time.

Most code can be written and tested completely independently of V2. Your code does need to depend on V2 emulator, but does not require you to change the emulator - you can write all your stuff in a single module separate from the existing emulator. Note that my sample parsing code github repo (linked elsewhere) has a dotnet core project that links to the emulator source code the way you will want to.

Module Ordering

F# has a policy of module compilation ordering where dependencies are strictly one-way from later modules to earlier ones. Normally every module is a distinct file. All the IDEs show F# source files in module compile order and allow you to change this. This tends to lead to well structured code. When designing F# if possible find a strict order that eliminates out of order dependencies where a function in an earlier module (e.g. functionA in First below) requires a function defined in a later module (functionB in Second below).

Out of order dependencies can be managed by passing function parameters. In the skeleton code below an unavoidable forward dependency exists in which functionA calls functionB but (we suppose) for reasons of other dependencies it is not possible to define functionB in or before module First.

This is managed by adding a parameter bFunc to functionA. Any call to functionA must pass functionB in via this parameter. You can see in this example that functionC, defined before functionB, is able to do this because functionD - the place from which functionC is called, is able to pass functionB into functionC and from their to functionA whwre it is needed.

It is much preferable to order code and avoid forward references entirely - but if not this provides an escape hatch.

For individual code definitions that add functionality to Visual2 it is useful to have an initial plan about where the additional module(s) will be placed in the Visual2 module ordering as shown in Visual Studio. Sometimes things change, and I don't require that this be exact.

module First

    let functionA (bFunc: int-> int) (x:float) (y:float) =
        let n = functionB 3  // unavioidable forward dependency
        failwithf "Not implemented"

    let functionC (bFunc: int -> int) p q =
        let functionAwithB = functionA bFunc
        functionAWithB 1.0 2.0
        failwithf "Not implemented"

Module Second

let functionB (x:int)  : int =
    failwithf "Not Implemented..."

let functionD() =
    let funcC = functionC functionB
    failwithf "Not implemented..."

Module Ordering Example in Visual2

The Testbench enhancement was added late in Visual2 development. It consists of two files:

Renderer/Testbench.fs
Emulator/TestLib.fs

The Testbench.fs code references the main GUI that runs assembly programs and therefore must be late in the compile order, however it has functions called from Renderer/Integration.fs and therefore must be before this. The TestLib.fs file contains types and functions used during assembler simulation and therefore must be much earlier in the compile order inside the emulator project and before the renderer. The main entry point to Testbench operation is from the GUI menus defined in Renderer/Renderer.fs which is last in compile order.

The testbench code can then be written without forward references, divided between these two files.

In the Integration module the code added to run testbench tests in the GUI was difficult to place: runTests is referenced in asmStepDisplay runTests references asmStepDisplay`

The solution was to add a parameter stepFunc passed to runTests that always contains the asmStepDisplay function. That allows runTests to be defined before asmStepDisplay even though it also calls it.

NB - Another solution in F# would be to use mutually recursive functions using the and keyword as documented here. they are however considered a code smell, to be used only when needed.

For a much longer discussion of module ordering - with a lot of references to similar problems in OOP code and dependency injection as a typical industry solution - see cyclic dependencies and its two linked pages removing cyclic dependencies and cycles and modularity in the wild.

Parsing

Many of you will be working on additions that involve parsing lines of assembly code. Although the existing Visual2 code contains lots of examples of parsers, they do not keep track of the position in the source line that the error occurred. I will provide a nice tokeniser and some example match-based parsing functions enough to get you started. Note - the parsing needed here is much simpler than recursively defined abstract syntax trees.

Sample code in github

Background on Partial Active Patterns in Visual2

Parsing methods: EIE students will possibly use various ad hoc methods for parsing - regex etc. (Check with me if you have problems getting regex to work under dotnet or FABLE - both allow them but the functions are not exactly identical). The best way in F# to implement a variety of parsers is using partial active patterns, as shown in the sample code parse-something demo. A (very complex) example of PAPs used for parsing can be found in Emulator.Expressions where recursive PAPs are used to parse assembly literal constant arithmetic expressions.

Hooking into the renderer

Displaying better parse errors

The idea here is that your code will - in some cases - replace the existing line-based parse error messages by better ones. The simple way to do this keeps close to the current error display mechanism, which is as follows:
- The line errors are displayed (as hovers and underlining) using top level Editor.makeErrorInEditor.
- This is called (after file parse) from Integration.highlightErrorParse which unfortunately does not currently get the text of the parsed line, though it does get the line number. This is called from Integration.tryParseAndIndentCode. That function is given the LoadImage lim, from which the program text can be extracted.
- This function calls (via some intermediates) ExecutionTop.reLoadProgram
- reLoadProgram does all the work (via subfunctions) to parse the current assembler and return a list of line errors on any error
You could hook into this in many different places, since all you need is the line text (to reparse) and the error message (to replace by a better one if possible). To highlight part of a line you need to pass more data down to highlightErrorParse - which currently does not understand anything except whole line errors
Given this your new parse function(s) could be implemented in various places, depending on where you decide to hook it (them) in. Perhaps it makes best sense to add to the emulator, and hook in to the executionTop parse in which case the module would be just before this. However in that case the sequence of above functions will need modification so that a line segment can be highlighted instead of whole line.
For more complex functionality, e.g. autocorrect options, you would need more info returned from the parser (such as an autocorrect suggestion list) and additional code in the renderer to process the autocorrect information.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly