Do we really need coroutines in a SV event driven test ?

[anrusal]

Hi @mballance ,
First thing congratulations for this project, I find it amazing and I think it could be very usefull for so many different use cases.
One of the usecases that I am actually analysing is use this as a way to run directly python scripts without needing to use async/await in the script.
Please correct me if I am wrong but the only reason you would need coroutines in a script would be in a scenarion where Python needs to generate stimulus to the DUT in paralel like driving a clock and then waiting the script to generate a transaction perhaps on a bus. On this scenario it makes sense to me to pause the execution of the threads and let the scheduler to generate stimulus in parallel.
However most of the "normal" scripts that interact with real hardware don't need multi-threading or any other multi-task mechanism since you just interact with the real hardware using a single thread. I wonder if pyhdl-if could be use on something like a UVM driven environment in which Python just works as a single thread stimulus generation for transactions while all the other taks are carried out by the SV/UVM envionment wihtout requiring to use the annoying async/await coroutine calls.
Regards, Toni

[mballance]

Hi Toni, thanks for reaching out. Happy you see ways to make the project work for you!

In order to run as a single thread, PyHDL-IF would need to be architected in a very different way with respect to how it uses the simulator's DPI interface. And, the rearchitecture would restrict to always only running a single thread. Many testbench environments will need to deal with more than one device interface, and these environments require multiple threads. Consequently, coroutines really are required any time behavior on the Python side must consume simulation time -- for example, start a bus cycle and wait for it complete.

Now, it certainly is possible to use Python in a non-blocking way from SystemVerilog. A common use case would be implementing a scoreboard in Python. Any time new data comes in on the SystemVerilog side, SV calls Python and passes the data. Since the scoreboard simply reacts to the data, these calls need not be async/coroutine calls.

Hope this helps!

Best Regards, Matthew

[anrusal]

Hi @mballance,
thanks for your response. I don't fully understand what do you mean with "Many testbench environments will need to deal with more than one device interface". I might be wrong but when working with real hardware usually you only have one single thread and you basically communciate with one interface board through USB that interacts with your hardware through GPIO, SPI, etc. For this case you can have a SV TB that deals with all the parallel stuff like clock generation, reset, etc and just implement the python usb calls by hooking these calls with the corresponding agent using your library. Thinking of an example I will just reuse this SRF08 example from instructables:

import smbus
import time
bus = smbus.SMBus(0)
address = 0x70

#SRF08 REQUIRES 5V

def write(value):
        bus.write_byte_data(address, 0, value)
        return -1

def lightlevel():
        light = bus.read_byte_data(address, 1)
        return light

def range():
        range1 = bus.read_byte_data(address, 2)
        range2 = bus.read_byte_data(address, 3)
        range3 = (range1 << 8) + range2
        return range3

while True:
        write(0x51)
        time.sleep(0.7)
        lightlvl = lightlevel()
        rng = range()
        print lightlvl
        print rng

For usecases like the one above I don't see the need of a multithread and therefore we should be able to remove the coroutines.
I have actually tried to implement all the code required to call SV tasks from functions but I am currently stuck getting the response in my custom "invoke_hdl_t_from_func" function in call_proxy_dpi.py. If I understood correctly how your library works you basically create a thread in SV that retrieves objects from a queue. These objects are created from Python using DPI calls and at some point they are placed in the queue using the functions in pyhdl_if_call_dpi.svh. Then once the task is completed a call from SV to Python is done. This call triggers the event and tells Python to retrieve the response from SV and continue with the execution.
My code replicates this behaviour but removing the coroutines. The problem is that it gets stuck after the pyhdl_pi_if_queue_runnable call and I am not sure how to move forward. I have included some parts of my code here to illustrate the issue, hopefully you could point me to where the problem is.

call_proxy_dpi.py

def invoke_hdl_t_from_func(
           self,
           method_name : str,
           args : tuple):
       from hdl_if.backend import Backend

       be = Backend.inst();
       evt = be.mkEvent()
       lock = PythonLock()

       self.ep.invoke_hdl_t_from_func(self.obj_id, lock, method_name, args)
       while(lock.get() == 0):        
           pass        

The self.ep.invoke_hdl_t_from_func() is just a duplicate of the self.ep.invoke_hdl_t() created just for debugging.
Regards, Toni

[mballance]

Thanks for the the example, @anrusal . It seems to me that you might be looking for a way to leverage existing code in a way that works with PyHDL-IF without requiring changes (eg add await) to that code. If that's the case, let's try to find an adapter that we can use to connect the existing code to PyHDL-IF.
While I haven't looked into it deeply, cocotb provides some functionality for bridging blocking/non-async to async. You might look into how they implement the cocotb.function decorator described here. I think an approach like this could be adopted in PyHDL-IF it it makes sense for your application.

[anrusal]

Hi @mballance , that could work. However I was thinking on something slightly different.
I've spent part of this week modifying your code in order to implement this feature and as I mentioned in my previous message I am stuck on transfering the control of the program from the function invoke_hdl_t_from_func to SV. I wonder if you could explain how exactly the control is transfered since I am not capable to figure out by myself. I've used the -dpi_funct_trail option in xcellium to trace the dpi/tasks execution and it seems than in the original function invoke_hdl_t the control is transfered from python to SV when the await is executed.
Thanks, Toni

[mballance]

Happy to explain, of course. This is one of the things that makes a single-threaded implementation difficult at best...
In order to make a task call through DPI, the simulator must 'own' the thread initiating the call and the caller (Python) must properly configure the DPI context to match the intended target. In practical terms, this means that Python must run inside a SystemVerilog thread and the Python side must maintain and periodically restore some DPI-context handles. I barely got it to work, and it was a mess. Doing this also meant that PyHDL-IF wouldn't be able to be used together with the cocotb library, which was a design goal.

So, what the library does instead is to make the task-calling API non-blocking from the perspective of Python.

• The Python code calls a DPI function (non-blocking) that queues a task call for SystemVerilog to execute. The Python code waits on an asyncio `event'.
• A thread on the SystemVerilog side picks up the behavior to execute, runs it, and then notifies the Python side that the call is complete. When SV notifies Python that the call is complete, it notifies the asyncio `event. The Python call now unblocks.

One thing to carefully check is that your SystemVerilog code starts the SV thread. This is done with pyhdl_if_start(). There's an example below.

pyhdl-if/examples/call/dpi/call_sv_bfm/call_sv_bfm.sv

Lines 45 to 52 in 1ec8d92

	initial begin
	automatic Test test;
	pyhdl_if_start();
	#50ns;
	reset = 0;

My best suggestion is to start a regular Python thread that uses the same non-blocking mechanism for interacting with SystemVerilog, but allows your user code to avoid async await calls.

Hope this helps!

-Matthew

[anrusal]

Hi @mballance ,
thanks for the explanation. It very similar to what I understood after studying your code.
I followed your indications and built a non async code, however I am struggling with the Python to SV synchronization.

My call_proxy_dpy.py runs the function call using the next function

class PythonLock():
    def __init__(self):
        self.status = 0

    def set(self, val) -> None:
        self.status = 1
        print("PythonLock: set " + str(self.status))

    def get(self):
        return self.status

    def clear(self):
        self.status = 0
...
def invoke_hdl_t_from_func(
           self,
           method_name : str,
           args : tuple):
       from hdl_if.backend import Backend

       be = Backend.inst();
       evt = be.mkEvent()
       lock = PythonLock()

       self.ep.invoke_hdl_t_from_func(self.obj_id, lock, method_name, args)
       while(lock.get() == 0):        
           pass  

The self.ep.invoke_hdl_t_from_func(self.obj_id, lock, method_name, args) function calls the dpi to create the PyHdlPiRunnable object and put it on the queue, after that the code gets stuck on the while loop. The lock.get() never changes because the execution is never released to SV. In your original code the await releases the control from python on certain way letting the __pyhdl_pi_if_run task to dequeue the PyHdlPiRunnable created from Python to be later processed. I wonder if you know any way to get rid of the while( lock.get() == 0) so I can get SV to retrieve the control of the execution, process the TaskCallClosure object and then notify back python to proceed with the rest of the script.
Thanks, Toni.