FEATURE: Support LIN Description File (LDF) as supplement to DBC

[MatinF]

Hi Daniel,

Since canmatrix now supports an initial implementation of LIN Description Files, I think it would be a great feature to add for an upcoming asammdf release. The LDF could be a supported format when a user seeks to decode LIN data. It would help further cement asammdf as a tool with great LIN support.

Happy to test this and I can provide sample LIN MF4 data and LDF.

[danielhrisca]

First there should be a canmatrix release with LDF support

[danielhrisca]

@MatinF the development branch has support now but you need to install the latest canmatrix development code and also ldfparser

[ghost]

I didn't get this to work on the development branch, but I came up with an intermediate solution that is working nicely so far:

• Use LDF parser, ldfparser 0.8.0, to parse the LDF file.
• Build a DBC file from the LDF parser for the signals/frames of interest. I build it per request and on the fly previous to extract_bus_logging. Ignore the scaling and units, only obtain the raw values as the DBC does not support physical conversions as the LDF does. The DBC file does not need to be complete, just the frames and signals (BO_, SG_) and some header stuff, use a complete one as a template and validate it with CANdb++ for example.
• Call extract_bus_logging with the DBC file.
• Get the signals as usual, values and timestamps.
• The LDF parser provides converters to convert the raw values to physical values.

It is too much code to share here but thought it could give you a few ideas, if I come across limitations I will let you know, I am able to extract signals, raw and physical values, and also diagnostics like this from MDF files.

[MatinF]

This is a minor issue so can be closed. However, I did get to test it with the attached file/LDF in 7.2.0 and I seem to be unable to get it to work. I get below error (I have latest canmatrix installed).

LIN_LDF_EXAMPLE.zip

Error: <class 'Exception'>	 
Traceback (most recent call last):
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\gui\utils.py", line 612, in run
    self.output = self._target(*self._args, **self._kwargs)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\mdf.py", line 4633, in extract_bus_logging
    out = self._extract_lin_logging(
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\mdf.py", line 5024, in _extract_lin_logging
    dbc = load_can_database(Path(dbc_name))
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\blocks\utils.py", line 1807, in load_can_database
    dbs = func(arg, import_type=import_type, key="db", **kwargs)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\canmatrix\formats\__init__.py", line 71, in loadp
    return load(fileObject, import_type, key, **options)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\canmatrix\formats\__init__.py", line 86, in load
    module_instance = sys.modules["canmatrix.formats." + import_type]
KeyError: 'canmatrix.formats.ldf'


Details: 2022-12-27, 15:32:51
--------------------------------------------------------------------------------
<class 'Exception'>	 
Traceback (most recent call last):
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\gui\utils.py", line 612, in run
    self.output = self._target(*self._args, **self._kwargs)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\mdf.py", line 4633, in extract_bus_logging
    out = self._extract_lin_logging(
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\mdf.py", line 5024, in _extract_lin_logging
    dbc = load_can_database(Path(dbc_name))
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\blocks\utils.py", line 1807, in load_can_database
    dbs = func(arg, import_type=import_type, key="db", **kwargs)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\canmatrix\formats\__init__.py", line 71, in loadp
    return load(fileObject, import_type, key, **options)
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\canmatrix\formats\__init__.py", line 86, in load
    module_instance = sys.modules["canmatrix.formats." + import_type]
KeyError: 'canmatrix.formats.ldf'

--------------------------------------------------------------------------------
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\gui\widgets\file.py", line 1661, in extract_bus_logging
    mdf = run_thread_with_progress(
  File "c:\users\marti\appdata\local\programs\python\python38\lib\site-packages\asammdf\gui\utils.py", line 562, in run_thread_with_progress
    raise Exception(thr.error)

[MatinF]

I wanted to revisit this as we have a number of users that would like to use LDF files to decode their raw LIN bus data in asammdf.

I tried the latest build below, but it looks like it does not currently support decoding of the test data I attached:
https://github.com/danielhrisca/asammdf/actions/runs/12254274873

[MatinF]

As a follow up on this, I seem to be able to load and parse an LDF file now with the latest canmatrix (and ldfparser installed) as outlined in this post: ebroecker/canmatrix#531 - but I am still unable to use this functionality to decode data in asammdf using the sample files attached.

I am doing a new article on LIN description files and it would be awesome to showcase asammdf for some of the basic decoding functionality if we can get this supported, in particular as it would allow for a like-for-like comparison vs. DBC files for LIN decoding, since asammdf also supports this.

[MatinF]

Hi again Daniel, just wanted to check if you'd have the chance to check if this can be supported? I think the base functionality is there, but for some reason the decoding is not working. Perhaps I am missing some dependency installation to get it right. As per my previous post, the below zip contains raw LIN bus data and an LDF, so it should enable the possibility to test if this is working in the latest version or not.

Let me know if I should take some specific steps to test this out

LIN_LDF_EXAMPLE.zip

[danielhrisca]

The decoding works on my PC

• ldfparser==0.25.0
• canmatrix==1.0
• asammdf development branch

[MatinF]

Hi Daniel, thanks a ton!

I tested this out as follows:

1. Installed the asammdf GUI development branch
2. Installed ldfparser
3. Installed canmatrix

I can verify that I can now decode via the LDF file and 'to some extent' get the expected result.

However, one critical challenge with LDF vs. DBC files is the support for 'multiple physical value ranges'. I explain this in my article on LIN Description Files, see the section "Example 2: Convert BatteryCurrent".

In short, the decoding scale/offset values will depend on the value of the raw data. Currently, the use of the canmatrix 'master' as a dependency means that there is no support for the multiple physical value ranges at all, implying that only the last entry in a range of decoding rules will be loaded at all into canmatrix. However, Eduard fixes this in the development branch of canmatrix, so that all the ranges are available in the matrix object.

However, it looks like asammdf only takes into account the last entry in the range, even if I install the development branch of canmatrix in the virtual environment. This is why in your screenshot, most of the values are 'invalid' - this is because essentially the incorrect decoding factor/offset is loaded.

To illustrate this, consider below the IBS_SOC frame from the canmatrix loaded LDF file:

...
Frame(
    name='IBS_SOC',
    arbitration_id=ArbitrationId(id=34, extended=False),
    size=6,
    transmitters=['IBS'],
    is_complex_multiplexed=False,
    is_fd=False,
    comment='',
    signals=[
        Signal(
            name='SOC',
            start_bit=0,
            size=16,
            is_little_endian=True,
            is_signed=False,
            offset=0.0,
            factor=1.0,
            unit='SOC_Error',
            receivers=['SG'],
            comment=None,
            multiplex=None,
            mux_value=None,
            is_float=False,
            is_ascii=False,
            type_label='',
            enumeration=None,
            comments={},
            attributes={},
            values={65535: 'LIN_INIT'},
            mux_val_grp=[],
            muxer_for_signal=None,
            calc_min_for_none=True,
            calc_max_for_none=True,
            cycle_time=0,
            initial_value=65535,
            scale_ranges=[
                {'min': 0, 'max': 1500, 'factor': 0.1, 'offset': 0.0, 'unit': '%'},
                {'min': 1501, 'max': 65534, 'factor': 1.0, 'offset': 0.0, 'unit': 'SOC_Error'},
            ],
            min=1501,
            max=65534,
        )
...

As per the scale_ranges section, the ideal behavior would be that asammdf applies the decoding rules based on the scale ranges. In other words, for raw data byte values in the range 0 to 1500, a factor of 0.1 should be used and a unit of %. For the range 1501 to 65534, a factor of 1 should be used and a unit of SOC_Error. Currently, it seems that asammdf applies the last entry of the scale_ranges when performing the decoding, which is why e.g. the SOC signal is 10x too high in value throughout the decoded data.

If this can be incorporated, I believe that it would be possible to fully support LDF files within asammdf, making it a game changer for LIN bus decoding use cases.

Let me know if anything is unclear.

[danielhrisca]

@MatinF please try the get_bits branch code

[MatinF]

@danielhrisca Awesome, this indeed fixes the scaling and offset from what I can tell, enabling the correct decoding of data! The only 'missing' aspect is that the unit still seems to be extracted as before, i.e. from the last entry, rather than being dependent on the range. This may be more difficult to fix, but you can see it e.g. based on the SOC value as below. The value 80.0 is correct, but the unit was supposed to be % in this area:

[danielhrisca]

It is not possible to change the unit because it would bring in too much complexity

[MatinF]

That makes sense, I understand - could an alternative then be to instead replace the unit with "" (i.e. blank) for any signal decoded via an LDF? I believe this will give a better result than the current one where the unit is loaded, as the unit will most likely be wrong in 50-80% of signals when using LDF files in this way. If this can be done, I think this branch would be ready for incorporation into the development branch.

[danielhrisca]

To improve this I've made the change to use the unit found in the first scale range

[MatinF]

I think that change works well and yields the relevant unit in the vast majority of cases - well spotted! If possible, it would be great to get this loaded into the development branch, along with (if possible) a dependency on the canmatrix development branch (as that is required in order to support the LDF functionality fully). If the latter is not possible, I'll try to hear Eduard from canmatrix if he can do a new release of canmatrix with this included.