Config validation

[Graf-Wronski]

Related Issue

Relates to #664 .

Description

This branch featues a draft for config validation and respective tests.

Checklist

• Documentation updated (docstrings, READMEs, user guides, inline comments, doc folder updates etc.)
• New unit/integration tests added (if applicable)
• Changes noted in release notes (if any)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

[jrasko]

maybe its best to move this to the add_market method so the simulation fails even earlier, when adding an invalid market 🤔

[Graf-Wronski]

I think you are right, that add_market would be the best place for this part if we organize validation locally. Regarding a possible DataValidation class, to me it sounded more like that the validation is intended to be done centrally for all data.

[jrasko]

see above, this may be moved to the add_unit_operator method

[jrasko]

I think this part is already ensured by the unit_operator constructor, where a new unit operator references all available markets automatically (i may be wrong though)

[Graf-Wronski]

Hmm. You are right, there is a similar logic in UnitsOperator.__init__. However (and this is why the corresponding test works): You theoretically can add a market to World after you added all UnitsOperators.

This makes we wonder what the purpose of the respective logic in UnitsOperator.__init__ is.

for market in self.available_markets:
            if market.market_id not in self.portfolio_strategies.keys():
                self.portfolio_strategies[market.market_id] = (
                    DirectUnitOperatorStrategy()
                )

[maurerle]

You theoretically can add a market to World after you added all UnitsOperators.

Yes, theoretically - it is currently not supported though. So one first has to create the markets and then add the participants.

Fixing this would require to somehow inject the information of available markets prior to the start of the simulation into all UnitsOperators, which would add another layer of magic/complexity to the simulation.

[Graf-Wronski]

I am a little confused. Does "Not supported" mean a code like this should raise an Exception?

world = setup_simple_world()

@pytest.fixture
def demand():
    return Demand(
        id="test_unit",
        unit_operator="test_operator",
        min_power=0,
        max_power=-1000,
        technology="demand",
        bidding_strategies={},
        forecaster=DemandForecaster(index, demand=-100),
    )


def test_addition_order(demand):
    """ Add participants before market. """
    
    
    world.add_unit_operator(
        "unit_operator",
        {"dispatch": "naive_eom"})
    world.add_unit_instance(
        operator_id="unit_operator",
        unit=demand)
    
    opening_dispatch = rr.rrule(rr.HOURLY, dtstart=world.start, until=world.end)
    
    market_products= [MarketProduct(
        duration=datetime.timedelta(hours=1),
        count=1,
        first_delivery=datetime.timedelta(hours=1))]
    
    dispatch_config = MarketConfig(
        "dispatch",
        opening_hours=opening_dispatch,
        market_products=market_products)
    
    world.add_market_operator("market_operator")
    world.add_market(
        market_operator_id="market_operator",
        market_config=dispatch_config)
    
    world.run()

[kim-mskw]

This seems to be perfect for the post_init method of the config, right?

[maurerle]

After creation of the market, you do not know if there will be participants.
After creation of a single agent, you do not know if there will be further agents.
So the only place to check if there is any UnitsOperator, which has a strategy for this market configured is right prior to the run execution call of the simulation, by iterating over all units of all UnitsOperators and checking the BiddingStrategies..

[kim-mskw]

Thanks @Graf-Wronski! I think the general discussion we have to have is whether this should happen centrally or decentralized. I noticed that @jrasko and I immediately suggested where these have to go in the framework.

Two things to consider for this:
What would be the best way to do it concerning the potential reusage in the GUI @jrasko. Or is reusage not an option? Remind me again.

What is the most natural way if one wants to extend these? Change a validation call in the unit-operator and market, and post_init, or in one validator class? What do you think @Graf-Wronski?

[jrasko]

@kim-mskw for now, it is irrelevant for the frontend when the validation takes place, since simulations are executed immeadiatly after creation.
However, I would suggest you consider the following:
Decentralized validation can prevent a World to be missconfigured at any time, so that future features (e.g. saving a created world and running it later or on a different machine) are ensured to use a valid configuration at all times. We also use this method already when creating units.
Centralized validation may provide a better overview on which validations exists.
I would therefore like to hear some more oppinions on this matter

[kim-mskw]

@kim-mskw for now, it is irrelevant for the frontend when the validation takes place, since simulations are executed immeadiatly after creation. However, I would suggest you consider the following: Decentralized validation can prevent a World to be missconfigured at any time, so that future features (e.g. saving a created world and running it later or on a different machine) are ensured to use a valid configuration at all times. We also use this method already when creating units. Centralized validation may provide a better overview on which validations exists. I would therefore like to hear some more oppinions on this matter

I am not sure I see the use case of creating a world and then saving it to deploy it somewhere else. When and why would I do that?

[Graf-Wronski]

Thanks @Graf-Wronski! I think the general discussion we have to have is whether this should happen centrally or decentralized. I noticed that @jrasko and I immediately suggested where these have to go in the framework.

Two things to consider for this: What would be the best way to do it concerning the potential reusage in the GUI @jrasko. Or is reusage not an option? Remind me again.

What is the most natural way if one wants to extend these? Change a validation call in the unit-operator and market, and post_init, or in one validator class? What do you think @Graf-Wronski?

I think with regards to extensibility, a central validation class would be advantageous since already implemented validations are more accesible. Also functions are shorter if validation is outsourced, which I (personally) prefer. The big question mark for central validation however is, where in the pipeline is it triggered.

[Graf-Wronski]

This logic will not work for markets that open multiple times.

x.opening_hours[0] + x.opening_duration

seems to be a applicable alterantive.

[kim-mskw]

Also, another thing to check. If one wants to use the compley_clearing but does not specify the param_dict it will just give a weird error that the nodes do not match because it initialisses with a default node

  market_mechanism: complex_clearing
  param_dict:
    network_path: .
    solver: highs
    zones_identifier: zone_id
    pricing_mechanism: pay_as_clear

[maurerle]

I would differentiate between independent misconfigurations and depending misconfigurations.

Generally, I would fail as early as possible - see also.
I think that it makes sense to have the validation at different places.
Otherwise, we would need to have all places which raise a ValueError in some kind of validator..?

I do see good use of a validation method for a pre-run check though.
This would cover

• Config validation #684 (comment)
• checks for the ordering of markets
• check for available demand and generation on markets
• ...

[maurerle]

@Graf-Wronski this issue is a little bit like a whish list and I guess that some of the comments here should rather be placed as additional entries in #664

This might be overwhelming at first, but remember to always do small steps without trying to solve everything at once :)

[kim-mskw]

@Graf-Wronski this issue is a little bit like a whish list and I guess that some of the comments here should rather be placed as additional entries in #664

This might be overwhelming at first, but remember to always do small steps without trying to solve everything at once :)

Moved them to the issue :)

[Graf-Wronski]

So I have been thinking a little a bit about the optimal placement for validation and why it appears a little controversial and tried to summarize our discussion. I think one potential source of controversy is the duality of file-based and script-based world creation.

If there was only file-based world creation, I think the best approach would be to load the data and immediately within the DataLoader validate the data (#failearly) before creating a world.

For the creation of a world step by step within a script, the case is more complex. There are some validations that appear a little more intuitive when conducted by the respective add_* method, e.g. when adding a market to the world, check if its 'start_time' fits simulation time. Others can not be conducted before world.run() as their validity depends on potential later world changes, e.g. as above when you want to check if a market is actually referenced.

Now there are these three possible validation locations:

1. Validation in DataLoader.
2. Validation in respective World.add_*
3. Valiation at the beginning of World.run()

Location 3 is the only one that is suitable as standalone method.

Location 1 can only work as a standalone method for file_based scenarios, while location 2 can only work in combination with location 3.

If I understand @maurerle correctly, you vote for location 2 where it is possible and location 3 where it is necessary.

I personally would go for validation only at location 3 but I could live with 2+3.

Are there any other opinions?

And does anyone see further need for validation for loaded data (at location 1)?