A global config and setup method

[AlanCoding]

Fixes #44

The argument for this is difficult and something I've struggled with. But it's a strong argument. We have 3 different actors:

1. The service itself, it runs tasks
2. The publisher, it submits tasks
3. Any dispatcherctl type command, exists in AWX, does control-and-reply to get debugging information

Because these are all invoked separately, we should requires the setup() method to be called, and if not, throw an error. Otherwise, it's very difficult to assure that the service is using the same config as the publisher, for example. On higher levels, this is easy to enforce using the Django .ready() method. But this will set the tone for how tests are written on the psycopg layer.

[Alex-Izquierdo]

Since this PR is a draft, I'm not sure if some of my questions are premature. I'n sorry in that case.

[AlanCoding]

Additional incoming change:

I've changed my tune on how __init__ arguments should be passed. We should be consistent, and this would result in needing to pass related objects as arguments when one object needs another. So the worker pool needs to be passed the process manager. Then the main object needs to be passed the worker pool. Then the factories will work out how to extract the kwargs from the config file. This suggests a new YAML structure like this:

service:
  pool_cls: dispatcher.pool.WorkerPool
  pool_kwargs:
    min_workers: 2
    max_workers: 12
  process_manager_cls: dispatcher.process.ForkServerManager
  process_manager_kwargs:
    preload_modules: my_app.hazmat

I was trying to make something work where we passed the process-manager kwargs into the main object... and this pattern doesn't scale well. And this all needs to be as "dumb", and straight-to-the-code as possible. Rephrasing, making the config file look messy is fine if it makes things simpler overall.

[Alex-Izquierdo]

Should have the config a schema from where it can be easily validated?

[AlanCoding]

Should have the config a schema from where it can be easily validated?

No and yes.

yes we can validate the schema. This is not jsonschema, but validating the content, which is a python dictionary by the point we would care to do this. Demo:

from dispatcher.config import setup, settings
setup(from_file='dispatcher.yml')
pkw = settings.service['pool_kwargs']

import inspect
from dispatcher.pool import WorkerPool
signature = inspect.signature(WorkerPool.__init__)
parameters = signature.parameters
valid_kwargs = [k for k, v in parameters.items() if v.kind != inspect.Parameter.VAR_POSITIONAL and v.kind != inspect.Parameter.VAR_KEYWORD]

>>> valid_kwargs
['self', 'worker_id', 'process']
>>> set(pkw.keys()) - set(valid_kwargs)
set()

This assures that all keys are valid. This doesn't do type validation, that would be a general problem that I would rather take to be solved, as opposed to solve myself.

I hope this answers the first part that no we don't want to write out a schema somewhere, because this general method defines the schema we want.

What we're lacking is an articulation of the value-add from doing fancy inspect stuff like this, as opposed to just letting the code go error. Possible answers:

1. We could use this inspection to construct a schema dynamically. Put it where then?
2. Validate the config schema in the publisher, not just in the service. But in that case, we could just initialize the objects and let the error happen that way, which would be simpler and more reliable.

[Alex-Izquierdo]

Should have the config a schema from where it can be easily validated?

No and yes.

yes we can validate the schema. This is not jsonschema, but validating the content, which is a python dictionary by the point we would care to do this. Demo:

from dispatcher.config import setup, settings
setup(from_file='dispatcher.yml')
pkw = settings.service['pool_kwargs']

import inspect
from dispatcher.pool import WorkerPool
signature = inspect.signature(WorkerPool.__init__)
parameters = signature.parameters
valid_kwargs = [k for k, v in parameters.items() if v.kind != inspect.Parameter.VAR_POSITIONAL and v.kind != inspect.Parameter.VAR_KEYWORD]

>>> valid_kwargs
['self', 'worker_id', 'process']
>>> set(pkw.keys()) - set(valid_kwargs)
set()

This assures that all keys are valid. This doesn't do type validation, that would be a general problem that I would rather take to be solved, as opposed to solve myself.

I hope this answers the first part that no we don't want to write out a schema somewhere, because this general method defines the schema we want.

What we're lacking is an articulation of the value-add from doing fancy inspect stuff like this, as opposed to just letting the code go error. Possible answers:

1. We could use this inspection to construct a schema dynamically. Put it where then?
2. Validate the config schema in the publisher, not just in the service. But in that case, we could just initialize the objects and let the error happen that way, which would be simpler and more reliable.

I agree that using JSON Schema might be overkill. However, my main concern is not how the validation is done but rather ensuring that validation happens effectively.

The primary argument for validation is the "fail faster, fail earlier" principle, which improves both error handling and user experience. If an error in the configuration can be detected early, we can provide a clear and specific error message explaining what is wrong and why it is not accepted. This prevents unnecessary execution and debugging effort.

I also disagree with the idea of simply initializing the objects and letting the error occur naturally. This approach may lead to errors that are harder to debug or, worse, result in silent failures or unexpected behaviors.

Your approach is certainly creative, but I see some potential issues. It would not account for optional parameters, and it relies on a strict 1:1 relationship between the configuration and function/object signatures. This limits flexibility and, in my opinion, introduces a partial coupling of concerns.

For these reasons, I advocate for an independent configuration that can be validated as early as possible.

[Alex-Izquierdo]

For these reasons, I advocate for an independent configuration that can be validated as early as possible.

I would like to clarify that I'm not going to block the PR on this matter. In one way or another, it is something that can be addressed perfectly in later iterations. I'm just sharing my PoV on how it could be. :)

[AlanCoding]

Auto-gen schema could help for documentation and versioning. We could put this in the repo and check for diff in checks. If there's a diff, then we can bump a version.

This is to help with a real problem, which is versioning. My intent was to cut a version (PyPI, or just a tag) before merging this because it requires changes to the eda-server branch. Auto-generating a spec file (that is enforced) would document when and how something changed to expected schema.

But failing earlier doesn't make sense to me. Initializing the objects from the factories (from settings) is "free", and can be done in any context.

[AlanCoding]

This is moving a bit fast, but I went ahead and pushed a commit that matches the description of my last comment.

I appreciate that this would be better as jsonschema. But that requires a general tool to convert type hints to jsonschema, there's no point in us maintaining that logic. Let's file an issue for it. Current format is kind of human readable, but jsonschema would be better.