Event-Driven-Architecture starter with Cement, Dramatiq, Grpc, NiceGUI

[TomFreudenberg]

Cement Issue Reporting and Feature Requests

Thanks for your work!

I have created a starter repository to use dramatiq and cement. Grpc is coming next.

Maybe this is interesting for you to see: https://github.com/tokeo/tokeo

[TomFreudenberg]

@derks could you please guide me a moment.

I want to implement a diskcache based app handler in the Tokeo cement app.

Would you decide to implement as Extension or as Plugin?

I am struggling when and for what to use plugin or extension.

Thanks in advance
Tom

[derks]

Quick reply (on mobile)... think of extensions as application agnostic, where plugins are application specific. Extensions extend the framework, plugins extend YOUR application.

[TomFreudenberg]

Thx! Ok, so implementing diskcache is extension like your's memcached and rediscache.

I checkout and send an PR when ready - maybe it is an extension in general

[TomFreudenberg]

Dear @derks another short question I didn't find yet.

Is there a way already defined by Cement (you) to access the initialized app object global from outside controller, handler etc.

E.g. I wrote some actors for dramatiq inside my app and there I need to access app.

I won't push the app always forward thru the different instances (too lazy). Currently I created a global_module to export app.

To export app in the main.py is not possible because of circular module errors afterwards.

Feels a bit quirky so I wanted to ask if you have something in mind?

[derks]

Apologies, on mobile. There is not officially. I would have to workout some ideas to give a good suggestion. If you had an MVCE it would help... I will try to work out an example idea though.

[TomFreudenberg]

Hi @derks thanks for quick feedback again.

So in case that you don't have something prepared, I will create an extension like 'share_app' and share the app object in that extension and register by hook.

[TomFreudenberg]

Hi @derks

it works with this very small extension, when enabled as extension.

I call it appshare :-)

In main.py just added to the extensions list:

    class Meta:

        extensions = [
            ...
            'tokeo.ext.appshare',
            ...
        ]

The file: tokeo/ext/appshare.py

"""

This module shares a globally access to the running app object

"""

class App():

    def __init__(self):
        self._app = None

    def __getattr__(self, key):
        # test _app object
        if self._app is None:
            raise AttributeError(f'\'App\' object has no attribute \'{key}\'')
        # return attribute
        return getattr(self._app, key)

app = App()

def load(app_to_share):
    app._app = app_to_share

and then it's accessible somewhere, e.g. actors by:

from tokeo.ext.appshare import app


def some_method():
    app.log.info('Here we are!')

[TomFreudenberg]

Hi @derks

after reading the foundation.py I realized to directly use the load() hook for extensions to directly activate the extensions. So now it's easy to setup just via extension.

Pretty fine

[TomFreudenberg]

Dear @derks

I switched over all my developments to this open source project:

https://github.com/tokeo/tokeo

I would be very happy if you could have a look on that especially about the extensions:

https://github.com/tokeo/tokeo/tree/master/tokeo/ext

Maybe you like also some of them to include in Cement core as well but not sure about this:

appshare.py | allow access to the main app object
diskcache.py | use diskcache.py as key/value cache and lock
dramatiq.py | include dramatiq
grpc.py | run a grpc server from proto files
pocketbase.py | use pocketbase as db
scheduler.py | integrate Apscheduler and have an interactive cron style scheduler
smtp.py | changes as you already know

---

I am very pleased, when you check the source codes and give me some feedback if I used the overall Cement framework conceptions in the right manner.

Thanks in advance for your feedback
Tom

[TomFreudenberg]

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

[TomFreudenberg]

@derks Sorry for coming up again, but could you please give me your guidiance about config:

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

Does ist make sense to add python-dotenv in addition or is there something similar already in place?

[derks]

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

I have had this use-case in other projects, but have not implemented anything officially. I'll have to try out your extension... I think one approach I would be interested in is something like profiles. Where you can have a set default_profile, but then be able to override:

# dynamically override profile
myapp -p <other_profile> ...


# set active profile
myapp profile set <other_profile>

# get active profile (default command)
myapp profile

default
profile-1
profile-2 **
other-profile


Something like that.... but I'll need to spend some time on it to consider a variety of use-cases.

[derks]

Dear @derks

I switched over all my developments to this open source project:

https://github.com/tokeo/tokeo

I would be very happy if you could have a look on that especially about the extensions:

https://github.com/tokeo/tokeo/tree/master/tokeo/ext

Maybe you like also some of them to include in Cement core as well but not sure about this:

appshare.py | allow access to the main app object diskcache.py | use diskcache.py as key/value cache and lock dramatiq.py | include dramatiq grpc.py | run a grpc server from proto files pocketbase.py | use pocketbase as db scheduler.py | integrate Apscheduler and have an interactive cron style scheduler smtp.py | changes as you already know

I am very pleased, when you check the source codes and give me some feedback if I used the overall Cement framework conceptions in the right manner.

Thanks in advance for your feedback Tom

Really impressed with the work you've done, and am happy you've chosen Cement for it. I definitely want to set aside time to review how you've used the framework, as well as the extensions you've built. I will do my best to provide feedback (now that 3.0.10 is done).

[TomFreudenberg]

Thanks @derks for your feedback.

I have created an appenv extension now for the config part and be very happy with that.

You may have a look here:

https://github.com/tokeo/tokeo/blob/master/tokeo/ext/appenv.py

That allows to have config like:

config/app.yaml
config/app.development.yaml
config/app.development.local.yaml
config/app.production.yaml
config/app.production.local.yaml

So you can put secrets as well into ENV_VARS like you defined by cement or put them into the .local yaml config.

You can start your app by:

MYAPP_ENV=dev myapp --help

or

MYAPP_ENV=production myapp --help

Settings get merged by cement config handler (last setting wins)

---

I really appriciate the work with cement - thanks for that piece of codes

[TomFreudenberg]

Dear @derks

to "navigate" in project folder, I need the location directory of the main.py from my cement app.

For that I have added an element to Meta but wonder if you already had something prepared I missed currently. That is the solution currently:

main.py

class Tokeo(App):
    """The Tokeo primary application."""

    class Meta:
        # this app name
        label = 'tokeo'

        # this app main path
        main_dir = os.path.dirname(fs.abspath(__file__))

        # configuration defaults
        config_defaults = dict(
            debug=False,
        )

It's just the line about main_dir of interest

In the appenv extension I can know prepare some APP_DIR_ vars relatively based on app._meta.main_dir

Do you have another suggestion or is this a proper way?

[TomFreudenberg]

Hi @derks

today I added a YamlConfigParser which will deep merge nested yaml configurations.

Maybe this would be an interesting feature to put directly into the framework at ConfigParser level?

Checkout: https://github.com/tokeo/tokeo/blob/master/tokeo/ext/yaml.py

[TomFreudenberg]

Hi @derks

hope you are fine.

I am making good progress using the Cement framework and our extensions. It is a pleasure to run work with this.

As a new extension there is a print output handler, but can run as normal print and will print all the args like:

app.print('Hello', a, b, 'Goodbye', sep=', ')

and a inspect method, to inspect given arguments with a number of keyword args like:

app.inspect(var_1, obj_1, var_2, system=True, methods=True, attributes=True, values=True, types=True)

This will do some little formatting and sending the inner information of object and vars.

If you like to add that per default, let me know that I create a PR you would like to add and match your style.

https://github.com/tokeo/tokeo/blob/master/tokeo/ext/print.py

In addition there is this Apscheduler extension you can use to create very detailed cron typed tasks. There is also an interactive command line included, that you can start and stop the tasks while running the scheduler.

Also there is a new "fire" command to instantely run a job whatever it is timed. That allows very good testing or ad hoc manoevers.

Last but not least the DiscCache Extension where you find also a throttle and a temper decoration to coordinate the amount of running tasks in parallel.

Looking forward to your next steps how to make that available for cement outside from tokeo

Cheers
Tom

[TomFreudenberg]

In addition let me ask this:

In your extensions and codes, you often create a new LOG like:

LOG = minimal_logger(__name__)
LOG.debug()

but in most of the times there is:

self.app.log.debug()

already existent.

Not using the app.log will put the log not thru the configured app handlers (like colorlog).

What is your reason in doing that, I prefer to use everything thru the app.

Thanks for your guidiance

Tom

[TomFreudenberg]

In addition let me ask this:
...
What is your reason in doing that, I prefer to use everything thru the app.

Hi @derks maybe you may give your advise on that, please

[TomFreudenberg]

Hi @derks

this is a new extension from the Tokeo

https://github.com/tokeo/tokeo/blob/master/tokeo/ext/automate.py

It is an automation extension to use invoke and fabric/paramikoSSH to run things by configuration on local and remote systems. There is a CLI call as well as an interactive shell, to fire the tasks. In conjunction with the scheduler extension those tasks could be fired manually and time controlled.

There are a kind of configurations to run the tasks only multiple hosts and groups:

hosts:
  dbhost:
    host: '192.168.0.5'
    user: 'dbuser'
    password: 'goahead'
  webhost:
    host: '192.168.0.8'
  proxy:
    host: '192.168.0.99'
    host_key: 'ssh-rsa ABCDEFEF...'

hostgroups:
  deployment:
    - dbhost
    - webhost
    - proxy

connections:
  port: 3322
  user: admin
  password: 'the_secret'
  sudo: 'allow_sudo_commands'

tasks:
  deploy:
    name: Handle the deployment
    module: 'mypackage.module'
    hosts: deployment
    kwargs:
      arg_named: value

mypackage.module

def deploy(app, connection, arg_named=''):
    app.log.info('Run the deployment ...')
    app.log.info(f'Parameter is set: {arg_named}')
    result = connection.run('call-my-deploment-script', hide=True, warn=False)
    return result

This can be fired now one by one or threaded parallel or interactive or ...

Some can integrate ansible in advanced for more ...

Maybe it will be usefull for others as well, I have to write the documentation for all that stuuf :-)

[TomFreudenberg]

A very simple local example is:

automate:
  tasks:
    module: tokeo.core.tasks.automate
    uname:
      name: uname automation
      kwargs:
        flags:
          - '-a'

tokeo.core.tasks.automate.py

from tokeo.core import tasks

def uname(app, connection, verbose=False, flags=None):
    app.log.info('Automation uname called')
    return connection.run(f'uname {" ".join(flags)}', hide=not verbose, warn=False)

using cli

yourapp automate run uname --verbose --as-json

result:

[[{"task_id": "uname", "connection_id": "_default", "host_id": "local", "stdout": "Darwin localhost.lan 22.6.0 Darwin Kernel Version 0.0.0: Thu Dec  1 12:00:01 PST 2099; root:xnu-8796.777~0/RELEASE_XOS risc64\n", "stderr": "", "command": "uname -a", "exited": 0, "values": null}]]

[derks]

In addition let me ask this:
...
What is your reason in doing that, I prefer to use everything thru the app.

Hi @derks maybe you may give your advise on that, please

@TomFreudenberg the reason for the minimal logger is so that parts of the app/framework/extensions can log before app.log is setup. It takes some thought by the extension developer, but basically... if you can use app.log, you should... but if you can't (yet) then you could use minimal logger (depending on where the logging is in the execution stack).