How to deploy extensions for Cement with own apps

[TomFreudenberg]

Hi @derks

I come up here to make our discussion streamlined from here #741 (comment). You wrote:

My thought is, that you can package up an extension with a module named cement_ext.<your_extension> and publish to PyPi as cement[your_extension] like we do for buit-in extensions. Curious what your thoughts are on that as you have built more third-party extensions than any other developer.

Primary focus is that I don't want developers like yourself to be limited by what I can/will add to the official sources. I'd much rather see a community build up around it.

I did a little research how to deploy Tokeo Toolset best for others. First I built a wheel package and run that in my projects instead a duplicate from tokeo. That's fine by now and helps me to do the things easy.

So for now I removed all the packages I wrote from my main.py (do not activate by default) and just deploy the ext code inside the tokeo package. So some can install tokeo and cement and use for e.g. tokeo.ext.appshare inside their main.py as extension to export the app object application wide. That's fine.

Activating other extensions like tokeo.ext.nicegui will need to activate the extension by developer but also there should be some starter as already developed in tokeo like here (https://github.com/tokeo/tokeo/tree/master/tokeo/core/pages). To give the developer access to my "Hints" or "Starters", I think about creating a tokeo setup (just working name) command, where templates can run then to create such things in the developers app. Having only all the extension code inside the package does at least not enforce the developer to install all other components while they are not in use.

So, I am not sure if there is really more necessary else in Cement than already exist? I think, that I could deploy all my work as seperate package in parallel to cement. What do you thing about that.

Cheers
Tom

[derks]

@TomFreudenberg I think that approach would work fine. You are building a parallel project that other projects will build upon, so it is basically another layer on top of Cement. Having a separate package/distribution for every extension multiplies the management overhead so it is better for you if you can continue to maintain them all as one project. You can always break them out into separate projects/packages later.

As for tokeo setup it sounds like you want to maybe copy the cement generate functionality, no? This is all in the Cement Generate Extension, if you're not already using that:

• https://docs.builtoncement.com/extensions/generate

The idea there is, you can cement generate project ... but also cement generate extension ... inside that project also.

My only suggestion is, test it from the perspective of a developer who wants to use something like tokeo.ext.nicegui with a Cement App, but not necessarily a Tokeo App. It should function the same as if it were a builtin Cement Extension (limited dependencies), so for that you might want to implement virtual packages like we did with cement.

Example Dependencies:

• cement
• cement[some_extension]
• tokeo[some_other_extension]

That should upll in the dependencies for tokeo[some_other_extension], but not all the dependencies for all of the extensions. Does that make sense?

[TomFreudenberg]

@derks thanks for your response.

It is exactly as you have written above. I am checking out cement generate already and want to continue the way you setup there.

And yes, I am looking from a developers perspective who is not developing tokeo but his own app. That is how I use the parts already. I have build now 4 apps running on the stack Cement + Tokeo extensions.

It's fun to work on this and I guess the extensions are helping others to get fast progress on their work. The most important will be a good documentation to get things out.

Question on that: What do you use for the documentation site of Cement? I like your doc site very much.

[TomFreudenberg]

from @derks advise:

https://docs.builtoncement.com/getting-started/developer-tools#customizing-templates

how to customize templates

[TomFreudenberg]

Hi @derks

I am doing the template for tokeo to generate a custom project.
For now it works fine but I have one question:

.generate.yml

    -   name: feature_dramatiq
        prompt: "Enable dramatiq [(Y)es/(N)o]"
        validate: "^[YN]$"
        case: "upper"
        default: "Y"

I use that in main.py for example:

{% if feature_dramatiq == "Y" %}
            'tokeo.ext.dramatiq',
{% endif -%}

So far so good.

But I want to generate additional files in case of feature enabled or not.

Do you have a tipp for me how to solve that, e.g.

templates/generate/project/{{app_label}}/core/tasks/actors.py

should only get created when feature_dramatiq == "Y"

Thanks
Tom

[derks]

@TomFreudenberg that is a good question. I have not run into that use-case on my own yet, but it definitely makes sense what you're looking for here. For what it is worth, that logic happens here in the base template handler:

• https://github.com/datafolklabs/cement/blob/main/cement/core/template.py#L193

I don't think there is a clean way to do it currently, however I have a question... aren't the files part of the extension? I assume that tokeo.ext.dramatiq is a single python file currently? (ex: tokeo/ext/dramatiq.py). I'm wondering if you don't just make it a directory instead:

├── tokeo
    └── ext
        └── dramatiq
            └── __init__.py
            └── actors.py

Basically, whatever you have in dramatiq.py, move to ext/dramatiq/__init__.py. I think this should load the same, as this is the same way that plugins work.

In that way, the file is part of the extension, and not "core" which is what I assume you want. There wouldn't be any reason to not create the file as part of generate.

[TomFreudenberg]

Hi @derks

I am not sure if I got your advise correctly but I try to explain how I run it currently:

Understand tokeo as an cement plus - just delivering some more (additional) framework extensions and some more (addition) generator templates.

So the current enabled extensions for running tokeo are:

        # load additional framework extensions
        extensions = [
            'colorlog',
            'generate',
            'tokeo.ext.print',
            'tokeo.ext.jinja2',
            'tokeo.ext.yaml',
        ]

ext.jinja2: because of

            'keep_trailing_newline': True,
            'trim_blocks': True,

you have not currently exposed in Cement jinja2

ext.yaml: because of deep merging and overlaying values and configs

ext.print: because of .print and .inspect not yet part of Cement.

---

The rest is you delivering the new extensions as a package.

---

If you install cement and tokeo

you can run:

cement generate
# or
tokeo generate

That's it.

---

Starting your own project you can use by now:

        # load additional framework extensions
        extensions = [
            'colorlog',
            'generate',
            'tokeo.ext.print',
            'tokeo.ext.jinja2',
            'tokeo.ext.yaml',
            'tokeo.ext.appshare',
        ]

a mix of extensions for the framework Cement coming from cement or tokeo.

That's it

---

I have created a project template you can run by tokeo generate project which enables (on request prompts) the extensions from tokeo and give examples as implementations just how to use the tokeo extensions.

The extension code still stays in the installed tokeo package like cement package and is NOT part of the new custom project. They just get enabled by the avtivate extensions in main.py. This should be similar as Cement, isn't it?

---

So I do not understand what might change and help if I use tokeo.ext.dramatiq.py or split it into a directory. I checked up your extensions and they exist in one file, so I choose that to be mostly equal for 3rd party developers.

---

What I am looking for is: If a 3rd party dev says: "Dramatiq = No" on generate project prompt, then I do not activate the extensions, I do not add the package to myproject.toml etc. - BUT - the generator will copy the file from template {{app_label}}/core/tasks/actors.py because it is part of the template but without enabled dramatiq extension - just not necessary.

Does this all makes sense to you?

---

In the last time, there is another issue with templates just why they are (jinja2) templates. I can't run pyink or flake8 over them anymore for formatting and linting, because of the jinja2 {{ var }}and {% if %}statements. I could leave them out from test but that does not prevent me from inserting bad code into template in the next round. That is not an issue for the 3rd party developer but for a template creator. For that I have no solution by now. Maybe some templatepreprocessing could help.

---

You opened the box of pandorra while letting Cement into the wild :-)

I am just a user of this :-D

[derks]

@TomFreudenberg I get it now. What if we supported "features" like this?

Example:

---
### top level - current functionality

ignore:
    - "^(.*)ignore-this(.*)$"
    - "^(.*)ignore-that(.*)$"

exclude:
    - "^(.*)exclude-this(.*)$"
    - "^(.*)exclude-that(.*)$"

variables:
    - name: 'my_variable_name'
      prompt: 'The Prompt Displayed to The User'


### optional features

features:
    - name: dramatiq
      default: true

      # if true - then add additional vars/ignore/exclude
      enabled:
          variables:
              - name: 'additional_dramatiq_var'
                 prompt: 'The Prompt Displayed to The User'

      # if false - then add additional vars/ignore/exclude
      disabled:
           exclude:
               - "^(.*)exclude-this(.*)$"
               - "^(.*)exclude-that(.*)$"

Features could auto prompt:

"Enable Feature: <feature_name> [Yn]"

Basically, features could just wrap around the same functionality but guarded by a boolean that triggers additional actions.

Thoughts?

[TomFreudenberg]

Hi @derks

that sounds fine to me. Same functionality just by additional level for features.