Netbox 3.5 script load changes cause script module load errors

[Omripresent]

NetBox version

v3.5.1

Python version

3.10

Steps to Reproduce

1. Create a custom script module using a subdirectory and init.py and a script class within this file
2. Mount the module under the scripts/ directory
3. Try to access the script via its API endpoint
4. Receive DoesNotExist at /api/extras/scripts/.

Expected Behavior

We should be able to make an API call to start a custom script execution when deployed as a module.

Observed Behavior

A ScriptModule matching query does not exist exception raised.

Traceback:

Environment:


Request Method: GET
Request URL: https://netbox-dev/api/extras/scripts/config.Render/

Django Version: 4.1.9
Python Version: 3.10.6
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'corsheaders',
 'debug_toolbar',
 'graphiql_debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'graphene_django',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'core',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'wireless',
 'django_rq',
 'drf_spectacular',
 'drf_spectacular_sidecar',
 'netbox_bgp.BGPConfig']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'netbox.middleware.RemoteUserMiddleware',
 'netbox.middleware.CoreMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 56, in inner
    response = get_response(request)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 55, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/opt/netbox/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/netbox/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/opt/netbox/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/netbox/netbox/extras/api/views.py", line 313, in retrieve
    module, script = self._get_script(pk)
  File "/opt/netbox/netbox/extras/api/views.py", line 285, in _get_script
    module, script = get_module_and_script(module_name, script_name)
  File "/opt/netbox/netbox/extras/scripts.py", line 465, in get_module_and_script
    module = ScriptModule.objects.get(file_path=f'{module_name}.py')
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/netbox/venv/lib/python3.10/site-packages/django/db/models/query.py", line 650, in get
    raise self.model.DoesNotExist(

Exception Type: DoesNotExist at /api/extras/scripts/config.Render/
Exception Value: ScriptModule matching query does not exist.

[Omripresent]

Note that this functionality worked until version 3.4, been working since 2.11

[kkthxbye-code]

Sadly the new script system is substantially different than the old system, and a lot of functionality that allows code reuse and better organization simply does not work anymore.

See:

#12432

#12431

It is still unclear if the new system will reach parity with pre-3.5 in these regards.

[Omripresent]

Noted, thank you for the quick response.

[Omripresent]

I found the cause of the issue I've been seeing and it doesn't seem to be related to #12432 or #12431.

There's a difference between how the Django view is looking up the ScriptModule compared to the API's view.

I have this proposed change to update the module based lookup on each instance Omripresent@f23d7e6

Should I open a new bug from this discussion and submit a PR for it?

[XioNoX]

I'm only seeing this thread now, I opened #16698 for an issue similar to what you're describing here and I'd say it's worth opening one for the issue you're facing.
I'm also wondering if it would be possible to upstream some of the script related features implemented by @kkthxbye-code in netbox-script-manager or if some features are already on the Netbox roadmap ? @jeremystretch maybe ?
Thanks

[cs-1]

I don't see any hints regarding the custom scripts changes in the release notes of v3.5.0 and v3.5.1 -- is there documentation describing what has changed and how to adjust custom scripts to the new script system?

[kkthxbye-code]

There is no documentation of the change no.

The changes are implemented mostly in these FR's by Jeremy:

#11890

#12068

#12081