Jupyter server telemetry and event logging

.github/workflows/codeql-analysis.yml

@@ -14,7 +14,7 @@ name: "CodeQL"
 
 on:
   push:
-    branches: [ master ]
+    branches: [ '*' ]
   pull_request:
     # The branches below must be a subset of the branches above
     branches: [ master ]

.github/workflows/main.yml

@@ -25,7 +25,7 @@ jobs:
         architecture: 'x64'
     - name: Upgrade packaging dependencies
       run: |
-        pip install --upgrade pip setuptools wheel
+        pip install --upgrade pip setuptools wheel --user
     - name: Get pip cache dir
       id: pip-cache
       run: |

.gitignore

@@ -1,20 +1,13 @@
 MANIFEST
+docs/source/operators/events
 build
 dist
 _build
 docs/man/*.gz
 docs/source/api/generated
 docs/source/config.rst
 docs/gh-pages
-notebook/i18n/*/LC_MESSAGES/*.mo
-notebook/i18n/*/LC_MESSAGES/nbjs.json
-notebook/static/components
-notebook/static/style/*.min.css*
-notebook/static/*/js/built/
-notebook/static/*/built/
-notebook/static/built/
-notebook/static/*/js/main.min.js*
-notebook/static/lab/*bundle.js
+docs/source/events
 node_modules
 *.py[co]
 __pycache__

CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.1.0] - 2020-12-11
+
+[Full Changelog](https://github.com/jupyter-server/jupyter_server/compare/1.0.10...1.1.0)
+
+**Merged pull requests:**
+
+- Restore pytest plugin from pytest-jupyter [\#360](https://github.com/jupyter-server/jupyter_server/pull/360) ([kevin-bates](https://github.com/kevin-bates))
+- Fix upgrade packaging dependencies build step [\#354](https://github.com/jupyter-server/jupyter_server/pull/354) ([mwakaba2](https://github.com/mwakaba2))
+- Await \_connect and inline read\_messages callback to \_connect [\#350](https://github.com/jupyter-server/jupyter_server/pull/350) ([ricklamers](https://github.com/ricklamers))
+- Update release instructions and dev version [\#348](https://github.com/jupyter-server/jupyter_server/pull/348) ([kevin-bates](https://github.com/kevin-bates))
+- Fix test\_trailing\_slash  [\#346](https://github.com/jupyter-server/jupyter_server/pull/346) ([kevin-bates](https://github.com/kevin-bates))
+- Apply security advisory fix to master [\#345](https://github.com/jupyter-server/jupyter_server/pull/345) ([kevin-bates](https://github.com/kevin-bates))
+- Allow toggling auth for prometheus metrics [\#344](https://github.com/jupyter-server/jupyter_server/pull/344) ([yuvipanda](https://github.com/yuvipanda))
+- Port Notebook PRs 5565 and 5588 - terminal shell heuristics [\#343](https://github.com/jupyter-server/jupyter_server/pull/343) ([kevin-bates](https://github.com/kevin-bates))
+- Port gateway updates from notebook \(PRs 5317 and 5484\) [\#341](https://github.com/jupyter-server/jupyter_server/pull/341) ([kevin-bates](https://github.com/kevin-bates))
+- add check\_origin handler to gateway WebSocketChannelsHandler [\#340](https://github.com/jupyter-server/jupyter_server/pull/340) ([ricklamers](https://github.com/ricklamers))
+- Remove pytest11 entrypoint and plugin, require tornado 6.1, remove asyncio patch, CI work [\#339](https://github.com/jupyter-server/jupyter_server/pull/339) ([bollwyvl](https://github.com/bollwyvl))
+- Switch fixtures to use those in pytest-jupyter to avoid collisions [\#335](https://github.com/jupyter-server/jupyter_server/pull/335) ([kevin-bates](https://github.com/kevin-bates))
+- Enable CodeQL runs on all pushed branches [\#333](https://github.com/jupyter-server/jupyter_server/pull/333) ([kevin-bates](https://github.com/kevin-bates))
+- Asynchronous Contents API [\#324](https://github.com/jupyter-server/jupyter_server/pull/324) ([mwakaba2](https://github.com/mwakaba2))
+
+
+
 ## [1.0.6] - 2020-11-18
 
 1.0.6 is a security release, fixing one vulnerability:

MANIFEST.in

@@ -8,6 +8,12 @@ include setupbase.py
 # include everything in package_data
 recursive-include jupyter_server *
 
+# Event Schemas
+graft jupyter_server/event-schemas
+
+# Event Schemas
+graft jupyter_server/event-schemas
+
 # Documentation
 graft docs
 exclude docs/\#*

docs/doc-requirements.txt

@@ -8,4 +8,5 @@ prometheus_client
 sphinxcontrib_github_alt
 sphinxcontrib-openapi
 sphinxemoji
-git+https://github.com/pandas-dev/pydata-sphinx-theme.git@master
+git+https://github.com/pandas-dev/pydata-sphinx-theme.git@master
+jupyter_telemetry_sphinxext

docs/environment.yml

@@ -13,4 +13,5 @@ dependencies:
   - sphinxcontrib_github_alt
   - sphinxcontrib-openapi
   - sphinxemoji
-  - git+https://github.com/pandas-dev/pydata-sphinx-theme.git@master
+  - git+https://github.com/pandas-dev/pydata-sphinx-theme.git@master
+  - sphinx-jsonschema

docs/source/_static/theme_overrides.css

@@ -0,0 +1,13 @@
+/* override table width restrictions */
+@media screen and (min-width: 767px) {
+
+   .wy-table-responsive table td {
+      /* !important prevents the common CSS stylesheets from overriding
+         this as on RTD they are loaded after this stylesheet */
+      white-space: normal !important;
+   }
+
+   .wy-table-responsive {
+      overflow: visible !important;
+   }
+}

docs/source/conf.py

@@ -70,8 +70,7 @@
     'sphinx.ext.mathjax',
     'IPython.sphinxext.ipython_console_highlighting',
     'sphinxcontrib_github_alt',
-    'sphinxcontrib.openapi',
-    'sphinxemoji.sphinxemoji'
+    'jupyter_telemetry_sphinxext'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -208,6 +207,12 @@
 #       since it is needed to properly generate _static in the build directory
 html_static_path = ['_static']
 
+html_context = {
+    'css_files': [
+            '_static/theme_overrides.css',  # override wide tables in RTD theme
+        ],
+}
+
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
 # directly to the root of the documentation.
@@ -371,3 +376,8 @@
 
 # import before any doc is built, so _ is guaranteed to be injected
 import jupyter_server.transutils
+
+# Jupyter telemetry configuration values.
+jupyter_telemetry_schema_source = "../jupyter_server/event-schemas"   # Path is relative to conf.py
+jupyter_telemetry_schema_output = "source/operators/events"           # Path is relative to conf.py
+jupyter_telemetry_index_title = "Telemetry Event Schemas"                # Title of the index page that lists all found schemas.

docs/source/developers/contents.rst

@@ -192,19 +192,19 @@ methods:
    ContentsManager.is_hidden
 
 You may be required to specify a Checkpoints object, as the default one,
-``FileCheckpoints``, could be incompatible with your custom 
+``FileCheckpoints``, could be incompatible with your custom
 ContentsManager.
 
 Customizing Checkpoints
 -----------------------
 .. currentmodule:: jupyter_server.services.contents.checkpoints
 
-Customized Checkpoint definitions allows behavior to be 
+Customized Checkpoint definitions allows behavior to be
 altered and extended.
 
 The ``Checkpoints`` and ``GenericCheckpointsMixin`` classes
 (from :mod:`jupyter_server.services.contents.checkpoints`)
-have reusable code and are intended to be used together, 
+have reusable code and are intended to be used together,
 but require the following methods to be implemented.
 
 .. autosummary::
@@ -220,7 +220,7 @@ No-op example
 ~~~~~~~~~~~~~
 
 Here is an example of a no-op checkpoints object - note the mixin
-comes first. The docstrings indicate what each method should do or 
+comes first. The docstrings indicate what each method should do or
 return for a more complete implementation.
 
 .. code-block:: python
@@ -238,7 +238,7 @@ return for a more complete implementation.
         def delete_checkpoint(self, checkpoint_id, path):
             """deletes a checkpoint for a file"""
         def list_checkpoints(self, path):
-            """returns a list of checkpoint models for a given file, 
+            """returns a list of checkpoint models for a given file,
             default just does one per file
             """
             return []
@@ -267,3 +267,24 @@ ContentsManager.
 .. _NBFormat: https://nbformat.readthedocs.io/en/latest/index.html
 .. _PGContents: https://github.com/quantopian/pgcontents
 .. _PostgreSQL: https://www.postgresql.org/
+
+Asynchronous Support
+--------------------
+
+An asynchronous version of the Contents API is available to run slow IO processes concurrently.
+
+- :class:`~manager.AsyncContentsManager`
+- :class:`~filemanager.AsyncFileContentsManager`
+- :class:`~largefilemanager.AsyncLargeFileManager`
+- :class:`~checkpoints.AsyncCheckpoints`
+- :class:`~checkpoints.AsyncGenericCheckpointsMixin`
+
+.. note::
+
+   .. _contentfree:
+
+   In most cases, the non-asynchronous Contents API is performant for local filesystems.
+   However, if the Jupyter Notebook web application is interacting with a high-latent virtual filesystem, you may see performance gains by using the asynchronous version.
+   For example, if you're experiencing terminal lag in the web application due to the slow and blocking file operations, the asynchronous version can reduce the lag.
+   Before opting in, comparing both non-async and async options' performances is recommended.
+

docs/source/developers/dependency.rst

@@ -1,12 +1,10 @@
 Depending on Jupyter Server
 ===========================
 
-If your project depends directly on Jupyter Server, be sure to watch Jupyter Server's :ref:`Change Log <changelog>` and pin your project to a version that works for your application. Major releases represent possible backwards-compatibility breaking API changes or features.
+If your project depends directly on Jupyter Server, be sure to watch Jupyter Server's ChangeLog and pin your project to a version that works for your application. Major releases represent possible backwards-compatibility breaking API changes or features.
 
 When a new major version in released on PyPI, a branch for that version will be created in this repository, and the version of the master branch will be bumped to the next major version number. That way, the master branch always reflects the latest un-released version.
 
-To see the changes between releases, checkout the :ref:`Change Log <changelog>`.
-
 To install the latest patch of a given version:
 
 .. code-block:: console

docs/source/developers/index.rst

@@ -13,4 +13,3 @@ These pages target people writing Jupyter Web applications and server extensions
    extensions
    savehooks
    contents
-   ../other/changelog

docs/source/eventlog.rst

@@ -0,0 +1,61 @@
+Eventlogging and Telemetry
+==========================
+
+The Notebook Server can be configured to record structured events from a running server using Jupyter's `Telemetry System`_. The types of events that the Notebook Server emits are defined by `JSON schemas`_ listed below_ emitted as JSON data, defined and validated by the JSON schemas listed below.
+
+
+.. _logging: https://docs.python.org/3/library/logging.html
+.. _`Telemetry System`: https://github.com/jupyter/telemetry
+.. _`JSON schemas`: https://json-schema.org/
+
+Emitting Server Events
+----------------------
+
+Event logging is handled by its ``Eventlog`` object. This leverages Python's standing logging_ library to emit, filter, and collect event data. 
+
+To begin recording events, you'll need to set two configurations:
+
+    1. ``handlers``: tells the EventLog *where* to route your events. This trait is a list of Python logging handlers that route events to 
+    2. ``allows_schemas``: tells the EventLog *which* events should be recorded. No events are emitted by default; all recorded events must be listed here.
+
+Here's a basic example for emitting events from the `contents` service:
+
+.. code-block::
+
+    import logging
+
+    c.EventLog.handlers = [
+        logging.FileHandler('event.log'),
+    ]
+
+    c.EventLog.allowed_schemas = [
+        'hub.jupyter.org/server-action'
+    ]
+
+The output is a file, ``"event.log"``, with events recorded as JSON data.
+
+`eventlog` endpoint
+-------------------
+
+The Notebook Server provides a public REST endpoint for external applications to validate and log events
+through the Server's Event Log. 
+
+To log events, send a `POST` request to the `/api/eventlog` endpoint. The body of the request should be a 
+JSON blog and is required to have the follow keys:
+
+    1. `'schema'` : the event's schema ID.
+    2. `'version'` : the version of the event's schema.
+    3. `'event'` : the event data in JSON format.
+
+Events that are validated by this endpoint must have their schema listed in the `allowed_schemas` trait listed above.
+
+.. _below:
+
+
+Server Event schemas
+--------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   events/index

docs/source/operators/index.rst

@@ -12,4 +12,5 @@ These pages are targeted at people using, configuring, and/or deploying multiple
    configuring-extensions
    migrate-from-nbserver
    public-server
-   security
+   security
+   telemetry

docs/source/operators/telemetry.rst

@@ -0,0 +1,61 @@
+Telemetry and Eventlogging
+==========================
+
+Jupyter Server can be configured to record structured events from a running server using Jupyter's `Telemetry System`_. The types of events that the Server emits are defined by `JSON schemas`_ listed below_ emitted as JSON data, defined and validated by the JSON schemas listed below.
+
+
+.. _logging: https://docs.python.org/3/library/logging.html
+.. _`Telemetry System`: https://github.com/jupyter/telemetry
+.. _`JSON schemas`: https://json-schema.org/
+
+Emitting Server Events
+----------------------
+
+Event logging is handled by its ``Eventlog`` object. This leverages Python's standing logging_ library to emit, filter, and collect event data.
+
+To begin recording events, you'll need to set two configurations:
+
+    1. ``handlers``: tells the EventLog *where* to route your events. This trait is a list of Python logging handlers that route events to
+    2. ``allows_schemas``: tells the EventLog *which* events should be recorded. No events are emitted by default; all recorded events must be listed here.
+
+Here's a basic example for emitting events from the `contents` service:
+
+.. code-block::
+
+    import logging
+
+    c.EventLog.handlers = [
+        logging.FileHandler('event.log'),
+    ]
+
+    c.EventLog.allowed_schemas = [
+        'hub.jupyter.org/server-action'
+    ]
+
+The output is a file, ``"event.log"``, with events recorded as JSON data.
+
+`eventlog` endpoint
+-------------------
+
+The Notebook Server provides a public REST endpoint for external applications to validate and log events
+through the Server's Event Log.
+
+To log events, send a `POST` request to the `/api/eventlog` endpoint. The body of the request should be a
+JSON blog and is required to have the follow keys:
+
+    1. `'schema'` : the event's schema ID.
+    2. `'version'` : the version of the event's schema.
+    3. `'event'` : the event data in JSON format.
+
+Events that are validated by this endpoint must have their schema listed in the `allowed_schemas` trait listed above.
+
+.. _below:
+
+
+Server Event schemas
+--------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   events/index