Merge pull request #178 from crim-ca/docs-pkg

Docs pkg

CHANGES.rst

@@ -13,8 +13,9 @@ Changes:
 - Add support of documentation RST file redirection to generated HTML for reference resolution in both Github source
   and Readthedocs served pages.
 - Improve documentation links, ReadTheDocs format and TOC references.
-- Avoid logging ``stdout/stderr`` in workflows
-- Add tests to make sure processes ``stdout/stderr`` are logged
+- Avoid logging ``stdout/stderr`` in workflows.
+- Add tests to make sure processes ``stdout/stderr`` are logged.
+- Remove Python 2.7 version as not *officially* supported.
 
 Fixes:
 ------
@@ -68,7 +69,7 @@ Changes:
 - Use ``weaver.request_options`` for `WPS GetCapabilities` and `WPS Check Status` requests under the running job.
 - Change default ``DOCKER_REPO`` value defined in ``Makefile`` to point to reference mentioned in ``README.md`` and
   considered as official deployment location.
-- Add ``application/x-cwl`` MIME-type supported with updated ``EDAM 1.24`` onthology.
+- Add ``application/x-cwl`` MIME-type supported with updated ``EDAM 1.24`` ontology.
 - Add ``application/x-yaml``  MIME-type to known formats.
 - Add ``application/x-tar`` and ``application/tar+gzip`` MIME-type (not official) but resolved as *synonym*
   ``application/gzip`` (official) to preserve compressed file support during `CWL` format validation.
@@ -409,4 +410,4 @@ Fixes:
 
 - Initial Release. Based off `Twitcher`_ tag `ogc-0.4.7`.
 
-.. _Twitcher: https://github.com/Ouranosinc/Twitcher
+.. _Twitcher: https://github.com/Ouranosinc/Twitcher

README.rst

@@ -26,6 +26,7 @@ For more details, see `Configuration`_ section.
 
 .. list-table::
     :stub-columns: 1
+    :widths: 20,80
 
     * - dependencies
       - | |py_ver| |requires| |pyup|
@@ -34,8 +35,8 @@ For more details, see `Configuration`_ section.
     * - releases
       - | |version| |commits-since| |license|
 
-.. |py_ver| image:: https://img.shields.io/badge/python-2.7%2C%203.6%2B-blue.svg
-    :alt: Requires Python 2.7, 3.6+
+.. |py_ver| image:: https://img.shields.io/badge/python-3.6%2B-blue.svg
+    :alt: Requires Python 3.6+
     :target: https://www.python.org/getit
 
 .. |commits-since| image:: https://img.shields.io/github/commits-since/crim-ca/weaver/1.10.1.svg

docs/_static/.gitignore

@@ -1,3 +1,4 @@
 # make this directory available as it is referenced by some extensions
 *.*
+!custom.css
 !.gitignore

docs/_static/custom.css

@@ -0,0 +1,26 @@
+
+/* override readthedocs theme to enforce using full-screen width for content */
+.wy-nav-content {
+    max-width: none;
+}
+
+/* force code-table to align their cells to top (align code line numbers between columns)
+    note: class attribute must be applied to match this specific type of table
+*/
+.code-table tbody tr td {
+    vertical-align: top !important;
+    /* max-width: min-content;*/
+}
+
+/* force code blocks to be a small as possible and centered
+   but also revert for tables that already does it across its full width
+   (without revert, some horizontal scrollbars for 1px move are displayed)
+*/
+/*
+div[class^="highlight"] {
+    max-width: min-content;
+}
+.code-table tbody tr td div {
+    max-width: none !important;
+}
+*/

docs/source/appendix.rst

@@ -11,6 +11,7 @@ Glossary
     `ADES`: Application Deployment and Execution Service
     `CWL`_: Common Workflow Language
     `EMS`: Execution Management Service
+    `HREF`: Hyperlink Reference
     I/O: Inputs and/or Outputs of CWL and/or WPS depending on context
     `OGC`_: Open Geospatial Consortium
     `WKT`: Well-Known Text geometry representation

docs/source/conf.py

@@ -235,7 +235,10 @@ def doc_redirect_include(file_path):
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-# html_static_path = ["_static"]
+html_static_path = ["../_static"]
+
+# override some styles of the selected theme
+html_css_files = ["custom.css"]
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied

docs/source/faq.rst

@@ -0,0 +1,117 @@
+.. _faq:
+.. include:: references.rst
+
+*************************
+FAQ
+*************************
+
+This section present some commonly encountered use-cases and basic solutions regarding `ADES`/`EMS` operation or
+more specifically related to `CWL` specification.
+
+.. contents::
+    :local:
+    :depth: 2
+
+
+How to specify the Docker image reference?
+==================================================
+
+In most situations, the ``CommonLineTool`` process will need to run a docker image. Doing so is as simple as adding the
+``DockerRequirement`` (`reference <cwl-docker-req>`_) as follows to the `Application Package` definition:
+
+.. code-block:: json
+
+    {
+      "cwlVersion": "v1.0",
+      "requirements": {
+        "DockerRequirement": {
+          "dockerPull": "<docker-url>"
+        }
+      },
+      "inputs": ["<...>"],
+      "outputs": ["<...>"],
+    }
+
+
+.. note::
+    The docker image reference must be publicly accessible to allow `CWL` to pull it. Alternatively, a private
+    docker reference can be used if the image is locally available. The process will fail to execute if it cannot
+    resolve the reference.
+
+The `Application Package` can be provided during process deployment. Please refer to below references for more details.
+
+.. seealso::
+
+    - :ref:`supported package locations <WPS-REST>`
+    - :ref:`Deploy` request
+
+
+Fixing permission error on input files
+==========================================
+
+Some processes expect their inputs to be writable (e.g.: ZIP files). When running an *Application Package* based on a
+`docker image`, `Weaver` mounts the input files as `volumes` in read-only mode for security reasons. This causes these
+processes to immediately fail as the running user cannot override nor write temporary files in the same directory
+(where the volume was mounted to), as it is marked with read permissions.
+
+To resolve this issue, the application developer should add the ``InitialWorkDirRequirement``
+(|cwl-wd-ref|_, |cwl-wd-ex|_) to his CWL package definition. This tells CWL to stage the files into the docker image
+into the running directory where the user will be allowed to generate outputs, and therefore, also allow edition of the
+inputs or generation of temporary files as when unpacking a compressed file.
+
+.. |cwl-wd-ref| replace:: reference
+.. _cwl-wd-ref: `cwl-workdir-req`_
+.. |cwl-wd-ex| replace:: example
+.. _cwl-wd-ex: `cwl-workdir-ex`_
+
+As example, the CWL definition could be similar to the following:
+
+.. code-block:: json
+
+    {
+      "cwlVersion": "v1.0",
+      "class": "CommandLineTool",
+      "requirements": {
+        "DockerRequirement": {
+          "dockerPull": "<docker-url>"
+        },
+        "InitialWorkDirRequirement": {
+          "listing": [{
+             "entry": "$(inputs.input_file)",
+             "writable": true
+            }
+          ]
+        }
+      },
+      "arguments": ["$(runtime.outdir)"],
+      "inputs": {
+        "input_file": {
+        "type": "File"
+      }
+    }
+
+Note that ``$(inputs.input_file)`` within ``InitialWorkDirRequirement`` tells which input to resolve for staging using
+the ``"writable": True`` parameter. All files listed there will be mounted with write permissions into working runtime
+directory of the executed docker container.
+
+
+Problem connecting workflow steps together
+==================================================
+
+
+.. seealso::
+
+    - :ref:`CWL Workflow`
+    - :ref:`Output File Format`
+
+
+Where can I find references to CWL specification and examples?
+================================================================
+
+There exist multiple sources, but official ones provided below have a create amount of examples and are being
+continuously improved by the developers (including being updated according to changes).
+
+- |cwl-guide|_
+- |cwl-cmdtool|_
+- |cwl-workflow|_
+