Add Sphinx-based documentation

To build the docs, move to the `docs` directory and run `make html`. The resulting docs will be in `docs/build/html`.

Author: Mr0grog

.editorconfig

@@ -6,3 +6,6 @@ trim_trailing_whitespace = true
 
 [*.py]
 max_line_length = 79
+
+[*.rst]
+indent_size = 2

.gitignore

@@ -1,4 +1,7 @@
 # Created by .gitignore support plugin (hsz.mobi)
+
+.vscode
+
 ### Python template
 # Byte-compiled / optimized / DLL files
 __pycache__/

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?= "-W"  # This flag turns warnings into errors.
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/requirements.txt

@@ -0,0 +1,15 @@
+-r ../requirements.txt
+-r ../requirements-experimental.txt
+-r ../requirements-dev.txt
+
+# ../requirements-server.txt includes some binary dependencies readthedocs
+# doesn't support, so just include the minimum here.
+# See `autodoc_mock_imports` in `docs/source/conf.py` for where other
+# dependencies we aren't actually installing get mocked.
+tornado >=6.0.0,<7
+
+# We need to tell readthedocs to install this package itself as well.
+# Confusingly, we have to specify this directory relative to the repo root
+# rather than relative to the directory where this file resides. That is
+# inconsistent with the lines above...the mysteries of pip!
+./

docs/source/api-reference.rst

@@ -0,0 +1,76 @@
+=============
+API Reference
+=============
+
+
+Diff Types
+----------
+
+*web-monitoring-diff* provides a variety of diff algorithms for use in comparing web content. They all follow a similar standardized signature and return format.
+
+**Diff Signatures**
+
+All diffs should have parameters named ``a_<body|text>`` and ``b_<body|text>`` as their first two arguments. These represent the two pieces of content to compare, where ``a`` represents the “from” or left-hand side and ``b`` represents the “to” or right-hand side of the comparison. The name indicates whether the function takes bytes (``a_body``/``b_body``) or a decoded string (``a_text``/``b_text``). The web server inspects argument names to determine what to pass to a given diff type.
+
+Additionally, diffs may take several other standardized parameters:
+
+* ``a_body``, ``b_body``: Raw HTTP reponse body (bytes), described above.
+* ``a_text``, ``b_text``: Decoded text of HTTP response body (str), described above.
+* ``a_url``, ``b_url``: URL at which the content being diffed is found. (This is useful when content contains location-relative information, like links.)
+* ``a_headers``, ``b_headers``: Dict of HTTP headers.
+
+Finally, some diffs take additional, diff-specific parameters.
+
+**Return Values**
+
+All diffs return a :class:`dict` with a key named ``"diff"``. The value of this dict entry varies by diff type, but is usually:
+
+- An array of changes. Each entry will be a 2-tuple, where the first item is an :class:`int` reprenting the type of change (``-1`` for removal, ``0`` for unchanged, ``1`` for addition, or other numbers for diff-specific meanings) and the second item is the data or string that was added/removed/unchanged.
+
+- A string representing a custom view of the diff, e.g. an HTML document.
+
+- A bytestring representing a custom binary view of the diff, e.g. an image.
+
+Each diff may add additional, diff-specifc keys to the dict. For example, :func:`web_monitoring_diff.html_diff_render` includes a ``"change_count"`` key indicating how many changes there were, since it’s tough
+to inspect the HTML of the resulting diff and count yourself.
+
+
+.. autofunction:: web_monitoring_diff.compare_length
+
+.. autofunction:: web_monitoring_diff.identical_bytes
+
+.. autofunction:: web_monitoring_diff.side_by_side_text
+
+.. autofunction:: web_monitoring_diff.html_text_diff
+
+.. autofunction:: web_monitoring_diff.html_source_diff
+
+.. autofunction:: web_monitoring_diff.links_diff
+
+.. autofunction:: web_monitoring_diff.links_diff_json
+
+.. autofunction:: web_monitoring_diff.links_diff_html
+
+.. autofunction:: web_monitoring_diff.html_diff_render
+
+.. automodule:: web_monitoring_diff.experimental
+
+    .. autofunction:: web_monitoring_diff.experimental.htmldiffer.diff
+
+    .. autofunction:: web_monitoring_diff.experimental.htmltreediff.diff
+
+
+Web Server
+----------
+
+.. autofunction:: web_monitoring_diff.server.make_app
+
+.. autofunction:: web_monitoring_diff.server.cli
+
+
+Exception Classes
+-----------------
+
+.. autoclass:: web_monitoring_diff.exceptions.UndecodableContentError
+
+.. autoclass:: web_monitoring_diff.exceptions.UndiffableContentError

docs/source/conf.py

@@ -0,0 +1,88 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'web-monitoring-diff'
+copyright = '2017-2020, Environmental Data & Governance Initiative'
+author = 'Environmental Data & Governance Initiative'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+import web_monitoring_diff
+# The short X.Y version.
+version = web_monitoring_diff.__version__
+# The full version, including alpha/beta/rc tags.
+release = web_monitoring_diff.__version__
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.githubpages',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.mathjax',
+    'sphinx.ext.viewcode',
+    'IPython.sphinxext.ipython_directive',
+    'IPython.sphinxext.ipython_console_highlighting',
+    'numpydoc',
+    'sphinx_copybutton',
+]
+
+# Generate the API documentation when building
+autosummary_generate = True
+numpydoc_show_class_members = False
+autodoc_mock_imports = ['html5_parser', 'pycurl', 'sentry']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {
+    'python': ('https://docs.python.org/3/', None),
+    'numpy': ('https://numpy.org/doc/stable/', None),
+    'scipy': ('https://docs.scipy.org/doc/scipy/reference/', None),
+    'pandas': ('https://pandas.pydata.org/pandas-docs/stable', None),
+    'matplotlib': ('https://matplotlib.org', None),
+}
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+# import sphinx_rtd_theme
+# html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']

docs/source/index.rst

@@ -0,0 +1,14 @@
+web-monitoring-diff
+===================
+
+*Web-monitoring-diff* is a suite of functions that *diff* (find the differences between) types of content commonly found on the web, such as HTML, text files, etc. in a variety of ways. It also includes an optional web server that generates diffs as an HTTP service.
+
+This package was originally built as a component of EDGI’s `Web Monitoring Project <https://github.com/edgi-govdata-archiving/web-monitoring>`_, but is also used by other organizations and tools.
+
+.. toctree::
+   :maxdepth: 2
+
+   installation
+   usage
+   api-reference
+   release-history

docs/source/installation.rst

@@ -0,0 +1,66 @@
+============
+Installation
+============
+
+*web-monitoring-diff* requires **Python 3.7 or newer**. Before anything else, make sure you’re using a supported version of Python. If you need to support different local versions of Python on your computer, we recommend using `pyenv`_ or `Conda`_.
+
+1. **System-level dependencies:** web-monitoring-diff depends on several system-level, non-Python libraries that you may need to install first. Specifically, you’ll need: ``libxml2``, ``libxslt``, ``openssl``, and ``libcurl``.
+
+  **On MacOS,** we recommend installing these with `Homebrew`_:
+
+  .. code-block:: bash
+
+    brew install libxml2
+    brew install libxslt
+    brew install openssl
+    # libcurl is built-in, so you generally don't need to install it
+
+  **On Debian Linux,** use ``apt``:
+
+  .. code-block:: bash
+
+    apt-get install libxml2-dev libxslt-dev libssl-dev openssl libcurl4-openssl-dev
+
+  **Other systems** may have different package managers or names for the packages, so you may need to look them up.
+
+2. **Install this package** with *pip*. Be sure to include the ``--no-binary lxml`` option:
+
+  .. code-block:: bash
+
+    pip install web-monitoring-diff --no-binary lxml
+
+  Or, to also install the web server for generating diffs on demand, install the ``server`` extras:
+
+  .. code-block:: bash
+
+      pip install web-monitoring-diff[server] --no-binary lxml
+
+  The ``--no-binary`` flag ensures that pip downloads and builds a fresh copy of ``lxml`` (one of web-monitoring-diff’s dependencies) rather than using a pre-built version. It’s slower to install, but is required for all the dependencies to work correctly together. **If you publish a package that depends on web-monitoring-diff, your package will need to be installed with this flag, too.**
+
+  **On MacOS,** you may need additional configuration to get ``pycurl`` use the Homebrew openssl. Try the following:
+
+  .. code-block:: bash
+
+    PYCURL_SSL_LIBRARY=openssl \
+      LDFLAGS="-L/usr/local/opt/openssl/lib" \
+      CPPFLAGS="-I/usr/local/opt/openssl/include" \
+      pip install web-monitoring-diff --no-binary lxml --no-cache-dir
+
+  The ``--no-cache-dir`` flag tells *pip* to re-build the dependencies instead of using versions it’s built already. If you tried to install once before but had problems with ``pycurl``, this will make sure pip actually builds it again instead of re-using the version it built last time around.
+
+  **For local development,** clone the git repository and then make sure to do an editable installation instead.
+
+  .. code-block:: bash
+
+      pip install .[server,dev] --no-binary lxml
+
+3. **(Optional) Install experimental diffs.** Some additional types of diffs are considered “experimental” — they may be new and still have lots of edge cases, may not be publicly available via PyPI or another package server, or may have any number of other issues. To install them, run:
+
+  .. code-block:: bash
+
+    pip install -r requirements-experimental.txt
+
+
+.. _pyenv: https://github.com/pyenv/pyenv
+.. _conda: https://docs.conda.io/en/latest/
+.. _Homebrew: https://brew.sh/

docs/source/release-history.rst

@@ -0,0 +1,8 @@
+===============
+Release History
+===============
+
+In Development: v0.1.0
+----------------------
+
+This project used to be a part of `web-monitoring-processing <https://github.com/edgi-govdata-archiving/web-monitoring-processing/>`_, which contains a wide variety of libraries, scripts, and other tools for working with data across all the various parts of EDGI’s Web Monitoring project. The goal of this initial release is to create a new, more focused package containing the diff-releated tools so they can be more easily used by others.