From dc54c1ff6eb3762ad4513a974c2ae57f73a19183 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Wed, 12 Jan 2022 17:55:10 -0800 Subject: [PATCH] v0.15.0: OpenAPI spec fixes, tail formula, etc. --- README.md | 539 +----------------- dactyl/config.py | 16 +- dactyl/default-config.yml | 19 +- dactyl/openapi.py | 11 +- dactyl/page.py | 9 +- dactyl/templates/breadcrumbs.html | 4 +- dactyl/templates/landing.html | 3 +- .../templates/template-openapi_data_type.md | 5 +- dactyl/templates/template-openapi_endpoint.md | 48 +- dactyl/version.py | 2 +- .../content/extending/extend-templates.md | 11 - examples/content/extending/templates.md | 109 ++++ .../content/filter-examples/multicode_tabs.md | 41 -- .../{filter-examples => filters}/badges.md | 0 .../{filter-examples => filters}/buttonize.md | 0 .../{filter-examples => filters}/callouts.md | 0 .../external_links.md | 0 examples/content/filters/filters.md | 38 ++ examples/content/filters/multicode_tabs.md | 75 +++ .../{filter-examples => filters}/xrefs.md | 4 +- examples/content/index.md | 22 + examples/content/usage/config.md | 103 ++++ examples/content/usage/editing.md | 55 ++ examples/content/usage/elasticsearch.md | 46 ++ examples/content/{ => usage}/link-checking.md | 5 +- examples/content/usage/openapi.md | 55 ++ examples/content/usage/style-checking.md | 108 ++++ examples/content/usage/usage.md | 128 +++++ examples/dactyl-config.yml | 46 +- examples/templates/template-extends.html | 22 +- releasenotes.md | 70 +-- tests/gfm-compat.md | 2 +- tests/integration.py | 34 +- tests/unit_target.py | 2 +- 34 files changed, 918 insertions(+), 714 deletions(-) delete mode 100644 examples/content/extending/extend-templates.md create mode 100644 examples/content/extending/templates.md delete mode 100644 examples/content/filter-examples/multicode_tabs.md rename examples/content/{filter-examples => filters}/badges.md (100%) rename examples/content/{filter-examples => filters}/buttonize.md (100%) rename examples/content/{filter-examples => filters}/callouts.md (100%) rename examples/content/{filter-examples => filters}/external_links.md (100%) create mode 100644 examples/content/filters/filters.md create mode 100644 examples/content/filters/multicode_tabs.md rename examples/content/{filter-examples => filters}/xrefs.md (92%) create mode 100644 examples/content/index.md create mode 100644 examples/content/usage/config.md create mode 100644 examples/content/usage/editing.md create mode 100644 examples/content/usage/elasticsearch.md rename examples/content/{ => usage}/link-checking.md (98%) create mode 100644 examples/content/usage/openapi.md create mode 100644 examples/content/usage/style-checking.md create mode 100644 examples/content/usage/usage.md diff --git a/README.md b/README.md index 6aa3139..688be69 100644 --- a/README.md +++ b/README.md @@ -18,543 +18,12 @@ Or a local install in a virtualenv: # Create an activate a virtualenv so the package and dependencies are localized virtualenv -p `which python3` venv_dactyl source venv_dactyl/bin/activate - -# Check out this repo -git clone https://github.com/ripple/dactyl - -# Install -pip3 install dactyl/ - -# Where 'dactyl/' is the top level directory of the repo, containing setup.py. -# And note the trailing '/' which tells pip to use a local directory to install it. -``` - -## Usage - -Simple ("Ad-Hoc") usage: - -```sh -$ dactyl_build --pages input1.md input2.md -``` - -By default, the resulting HTML pages are written to a folder called `out/` in the current working directory. You can specify a different output path in the config file or by using the `-o` parameter. - -### Building PDF - -Dactyl generates PDFs by making temporary HTML files and running [Prince][]. Use the `--pdf` command to generate a PDF. Dactyl tries to come up with a sensible output filename by default, or you can provide one (which must end in `.pdf`): - -```sh -$ dactyl_build --pages input1.md input2.md --pdf MyGuide.pdf -``` - -### Advanced Usage - -Dactyl is intended to be used with a config file containing a list of pages to parse. Pages are grouped into "targets" that represent a group of documents to be built together; a page can belong to multiple targets, and can even contain conditional syntax so that it builds slightly different depending on the target in question. Targets and pages can also use different templates from each other, and pages can inherit semi-arbitrary key/value pairs from the targets. - -For more information on configuration, see the `default-config.yml` and the [examples](examples/) folder. - -The input pages in the config file should be specified relative to the `content_path`, which is `content/` by default. You can also specify a URL to pull in a markdown file from a remote source, but if you do, Dactyl won't run any pre-processing on it. - -For a full list of Dactyl options, use the `-h` parameter. - -#### Specifying a Config File - -By default, Dactyl looks for a config file named `dactyl-config.yml` in the current working directory. You can specify an alternate config file with the `-c` or `--config` parameter: - -```sh -$ dactyl_build -c path/to/alt-config.yml -``` - -For more information on configuration, see the `default-config.yml` and the [examples](examples/) folder. - -#### Specifying a Target - -If your config file contains more than one **target**, Dactyl builds the first one by default. You can specify a different target by passing its `name` value with the `-t` parameter: - -```sh -$ dactyl_build -t non-default-target -``` - -#### Static Files - -Your templates may require certain static files (such as JavaScript, CSS, and images) to display properly. Your content may have its own static files (such as diagrams and figures). By default, Dactyl assumes that templates have static files in the `assets/` folder. You can configure this path and also specify one or more paths to static files referenced by your content. When you build, Dactyl copies files from these folders to the output folder by default depending on which mode you're building: - -| Build Mode | Files copied to output folder by default | -|:-------------------|:--------------------------------------------------------| -| HTML | Both template and content static files | -| PDF | Neither template nor content static files (cannot be overridden) | -| Markdown | Content static files only | -| ElasticSearch JSON | Neither template nor content static files | - -You can use a commandline flag to explicitly specify what gets copied to the output folder, except in the case of PDF. (In PDF mode, Dactyl writes only the final PDF to the output folder.) The flags are as follows: - -| Flag (long version) | Short version | Meaning | -|:--------------------|:--------------|:---------------------------------------| -| `--copy_static` | `-s` | Copy all static files to the out dir. | -| `--no_static` | `-S` | Don't copy any static files to the out dir. | -| `--template_static` | `-T` | Copy only templates' static files to the out dir | -| `--content_static` | `-C` | Copy only the content's static files to the out dir | - -The following config file parameters control what paths Dactyl checks for static content: - -| Field | Default | Description | -|---|---|---| -| `template_static_path` | `assets/` | Static files belonging to the templates. | -| `content_static_path` | (None) | Static files belonging to content. This can be a single folder path, as a string, or an array of paths to files or folders. Dactyl copies all files and folders (regardless of whether the current target uses them). | - -#### Listing Available Targets - -If you have a lot of targets, it can be hard to remember what the short names for each are. If you provide the `-l` flag, Dactyl will list available targets and then quit without doing anything: - -```sh -$ dactyl_build -l -tests Dactyl Test Suite -rc-install Ripple Connect v2.6.3 Installation Guide -rc-release-notes -kc-rt-faq Ripple Trade Migration FAQ -``` - -#### Building Markdown - -This mode runs the pre-processor only, so you can generate Markdown files that are more likely to display properly in conventional Markdown parsers (like the one built into GitHub). Use the `--md` flag to output Markdown files, skipping the HTML/PDF templates entirely. - -```sh -$ dactyl_build --md -``` - -#### Building Only One Page - -If you only want to build a single page, you can use the `--only` flag, followed by the filename you want to build (either the input filename ending in `.md` or the output filename ending in `.html`): - -```sh -dactyl_build --only index.html --pdf -``` - -This command can be combined with the `--pdf` or `--md` flags. You can also use it with the `--target` setting (in case you want the context from the target even though you're only building one page.) - -#### Watch Mode - -You can use the `-w` flag to make Dactyl run continuously, watching for changes to its input templates or markdown files. Whenever it detects that a file has changed, Dactyl automatically rebuilds the output in whatever the current mode is, (HTML, PDF, or Markdown). - -To be detected as a change, the file has to match one of the following patterns: - -``` -*.md -*/code_samples/* -template-*.html -``` - -Beware: some configurations can lead to an infinite loop. (For example, if your output directory is a subdirectory of your content directory and you use Dactyl in `--md` mode.) - -**Limitations:** Watch mode can be combined with `--only`, but re-builds the page even when it detects changes to unrelated pages. Watch mode doesn't detect changes to the config file, static files, or filters. - -To stop watching, interrupt the Dactyl process (Ctrl-C in most terminals). - -#### ElasticSearch Compatibility - -Dactyl has the ability to build JSON formatted for upload to [ElasticSearch](https://www.elastic.co/products/elasticsearch) and even upload it directly. - -To build JSON files for upload to ElasticSearch, use the `--es` mode: - -``` -dactyl_build --es -``` - -This writes files to the usual output directory using an ElasticSearch JSON template. Dactyl skips any files that do not have a `md` source parameter in this mode. The output filenames are the pages' `html` filenames, except ending in `.json` instead of `.html`. You can specify a custom template for these JSON files using the top-level `default_es_template` field in the config file. This template must be a valid JSON file and has several special properties as described in [ElasticSearch JSON Templates](#elasticsearch-json-templates). - -Dactyl can also upload these files directly to an ElasticSearch instance, even when building for another mode. For example, to build the HTML version of a target named `filterdemos` but also upload that target's JSON-formatted data to an ElasticSearch instance: - -``` -dactyl_build -t filterdemos --html --es_upload https://my-es-instance.example.com:9200 -``` - -The parameter to `--es_upload` should be the base URL of your ElasticSearch index. You can omit the parameter to use the default base URL of `http://localhost:9200`. - - -#### ElasticSearch JSON Templates - -Dactyl has a special format for JSON templates meant for creating ElasticSearch data. These templates must be valid JSON and are processed according to the following rules: - -- Any strings in the fields' values are "preprocessed" in a similar context to the Jinja2-based Markdown preprocessor. For example, the string `{{currentpage.name}}` evaluates to the page's name. -- Any object containing the key `__dactyl_eval__` is evaluated as a Python expression. The object is replaced with the results of the expression, with lists becoming JSON arrays and dictionaries becoming JSON objects. -- The above rules apply recursively to values nested in arrays and objects. All other values are preserved literally. - -The context provided to the preprocessing and to the `__dactyl_eval__` expressions is the same and contains the following: - -| Field | Python Type | Description | -|:----------------|:------------|:---------------------------------------------| -| `currentpage` | `dict` | The current page definition (usually derived from the config file) | -| `target` | `dict` | The current target definition (usually derived from the config file) | -| `categories` | `list` | A list of unique `category` values used by pages in the current target, in order of appearance. | -| `page_filters` | `list` | A list of the names of Dactyl filters applied to the current page. | -| `mode` | `str` | Always equal to `es` in this context | -| `current_time` | `str` | The current time, in the `time_format` specified in the config. (Defaults to YYYY-MM-DD) | -| `bypass_errors` | `bool` | If `true`, this build is running with the option to continue through errors where possible. | - - -### OpenAPI Specification Parsing - -Dactyl contains experimental support for automatically generating documentation from an [OpenAPI Specification](https://github.com/OAI/OpenAPI-Specification). Dactyl has partial support for **v3.0.x** of the OpenAPI spec. - -From the commandline, you can generate documentation for a spec using the `--openapi` option, providing a file path or URL to the spec. For example: - -``` -dactyl_build --openapi https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml -``` - -You can also [add an OpenAPI specification to a config file](#openapi-specifications), where the generated documentation can be part of a larger target that includes other files. - - - -### Link Checking - -The link checker is a separate script. It assumes that you've already built some documentation to an output path. Use it as follows: - -```sh -$ dactyl_link_checker +pip3 install dactyl ``` -This checks all the files in the output directory for links and confirms that any HTTP(S) links, including relative links to other files, are valid. For anchor links, it checks that an element with the correct ID exists in the target file. It also checks that the `src` of all image tags exists. - -If there are links that are always reported as broken but you don't want to remove (for example, URLs that block Python's user-agent) you can add them to the `known_broken_links` array in the config. - -In quiet mode (`-q`), the link checker still reports in every 30 seconds just so that it doesn't get treated as stalled and killed by continuous integration software (e.g. Jenkins). - -To reduce the number of meaningless failure reports (because a particular website happened to be down momentarily while you ran the link checker), if there are any broken remote links, the link checker waits 2 minutes after finishing and then retries those links in case they came back up. (If they did, they're not considered broken for the link checker's final report.) - -You can also run the link checker in offline mode (`-o`) to skip any remote links and just check that the files and anchors referenced exist in the output directory. - -If you have a page that uses JavaScript or something to generate anchors dynamically, the link checker can't find those anchors (since it doesn't run any JS). You can add such pages to the `ignore_anchors_in` array in your config to skip checking for links that go to anchors in such pages. - - -## Style Checking - -The style checker is experimental. It is only suitable for English text. It reports several details about document contents that may be helpful for identifying documents whose readability you could improve. These details are: - -- Discouraged words and phrases. -- Page length details. -- Readability scores. -- Spell-checking. - -Example usage: - -```sh -$ dactyl_style_checker -``` - -The style checker re-generates contents in-memory (never writing it out), unlike the link checker which requires you to run `dactyl_build` first. It only checks contents that come from Markdown, not from HTML templates. - -The style checker uses the first target in the config file unless you specify another target with `-t`. You can check just one file by passing its HTML path in the `--only` parameter. - -The exit code of the command is 0 (success) if it found no discouraged words, the spell checker found no unknown words, and no pages failed their configured readability goals. Otherwise, the exit code of the command is 1 (failure). - - -### Discouraged Words and Phrases - -You can suggest specific words or phrases to discourage. The style checker checks for instances of these words and phrases in documents' content, and suggests alternatives based on the phrase file. Dactyl does not check text in ``, `
`, and `` elements since those are intended to be code samples.
-
-To configure lists of discouraged words and phrases, add the following config options:
-
-| Field                       | Value  | Description                           |
-|:----------------------------|:-------|:--------------------------------------|
-| `word_substitutions_file`   | String | The path to a YAML file with a single top-level map. The keys are the words to discourage and the values are suggestions of words to replace them with. |
-| `phrase_substitutions_file` | String | The path to a YAML file with a single top-level map. The keys are phrases to discourage and the values are suggestions of phrases to replace them with. |
-
-You can add an exemption to a specific discouraged word/phrase rule with an HTML comment. The exemption applies to the whole output (HTML) file in which it appears.
-
-```html
-Maybe the word "will" is a discouraged word, but you really want to use it here without flagging it as a violation? Adding a comment like this  makes it so.
-```
-
-### Spell Checking
-
-Dactyl uses [pyspellchecker](https://pyspellchecker.readthedocs.io/en/latest/) to report possible spelling errors and suggest corrections. The built-in dictionary is not very thorough; you can extend it by providing a dictionary file with more words. Spell checking is case-insensitive.
-
-If you want the spell checker to skip a page, put `skip_spell_checker: true` in the page definition.
-
-If you want to ignore one or more words on a single page only, add a comment such as the following anywhere in the page:
-
-```html
-
-```
-
-To extend the built-in dictionary used for all files, add the following field to the config. (You cannot remove words from the built-in dictionary.)
-
-| Field           | Value  | Description                                       |
-|:----------------|:-------|:--------------------------------------------------|
-| `spelling_file` | String | Path to a text file with words to add to the dictionary. Each line of the file should contain a single word (case-insensitive). |
-
-
-### Length Metrics
-
-Dactyl reports the number of characters of text, number of sentences, and number of words in each document. These counts only include text contents (the parts generated from Markdown). They do not include code samples (not even inlined code words), or images/figures. The sentence counts are estimates. Headings, list items, and table cells each count as one sentence in these metrics. The summary includes the averages across all pages, and the stats for the three longest and shortest pages.
-
-These metrics are intended to be helpful for choosing documents that would be better off combined or split up. They can also be useful for interpreting readability scores, which tend to be less reliable for very short documents.
-
-
-### Readability Scores
-
-The style checker reports readability scores based on several formulas implemented in the [textstat library](https://github.com/shivam5992/textstat). These can help you identify documents with a high proportion of big words and long sentences.
-
-> **Caution:** Readability formulas are not very smart. Trying to get a high readability score can actually decrease the clarity of your writing if you aren't mindful of other factors. Things readability formulas usually don't take into account include: brevity; complexity of the high-level structure; logical connections such as cause and effect; and precise use of language. They tend to score tables and bulleted lists badly even though those structure are very helpful for actual readability.
-
-By default, Dactyl prints the readability scores for each page as it analyzes them. The `-q` option hides this output. The summary at the end lists the average scores for all pages analyzed and the three pages with the worst Flesch Reading Ease scores.
-
-#### Readability Goals
-
-You can set readability goals for individual pages or an entire target by adding the `readability_goals` field. This field should contain a map of readability metrics to target scores. Goals defined for individual pages override goals set for the entire target. Dactyl compares a page's readability scores to any goals set and reports a list of pages that failed their goals in the summary. The goal passes if the page's score is equal better (easier to read) than the stated goal value, and fails otherwise. For Flesch Reading Ease, higher scores represent better readability; for the other tests, _lower_ scores represent better readability.
-
-**Note:** Since very short pages tend to have inconsistent and unreliable readability scores, Dactyl does not calculate readability scores for pages with fewer than 10 "sentences". (Bullet points, headings, and table cells each count as separate "sentences" for this purpose.)
-
-Example configuration:
-
-```yaml
-targets:
-
-  - name: my-target
-    display_name: Example Target
-    readability_goals:
-        flesch_reading_ease: 50
-        automated_readability_index: 12
-```
-
-The available readability tests are:
-
-| Field Name                     | Details                                     |
-|:-------------------------------|:--------------------------------------------|
-| `flesch_reading_ease`          | [Flesch reading ease](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests#Flesch_reading_ease). Maximum score 121.22; no limit on how negative the score can be. |
-| `smog_index`                   | [SMOG grade](https://en.wikipedia.org/wiki/SMOG). Gives an estimated grade level. |
-| `coleman_liau_index`           | [Coleman-Liau index](https://en.wikipedia.org/wiki/Coleman%E2%80%93Liau_index). Gives an estimated grade level. |
-| `automated_readability_index`  | [Automated readability index](https://en.wikipedia.org/wiki/Automated_readability_index). Gives an estimated grade level. |
-| `dale_chall_readability_score` | [Dale-Chall readability formula](https://en.wikipedia.org/wiki/Dale%E2%80%93Chall_readability_formula). Decimal representing difficulty; lower values map to lower grade levels. |
-| `linsear_write_formula`        | [Linsear Write formula](https://en.wikipedia.org/wiki/Linsear_Write). Gives an estimated grade level. |
-| `gunning_fog`                  | [Gunning fog index](https://en.wikipedia.org/wiki/Gunning_fog_index). Gives an estimated grade level. |
-
-Estimated grade levels are based on the United States school system and are given as decimal approximations. For example, `11.5` represents somewhere between 11th and 12th grade (high school junior to senior).
-
-
-## Configuration
-
-Many parts of Dactyl are configurable. An advanced setup would probably have a directory structure such as the following:
-
-```
-./                      # Top-level dir; this is where you run dactyl_*
-./dactyl-config.yml     # Default config file name
-./content               # Dir containing your .md source files
----------/*/*.md        # You can sort .md files into subdirs if you like
----------/static/*      # Static images referencd in your .md files
-./templates/template-*.html # Custom HTML Templates
-./assets                # Directory for static files referenced by templates
-./out                   # Directory where output gets generated. Can be deleted
-```
-
-All of these paths can be configured.
-
-### Targets
-
-A target represents a group of pages, which can be built together or concatenated into a single PDF. You should have at least one target defined in the `targets` array of your Dactyl config file. A target definition should consist of a short `name` (used to specify the target in the commandline and elsewhere in the config file) and a human-readable `display_name` (used mostly by templates but also when listing targets on the commandline).
-
-A simple target definition:
-
-```
-targets:
-    -   name: kc-rt-faq
-        display_name: Ripple Trade Migration FAQ
-```
-
-In addition to `name` and `display_name`, a target definition can contain arbitrary key-values to be inherited by all pages in this target. Dictionary values are inherited such that keys that aren't set in the page are carried over from the target, recursively. The rest of the time, fields that appear in a page definition take precedence over fields that appear in a target definition.
-
-Some things you may want to set at the target level include `filters` (an array of filters to apply to pages in this target), `template` (template to use when building HTML), and `pdf_template` (template to use when building PDF). You can also use the custom values in templates and preprocessing. Some filters define additional fields that affect the filter's behavior.
-
-The following field names cannot be inherited: `name`, `display_name`, and `pages`.
-
-### Pages
-
-Each page represents one HTML file in your output. A page can belong to one or more targets. When building a target, all the pages belonging to that target are built in the order they appear in the `pages` array of your Dactyl config file.
-
-Example of a pages definition with two files:
-
-```
-pages:
-    -   name: RippleAPI
-        category: References
-        html: reference-rippleapi.html
-        md: https://raw.githubusercontent.com/ripple/ripple-lib/0.17.2/docs/index.md
-        filters:
-            - remove_doctoc
-            - add_version
-        targets:
-            - local
-            - ripple.com
-
-    -   name: rippled
-        category: References
-        html: reference-rippled.html
-        md: reference-rippled.md
-        targets:
-            - local
-            - ripple.com
-```
-
-Each individual page definition can have the following fields:
-
-| Field                    | Type      | Description                           |
-|:-------------------------|:----------|:--------------------------------------|
-| `targets`                | Array     | The short names of the targets that should include this page. |
-| `html`                   | String    | _(Optional)_ The filename where this file should be written in the output directory. If omitted, Dactyl chooses a filename based on the `md` field (if provided), the `name` field (if provided), or the current time (as a last resort). By default, generated filenames flatten the folder structure of the md files. To instead replicate the folder structure of the source documents in auto-generated filenames, add `flatten_default_html_paths: true` to the top level of your Dactyl config file. |
-| `name`                   | String    | _(Optional)_ Human-readable display name for this page. If omitted but `md` is provided, Dactyl tries to guess the right file name by looking at the first two lines of the `md` source file. |
-| `md`                     | String    | _(Optional)_ The markdown filename to parse to generate this page, relative to the **content_path** in your config. If this is not provided, the source file is assumed to be empty. (You might do that if you use a nonstandard `template` for this page.) |
-| `openapi_specification`  | String    | _(Optional)_ The file path or http(s) URL to an OpenAPI v3.0 specification to be parsed into generated documentation. If provided, this entry becomes expanded into a set of several pages that describe the methods and data types defined for the given API. The generated pages inherit the other fields of this page object. **Experimental.** If the path is a relative path, it is evaluated based on the directory Dactyl is called from, not the content directory. |
-| `api_slug`               | String    | _(Optional)_ If this is an `openapi_specification` entry,
-| `category` | String | _(Optional)_ The name of a category to group this page into. This is used by Dactyl's built-in templates to organize the table of contents. |
-| `template`               | String    | _(Optional)_ The filename of a custom [Jinja][] HTML template to use when building this page for HTML, relative to the **template_path** in your config. |
-| `pdf_template`           | String    | _(Optional)_ The filename of a custom [Jinja][] HTML template to use when building this page for PDF, relative to the **template_path** in your config. |
-| `openapi_md_template_path` | String | _(Optional)_ Path to a folder containing [templates to be used for OpenAPI spec parsing](#openapi-spec-templates). If omitted, use the [built-in templates](dactyl/templates/). |
-| `parent`                 | String    | _(Optional)_ The HTML filename of the page to treat as a parent of this one for purposes of hierarchy. If omitted, treat the page as a "top-level" page. |
-| ...                      | (Various) | Additional arbitrary key-value pairs as desired. These values can be used by templates or pre-processing. |
-
-If the file specified by `md` begins with YAML frontmatter, separated by a line of exactly `---`, the frontmatter is used as a basis for these fields. Certain frontmatter fields are adapted from Jekyll format to Dactyl format: for example, `title` gets copied to `name` if the page does not have a `name`.
-
-The following fields are automatically added after a page has been parsed to HTML. (They're not available when preprocessing or rendering Markdown to HTML, but _are_ available when rendering HTML templates.)
-
-| Field                    | Type      | Description                           |
-|:-------------------------|:----------|:--------------------------------------|
-| `plaintext` | String     | A plaintext-only version of the page's markdown content, with all Markdown and HTML syntax removed. |
-| `headermap` | Dictionary | A mapping of the page's headers to the unique IDs of those headers in the generated HTML version. |
-| `blurb`     | String     | An introductory blurb generated from the page's first paragraph of text. |
-| `children`  | List       | A list of pages, in order of appearance, that refer to this page as their `parent`. Each of these "child" pages is a reference to the page definition (dictionary) for that child. |
-| `is_ancestor_of` | Function | A function that takes one argument, the string identifying a potential child page by that child's `html` field. This function returns `True` if this page is a direct or indirect parent of the child page. |
-
-[Jinja]: http://jinja.pocoo.org/
-
-#### OpenAPI Specifications
-
-You can add a special entry to the `pages` array to represent an OpenAPI v3.0 specification, which will be expanded into several pages representing the methods and data types specified in the file. For example:
-
-```
--   openapi_specification: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml
-    api_slug: petstore
-    foo: bar
-    targets:
-        - baz
-```
-
-The `api_slug` field is optional, and provides a prefix that gets used for a bunch of file names and stuff. Other fields are inherited by all of the pages the specification builds, which are:
-
-- An "All Methods" table of contents, listing every path operation in the `paths` of the OpenAPI specification.
-- "Tag Methods" table of contents pages for each `tag` used in the OpenAPI specification.
-- Pages for all "API Methods" (path operations) in `paths` of the OpenAPI specification.
-- A "Data Types" table of contents, listing every data type defined in the `schema` section of the OpenAPI specification.
-- Individual pages for each data type in the OpenAPI specification's `schema` section.
-
-You can override the [templates used for generated OpenAPI pages](#openapi-spec-templates) to adjust how the Markdown is generated by providing the path to a different set of templates in the `openapi_md_template_path` field.
-
-
-## Editing
-
-Dactyl supports extended Markdown syntax with the [Python-Markdown Extra](https://pythonhosted.org/Markdown/extensions/extra.html) module. This correctly parses most GitHub-Flavored Markdown syntax (such as tables and fenced code blocks) as well as a few other features.
-
-### Pre-processing
-
-Dactyl pre-processes Markdown files by treating them as [Jinja][] Templates, so you can use [Jinja's templating syntax](http://jinja.pocoo.org/docs/dev/templates/) to do advanced stuff like include other files or pull in variables from the config or commandline. Dactyl passes the following fields to Markdown files when it pre-processes them:
-
-| Field             | Value                                                    |
-|:------------------|:---------------------------------------------------------|
-| `target`          | The [target](#targets) definition of the current target. |
-| `pages`           | The [array of page definitions](#pages) in the current target. Use this to generate navigation across pages. (The default templates don't do this, but you should.) |
-| `currentpage`     | The definition of the page currently being rendered. This inherits most fields defined at the target level as well. |
-| `categories`      | A de-duplicated array of categories that are used by at least one page in this target, sorted in the order they first appear. |
-| `config`          | The global Dactyl config object. |
-| `content`         | The parsed HTML content of the page currently being rendered. |
-| `current_time`    | The current date as of rendering. The format is YYYY-MM-DD by default; you can also set the `time_format` field to a custom [stftime format string](http://strftime.org/). |
-| `mode`            | The output format: either `html` (default), `pdf`, or `md`. |
-
-
-### Adding Variables from the Commandline
-
-You can pass in a JSON or YAML-formatted list of variables using `--vars` commandline switch. Any such variables get added as fields of `target` and inherited by `currentpage` in any case where `currentpage` does not already have the same variable name set. For example:
-
-```sh
-$ cat md/vartest.md
-Myvar is: '{{ target.myvar }}'
-
-$ dactyl_build --vars '{"myvar":"foo"}'
-rendering pages...
-writing to file: out/index.html...
-Preparing page vartest.md
-reading markdown from file: vartest.md
-... parsing markdown...
-... modifying links for target: default
-... re-rendering HTML from soup...
-writing to file: out/test_vars.html...
-done rendering
-copying static pages...
-
-$ cat out/test_vars.html | grep Myvar
-

Myvar is: 'foo'

-``` - -If argument to `--vars` ends in `.yaml` or `.json`, Dactyl treats the argument as a filename and opens it as a YAML file. (YAML is a superset of JSON, so this works for JSON files.) Otherwise, Dactyl treats the argument as a YAML/JSON object directly. Be sure that the argument is quoted and escaped as necessary based on the commandline shell you use. - -You cannot set the following reserved keys: - -- `name` -- `display_name` (Instead, use the `--title` argument to set the display name of the target on the commandline.) -- `pages` - - -### Filters - -Furthermore, Dactyl supports additional custom post-processing through the use of filters. Filters can operate on the markdown (after it's been pre-processed), on the raw HTML (after it's been parsed), or on a BeautifulSoup object representing the output HTML. Filters can also export functions and values that are available to the preprocessor. - -Dactyl comes with several filters, which you can enable in your config file. You can also write your own filters. If you do, you must specify the paths to the folder(s) containing your filter files in the `filter_paths` array of the config file. - -To enable a filter for a target or page, set the `filters` field of the config to be an array of filter names, where the filter names are derived from the Python source files in the format `filter_.py`. Filter names must be valid Python variable names, so they can't start with a numeral and must contain only alphanumeric and underscore characters. - -Dactyl automatically runs the following functions from filter files (skipping any that aren't defined): - -1. Before running the preprocessor on a page, Dactyl adds all items from each filter's `export` global dictionary to the preprocessor environment. -2. Dactyl runs the `filter_markdown(md, **kwargs)` function of each filter after the preprocessor. This function receives the preprocessed markdown as a string in the `md` argument and must return a string with the markdown as filtered. -3. Dactyl runs the `filter_html(html, **kwargs)` function after the markdown processor. This function receives the parsed markdown content as an HTML string in the `html` argument and must return a string with the HTML as filtered. -4. Dactyl runs the `filter_soup(soup, **kwargs)` function after the HTML filters. This function is expected to directly modify the `soup` argument, which contains a [BeautifulSoup 4 object](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) representing the HTML contents. - -The keyword arguments (`**kwargs`) for the functions may change in future versions. As of Dactyl 0.5.0, the arguments are as follows: - -| Field | Type | Description | -|:---------------|:-----------|:-----------------------------------------------| -| `currentpage` | Dict | The current page, as defined in the config file plus values inherited from the current target and any processing or calculations. (For example, Dactyl automatically adds a `name` field if one isn't present.) | -| `categories` | List | A de-duplicated, ordered list of `category` fields present among pages in this target. | -| `pages` | List | A list of page objects for all pages in the current target, in the same order they appear in the config file. | -| `target` | Dict | The current target definition, as derived from the config file. | -| `current_time` | String | The time this build was started. The format is defined by your config's global `time_format` field (in [stftime format](http://strftime.org/)), defaulting to YYYY-MM-DD. | -| `mode` | String | Either `html`, `pdf`, or `md` depending on what output Dactyl is building. | -| `config` | Dict | The global config object, based on the config file plus any commandline switches. | -| `logger` | [Logger][] | The logging object Dactyl uses, with the verbosity set to match user input. | - -[Logger]: https://docs.python.org/3/library/logging.html#logger-objects - - -See the [examples](examples/) for examples of how to do many of these things. - -## Templates - -Dactyl provides the following information to templates, which you can access with Jinja's templating syntax (e.g. `{{ target.display_name }}`): - -| Field | Value | -|:------------------|:---------------------------------------------------------| -| `target` | The [target](#targets) definition of the current target. | -| `pages` | The [array of page definitions](#pages) in the current target. Use this to generate navigation across pages. (The default templates don't do this, but you should.) | -| `currentpage` | The definition of the page currently being rendered. | -| `categories` | A de-duplicated array of categories that are used by at least one page in this target, sorted in the order they first appear. | -| `config` | The global Dactyl config object. | -| `content` | The parsed HTML content of the page currently being rendered. | -| `current_time` | The current date as of rendering. The format is YYYY-MM-DD by default; you can also set the `time_format` field to a custom [stftime format string](http://strftime.org/). | -| `mode` | The output format: either `html` (default), `pdf`, or `md`. | -| `page_toc` | A table of contents generated from the current page's headers. Wrap this in a `
    ` element. | -| `sidebar_content` | (Deprecated alias for `page_toc`.) | -### OpenAPI Spec Templates +## Documentation -When generating docs from an API specification, Dactyl generates a Markdown version for each of these pages first, then uses that as the content for the HTML version as it does when parsing normal Markdown pages. You can use your own templates instead. The templates must use the same filenames as the [built-in OpenAPI templates](dactyl/templates/), which match the pattern `template-openapi_*.md`. +Dactyl's documentation is built from this repository's [examples/](examples/) section. You can see the finished version at: - +> diff --git a/dactyl/config.py b/dactyl/config.py index a611116..0f190ae 100644 --- a/dactyl/config.py +++ b/dactyl/config.py @@ -35,7 +35,6 @@ def __init__(self, cli_args): else: logger.debug("No config file specified, trying ./dactyl-config.yml") self.load_config_from_file(DEFAULT_CONFIG_FILE) - self.check_consistency() self.load_filters() self.page_cache = [] @@ -53,6 +52,8 @@ def load_config_from_file(self, config_file): try: with open(config_file, "r", encoding="utf-8") as f: loaded_config = yaml.load(f) + if loaded_config is None: + loaded_config = {} except FileNotFoundError as e: if config_file == DEFAULT_CONFIG_FILE: logger.info("Couldn't read a config file; using generic config") @@ -75,6 +76,12 @@ def load_config_from_file(self, config_file): logger.warning("Deprecation warning: Global field pdf_template has " +"been renamed default_pdf_template") + if "flatten_default_html_paths" in loaded_config: + if loaded_config["flatten_default_html_paths"] == True: + loaded_config["default_html_names"] = "flatten" + else: + loaded_config["default_html_names"] = "path" + self.config.update(loaded_config) def check_consistency(self): @@ -96,7 +103,7 @@ def check_consistency(self): # Check page list for consistency for page in self.config["pages"]: - if "targets" not in page: + if "targets" not in page: #CURSOR if "name" in page: logger.warning("Page %s is not part of any targets." % page["name"]) @@ -132,6 +139,11 @@ def load_pages(self): # OpenAPI specs are too much work to load at this time self.page_cache.append(OPENAPI_SPEC_PLACEHOLDER) + # Check consistency here instead of earlier so that we have frontmatter + # from pages first. (Otherwise, we raise false alarms about pages not + # being part of any target.) + self.check_consistency() + def load_filters(self): # Figure out which filters we need filternames = set(self.config["default_filters"]) diff --git a/dactyl/default-config.yml b/dactyl/default-config.yml index a2764ca..df3c081 100644 --- a/dactyl/default-config.yml +++ b/dactyl/default-config.yml @@ -51,10 +51,21 @@ skip_preprocessor: false template_allow_undefined: true preprocessor_allow_undefined: true -## By default, generates HTML paths from md paths by replacing / with - -## Set this to false to make default HTML paths mirror the folder structure of -## the input md files. -flatten_default_html_paths: true +## How to generate default HTML paths from Markdown files. These can be +## overwritten by the "html" field of the page definition. +## In all cases, the file extension is changed from .md to .html. +## Available formulas include: +## "flatten": The default. Use the full path to the md file, except replace / +## with - so all files end up in one output folder. +## "path": Use the full path to the md file including folders. This can make +## for prettier URLs but requires you to handle absolute paths correctly +## "tail": Use just the filename, not the whole path. This can result in +## duplicates if you have files with the same name in different folders, +## for example if you have multiple "overview.md" docs. You should +## change the duplicates individually using "html" parameters in the +## page definitions, as needed. +# Legacy parameter: flatten_default_html_paths: true +default_html_names: flatten ## Set this to true to disable Dactyl's built-in syntax highlighting no_highlighting: false diff --git a/dactyl/openapi.py b/dactyl/openapi.py index 5b78d59..84f9a27 100644 --- a/dactyl/openapi.py +++ b/dactyl/openapi.py @@ -167,7 +167,7 @@ def clean_up_swag(self): schema["title"] = title if "example" in schema: try: - j = json.dumps(schema["example"], indent=4, default=self.json_default) + j = self.json_pp(schema["example"]) schema["example"] = j except Exception as e: logger.debug("%s example isn't json: %s"%(title,j)) @@ -245,7 +245,7 @@ def get_x_example_request_body(self, path, method, endpoint): return "" try: - ex_pp = json.dumps(ex, indent=4, separators=(',', ': '), default=self.json_default) + ex_pp = self.json_pp(ex) except TypeError as e: traceback.print_tb(e.__traceback__) logger.debug("json dumps failed on example '%s'"%ex) @@ -387,6 +387,7 @@ def new_context(self): "debug": logger.debug, "slugify": slugify, "md_escape": self.md_escape, + "json_pp": self.json_pp } @staticmethod @@ -403,6 +404,12 @@ def md_escape(text): s += c return s + def json_pp(self, j): + """ + Pretty-print function for JSON + """ + return json.dumps(j, indent=4, default=self.json_default) + @staticmethod def json_default(o): """ diff --git a/dactyl/page.py b/dactyl/page.py index 9ec8a9b..50fe9ce 100644 --- a/dactyl/page.py +++ b/dactyl/page.py @@ -159,10 +159,13 @@ def provide_default_filename(self): logger.debug("Need to generate html filename for page %s" % self) if "md" in self.data: - # TODO: support other formulas including "tail" or "pretty" new_filename = re.sub(r"[.]md$", ".html", self.data["md"]) - if self.config.get("flatten_default_html_paths", True): + name_formula = self.config.get("default_html_names", "flatten") + if name_formula == "flatten": new_filename = new_filename.replace(os.sep, "-") + elif name_formula == "tail": + new_filename = new_filename.split(os.sep)[-1] + # the "path" formula is a no-op here. self.data["html"] = new_filename elif "name" in self.data: new_filename = slugify(self.data["name"]).lower()+".html" @@ -173,7 +176,7 @@ def provide_default_filename(self): self.data[PROVIDED_FILENAME_KEY] = True logger.debug("Generated html filename '%s' for page: %s" % - (new_filename, self)) + (self.data["html"], self)) def provide_name(self): """ diff --git a/dactyl/templates/breadcrumbs.html b/dactyl/templates/breadcrumbs.html index 2a2c4bb..b53bb63 100644 --- a/dactyl/templates/breadcrumbs.html +++ b/dactyl/templates/breadcrumbs.html @@ -15,7 +15,9 @@ {% if currentpage.html != "index.html" %} {%- for page in ns.crumbs %} - + {% endfor %} {% endif %} diff --git a/dactyl/templates/landing.html b/dactyl/templates/landing.html index f26e011..f8a57c4 100644 --- a/dactyl/templates/landing.html +++ b/dactyl/templates/landing.html @@ -4,9 +4,10 @@ {% if content %}
    -
    +
    {{ content }} +

    Read More

    {% set show_blurbs = True %} {% set depth= 1 %} {% include 'children.html' %} diff --git a/dactyl/templates/template-openapi_data_type.md b/dactyl/templates/template-openapi_data_type.md index f7cfb20..a3884dd 100644 --- a/dactyl/templates/template-openapi_data_type.md +++ b/dactyl/templates/template-openapi_data_type.md @@ -45,7 +45,7 @@ {% endif -%} {% if example is defined %}- **Example:** - {{example|indent(8,indentfirst=False)}} + {{example|indent(8,first=False)}} {% endif -%} @@ -54,8 +54,9 @@ This type can contain the following fields: | Field | Type | Required? | Description | |-------|------|-----------|-------------| +{%- set required_fields = required if required is defined else [] -%} {%- for name,field in properties.items() %} -| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{"Required" if field.required else "Optional"}} | {{field.description}} | +| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{"Required" if name in required_fields else "Optional"}} | {{field.description}} | {%- endfor %} {% if additionalProperties is defined and additionalProperties == True %}This type MUST NOT contain any additional fields.{% endif %} diff --git a/dactyl/templates/template-openapi_endpoint.md b/dactyl/templates/template-openapi_endpoint.md index 20163da..8e3c1c5 100644 --- a/dactyl/templates/template-openapi_endpoint.md +++ b/dactyl/templates/template-openapi_endpoint.md @@ -1,18 +1,15 @@ # {{summary}} -{{description}} - -## Request Format - ``` {{method|upper}} {{path}} -{%- if method in ["post","put","delete"] and requestBody is defined %} - -{{ x_example_request_body }} -{% endif %} ``` +{{description}} + +## Request Format + {% if path_params|length %} +#### Path Parameters This API method uses the following path parameters: | Field | Value | Required? | Description | @@ -24,6 +21,7 @@ This API method uses the following path parameters: {% endif %} {% if query_params|length %} +#### Query Parameters This API method uses the following query parameters: | Field | Value | Required? | Description | @@ -35,11 +33,28 @@ This API method uses the following query parameters: {% endif %} {% if requestBody is defined %} +### Request Body {{requestBody.description}} {% if requestBody.content is defined %} {% for mediatype,thisbody in requestBody.content.items() %} +{% if thisbody.examples is defined and thisbody.examples|length > 0 %} + + + +{% for body_name,body_sample in thisbody.examples.items() %} +_{{body_name}}_ + +```{%if mediatype == "application/json"%}json{%endif%} +{{json_pp(body_sample)}} +``` +{% endfor %} + + + +{% endif %} + {% if thisbody.schema is defined %} **Media type:** {{mediatype|replace("*","\*")}} @@ -51,8 +66,9 @@ The request uses the following fields: | Field | Type | Required? | Description | |-------|------|-----------|-------------| +{%- set required_fields = thisbody.schema.required if thisbody.schema.required is defined else [] -%} {%- for name,field in thisbody.schema.properties.items() %} -| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{"Required" if field.required else "May be omitted"}} | {% if field.description is defined %}{{field.description}}{% endif %} | +| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{"Required" if name in required_fields else "Optional"}} | {% if field.description is defined %}{{field.description}}{% endif %} | {%- endfor %} {% endif %} @@ -81,21 +97,27 @@ The response uses the following fields: | Field | Type | Required? | Description | |-------|------|-----------|-------------| +{%- set required_fields = thisbody.schema.required if thisbody.schema.required is defined else [] -%} {%- for name,field in thisbody.schema.properties.items() %} -| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{ "Required" if field.required else "May be omitted"}} | {% if field.description is defined %}{{field.description}}{% endif %} | +| `{{name}}` | {{field.type|title}}{% if "items" in field.keys() and "title" in field["items"].keys() %} of [{{field["items"].title}}]({{type_link(field["items"].title)}}){% endif %} {% if field["title"] is defined %}([{{field.title}}]({{type_link(field.title)}})){% endif %} | {{ "Required" if name in required_fields else "Optional"}} | {% if field.description is defined %}{{field.description}}{% endif %} | {%- endfor %} {% endif %}{# TODO: handle allOf, etc. #} {% if thisbody.examples is defined and thisbody.examples|length > 0 %} #### Example Response(s) -{% for body_name,body_sample in thisbody.examples %} + + +{% for body_name,body_sample in thisbody.examples.items() %} _{{body_name}}_ -``` -{{body_sample|pprint}} +```{%if mediatype == "application/json"%}json{%endif%} +{{json_pp(body_sample)}} ``` {% endfor %} + + + {% endif %} {% endif %} diff --git a/dactyl/version.py b/dactyl/version.py index a95d4fe..a842d05 100644 --- a/dactyl/version.py +++ b/dactyl/version.py @@ -1 +1 @@ -__version__ = '0.14.5' +__version__ = '0.15.0' diff --git a/examples/content/extending/extend-templates.md b/examples/content/extending/extend-templates.md deleted file mode 100644 index 7b38bde..0000000 --- a/examples/content/extending/extend-templates.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -template: template-extends.html -related_links: - - with-frontmatter.html - - code-highlighting.html ---- -# Template Extension - -You can extend or include the base templates built into Dactyl. - -For example, this page replaces the right-hand sidebar with a "related pages" module. diff --git a/examples/content/extending/templates.md b/examples/content/extending/templates.md new file mode 100644 index 0000000..7a16acd --- /dev/null +++ b/examples/content/extending/templates.md @@ -0,0 +1,109 @@ +--- +template: template-extends.html +related_links: + - frontmatter.html + - code-highlighting.html +--- +# Templates + +Dactyl provides built-in templates that are fairly full-featured, but you can also write your own, including ones that extent or repurpose parts of the built-in templates, and get updates automatically when Dactyl is updated. As an example, this page uses a custom template that addsa "related pages" module to the right-hand sidebar. + +## Built-In Templates + +Many of the built-in templates work better if you set certain fields on your target or page definitions. Here's a list of those fields: + +| Field | Description | +|---|---| +| `prefix` | The base path for this site. Use `/` for a site served from the top-level of its domain. Required for navigation if your site has subfolders or "pretty URLs". | +| `logo` | URL or path for a logo image to use in the top-left. If not defined, uses a text header instead. | +| `google_analytics_tag` | String tag to use with Google Analytics, e.g. "UA-00000000-0". If not defined, doesn't load Google Analytics. | +| `repository` | URL to this site's source repository on GitHub. Required for the "Edit on GitHub" button. | +| `url` | The fully-qualified URL for the base of the site. Required by the sitemap and Google Search templates. | +| `stylesheet` | URL or path to the default stylesheet. The default includes Bootstrap 4.5 as well as custom CSS for code tabs, callouts, and the page layout. See the [styles dir](./dactyl/styles/) for the source SCSS. The default is served by dactyl.link. | +| `dactyljs` | URL or path to the Dactyl JavaScript file to use. This defines "jump to top" and code tab behavior. The default is served by dactyl.link. | +| `bootstrapjs` | URL or path to the Bootstrap JavaScript file to use. The default is served by BootstrapCDN. | +| `fontawesomecss` | URL or path to FontAwesome (v4) CSS file to use. The default is served by BootstrapCDN. | + +### Page Templates + +The following built-in templates represent **full pages**, so you can use them with the `default_template:` and `template:` settings in the config file or frontmatter. You can also derive your own templates from these templates using `{{"{%"}} extends 'template' {{"%}"}}` syntax. + +| Template | Description | +|---|---| +| `404.html` | Contains an error message intended to be used as a custom 404 page. | +| `base.html` | A general purpose template with a 3-column layout, fixed header, and a footer. The navigation uses the hierarchy (`parent` and `child`) fields introduced by Dactyl v0.10.0. This uses [Bootstrap 4.5.0](https://getbootstrap.com/docs/4.5/). Most of the other built-in templates are derived from this template. | +| `doc.html` | Specialized for individual documents. This is the new default template. The right sidebar has an in-page table of contents, and this runs code tab and syntax highlighting JavaScript by default. (You still need to enable the multicode_tabs filter in your `dactyl-config.yml` file to get code tab syntax.) | +| `landing.html` | A landing page that displays a list of child pages in the center column. | +| `pdf-cover.html` | A cover page and table of contents for PDFs. | +| `redirect.html` | Redirects the user to another URL, as set by the page's `redirect_url` field. Useful for deprecating pages. | +| `simple.html` | A minimal template with no dependencies. | +| `template-sitemap.txt` | A template for a text [sitemap](https://support.google.com/webmasters/answer/183668?hl=en) for use by search engines. | + +When extending the default templates, you many of them have blocks you can replace. For the full list, see [the templates](./dactyl/templates/) directly. + + +### Module Templates + +The following built-in templates are partial modules you can use with `{{"{%"}} include 'templatehere.html' {{"%}"}}` blocks from other templates. Many of these pieces are used by the page templates above, as well: + +| Template | Description | +|---|---| +| `algolia-docsearch.html` | Provides a search box (and accompanying resources) powered by [Algolia DocSearch](https://docsearch.algolia.com/). To use this, you must provide your Algolia API key in the target's `algolia_api_key` field provide your index name in the target's `algolia_index_name` field. | +| `breadcrumbs.html` | Provides [breadcrumbs](https://getbootstrap.com/docs/4.5/components/breadcrumb/) to the current page, based on the hierarchy fields. | +| `children.html` | Displays a bulleted list of children of the current page. You can modify the behavior by setting certain properties before including this template. (See below for an example.) | +| `footer.html` | A footer containing a copyright notice, license link, and language selector (if you have the right fields defined). | +| `github-edit.html` | A button that links to edit the current page's source file on GitHub. Requires the target's `repository` field to be the URL of the site's repository on GitHub. | +| `header.html` | A fixed header containing a logo, navigation to top-level pages, search, and Edit on GitHub buttons if the right fields are defined. | +| `language-dropdown.html` | A language-selector dropdown that points to the equivalent page in other languages, if you have multiple languages defined. (This is the one used in the header.) | +| `language-dropdown.html` | A horizontal language selector that points to the equivalent page in other languages, if you have multiple languages defined. (This is the one used in the footer.) | +| `page-toc.html` | A Bootstrap/ScrollSpy-ready table of contents based on the headers in the current page's Markdown contents. | +| `tree-nav.html` | Tree-style site navigation with collapsible levels. You can set a custom page to be the "top" of the tree to show only a subset of your site. Otherwise it uses the first page (usually the auto-provided cover page) as the top of the tree. Set `nav_omit: true` on a page to hide that page from this navigation. | + +#### Children Module + +The following demonstrates how to use the `children.html` template to display a list of children of a page (including links): + +{% raw %} +```html +{ % set parent_html = 'some-parent.html' %} +{ % set show_blurbs = True %} +{ % set depth = 3 %} +{ % include 'children.html' %} +``` +{% endraw %} + +You can omit any or all of the `{{"{%"}} set ... {{"%}"}}` statements to use the defaults: + +| Setting | Description | Default | +|---|---|---| +| `parent_html` | Which page's children to show. The HTML filename of that page. | The current page. | +| `show_blurbs` | If True, add the child page's `blurb` attribute next to its link. | False | +| `depth` | How many levels in the hierarchy to show below the parent. | 5 | + +#### Tree Nav Module + +The following shows how to display a subset of the tree nav (starting with the file `some_parent.html`) instead of the full tree: + +{% raw %} +```html +{ % set tree_top = pages|selectattr('html', 'defined_and_equalto', 'some_parent.html')|list|first %} +{ % include 'tree-nav.html' %} +``` +{% endraw %} + +## Template Data + +Dactyl provides the following information to templates, which you can access with Jinja's templating syntax (e.g. `{{"{{"}} target.display_name {{"}}"}}`): + +| Field | Value | +|:------------------|:---------------------------------------------------------| +| `target` | The [target](#targets) definition of the current target. | +| `pages` | The [array of page definitions](#pages) in the current target. Use this to generate navigation across pages. (The default templates don't do this, but you should.) | +| `currentpage` | The definition of the page currently being rendered. | +| `categories` | A de-duplicated array of categories that are used by at least one page in this target, sorted in the order they first appear. | +| `config` | The global Dactyl config object. | +| `content` | The parsed HTML content of the page currently being rendered. | +| `current_time` | The current date as of rendering. The format is YYYY-MM-DD by default; you can also set the `time_format` field to a custom [stftime format string](http://strftime.org/). | +| `mode` | The output format: either `html` (default), `pdf`, or `md`. | +| `page_toc` | A table of contents generated from the current page's headers. Wrap this in a `
      ` element. | +| `sidebar_content` | (Deprecated alias for `page_toc`.) | diff --git a/examples/content/filter-examples/multicode_tabs.md b/examples/content/filter-examples/multicode_tabs.md deleted file mode 100644 index 1e7171c..0000000 --- a/examples/content/filter-examples/multicode_tabs.md +++ /dev/null @@ -1,41 +0,0 @@ -# Multi Code-Tabs Filter - -The multi code-tabs filter lets you list multiple code samples and have them appear as tabs in the HTML version. It looks like this: - - - -*Tab 1 Name* - -```json -{ - "id": 2, - "command": "account_info", - "account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59", - "strict": true, - "ledger_index": "validated" -} -``` - -*Tab 2 Name* - -```json -POST http://s1.ripple.com:51234/ -{ - "method": "account_info", - "params": [ - { - "account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59", - "strict": true, - "ledger_index": "validated" - } - ] -} -``` - -*Tab 3 Name* - -```bash -rippled account_info r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59 validated true -``` - - diff --git a/examples/content/filter-examples/badges.md b/examples/content/filters/badges.md similarity index 100% rename from examples/content/filter-examples/badges.md rename to examples/content/filters/badges.md diff --git a/examples/content/filter-examples/buttonize.md b/examples/content/filters/buttonize.md similarity index 100% rename from examples/content/filter-examples/buttonize.md rename to examples/content/filters/buttonize.md diff --git a/examples/content/filter-examples/callouts.md b/examples/content/filters/callouts.md similarity index 100% rename from examples/content/filter-examples/callouts.md rename to examples/content/filters/callouts.md diff --git a/examples/content/filter-examples/external_links.md b/examples/content/filters/external_links.md similarity index 100% rename from examples/content/filter-examples/external_links.md rename to examples/content/filters/external_links.md diff --git a/examples/content/filters/filters.md b/examples/content/filters/filters.md new file mode 100644 index 0000000..15fdf25 --- /dev/null +++ b/examples/content/filters/filters.md @@ -0,0 +1,38 @@ +--- +parent: index.html +category: Filters +section_header: true +template: landing.html +targets: + - everything + - filterdemos +--- +# Filters + +Dactyl supports additional custom post-processing through the use of **filters** which are essentially custom plugins. Filters can operate on the markdown (after it's been pre-processed), on the raw HTML (after it's been parsed), or on a BeautifulSoup object representing the output HTML. Filters can also export functions and values that are available to the preprocessor. + +Dactyl comes with several filters, which you can enable in your config file. You can also write your own filters. If you do, you must specify the paths to the folder(s) containing your filter files in the `filter_paths` array of the config file. + +To enable a filter for a target or page, set the `filters` field of the config to be an array of filter names, where the filter names are derived from the Python source files in the format `filter_.py`. Filter names must be valid Python variable names, so they can't start with a numeral and must contain only alphanumeric and underscore characters. + +Dactyl automatically runs the following functions from filter files (skipping any that aren't defined): + +1. Before running the preprocessor on a page, Dactyl adds all items from each filter's `export` global dictionary to the preprocessor environment. +2. Dactyl runs the `filter_markdown(md, **kwargs)` function of each filter after the preprocessor. This function receives the preprocessed markdown as a string in the `md` argument and must return a string with the markdown as filtered. +3. Dactyl runs the `filter_html(html, **kwargs)` function after the markdown processor. This function receives the parsed markdown content as an HTML string in the `html` argument and must return a string with the HTML as filtered. +4. Dactyl runs the `filter_soup(soup, **kwargs)` function after the HTML filters. This function is expected to directly modify the `soup` argument, which contains a [BeautifulSoup 4 object](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) representing the HTML contents. + +The keyword arguments (`**kwargs`) for the functions may change in future versions. As of Dactyl 0.5.0, the arguments are as follows: + +| Field | Type | Description | +|:---------------|:-----------|:-----------------------------------------------| +| `currentpage` | Dict | The current page, as defined in the config file plus values inherited from the current target and any processing or calculations. (For example, Dactyl automatically adds a `name` field if one isn't present.) | +| `categories` | List | A de-duplicated, ordered list of `category` fields present among pages in this target. | +| `pages` | List | A list of page objects for all pages in the current target, in the same order they appear in the config file. | +| `target` | Dict | The current target definition, as derived from the config file. | +| `current_time` | String | The time this build was started. The format is defined by your config's global `time_format` field (in [stftime format](http://strftime.org/)), defaulting to YYYY-MM-DD. | +| `mode` | String | Either `html`, `pdf`, or `md` depending on what output Dactyl is building. | +| `config` | Dict | The global config object, based on the config file plus any commandline switches. | +| `logger` | [Logger][] | The logging object Dactyl uses, with the verbosity set to match user input. | + +[Logger]: https://docs.python.org/3/library/logging.html#logger-objects diff --git a/examples/content/filters/multicode_tabs.md b/examples/content/filters/multicode_tabs.md new file mode 100644 index 0000000..17eb1c5 --- /dev/null +++ b/examples/content/filters/multicode_tabs.md @@ -0,0 +1,75 @@ +# Multi Code-Tabs Filter + +The multi code-tabs filter lets you list multiple code samples and have them appear as tabs in the HTML version. It looks like this: + + + +*Tab 1 Name* + +```json +{ + "id": 2, + "command": "account_info", + "account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59", + "strict": true, + "ledger_index": "validated" +} +``` + +*Tab 2 Name* + +```json +POST http://s1.ripple.com:51234/ +{ + "method": "account_info", + "params": [ + { + "account": "r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59", + "strict": true, + "ledger_index": "validated" + } + ] +} +``` + +*Tab 3 Name* + +```bash +rippled account_info r9cZA1mLK5R5Am25ArfXFmqgNwjZgnfk59 validated true +``` + + + + +## Syntax + +To display code in tabs, use the following syntax: + + + + _First Tab Name_ + + ```js + console.log("First tab contents"); + ``` + + _Second Tab Name_ + + ```py + print("Second tab contents") + ``` + + + +There is no hard limit on the number of tabs you can have. If the total width of the tab names gets too long for the viewscreen, they get broken into multiple lines, which may not look that good depending on your CSS. + +Tabs can only contain code blocks, not other contents. + +## With Custom Templates + +These tabs require some CSS and JavaScript to work: + +- [multicode_tabs.js](https://dactyl.link/template_assets/multicode_tabs.js) +- [dactyl-multicode_tabs.css](https://dactyl.link/template_assets/dactyl-multicode_tabs.css) + +If you're writing your own templates, you need to include these lines of code in your project to get the tabs to display properly. If you're using the built-in templates, they load this code automatically, so you don't need to do anything special. diff --git a/examples/content/filter-examples/xrefs.md b/examples/content/filters/xrefs.md similarity index 92% rename from examples/content/filter-examples/xrefs.md rename to examples/content/filters/xrefs.md index cc95a66..21b6106 100644 --- a/examples/content/filter-examples/xrefs.md +++ b/examples/content/filters/xrefs.md @@ -2,14 +2,14 @@ Intelligent cross-references are a feature designed to be a link when they can be, and text directing people to a different target (guide) if not. Here's some examples for testing: -[This should link to the callouts demo page.](XREF: filter-examples-callouts.html) +{#[This should link to the callouts demo page.](XREF: callouts.html)#} You can link by HTML filename: [ ](XREF: lists-and-codeblocks.html) Or by MD filename: [](XREF:lists-and-codeblocks.md) If the MD filename is ambiguous, you can specify the full path: -[](xref: filter-examples/xrefs.md) +[](xref: filters/xrefs.md) For an example of an inline cross reference, see [](xref:gfm-compat.html). diff --git a/examples/content/index.md b/examples/content/index.md new file mode 100644 index 0000000..08e0596 --- /dev/null +++ b/examples/content/index.md @@ -0,0 +1,22 @@ +# Dactyl + +Documentation tools for enterprise-quality documentation from Markdown source. Dactyl has advanced features to enable [single-sourcing](https://en.wikipedia.org/wiki/Single_source_publishing) and an extensible syntax for building well-organized, visually attractive docs. It generates output in HTML (natively), and can make PDFs if you have [Prince][] installed. + +[Prince]: http://www.princexml.com/ + +## Installation + +Dactyl requires [Python 3](https://python.org/). Install with [pip](https://pip.pypa.io/en/stable/): + +``` +sudo pip3 install dactyl +``` + +Or a local install in a virtualenv: + +```sh +# Create an activate a virtualenv so the package and dependencies are localized +virtualenv -p `which python3` venv_dactyl +source venv_dactyl/bin/activate +pip3 install dactyl +``` diff --git a/examples/content/usage/config.md b/examples/content/usage/config.md new file mode 100644 index 0000000..b760d3c --- /dev/null +++ b/examples/content/usage/config.md @@ -0,0 +1,103 @@ +--- +parent: usage.html +targets: + - everything +--- +# Configuration + +Dactyl is intended to be used with a config file containing a list of pages to parse. Pages are grouped into "targets" that represent a group of documents to be built together; a page can belong to multiple targets, and can even contain conditional syntax so that it builds differently depending on the target in question. Targets and pages can also use different templates from each other, and pages can inherit semi-arbitrary key/value pairs from the targets. + +The input pages in the config file should be specified relative to the `content_path`, which is `content/` by default. You can also specify a URL to pull in a Markdown file from a remote source, but if you do, Dactyl won't run any pre-processing on it. + +## Directory Paths +An advanced setup would probably have a directory structure such as the following: + +``` +./ # Top-level dir; this is where you run dactyl_* +./dactyl-config.yml # Default config file name +./content # Dir containing your .md source files +---------/*/*.md # You can sort .md files into subdirs if you like +---------/static/* # Static images referencd in your .md files +./templates/template-*.html # Custom HTML Templates +./assets # Directory for static files referenced by templates +./out # Directory where output gets generated. Can be deleted +``` + +All of these paths can be configured. + +## Targets + +A target represents a group of pages, which can be built together or concatenated into a single PDF. You should have at least one target defined in the `targets` array of your Dactyl config file. A target definition should consist of a short `name` (used to specify the target in the commandline and elsewhere in the config file) and a human-readable `display_name` (used mostly by templates but also when listing targets on the commandline). + +A simple target definition: + +```yaml +targets: + - name: kc-rt-faq + display_name: Ripple Trade Migration FAQ +``` + +In addition to `name` and `display_name`, a target definition can contain arbitrary key-values to be inherited by all pages in this target. Dictionary values are inherited such that keys that aren't set in the page are carried over from the target, recursively. The rest of the time, fields that appear in a page definition take precedence over fields that appear in a target definition. + +Some things you may want to set at the target level include `filters` (an array of filters to apply to pages in this target), `template` (template to use when building HTML), and `pdf_template` (template to use when building PDF). You can also use the custom values in templates and preprocessing. Some filters define additional fields that affect the filter's behavior. + +The following field names cannot be inherited: `name`, `display_name`, and `pages`. + +## Pages + +Each page represents one HTML file in your output. A page can belong to one or more targets. When building a target, all the pages belonging to that target are built in the order they appear in the `pages` array of your Dactyl config file. + +Example of a pages definition with two files: + +```yaml +pages: + - name: RippleAPI + category: References + html: reference-rippleapi.html + md: https://raw.githubusercontent.com/ripple/ripple-lib/0.17.2/docs/index.md + filters: + - remove_doctoc + - add_version + targets: + - local + - ripple.com + + - name: rippled + category: References + html: reference-rippled.html + md: reference-rippled.md + targets: + - local + - ripple.com +``` + +Each individual page definition can have the following fields: + +| Field | Type | Description | +|:-------------------------|:----------|:--------------------------------------| +| `targets` | Array | The short names of the targets that should include this page. | +| `html` | String | _(Optional)_ The filename where this file should be written in the output directory. If omitted, Dactyl chooses a filename based on the `md` field (if provided), the `name` field (if provided), or the current time (as a last resort). By default, generated filenames flatten the folder structure of the md files. To instead replicate the folder structure of the source documents in auto-generated filenames, add `flatten_default_html_paths: true` to the top level of your Dactyl config file. | +| `name` | String | _(Optional)_ Human-readable display name for this page. If omitted but `md` is provided, Dactyl tries to guess the right file name by looking at the first two lines of the `md` source file. | +| `md` | String | _(Optional)_ The markdown filename to parse to generate this page, relative to the **content_path** in your config. If this is not provided, the source file is assumed to be empty. (You might do that if you use a nonstandard `template` for this page.) | +| `openapi_specification` | String | _(Optional)_ The file path or http(s) URL to an OpenAPI v3.0 specification to be parsed into generated documentation. If provided, this entry becomes expanded into a set of several pages that describe the methods and data types defined for the given API. The generated pages inherit the other fields of this page object. **Experimental.** If the path is a relative path, it is evaluated based on the directory Dactyl is called from, not the content directory. | +| `api_slug` | String | _(Optional)_ If this is an `openapi_specification` entry, +| `category` | String | _(Optional)_ The name of a category to group this page into. This is used by Dactyl's built-in templates to organize the table of contents. | +| `template` | String | _(Optional)_ The filename of a custom [Jinja][] HTML template to use when building this page for HTML, relative to the **template_path** in your config. | +| `pdf_template` | String | _(Optional)_ The filename of a custom [Jinja][] HTML template to use when building this page for PDF, relative to the **template_path** in your config. | +| `openapi_md_template_path` | String | _(Optional)_ Path to a folder containing [templates to be used for OpenAPI spec parsing](#openapi-spec-templates). If omitted, use the [built-in templates](dactyl/templates/). | +| `parent` | String | _(Optional)_ The HTML filename of the page to treat as a parent of this one for purposes of hierarchy. If omitted, treat the page as a "top-level" page. | +| ... | (Various) | Additional arbitrary key-value pairs as desired. These values can be used by templates or pre-processing. | + +If the file specified by `md` begins with YAML frontmatter, separated by a line of exactly `---`, the frontmatter is used as a basis for these fields. Certain frontmatter fields are adapted from Jekyll format to Dactyl format: for example, `title` gets copied to `name` if the page does not have a `name`. + +The following fields are automatically added after a page has been parsed to HTML. (They're not available when preprocessing or rendering Markdown to HTML, but _are_ available when rendering HTML templates.) + +| Field | Type | Description | +|:-------------------------|:----------|:--------------------------------------| +| `plaintext` | String | A plaintext-only version of the page's markdown content, with all Markdown and HTML syntax removed. | +| `headermap` | Dictionary | A mapping of the page's headers to the unique IDs of those headers in the generated HTML version. | +| `blurb` | String | An introductory blurb generated from the page's first paragraph of text. | +| `children` | List | A list of pages, in order of appearance, that refer to this page as their `parent`. Each of these "child" pages is a reference to the page definition (dictionary) for that child. | +| `is_ancestor_of` | Function | A function that takes one argument, the string identifying a potential child page by that child's `html` field. This function returns `True` if this page is a direct or indirect parent of the child page. | + +[Jinja]: http://jinja.pocoo.org/ diff --git a/examples/content/usage/editing.md b/examples/content/usage/editing.md new file mode 100644 index 0000000..f38881a --- /dev/null +++ b/examples/content/usage/editing.md @@ -0,0 +1,55 @@ +--- +parent: usage.html +targets [everything] +--- +# Editing + +Dactyl supports extended Markdown syntax with the [Python-Markdown Extra](https://pythonhosted.org/Markdown/extensions/extra.html) module. This correctly parses most GitHub-Flavored Markdown syntax (such as tables and fenced code blocks) as well as a few other features. + +## Pre-processing + +Dactyl pre-processes Markdown files by treating them as [Jinja][] Templates, so you can use [Jinja's templating syntax](http://jinja.pocoo.org/docs/dev/templates/) to do advanced stuff like include other files or pull in variables from the config or commandline. Dactyl passes the following fields to Markdown files when it pre-processes them: + +| Field | Value | +|:------------------|:---------------------------------------------------------| +| `target` | The [target](#targets) definition of the current target. | +| `pages` | The [array of page definitions](#pages) in the current target. Use this to generate navigation across pages. (The default templates don't do this, but you should.) | +| `currentpage` | The definition of the page currently being rendered. This inherits most fields defined at the target level as well. | +| `categories` | A de-duplicated array of categories that are used by at least one page in this target, sorted in the order they first appear. | +| `config` | The global Dactyl config object. | +| `content` | The parsed HTML content of the page currently being rendered. | +| `current_time` | The current date as of rendering. The format is YYYY-MM-DD by default; you can also set the `time_format` field to a custom [stftime format string](http://strftime.org/). | +| `mode` | The output format: either `html` (default), `pdf`, or `md`. | + + +## Adding Variables from the Commandline + +You can pass in a JSON or YAML-formatted list of variables using `--vars` commandline switch. Any such variables get added as fields of `target` and inherited by `currentpage` in any case where `currentpage` does not already have the same variable name set. For example: + +```sh +$ cat md/vartest.md +Myvar is: '{{ target.myvar }}' + +$ dactyl_build --vars '{"myvar":"foo"}' +rendering pages... +writing to file: out/index.html... +Preparing page vartest.md +reading markdown from file: vartest.md +... parsing markdown... +... modifying links for target: default +... re-rendering HTML from soup... +writing to file: out/test_vars.html... +done rendering +copying static pages... + +$ cat out/test_vars.html | grep Myvar +

      Myvar is: 'foo'

      +``` + +If argument to `--vars` ends in `.yaml` or `.json`, Dactyl treats the argument as a filename and opens it as a YAML file. (YAML is a superset of JSON, so this works for JSON files.) Otherwise, Dactyl treats the argument as a YAML/JSON object directly. Be sure that the argument is quoted and escaped as necessary based on the commandline shell you use. + +You cannot set the following reserved keys: + +- `name` +- `display_name` (Instead, use the `--title` argument to set the display name of the target on the commandline.) +- `pages` diff --git a/examples/content/usage/elasticsearch.md b/examples/content/usage/elasticsearch.md new file mode 100644 index 0000000..30bce3d --- /dev/null +++ b/examples/content/usage/elasticsearch.md @@ -0,0 +1,46 @@ +--- +parent: usage.html +html: elasticsearch.html +targets: + - everything +--- +# ElasticSearch Compatibility + +Dactyl has the ability to build JSON formatted for upload to [ElasticSearch](https://www.elastic.co/products/elasticsearch) and even upload it directly. + +To build JSON files for upload to ElasticSearch, use the `--es` mode: + +```sh +dactyl_build --es +``` + +This writes files to the usual output directory using an ElasticSearch JSON template. Dactyl skips any files that do not have a `md` source parameter in this mode. The output filenames are the pages' `html` filenames, except ending in `.json` instead of `.html`. You can specify a custom template for these JSON files using the top-level `default_es_template` field in the config file. This template must be a valid JSON file and has several special properties as described in [ElasticSearch JSON Templates](#elasticsearch-json-templates). + +Dactyl can also upload these files directly to an ElasticSearch instance, even when building for another mode. For example, to build the HTML version of a target named `filterdemos` but also upload that target's JSON-formatted data to an ElasticSearch instance: + +```sh +dactyl_build -t filterdemos --html --es_upload https://my-es-instance.example.com:9200 +``` + +The parameter to `--es_upload` should be the base URL of your ElasticSearch index. You can omit the parameter to use the default base URL of `http://localhost:9200`. + + +## ElasticSearch JSON Templates + +Dactyl has a special format for JSON templates meant for creating ElasticSearch data. These templates must be valid JSON and are processed according to the following rules: + +- Any strings in the fields' values are "preprocessed" in a similar context to the Jinja2-based Markdown preprocessor. For example, the string `{{currentpage.name}}` evaluates to the page's name. +- Any object containing the key `__dactyl_eval__` is evaluated as a Python expression. The object is replaced with the results of the expression, with lists becoming JSON arrays and dictionaries becoming JSON objects. +- The above rules apply recursively to values nested in arrays and objects. All other values are preserved literally. + +The context provided to the preprocessing and to the `__dactyl_eval__` expressions is the same and contains the following: + +| Field | Python Type | Description | +|:----------------|:------------|:---------------------------------------------| +| `currentpage` | `dict` | The current page definition (usually derived from the config file) | +| `target` | `dict` | The current target definition (usually derived from the config file) | +| `categories` | `list` | A list of unique `category` values used by pages in the current target, in order of appearance. | +| `page_filters` | `list` | A list of the names of Dactyl filters applied to the current page. | +| `mode` | `str` | Always equal to `es` in this context | +| `current_time` | `str` | The current time, in the `time_format` specified in the config. (Defaults to YYYY-MM-DD) | +| `bypass_errors` | `bool` | If `true`, this build is running with the option to continue through errors where possible. | diff --git a/examples/content/link-checking.md b/examples/content/usage/link-checking.md similarity index 98% rename from examples/content/link-checking.md rename to examples/content/usage/link-checking.md index 575794c..9bca7d9 100644 --- a/examples/content/link-checking.md +++ b/examples/content/usage/link-checking.md @@ -1,6 +1,7 @@ --- -html: link-checking.html -parent: features.html +parent: usage.html +targets: + - everything --- # Link Checking diff --git a/examples/content/usage/openapi.md b/examples/content/usage/openapi.md new file mode 100644 index 0000000..475c07a --- /dev/null +++ b/examples/content/usage/openapi.md @@ -0,0 +1,55 @@ +--- +parent: usage.html +targets: [everything] +--- +# OpenAPI Specifications + +Dactyl can parse an [OpenAPI v3.0 specification](https://github.com/OAI/OpenAPI-Specification) to generate API documentation for the API that specification describes, including API methods and object schemas. + +## Ad-Hoc Usage + +You can use the `--openapi ` parameter to build a single target from + +```sh +dactyl_build --openapi openapi.yaml +``` + +You can combine this with `--md` or `--pdf` to output the generated documentation in Markdown or PDF format, respectively. + + +## Config File +You can add a special entry to the `pages` array to represent an API reference; Dactyl will expand that file into multiple pages that each inherit any fields of this entry. For example: + +``` +- openapi_specification: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml + api_slug: petstore + foo: bar + targets: + - baz +``` + +The `api_slug` field is optional, and provides a prefix that gets used for a bunch of file names and stuff. You can use the other fields to specify which HTML templates to use or to pass more info to those templates to control how they display. + +The generated pages are: + +- An "All Methods" table of contents, listing every path operation in the `paths` of the OpenAPI specification. +- "Tag Methods" table of contents pages for each `tag` used in the OpenAPI specification. +- Pages for all "API Methods" (path operations) in `paths` of the OpenAPI specification. +- A "Data Types" table of contents, listing every data type defined in the `schema` section of the OpenAPI specification. +- Individual pages for each data type in the OpenAPI specification's `schema` section. + +## Custom Templates + +You can override the [templates used for generated OpenAPI pages](#openapi-spec-templates) to adjust how the Markdown is generated. + +In the page or target definition of your config file, set the `openapi_md_template_path` field to a path that contains the following templates: + +| Filename | Template for... | +|:---------------------------------------|:------------------------------------| +| `template-openapi_data_type.md` | Each individual data type in the spec's `schema` section. | +| `template-openapi_data_types_toc.md` | Table of contents for the data types. | +| `template-openapi_endpoint_tag_toc.md` | Table of contents for each endpoint `tag` in the spec. | +| `template-openapi_endpoint_toc.md` | Table of contents for all endpoints and tags. | +| `template-openapi_endpoint.md` | Table of contents for each individual endpoint in the `paths` section. | + +These templates use [Jinja](http://jinja.pocoo.org/) syntax. diff --git a/examples/content/usage/style-checking.md b/examples/content/usage/style-checking.md new file mode 100644 index 0000000..7c54c24 --- /dev/null +++ b/examples/content/usage/style-checking.md @@ -0,0 +1,108 @@ +--- +parent: usage.html +targets: [everything] +--- +# Style Checking + +The style checker is experimental. It is only suitable for English text. It reports several details about document contents that may be helpful for identifying documents whose readability you could improve. These details are: + +- Discouraged words and phrases. +- Page length details. +- Readability scores. +- Spell-checking. + +Example usage: + +```sh +$ dactyl_style_checker +``` + +The style checker re-generates contents in-memory (never writing it out), unlike the link checker which requires you to run `dactyl_build` first. It only checks contents that come from Markdown, not from HTML templates. + +The style checker uses the first target in the config file unless you specify another target with `-t`. You can check just one file by passing its HTML path in the `--only` parameter. + +The exit code of the command is 0 (success) if it found no discouraged words, the spell checker found no unknown words, and no pages failed their configured readability goals. Otherwise, the exit code of the command is 1 (failure). + + +## Discouraged Words and Phrases + +You can suggest specific words or phrases to discourage. The style checker checks for instances of these words and phrases in documents' content, and suggests alternatives based on the phrase file. Dactyl does not check text in ``, `
      `, and `` elements since those are intended to be code samples.
      +
      +To configure lists of discouraged words and phrases, add the following config options:
      +
      +| Field                       | Value  | Description                           |
      +|:----------------------------|:-------|:--------------------------------------|
      +| `word_substitutions_file`   | String | The path to a YAML file with a single top-level map. The keys are the words to discourage and the values are suggestions of words to replace them with. |
      +| `phrase_substitutions_file` | String | The path to a YAML file with a single top-level map. The keys are phrases to discourage and the values are suggestions of phrases to replace them with. |
      +
      +You can add an exemption to a specific discouraged word/phrase rule with an HTML comment. The exemption applies to the whole output (HTML) file in which it appears.
      +
      +```html
      +Maybe the word "will" is a discouraged word, but you really want to use it here without flagging it as a violation? Adding a comment like this  makes it so.
      +```
      +
      +## Spell Checking
      +
      +Dactyl uses [pyspellchecker](https://pyspellchecker.readthedocs.io/en/latest/) to report possible spelling errors and suggest corrections. The built-in dictionary is not very thorough; you can extend it by providing a dictionary file with more words. Spell checking is case-insensitive.
      +
      +If you want the spell checker to skip a page, put `skip_spell_checker: true` in the page definition.
      +
      +If you want to ignore one or more words on a single page only, add a comment such as the following anywhere in the page:
      +
      +```html
      +
      +```
      +
      +To extend the built-in dictionary used for all files, add the following field to the config. (You cannot remove words from the built-in dictionary.)
      +
      +| Field           | Value  | Description                                       |
      +|:----------------|:-------|:--------------------------------------------------|
      +| `spelling_file` | String | Path to a text file with words to add to the dictionary. Each line of the file should contain a single word (case-insensitive). |
      +
      +
      +## Length Metrics
      +
      +Dactyl reports the number of characters of text, number of sentences, and number of words in each document. These counts only include text contents (the parts generated from Markdown). They do not include code samples (not even inlined code words), or images/figures. The sentence counts are estimates. Headings, list items, and table cells each count as one sentence in these metrics. The summary includes the averages across all pages, and the stats for the three longest and shortest pages.
      +
      +These metrics are intended to be helpful for choosing documents that would be better off combined or split up. They can also be useful for interpreting readability scores, which tend to be less reliable for very short documents.
      +
      +
      +## Readability Scores
      +
      +The style checker reports readability scores based on several formulas implemented in the [textstat library](https://github.com/shivam5992/textstat). These can help you identify documents with a high proportion of big words and long sentences.
      +
      +> **Caution:** Readability formulas are not very smart. Trying to get a high readability score can actually decrease the clarity of your writing if you aren't mindful of other factors. Things readability formulas usually don't take into account include: brevity; complexity of the high-level structure; logical connections such as cause and effect; and precise use of language. They tend to score tables and bulleted lists badly even though those structure are very helpful for actual readability.
      +
      +By default, Dactyl prints the readability scores for each page as it analyzes them. The `-q` option hides this output. The summary at the end lists the average scores for all pages analyzed and the three pages with the worst Flesch Reading Ease scores.
      +
      +### Readability Goals
      +
      +You can set readability goals for individual pages or an entire target by adding the `readability_goals` field. This field should contain a map of readability metrics to target scores. Goals defined for individual pages override goals set for the entire target. Dactyl compares a page's readability scores to any goals set and reports a list of pages that failed their goals in the summary. The goal passes if the page's score is equal better (easier to read) than the stated goal value, and fails otherwise. For Flesch Reading Ease, higher scores represent better readability; for the other tests, _lower_ scores represent better readability.
      +
      +**Note:** Since very short pages tend to have inconsistent and unreliable readability scores, Dactyl does not calculate readability scores for pages with fewer than 10 "sentences". (Bullet points, headings, and table cells each count as separate "sentences" for this purpose.)
      +
      +Example configuration:
      +
      +```yaml
      +targets:
      +
      +  - name: my-target
      +    display_name: Example Target
      +    readability_goals:
      +        flesch_reading_ease: 50
      +        automated_readability_index: 12
      +```
      +
      +The available readability tests are:
      +
      +| Field Name                     | Details                                     |
      +|:-------------------------------|:--------------------------------------------|
      +| `flesch_reading_ease`          | [Flesch reading ease](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests#Flesch_reading_ease). Maximum score 121.22; no limit on how negative the score can be. |
      +| `smog_index`                   | [SMOG grade](https://en.wikipedia.org/wiki/SMOG). Gives an estimated grade level. |
      +| `coleman_liau_index`           | [Coleman-Liau index](https://en.wikipedia.org/wiki/Coleman%E2%80%93Liau_index). Gives an estimated grade level. |
      +| `automated_readability_index`  | [Automated readability index](https://en.wikipedia.org/wiki/Automated_readability_index). Gives an estimated grade level. |
      +| `dale_chall_readability_score` | [Dale-Chall readability formula](https://en.wikipedia.org/wiki/Dale%E2%80%93Chall_readability_formula). Decimal representing difficulty; lower values map to lower grade levels. |
      +| `linsear_write_formula`        | [Linsear Write formula](https://en.wikipedia.org/wiki/Linsear_Write). Gives an estimated grade level. |
      +| `gunning_fog`                  | [Gunning fog index](https://en.wikipedia.org/wiki/Gunning_fog_index). Gives an estimated grade level. |
      +
      +Estimated grade levels are based on the United States school system and are given as decimal approximations. For example, `11.5` represents somewhere between 11th and 12th grade (high school junior to senior).
      diff --git a/examples/content/usage/usage.md b/examples/content/usage/usage.md
      new file mode 100644
      index 0000000..ccf5cfb
      --- /dev/null
      +++ b/examples/content/usage/usage.md
      @@ -0,0 +1,128 @@
      +---
      +parent: index.html
      +targets:
      +  - everything
      +---
      +# Usage
      +
      +Dactyl is intended to be used with a config file (typically `dactyl-config.yml`) that specifies a full list of files to build with necessary metadata and other config. So, if you've got this set up, you can run Dactyl like this:
      +
      +```sh
      +dactyl_build
      +```
      +
      +By default, this outputs HTML. You can also build a PDF or even [Markdown](#building-markdown). For a full list of Dactyl's commandline options, use the `-h` parameter.
      +
      +## Ad-Hoc Usage
      +
      +Simple ("Ad-Hoc") usage:
      +
      +```sh
      +$ dactyl_build --pages input1.md input2.md
      +```
      +
      +By default, the resulting HTML pages are written to a folder called `out/` in the current working directory. You can specify a different output path in the config file or by using the `-o` parameter.
      +
      +### Building PDF
      +
      +Dactyl generates PDFs by making temporary HTML files and running [Prince][]. Use the `--pdf` command to generate a PDF. Dactyl tries to come up with a sensible output filename by default, or you can provide one (which must end in `.pdf`):
      +
      +```sh
      +$ dactyl_build --pages input1.md input2.md --pdf MyGuide.pdf
      +```
      +
      +[Prince]: http://www.princexml.com/
      +
      +
      +## Specifying a Config File
      +
      +By default, Dactyl looks for a config file named `dactyl-config.yml` in the current working directory. You can specify an alternate config file with the `-c` or `--config` parameter:
      +
      +```sh
      +$ dactyl_build -c path/to/alt-config.yml
      +```
      +
      +For more information on configuration, see the `default-config.yml` and the [examples](examples/) folder.
      +
      +## Specifying a Target
      +
      +If your config file contains more than one **target**, Dactyl builds the first one by default. You can specify a different target by passing its `name` value with the `-t` parameter:
      +
      +```sh
      +$ dactyl_build -t non-default-target
      +```
      +
      +## Static Files
      +
      +Your templates may require certain static files (such as JavaScript, CSS, and images) to display properly. Your content may have its own static files (such as diagrams and figures). By default, Dactyl assumes that templates have static files in the `assets/` folder. You can configure this path and also specify one or more paths to static files referenced by your content. When you build, Dactyl copies files from these folders to the output folder by default depending on which mode you're building:
      +
      +| Build Mode         | Files copied to output folder by default                |
      +|:-------------------|:--------------------------------------------------------|
      +| HTML               | Both template and content static files                  |
      +| PDF                | Neither template nor content static files (cannot be overridden) |
      +| Markdown           | Content static files only                               |
      +| ElasticSearch JSON | Neither template nor content static files               |
      +
      +You can use a commandline flag to explicitly specify what gets copied to the output folder, except in the case of PDF. (In PDF mode, Dactyl writes only the final PDF to the output folder.) The flags are as follows:
      +
      +| Flag (long version) | Short version | Meaning                                |
      +|:--------------------|:--------------|:---------------------------------------|
      +| `--copy_static`     | `-s`          | Copy all static files to the out dir.  |
      +| `--no_static`       | `-S`          | Don't copy any static files to the out dir. |
      +| `--template_static` | `-T`          | Copy only templates' static files to the out dir |
      +| `--content_static`  | `-C`          | Copy only the content's static files to the out dir |
      +
      +The following config file parameters control what paths Dactyl checks for static content:
      +
      +| Field | Default | Description |
      +|---|---|---|
      +| `template_static_path` | `assets/` | Static files belonging to the templates. |
      +| `content_static_path` | (None) | Static files belonging to content. This can be a single folder path, as a string, or an array of paths to files or folders. Dactyl copies all files and folders (regardless of whether the current target uses them). |
      +
      +## Listing Available Targets
      +
      +If you have a lot of targets, it can be hard to remember what the short names for each are. If you provide the `-l` flag, Dactyl will list available targets and then quit without doing anything:
      +
      +```sh
      +$ dactyl_build -l
      +tests        Dactyl Test Suite
      +rc-install        Ripple Connect v2.6.3 Installation Guide
      +rc-release-notes        
      +kc-rt-faq        Ripple Trade Migration FAQ
      +```
      +
      +## Building Markdown
      +
      +This mode runs the pre-processor only, so you can generate Markdown files that are more likely to display properly in conventional Markdown parsers (like the one built into GitHub). Use the `--md` flag to output Markdown files, skipping the HTML/PDF templates entirely.
      +
      +```sh
      +$ dactyl_build --md
      +```
      +
      +## Building Only One Page
      +
      +If you only want to build a single page, you can use the `--only` flag, followed by the filename you want to build (either the input filename ending in `.md` or the output filename ending in `.html`):
      +
      +```sh
      +dactyl_build --only index.html --pdf
      +```
      +
      +This command can be combined with the `--pdf` or `--md` flags. You can also use it with the `--target` setting (in case you want the context from the target even though you're only building one page.)
      +
      +## Watch Mode
      +
      +You can use the `-w` flag to make Dactyl run continuously, watching for changes to its input templates or markdown files. Whenever it detects that a file has changed, Dactyl automatically rebuilds the output in whatever the current mode is, (HTML, PDF, or Markdown).
      +
      +To be detected as a change, the file has to match one of the following patterns:
      +
      +```
      +*.md
      +*/code_samples/*
      +template-*.html
      +```
      +
      +Beware: some configurations can lead to an infinite loop. (For example, if your output directory is a subdirectory of your content directory and you use Dactyl in `--md` mode.)
      +
      +**Limitations:** Watch mode can be combined with `--only`, but re-builds the page even when it detects changes to unrelated pages. Watch mode doesn't detect changes to the config file, static files, or filters.
      +
      +To stop watching, interrupt the Dactyl process (Ctrl-C in most terminals).
      diff --git a/examples/dactyl-config.yml b/examples/dactyl-config.yml
      index 9504205..aca6b5d 100644
      --- a/examples/dactyl-config.yml
      +++ b/examples/dactyl-config.yml
      @@ -7,12 +7,21 @@ filter_paths:
       
       defaults: &defaults
           hover_anchors: "ΒΆ"
      +    skip_cover: true
       
       spelling_file: spelling.txt
      +default_html_names: tail
      +
      +cover_page:
      +    html: index.html
      +    md: index.md
      +    name: Index
      +    pdf_template: pdf-cover.html
      +    template: landing.html
       
       targets:
           -   name: everything
      -        display_name: Dactyl Examples
      +        display_name: "Dactyl: A Heroic Doc Tool"
               filters:
                   - demote_headers
               demote_headers_pdf_only: true
      @@ -30,6 +39,14 @@ targets:
               <<: *defaults
       
       pages:
      +    -   md: usage/usage.md
      +    -   md: usage/config.md
      +    -   md: usage/editing.md
      +    -   md: usage/link-checking.md
      +    -   md: usage/style-checking.md
      +    -   md: usage/elasticsearch.md
      +    -   md: usage/openapi.md
      +
           -   name: Features
               html: features.html
               category: Features
      @@ -63,10 +80,6 @@ pages:
               targets:
                   - everything
       
      -    -   md: link-checking.md
      -        targets:
      -            - everything
      -
           -   md: conditionals.md
               category: Features
               parent: features.html
      @@ -109,16 +122,9 @@ pages:
               targets:
                   - everything
       
      -    -   name: Filters
      -        category: Filters
      -        section_header: true
      -        parent: index.html
      -        template: landing.html
      -        targets:
      -            - everything
      -            - filterdemos
      +    -   md: filters/filters.md
       
      -    -   md: filter-examples/callouts.md
      +    -   md: filters/callouts.md
               category: Filters
               parent: filters.html
               targets:
      @@ -127,7 +133,7 @@ pages:
               filters:
                   - callouts
       
      -    -   md: filter-examples/xrefs.md
      +    -   md: filters/xrefs.md
               category: Filters
               parent: filters.html
               targets:
      @@ -136,7 +142,7 @@ pages:
               filters:
                   - xrefs
       
      -    -   md: filter-examples/buttonize.md
      +    -   md: filters/buttonize.md
               category: Filters
               parent: filters.html
               targets:
      @@ -145,7 +151,7 @@ pages:
               filters:
                   - buttonize
       
      -    -   md: filter-examples/badges.md
      +    -   md: filters/badges.md
               category: Filters
               parent: filters.html
               targets:
      @@ -154,7 +160,7 @@ pages:
               filters:
                   - badges
       
      -    -   md: filter-examples/multicode_tabs.md
      +    -   md: filters/multicode_tabs.md
               category: Filters
               parent: filters.html
               targets:
      @@ -163,7 +169,7 @@ pages:
               filters:
                   - multicode_tabs
       
      -    -   md: filter-examples/external_links.md
      +    -   md: filters/external_links.md
               targets:
                   - everything
                   - filterdemos
      @@ -177,7 +183,7 @@ pages:
               targets:
                   - everything
       
      -    -   md: extending/extend-templates.md
      +    -   md: extending/templates.md
               category: Extending
               parent: extending-index.html
               targets:
      diff --git a/examples/templates/template-extends.html b/examples/templates/template-extends.html
      index 5fa5276..98778a0 100644
      --- a/examples/templates/template-extends.html
      +++ b/examples/templates/template-extends.html
      @@ -2,18 +2,16 @@
       
       {% block right_sidebar %}
       
      -