Markdown parsing in doc comments for help messages Tracking Issue #5900

ModProg · 2025-02-03T22:20:38Z

This tracks the feature unstable-markdown that implements compile time markdown parsing in the derive macros.

Implemented Markdown features

markdown	output	comment
paragraph	joined on one line, surrounded by empty line
`# heading`	bold + underline + surrounded by empty line	default style for heading
`![image]()`	alt text	anyone feeling sixel?!
`<html>`	verbatim
`> blockquote`		alternatively could use `>`
code block		same styling as `code`
(un)ordered lists
`emphasis`	italic
`strong`	bold
`~stikethrough~`	strikethrough
`[link](url)`	underlined alt text + OSC 8
`inline code `	bold	default style for literal
`---` hrule	`---` surrounded by empty lines
`\| tables \|`	verbatim
`\\\n` and `\n` hard break	ends line

Open Questions

unicode support i.e. • for bullet lists, Options::ENABLE_SMART_PUNCTUATION for pulldown_cmark, unicode lines for blockquotes
using the default clap styling for code makes it the same to bold.
should we make images a link to the image source
should we add custom html tags a la https://docs.rs/color-print/latest/color_print/ this would allow users to use colors without us actually supporting runtime style information.
should we use other enumeration for nested lists? e.g. a., b. and i., ii.? we could use https://docs.rs/nominals for that
styling for code blocks and block quotes
separate control of OSC 8 for links & colors
option to display URLs next to links.
option to diplay URLs at the end of the help.

Markdown Parser

See https://github.com/rosetta-rs/md-rosetta-rs for a comparison of parsers.

`pulldown-cmark`

Same behavior as rustdoc (options for reference https://github.com/rust-lang/rust/blob/59588250ad973ce69bd15879314c9769e65f36b3/compiler/rustc_resolve/src/rustdoc.rs#L244-L250).

But higher compile times, runtime performance probably not as impactful as we are doing it at compile time.

`minimad`

Had some parsing issues but might be fixable.

The text was updated successfully, but these errors were encountered:

ilyagr · 2025-02-04T00:43:01Z

(I initially posted this in #5891 (comment) , but I think this is a better place)

Thanks for a cool feature!

I haven't looked at this PR carefully, but I'd appreciate an option to print hyperlink targets normally in addition to possible terminal hyperlinks, so that the user can copy-paste them normally. Even if the terminal understands hyperlinks, users might not expect a terminal to behave like a web browser.

Also, some terminals make it hard to use hyperlinks in spite of supporting them. For example, in Ghostty 1.1.0 on a Mac, if mouse detection is on (the program inside the terminal requests mouse events, which for me is tmux), you need to hold both Cmd and Shift to be able to click on a link. I think that might be a bug (I might report it to them), but they might consider it a feature, and in any case it illustrates a problem.

ModProg · 2025-02-04T10:17:34Z

I haven't looked at this PR carefully, but I'd appreciate an option to print hyperlink targets normally in addition to possible terminal hyperlinks, so that the user can copy-paste them normally.

Currently you could achieve this by using the <https://example.com> syntax.

What would you like to see here additionally?

epage · 2025-02-04T18:04:53Z

One question is what parser to use for this. rustdoc compatibility is great but so is having good build times. I would say that my bare minimum requirement is that a subset of markdown can be parsed by rustdoc and clap.

In #5891, @dpc mentioned the idea of using a djot parser. It was added to https://github.com/rosetta-rs/md-rosetta-rs. Not as fast to build as minimad but faster than pulldown-cmark. Release parse times are all in the same ballpark. Looking over their quick start, it mostly looks like a subset would work. The one question I had was about **strong** as they only mention *strong* support which is *emphasis*/_emphasis_ in markdown. Looking over the reference and using the play ground, it looks like it can work because **strong** creates <strong><strong>strong</strong></strong>

It would be good to evaluate minimad further to see if it has a subset of markdown that renders that same as rustdoc and how that compares to djot.

ilyagr · 2025-02-04T20:37:51Z

I haven't looked at this PR carefully, but I'd appreciate an option to print hyperlink targets normally in addition to possible terminal hyperlinks, so that the user can copy-paste them normally.

Currently you could achieve this by using the <https://example.com> syntax.

What would you like to see here additionally?

If I write

You must [frobnicate] all inputs!

[frobnicate]: https://example.com

I might want it to keep looking like that on output. I'd be quite happy for "frobnicate" to also become a terminal hyperlink, but I'd want to keep the last line in the output in the help text (if it goes to the terminal) for copy-pasting ease.

I'd probably want the same for

You must [frobnicate](https://example.com) all the inputs.

OTOH, when the same text is used to generate HTML for a CLI Reference (which I currently do with https://github.com/ConnorGray/clap-markdown, which has some limitations), then of course I would want only a normal hyperlink.

ModProg · 2025-02-06T14:05:19Z

OTOH, when the same text is used to generate HTML for a CLI Reference (which I currently do with https://github.com/ConnorGray/clap-markdown, which has some limitations), then of course I would want only a normal hyperlink.

This is difficult to impossible with the current approach (actually this feature in its current state will break clap-markdown unless it starts parsing ANSI escapes), the problem is we do the markdown handling at compile time, i.e. we basically cannot change the output at runtime.

If we want to make it configurable, we'd need to add flags to #[clap(...)].

Not an actual solution but a workaround for what you want to render:

#[derive(Debug, Parser)]
struct Args {
    /// You must [frobnicate] all inputs!
    ///
    /// [frobnicate]: https://example.com
    first: String,
    /// You must [frobnicate](https://example.com) all inputs!
    second: String,
    /// You must [frobnicate] all inputs!
    ///
    /// \[frobnicate]: <https://example.com>
    /// 
    /// [frobnicate]: https://example.com
    working_first: String,
    /// You must [frobnicate](https://example.com)(<https://example.com>) all inputs!
    working_second: String,
}

ModProg · 2025-02-06T14:09:31Z

It would be good to evaluate minimad further to see if it has a subset of markdown that renders that same as rustdoc and how that compares to djot.

We should probably create some test cases to compare the “correctness” of different non pulldowncmark implementations to decide whether we can accept/fix them.

ilyagr · 2025-02-06T22:35:37Z

Thanks for the workaround!

I appreciate the difficulty; I'm not sure what the best solution is. It seems like this might be a blocker for us for turning on the markdown rendering (which is not the end of the world), but maybe we'll use the workaround, or figure something else out.

I also hope that if markdown rendering stays off, clap_markdown won't break.

ilyagr · 2025-02-06T22:59:32Z

Here's one possible approach. Instead of trying to reproduce the exact formatting of the input, just detect all the links in the output, and have the option to display them after the text.

So, even if the input is

    /// You must [frobnicate](https://example.com) all inputs!
    second: String,

the output to the terminal could have the link shown in the footnote style:

<SECOND>
You must frobnicate all the inputs!

[frobnicate]: https://example.com

There could also be square brackets around the first "frobnicate", but I don't think they would be very useful¹. OTOH, the output style could look even less like Markdown, this look is just the first one that came to mind and seemed OK.

if you are sure the underline gets shown in the terminal, at least. Actually, another upside of this approach is that you could eventually use slightly different output format depending on what you know about the terminal capabilities. ↩

ModProg mentioned this issue Feb 3, 2025

Markdown parsing in doc comments #5891

Merged

7 tasks

epage added A-help Area: documentation, including docs.rs, readme, examples, etc... C-tracking-issue Category: A tracking issue for an unstable feature S-waiting-on-design Status: Waiting on user-facing design to be resolved before implementing labels Feb 3, 2025

epage mentioned this issue Feb 3, 2025

Clap derive should parse markdown doc comment into normal text #2389

Closed

2 tasks

dpc mentioned this issue Feb 4, 2025

Smile, you're in a benchmark hellux/jotdown#80

Open

ilyagr mentioned this issue Feb 6, 2025

roff manpages contain literal ```text blocks jj-vcs/jj#5605

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Markdown parsing in doc comments for help messages Tracking Issue #5900

Markdown parsing in doc comments for help messages Tracking Issue #5900

ModProg commented Feb 3, 2025 •

edited

Loading

ilyagr commented Feb 4, 2025 •

edited

Loading

ModProg commented Feb 4, 2025

epage commented Feb 4, 2025

ilyagr commented Feb 4, 2025 •

edited

Loading

ModProg commented Feb 6, 2025

ModProg commented Feb 6, 2025

ilyagr commented Feb 6, 2025 •

edited

Loading

ilyagr commented Feb 6, 2025 •

edited

Loading

Markdown parsing in doc comments for help messages Tracking Issue #5900

Markdown parsing in doc comments for help messages Tracking Issue #5900

Comments

ModProg commented Feb 3, 2025 • edited Loading

Implemented Markdown features

Open Questions

Markdown Parser

pulldown-cmark

minimad

ilyagr commented Feb 4, 2025 • edited Loading

ModProg commented Feb 4, 2025

epage commented Feb 4, 2025

ilyagr commented Feb 4, 2025 • edited Loading

ModProg commented Feb 6, 2025

ModProg commented Feb 6, 2025

ilyagr commented Feb 6, 2025 • edited Loading

ilyagr commented Feb 6, 2025 • edited Loading

Footnotes

ModProg commented Feb 3, 2025 •

edited

Loading

`pulldown-cmark`

`minimad`

ilyagr commented Feb 4, 2025 •

edited

Loading

ilyagr commented Feb 4, 2025 •

edited

Loading

ilyagr commented Feb 6, 2025 •

edited

Loading

ilyagr commented Feb 6, 2025 •

edited

Loading