[Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) #8886

wbamberg · 2021-09-13T17:21:05Z

Fixes #8741 .

This PR converts all the docs under Web/API into Markdown.

wbamberg · 2021-09-13T17:41:18Z

Here's a Google sheet listing 2000 randomly selected pages under Web/API: https://docs.google.com/spreadsheets/d/1kKeUNNZlxaLUV2KsDtnflHNedm9Jm2EwDvdgb64QnPw/edit#gid=0 . I've shared with possible reviewers.

The purpose of this is just to ensure different people review different pages.

To use it:

put your name beside some set of pages, in the "Reviewer" column
check out this branch, and run yarn start
open the pages you claimed, and see how they look. Look especially for mangled formatting and live samples that don't work properly.
you could also check the Markdown source, although I think that is less likely to be a problem (since the conversion report tells us that all expected elements got converted)
capture any problems as comments to this PR

Thanks!

teoli2003 · 2021-09-13T18:50:19Z

@wbamberg : There is something strange with http://localhost:5000/en-US/docs/Web/API/Document_object_model/Using_the_W3C_DOM_Level_1_Core/Example. It gets a new flaw: unsafe_html…

wbamberg · 2021-09-13T19:01:12Z

@wbamberg : There is something strange with http://localhost:5000/en-US/docs/Web/API/Document_object_model/Using_the_W3C_DOM_Level_1_Core/Example. It gets a new flaw: unsafe_html…

The converter seems to be converting something like:

<pre> &lt;html&gt;
...
</pre>

...into:

<html>
...

...which is... surprising?

wbamberg · 2021-09-13T21:51:08Z

It seems like if there's a <pre> block containing HTML, and there are no elements before it, then it's getting converted as raw HTML.

files/en-us/web/api/document_object_model/using_the_w3c_dom_level_1_core/example/index.html is the only page in Web/API whose first element is a <pre> .

I wonder if this is connected with the converter using indentation for plain <pre> blocks rather than backticks (and then the indentation is getting destroyed somehow).

I can fix this by adding class="brush: html" to the <pre> block. I'm going to wait to see whether any other issues are found in these pages before making that fix though.

hamishwillee · 2021-09-14T00:14:55Z

A few notes on problems that are not to do with conversion:

remember to report that {{InheritanceDiagram}} does not have a sensible default rendering. i.e. it aways needs to be scrolled by default.
https://developer.mozilla.org/en-US/docs/web/api/indexeddb_api/using_indexeddb - the online demo link at very end broken
Wherever we see a comment like "Gecko-specific notes" this probably means cleaning the doc and possibly pushing into BCD.

sideshowbarker · 2021-09-14T01:38:16Z

http://localhost:5000/en-US/docs/web/api/offscreencanvas/converttoblob has this:

**`OffscreenCanvas.convertToBlob()`**method

…because the source has this:

The <strong><code>OffscreenCanvas.convertToBlob()</code></strong>method

That HTML renders as expected but the lack of a space in `**method causes the markdown to not render as expected.

What should I do for these kinds of cases? Ignore it?

I could do a regex replace across all files/en-us/web/api/ files in this branch to do s/\*\*(\w)/** $1/ — and also s/(\w)**/$1 \*\*/ — and that would make the markdown render as expected.

rg -P '`\*\*\w' files/en-us/web/api/ | wc -l
      14

So that’s 14 cases of the pattern like convertToBlob()**method`

$ rg -P '\w\*\*`' files/en-us/web/api/
files/en-us/web/api/bluetoothremotegattcharacteristic/writevaluewithresponse/index.md
15:The**`BluetoothRemoteGATTCharacteristic.writeValueWithResponse()`** method sets a {{domxref("BluetoothRemoteGATTCharacteristic")}} object’s `value` property to the bytes contained in a given {{JSxRef("ArrayBuffer")}}, calls [`WriteCharacteristicValue`(_this_=`this`, _value=value_, _response_=`"required"`)](https://webbluetoothcg.github.io/web-bluetooth/#writecharacteristicvalue), and returns the resulting {{JSxRef("Promise")}}.

files/en-us/web/api/bluetoothremotegattcharacteristic/writevalue/index.md
19:The**`BluetoothRemoteGATTCharacteristic.writeValue()`** method sets a {{domxref("BluetoothRemoteGATTCharacteristic")}} object’s `value` property to the bytes contained in a given {{JSxRef("ArrayBuffer")}}, calls [`WriteCharacteristicValue`(_this_=`this`, _value=value_, _response_=`"optional"`)](https://webbluetoothcg.github.io/web-bluetooth/#writecharacteristicvalue), and returns the resulting {{JSxRef("Promise")}}.

files/en-us/web/api/rtcpeerconnection/icegatheringstate/index.md
15:The read-only property**`RTCPeerConnection.iceGatheringState`** returns a string

So that’s 3 cases of the flip-side pattern like **`BluetoothRemoteGATTCharacteristic.

If we want, I can push a commit with those fixes to this branch.

Or we could also just change this in a follow-up PR post-merge.

It’s not strictly a problem with the conversion — instead it’s a problem with broken markup in the source (garbage in, garbage out…)

wbamberg · 2021-09-14T01:48:01Z

Great catch @sideshowbarker ! If you could push a fix for these 17 cases to this PR I think that would be ideal. (I assume you mean fix the converted Markdown, not fix the upstream HTML?)

sideshowbarker · 2021-09-14T02:51:09Z

(I assume you mean fix the converted Markdown, not fix the upstream HTML?)

Yup, I meant fixing the Markdown output.

Will run into and write up a commit.

This change fixes some cases where, because the HTML source markup lacked a space before a start tag or after a end tag, the resulting markdown output didn’t render as expected.

wbamberg · 2021-09-14T04:23:48Z

It seems like if there's a <pre> block containing HTML, and there are no elements before it, then it's getting converted as raw HTML.

files/en-us/web/api/document_object_model/using_the_w3c_dom_level_1_core/example/index.html is the only page in Web/API whose first element is a <pre> .

I wonder if this is connected with the converter using indentation for plain <pre> blocks rather than backticks (and then the indentation is getting destroyed somehow).

I can fix this by adding class="brush: html" to the <pre> block. I'm going to wait to see whether any other issues are found in these pages before making that fix though.

I just pushed a fix for this "unsafe HTML" issue.

wbamberg · 2021-09-14T04:54:37Z

I've found a few places where conversion of <kdb> elements inside <dd> was getting mangled, and have (I think) found and fixed all of these in wbamberg@830104e.

wbamberg · 2021-09-14T05:04:01Z

I found one <dd> containing just a space, which the converter did not make sense of. I fixed that in d52e248 but couldn't find any other instances of this issue.

sideshowbarker · 2021-09-14T05:14:02Z

http://localhost:5000/en-US/docs/web/api/document/createelement#parameters has this:

- _options _{{optional_inline}}

…because the HTML it was converted from has this:

 <dt><var>options </var>{{optional_inline}}</dt>

I’ll try to see if I can put together a regex pattern to catch any other possible cases of that (without catching a bunch of false positives)

teoli2003 · 2021-09-14T05:18:36Z

In http://localhost:5000/en-US/docs/web/api/webglrenderbuffer, there was a <code>missing at the beginning. Do you want me to fix the output here or in a follow-up (not a conversion problem per se)?

wbamberg · 2021-09-14T06:14:16Z

In http://localhost:5000/en-US/docs/web/api/webglrenderbuffer, there was a <code>missing at the beginning. Do you want me to fix the output here or in a follow-up (not a conversion problem per se)?

I think this is strictly optional. On the one hand it's a small low-risk change that's a definite improvement. On the other it's out of scope for this PR, isn't a regression from the HTML and is one of probably many bits of broken markup (we have fixed many in this work, but I'm sure there are many left).

teoli2003 · 2021-09-14T07:30:17Z

I think this is strictly optional. On the one hand it's a small low-risk change that's a definite improvement. On the other it's out of scope for this PR, isn't a regression from the HTML and is one of probably many bits of broken markup (we have fixed many in this work, but I'm sure there are many left).

I will do follow-ups afterward then.

This change fixes a bunch of cases of misplaced-space borkage found by markdownlint.

This change simplifies a few cases where the source was using emphasis inside code. It simplifies them by dropping the emphasis.

This change fixes some cases where the ">" blockquote marker ended up in the middle of link text of a hyperlink — which caused markdownlint to complain.

This change fixes a few cases with superfluous/redundant emphasis markup — as well as one case where for some reason the converter failed to convert some ... marked to markdown.

This fixes a case where apparently the source has both a macro call and an HTML hyperlink for the same thing. So this just drops the hyperlink and keeps the macro call.

This fixes a few cases where the sources had excessive code markup that didn’t serve a real informational purpose (and that made markdownlint confused). It just drops or tweaks some of the markdown.

wbamberg · 2021-09-14T15:02:20Z

We've reviewed the 2000 pages in the spreadsheet, about 1/3 of the total number of pages. We've found a smattering of small issues, mostly to do with badly formed inline markup. We've tried to make generalized fixes across the Web/API docs (especially @sideshowbarker !) I think there's a good chance that there will be a few more of these little issues in the docs, but I don't think it's likely that there are big problems with this conversion, so I think we should merge it.

sideshowbarker · 2021-09-14T15:39:16Z

I think there's a good chance that there will be a few more of these little issues in the docs, but I don't think it's likely that there are big problems with this conversion, so I think we should merge it.

+1 — I think we can be confident at this point that we’ve scrutinized everything pretty thoroughly

wbamberg added 2 commits September 13, 2021 10:03

Rename HTML files to MD

11723e2

Convert file contents to Markdown

22a0144

wbamberg requested a review from a team as a code owner September 13, 2021 17:21

wbamberg requested review from sideshowbarker and removed request for a team September 13, 2021 17:21

wbamberg changed the title ~~Convert webapi to markdown (DO NOT SQUASH MERGE)~~ [Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) Sep 13, 2021

sideshowbarker and others added 2 commits September 14, 2021 12:40

Fix problematic strong+code Markdown output

b019995

This change fixes some cases where, because the HTML source markup lacked a space before a start tag or after a end tag, the resulting markdown output didn’t render as expected.

Fix unsafe HTML leaking into page

54bc4a4

wbamberg added 2 commits September 13, 2021 21:28

Fix broken strong markup

003d04e

Fix mangled kbd conversion

830104e

Fix a mangled dd

d52e248

Fix a couple of cases of problematic emphasis

107cf44

sideshowbarker added 3 commits September 14, 2021 16:49

Fix some misplaced spaces

9b9d8f1

This change fixes a bunch of cases of misplaced-space borkage found by markdownlint.

Fix a few emphasis-in-code cases

2c8ac49

This change simplifies a few cases where the source was using emphasis inside code. It simplifies them by dropping the emphasis.

Fix some oddly-placed-blockquote-marker cases

3096bb3

This change fixes some cases where the ">" blockquote marker ended up in the middle of link text of a hyperlink — which caused markdownlint to complain.

sideshowbarker and others added 5 commits September 14, 2021 16:57

Fix various superfluous/redundant-emphasis cases

603550e

This change fixes a few cases with superfluous/redundant emphasis markup — as well as one case where for some reason the converter failed to convert some ... marked to markdown.

Fix one oddball markdown case

cc1c42d

This fixes a case where apparently the source has both a macro call and an HTML hyperlink for the same thing. So this just drops the hyperlink and keeps the macro call.

Fix some “too much code markup” cases

ca0082f

This fixes a few cases where the sources had excessive code markup that didn’t serve a real informational purpose (and that made markdownlint confused). It just drops or tweaks some of the markdown.

Fix misplaced end-of-code mark

72e0805

Fix another misplace end-of-code mark

3c77131

sideshowbarker approved these changes Sep 14, 2021

View reviewed changes

wbamberg merged commit 0a16b27 into mdn:main Sep 14, 2021

wbamberg deleted the convert-webapi-to-markdown branch September 28, 2021 04:00

wbamberg mentioned this pull request Oct 24, 2021

[Markdown][Add-ons] Convert Add-ons docs to Markdown - DO NOT SQUASH MERGE #10057

Merged

github-actions bot locked as resolved and limited conversation to collaborators Sep 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) #8886

[Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) #8886

wbamberg commented Sep 13, 2021

wbamberg commented Sep 13, 2021

teoli2003 commented Sep 13, 2021

wbamberg commented Sep 13, 2021

wbamberg commented Sep 13, 2021

hamishwillee commented Sep 14, 2021 •

edited

Loading

sideshowbarker commented Sep 14, 2021 •

edited

Loading

wbamberg commented Sep 14, 2021

sideshowbarker commented Sep 14, 2021

wbamberg commented Sep 14, 2021

wbamberg commented Sep 14, 2021

wbamberg commented Sep 14, 2021

sideshowbarker commented Sep 14, 2021

teoli2003 commented Sep 14, 2021

wbamberg commented Sep 14, 2021

teoli2003 commented Sep 14, 2021

wbamberg commented Sep 14, 2021 •

edited

Loading

sideshowbarker commented Sep 14, 2021

[Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) #8886

[Markdown] [Web/API] Convert webapi to markdown (DO NOT SQUASH MERGE) #8886

Conversation

wbamberg commented Sep 13, 2021

wbamberg commented Sep 13, 2021

teoli2003 commented Sep 13, 2021

wbamberg commented Sep 13, 2021

wbamberg commented Sep 13, 2021

hamishwillee commented Sep 14, 2021 • edited Loading

sideshowbarker commented Sep 14, 2021 • edited Loading

wbamberg commented Sep 14, 2021

sideshowbarker commented Sep 14, 2021

wbamberg commented Sep 14, 2021

wbamberg commented Sep 14, 2021

wbamberg commented Sep 14, 2021

sideshowbarker commented Sep 14, 2021

teoli2003 commented Sep 14, 2021

wbamberg commented Sep 14, 2021

teoli2003 commented Sep 14, 2021

wbamberg commented Sep 14, 2021 • edited Loading

sideshowbarker commented Sep 14, 2021

hamishwillee commented Sep 14, 2021 •

edited

Loading

sideshowbarker commented Sep 14, 2021 •

edited

Loading

wbamberg commented Sep 14, 2021 •

edited

Loading