Refactor the the ParsedTypeDocstring #874

tristanlatr · 2025-02-02T19:35:58Z

Refactor the the ParsedTypeDocstring - used in numpy and google docformat as well as when --process-types option is passed.

This gets rid of some unfortunate code that was not doing it's thing right. We replace that dirty processing by legitimately creating a docutils tree for these types are rendering correctly. In order to do that, the linker had to be adjusted not to wrap links inside <code> tags anymore (see the preliminary refactor commit ec6c907).

Make possible for PR #723 to be complete
Fixes #581
Fixes #873

Before:

After:

The warning diff was unexpected... it appears that pydoctor was reporting link not found at several unrelated places, increasing the number of warnings uselessly.

…de> tags are now added by the html translator when the document is a docstring. Otherwise it does not add the enclosing <code> tags because we're already in the middle of a code tag or similar <span class="rst-literal">. Adjust the themes so the <code> tags and <span class="rst-literal"> are really equivalent.

tristanlatr · 2025-02-02T19:36:44Z

pydoctor/epydoc/markup/_types.py

-
-        combined_tokens: list[tuple[Any, TokenType]] = []
-
-        open_parenthesis = 0
-        open_square_braces = 0
-
-        for _token, _type in tokens:
-            # The actual type of_token is str | Tag | Node. 
-
-            if (_type is TokenType.DELIMITER and _token in ('[', '(', ')', ']')) \
-               or _type is TokenType.OBJ: 
-                if _token == "[": open_square_braces += 1
-                elif _token == "(": open_parenthesis += 1
-
-                if _type is TokenType.OBJ:
-                    _token = docstring_linker.link_xref(
-                                _token, _token, self._lineno)
-
-                if open_square_braces + open_parenthesis > 0:
-                    try: last_processed_token = combined_tokens[-1]
-                    except IndexError:
-                        combined_tokens.append((_token, _type))
-                    else:
-                        if last_processed_token[1] is TokenType.OBJ \
-                           and isinstance(last_processed_token[0], Tag):
-                            # Merge with last Tag
-                            if _type is TokenType.OBJ:
-                                assert isinstance(_token, Tag)
-                                last_processed_token[0](*_token.children)
-                            else:
-                                last_processed_token[0](_token)
-                        else:
-                            combined_tokens.append((_token, _type))
-                else:
-                    combined_tokens.append((_token, _type))
-
-                if _token == "]": open_square_braces -= 1
-                elif _token == ")": open_parenthesis -= 1
-
-            else:
-                # the token will be processed in _convert_type_spec_to_stan() method.
-                combined_tokens.append((_token, _type))
-
-        return combined_tokens


This was the unfortunate code

codecov · 2025-02-02T19:37:43Z

Codecov Report

Attention: Patch coverage is 96.22642% with 2 lines in your changes missing coverage. Please review.

Project coverage is 92.77%. Comparing base (3357f21) to head (7eb22b7).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pydoctor/node2stan.py	71.42%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #874      +/-   ##
==========================================
- Coverage   92.79%   92.77%   -0.02%     
==========================================
  Files          47       47              
  Lines        8468     8448      -20     
  Branches     1550     1542       -8     
==========================================
- Hits         7858     7838      -20     
  Misses        350      350              
  Partials      260      260

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This reverts commit c6ffcef.

…pected warnings

tristanlatr · 2025-02-02T21:52:15Z

The warning diff is OK, I've double checked it and it appears that pydoctor was reporting link not found at several unrelated places, increasing the number of warnings uselessly.

docs/source/conf.py

tristanlatr · 2025-02-13T23:37:04Z

pydoctor/test/test_napoleon_docstring.py

+            ['defaultdict', ', and '],
+            ['defaultdict', ', or '],
+            ['defaultdict', ' of '],
+            ['defaultdict', ' of ', 'x', ' to '],
+            ['defaultdict', ', ', 'and'],
+            ['defaultdict', ', ', 'or'],


These are edge cases where a natural language delimiter is trailing in a type expression from docstrings.

It is not valid english so the token should not be interpreted as a delimiter when it expect a non-existent following-up token. It should end up in the same token just like 'defaultdict of' below. Though, emitting more unknown tokens kinds should be done with care since it will alter the behaviour of numpy and google style docstring's Return section and friends. But these input should really be treated as a single unknown kind of token instead of being tokenized wrongly. The only exception is the comma delimiter that is special because the adding a trailing coma should not make the preceding token to be merged with the comma.

This can be addressed in a following-up PR.

tristanlatr · 2025-02-13T23:38:24Z

pydoctor/test/test_templatewriter.py

-test:36: bad rendering of class signature: SAXParseException: <unknown>.+ undefined entity
-'''.splitlines()
-
-    # Some how the type processing get rid of the non breaking spaces, but it's more an implementation


this implementation detail is now gone.

tristanlatr · 2025-02-13T23:42:31Z

pydoctor/linker.py

        else:
            if isinstance(resolved, str):
                xref = intersphinx_link(label, url=resolved)
            else:
                xref = taglink(resolved, self.page_url, label)

-        return tags.code(xref)
+        return xref


This change is rather important: it makes the linker always produce a <a> tag only and never wrap it in a <code> tag. This is good because having the code tag is rather a presentation matter and the linker should,.. well link. This transfers the presentation responsibility to the node2stan.py module that will include <code> or not based on the source of the document.

tristanlatr · 2025-02-13T23:44:35Z

pydoctor/linker.py

-        return tags.transparent(label)
+        return tags.a(label)

    def link_xref(self, target: str, label: "Flattenable", lineno: int) -> Tag:
-        return tags.code(label)
+        return tags.a(label)


This explains the major HTML diffs in the tests. It's Important that the NotFoundLinker emits a <a> tag (in this case with no href attribute) rather than a transparent or code tag because it's closer to what the actual linker does.

tristanlatr · 2025-02-13T23:49:03Z

pydoctor/test/testpackages/numpy/_machar.py

I'm gonna dig for a better reproducer of the issue and remove this huge file from the tests.

pydoctor/epydoc/markup/_types.py

- fix the linenumber of reported type docstring issues by 1. - the nested warnings for unknown token are now properly propagated and reported.

…m:twisted/pydoctor into 873-implement-parsedtypedocstring.to_node

…inenumber correct for epytext and restructuredtext.

This reverts commit 2681285.

tristanlatr

This is a hell of a refactor... But it's in the right direction.

tristanlatr · 2025-02-17T15:58:08Z

pydoctor/epydoc/markup/_types.py

+
+            TokenType.CONTROL: lambda _token, _, __: \
+                nodes.emphasis(_token, _token),
+


Suggested change

github-actions · 2025-02-20T00:03:31Z

Diff from pydoctor_primer, showing the effect of this PR on open source code:

numpy (https://github.com/numpy/numpy)
- /projects/numpy/numpy/_core/_ufunc_config.py:389: bad docstring: invalid value set (missing closing brace): {divide
+ /projects/numpy/numpy/_core/_ufunc_config.py:388: bad docstring: invalid value set (missing closing brace): {divide
- /projects/numpy/numpy/_core/_ufunc_config.py:389: bad docstring: invalid value set (missing opening brace): invalid}
+ /projects/numpy/numpy/_core/_ufunc_config.py:388: bad docstring: invalid value set (missing opening brace): invalid}
- /projects/numpy/numpy/_core/fromnumeric.py:387: bad docstring: invalid value set (missing closing brace): {'raise'(
+ /projects/numpy/numpy/_core/fromnumeric.py:386: bad docstring: invalid value set (missing closing brace): {'raise'(
- /projects/numpy/numpy/_core/fromnumeric.py:387: bad docstring: invalid value set (missing opening brace): }
+ /projects/numpy/numpy/_core/fromnumeric.py:386: bad docstring: invalid value set (missing opening brace): }
- /projects/numpy/numpy/_core/fromnumeric.py:387: bad docstring: unbalanced parenthesis in type expression
+ /projects/numpy/numpy/_core/fromnumeric.py:386: bad docstring: unbalanced parenthesis in type expression
- /projects/numpy/numpy/_core/fromnumeric.py:3899: bad docstring: invalid value set (missing closing brace): {int
+ /projects/numpy/numpy/_core/fromnumeric.py:3898: bad docstring: invalid value set (missing closing brace): {int
- /projects/numpy/numpy/_core/fromnumeric.py:3899: bad docstring: invalid value set (missing opening brace): float}
+ /projects/numpy/numpy/_core/fromnumeric.py:3898: bad docstring: invalid value set (missing opening brace): float}
- /projects/numpy/numpy/_core/fromnumeric.py:3926: bad docstring: invalid value set (missing closing brace): {int
+ /projects/numpy/numpy/_core/fromnumeric.py:3925: bad docstring: invalid value set (missing closing brace): {int
- /projects/numpy/numpy/_core/fromnumeric.py:3926: bad docstring: invalid value set (missing opening brace): float}
+ /projects/numpy/numpy/_core/fromnumeric.py:3925: bad docstring: invalid value set (missing opening brace): float}
- /projects/numpy/numpy/_core/fromnumeric.py:4103: bad docstring: invalid value set (missing closing brace): {int
+ /projects/numpy/numpy/_core/fromnumeric.py:4102: bad docstring: invalid value set (missing closing brace): {int
- /projects/numpy/numpy/_core/fromnumeric.py:4103: bad docstring: invalid value set (missing opening brace): float}
+ /projects/numpy/numpy/_core/fromnumeric.py:4102: bad docstring: invalid value set (missing opening brace): float}
- /projects/numpy/numpy/_core/fromnumeric.py:4130: bad docstring: invalid value set (missing closing brace): {int
+ /projects/numpy/numpy/_core/fromnumeric.py:4129: bad docstring: invalid value set (missing closing brace): {int
- /projects/numpy/numpy/_core/fromnumeric.py:4130: bad docstring: invalid value set (missing opening brace): float}
+ /projects/numpy/numpy/_core/fromnumeric.py:4129: bad docstring: invalid value set (missing opening brace): float}
- /projects/numpy/numpy/_core/einsumfunc.py:776: bad docstring: invalid value set (missing closing brace): {bool
+ /projects/numpy/numpy/_core/einsumfunc.py:775: bad docstring: invalid value set (missing closing brace): {bool
- /projects/numpy/numpy/_core/einsumfunc.py:776: bad docstring: invalid value set (missing opening brace): }
+ /projects/numpy/numpy/_core/einsumfunc.py:775: bad docstring: invalid value set (missing opening brace): }
- /projects/numpy/numpy/_core/einsumfunc.py:1090: bad docstring: invalid value set (missing closing brace): {data-type
+ /projects/numpy/numpy/_core/einsumfunc.py:1089: bad docstring: invalid value set (missing closing brace): {data-type
- /projects/numpy/numpy/_core/einsumfunc.py:1090: bad docstring: invalid value set (missing opening brace): None}
+ /projects/numpy/numpy/_core/einsumfunc.py:1089: bad docstring: invalid value set (missing opening brace): None}
- /projects/numpy/numpy/_core/einsumfunc.py:1114: bad docstring: invalid value set (missing closing brace): {False
+ /projects/numpy/numpy/_core/einsumfunc.py:1113: bad docstring: invalid value set (missing closing brace): {False
- /projects/numpy/numpy/_core/einsumfunc.py:1114: bad docstring: invalid value set (missing opening brace): }
+ /projects/numpy/numpy/_core/einsumfunc.py:1113: bad docstring: invalid value set (missing opening brace): }
- /projects/numpy/numpy/lib/_datasource.py:175: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:174: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:175: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:174: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_datasource.py:177: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:176: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:177: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:176: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:499: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:499: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_datasource.py:500: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:501: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:500: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:501: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_datasource.py:502: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:502: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_datasource.py:669: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:668: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:669: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:668: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_datasource.py:671: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_datasource.py:670: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_datasource.py:671: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_datasource.py:670: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_npyio_impl.py:333: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_npyio_impl.py:332: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_npyio_impl.py:333: bad docstring: invalid value set (missing opening brace): }
+ /projects/numpy/numpy/lib/_npyio_impl.py:332: bad docstring: invalid value set (missing opening brace): }
- /projects/numpy/numpy/lib/_npyio_impl.py:1455: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_npyio_impl.py:1454: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_npyio_impl.py:1455: bad docstring: invalid value set (missing opening brace): str}
+ /projects/numpy/numpy/lib/_npyio_impl.py:1454: bad docstring: invalid value set (missing opening brace): str}
- /projects/numpy/numpy/lib/_npyio_impl.py:1809: bad docstring: invalid value set (missing closing brace): {None
+ /projects/numpy/numpy/lib/_npyio_impl.py:1808: bad docstring: invalid value set (missing closing brace): {None
- /projects/numpy/numpy/lib/_npyio_impl.py:1809: bad docstring: invalid value set (missing opening brace): sequence}
+ /projects/numpy/numpy/lib/_npyio_impl.py:1808: bad docstring: invalid value set (missing opening brace): sequence}
- /projects/numpy/numpy/lib/_npyio_impl.py:1827: bad docstring: invalid value set (missing closing brace): {True
+ /projects/numpy/numpy/lib/_npyio_impl.py:1826: bad docstring: invalid value set (missing closing brace): {True
- /projects/numpy/numpy/lib/_npyio_impl.py:1827: bad docstring: invalid value set (missing opening brace): }
+ /projects/numpy/numpy/lib/_npyio_impl.py:1826: bad docstring: invalid value set (missing opening brace): }
- /projects/numpy/numpy/lib/_function_base_impl.py:1005: bad docstring: invalid value set (missing closing brace): {1
+ /projects/numpy/numpy/lib/_function_base_impl.py:1004: bad docstring: invalid value set (missing closing brace): {1
- /projects/numpy/numpy/lib/_function_base_impl.py:1005: bad docstring: invalid value set (missing opening brace): 2}
+ /projects/numpy/numpy/lib/_function_base_impl.py:1004: bad docstring: invalid value set (missing opening brace): 2}
- /projects/numpy/numpy/lib/_function_base_impl.py:3935: bad docstring: invalid value set (missing closing brace): {int
+ /projects/numpy/numpy/lib/_function_base_impl.py:3934: bad docstring: invalid value set (missing closing brace): {int
- /projects/numpy/numpy/lib/_function_base_impl.py:3935: bad docstring: invalid value set (missing opening brace): None}
+ /projects/numpy/numpy/lib/_function_base_impl.py:3934: bad docstring: invalid value set (missing opening brace): None}
- /projects/numpy/numpy/lib/_function_base_impl.py:4096: bad docstring: invalid value set (missing closing brace): {int

... (truncated 502 lines) ...

tristanlatr added 2 commits February 2, 2025 14:26

Fix #723 and #581

f06cd12

tristanlatr commented Feb 2, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

tristanlatr added 2 commits February 2, 2025 15:24

Fix the linenumber issue in the new references

788957d

Help mypy

aefe3c7

This comment has been minimized.

Sign in to view

trying this...

c6ffcef

This comment has been minimized.

Sign in to view

Revert "trying this..."

a8d9af8

This reverts commit c6ffcef.

This comment has been minimized.

Sign in to view

Turns out this refactors fixes an obscure bug that would trigger unex…

30bdf2b

…pected warnings

Try to fix mypy

ec37bff

This comment has been minimized.

Sign in to view

Fix tests of docs

36f6eba

tristanlatr added the maintenance label Feb 2, 2025

This comment has been minimized.

Sign in to view

tristanlatr mentioned this pull request Feb 4, 2025

Wrap signatures onto several lines when function len is over a treshold and function has the focus #831

Open

Showcase the literak choices of google/numpy in the demo

bd9d457

tristanlatr commented Feb 13, 2025

View reviewed changes

docs/source/conf.py Outdated Show resolved Hide resolved

Re-enable spelling extension

ed1e6d9

This comment has been minimized.

Sign in to view

tristanlatr commented Feb 13, 2025

View reviewed changes

pydoctor/epydoc/markup/_types.py Show resolved Hide resolved

tristanlatr added 3 commits February 14, 2025 10:37

Fix the numpy-style type in the demo

5f5542a

Actually fix a couple of issues:

042f9dc

- fix the linenumber of reported type docstring issues by 1. - the nested warnings for unknown token are now properly propagated and reported.

Merge branch '873-implement-parsedtypedocstring.to_node' of github.co…

c68853b

…m:twisted/pydoctor into 873-implement-parsedtypedocstring.to_node

This comment has been minimized.

Sign in to view

This changes simplify the parsed type docstring and makes the logic l…

6703a1f

…inenumber correct for epytext and restructuredtext.

tristanlatr mentioned this pull request Feb 14, 2025

Use composition instead of multiple supertypes for ParsedTypeDocstring #877

Open

This comment has been minimized.

Sign in to view

tristanlatr added 2 commits February 14, 2025 16:25

Properly add regression test for the duplicated type attribute bug

a554adc

fix pyflakes

742850a

This comment has been minimized.

Sign in to view

tristanlatr added 3 commits February 17, 2025 17:47

get_lineno refactor

2681285

Revert "get_lineno refactor"

dace95f

This reverts commit 2681285.

Simplification now that the nested warnings are not useful

40a8623

This comment has been minimized.

Sign in to view

add a comment to get_lineno

32cbcfa

This comment has been minimized.

Sign in to view

Merge branch 'master' into 873-implement-parsedtypedocstring.to_node

8359c43

tristanlatr commented Feb 19, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

Merge branch 'master' into 873-implement-parsedtypedocstring.to_node

cba36ab

This comment has been minimized.

Sign in to view

Remove unused import

7eb22b7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor the the ParsedTypeDocstring #874

Refactor the the ParsedTypeDocstring #874

tristanlatr commented Feb 2, 2025 •

edited

Loading

tristanlatr Feb 2, 2025

codecov bot commented Feb 2, 2025 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr commented Feb 2, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr Feb 13, 2025

tristanlatr Feb 13, 2025

tristanlatr Feb 13, 2025

tristanlatr Feb 13, 2025

tristanlatr Feb 13, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr left a comment

tristanlatr Feb 17, 2025

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 20, 2025


		TokenType.CONTROL: lambda _token, _, __: \
		nodes.emphasis(_token, _token),

Refactor the the ParsedTypeDocstring #874

Are you sure you want to change the base?

Refactor the the ParsedTypeDocstring #874

Conversation

tristanlatr commented Feb 2, 2025 • edited Loading

tristanlatr Feb 2, 2025

Choose a reason for hiding this comment

codecov bot commented Feb 2, 2025 • edited Loading

Codecov Report

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr commented Feb 2, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr Feb 13, 2025

Choose a reason for hiding this comment

tristanlatr Feb 13, 2025

Choose a reason for hiding this comment

tristanlatr Feb 13, 2025

Choose a reason for hiding this comment

tristanlatr Feb 13, 2025

Choose a reason for hiding this comment

tristanlatr Feb 13, 2025

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

tristanlatr left a comment

Choose a reason for hiding this comment

tristanlatr Feb 17, 2025

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 20, 2025

tristanlatr commented Feb 2, 2025 •

edited

Loading

codecov bot commented Feb 2, 2025 •

edited

Loading