Summary
Text render mode is part of the graphics state and persists across text objects (ISO 32000-1, 8.4.1 and 9.3.6): a 3 Tr (invisible) set in one BT … ET still applies to the next text object, and one set outside any text object applies to all that follow. The extractor forgets the mode at the next BT and ignores one set outside a text object, so with the default include_invisible: false it serves text that no viewer shows.
Reproduction
Two one-page PDFs, Helvetica as /F1:
-
Mode set inside the first text object, second text object relying on it:
BT /F1 12 Tf 72 700 Td 3 Tr (alpha) Tj ET
BT /F1 12 Tf 72 680 Td (beta) Tj ET
Expected (invisible text skipped): nothing. Actual: beta.
-
Mode set outside any text object:
3 Tr
BT /F1 12 Tf 72 700 Td (alpha) Tj ET
BT /F1 12 Tf 72 680 Td (beta) Tj ET
Expected: nothing. Actual: alpha beta.
A viewer renders both pages blank (rendered with MuPDF: 0 dark pixels in the text area, against 597 for the same text under 0 Tr).
Where
The page walker in src/extractor/content_stream.rs tracks render_mode per text object rather than as graphics state (and the q/Q stack). The detector's executed-content scan (src/detector/content_scan.rs) already follows Tr through q/Q and across text objects; the extractor should read it the same way.
Impact
Searchable-scan producers that set 3 Tr once per page in the first text object (or before it) have their invisible layer served as the page's text. Seen on main at 6f24415 and 1.22.1.
Suggested fix
Keep text_rendering_mode as graphics state in the page walker: initialise it once per content stream (a form XObject inherits the invoker's value, as src/extractor/xobjects.rs already does), leave it untouched at BT/ET, apply Tr wherever it appears, and keep the existing q/Q save and restore. The comment at the BT handler notes that the separate paint state already retains Tr for weight inference, so the two can share one reading. The form-XObject walker does not reset the mode at BT, so today the page walker and the form walker disagree on the same operators.
Summary
Text render mode is part of the graphics state and persists across text objects (ISO 32000-1, 8.4.1 and 9.3.6): a
3 Tr(invisible) set in oneBT … ETstill applies to the next text object, and one set outside any text object applies to all that follow. The extractor forgets the mode at the nextBTand ignores one set outside a text object, so with the defaultinclude_invisible: falseit serves text that no viewer shows.Reproduction
Two one-page PDFs, Helvetica as
/F1:Mode set inside the first text object, second text object relying on it:
Expected (invisible text skipped): nothing. Actual:
beta.Mode set outside any text object:
Expected: nothing. Actual:
alpha beta.A viewer renders both pages blank (rendered with MuPDF: 0 dark pixels in the text area, against 597 for the same text under
0 Tr).Where
The page walker in
src/extractor/content_stream.rstracksrender_modeper text object rather than as graphics state (and theq/Qstack). The detector's executed-content scan (src/detector/content_scan.rs) already followsTrthroughq/Qand across text objects; the extractor should read it the same way.Impact
Searchable-scan producers that set
3 Tronce per page in the first text object (or before it) have their invisible layer served as the page's text. Seen on main at 6f24415 and 1.22.1.Suggested fix
Keep
text_rendering_modeas graphics state in the page walker: initialise it once per content stream (a form XObject inherits the invoker's value, assrc/extractor/xobjects.rsalready does), leave it untouched atBT/ET, applyTrwherever it appears, and keep the existingq/Qsave and restore. The comment at theBThandler notes that the separate paint state already retainsTrfor weight inference, so the two can share one reading. The form-XObject walker does not reset the mode atBT, so today the page walker and the form walker disagree on the same operators.