Feat/pandoc ref - #707
Merged
Merged
Conversation
Standalone HTML prototype verified against Obsidian's app.css in both light and dark themes. Decided: flat bibliography in citeproc order with navigate-first interaction (click cycles occurrences, hover chevron opens the Literature Note), quiet baseline-aligned ref numbers, and CSL-formatted read-only editor widgets. Resolves #604.
Specify the stable interface between `zotlit:resolve` (Obsidian CLI handler) and `zotlit-cite.lua` (Pandoc Lua filter). The handler owns link resolution, `zotero-key` frontmatter reading, and live DB citation key lookup; the filter owns fragment parsing, Citation Run detection, and Cite AST construction. Both sides percent-decode linkpaths to support wikilinks and markdown-format links. All-or-nothing error handling with six handler error codes. Verified Pandoc 3.6.4 AST behavior for target encoding, fragment preservation, and run patterns. Resolves #611.
Publish the Lua filter and defaults through versioned Resource Releases. Define agent installation, update, compatibility, and guide contracts. Resolves #610.
Extend Resource Releases to include agent-downloaded Pandoc assets. Define the Pandoc CLI Guide as the matching-version locator. Refs #610.
Add itemToCsl and its supporting modules (zt-csl-name for structured creator-name normalization, zt-extra-to-csl for Extra-field CSL overrides) plus a generated field/type map in @zotlit/zotero-types/csl, extended with text-field candidate lists, date fields, and the item-type -> CSL-type inversion needed to drive the mapper. itemUri (packages/db/src/lib/zt-uri.ts) builds the persistent http://zotero.org/ Item URI CSL cites items by, which requires the signed-in account's userID/localUserKey alongside its username. getCurrentUsername is replaced by getZoteroIdentity, returning all three so callers can resolve both the URI and the existing weblink/annotation username uses from one query; call sites across apps/obsidian and packages/db are updated accordingly.
Add a committed fixture that pairs Zotero 9.0.3 native item JSON with the CSL-JSON Zotero produces from it, one case per regular item type. The generator downloads both upstream corpora from pinned commit URLs, verifies a pinned SHA-256 for each, checks them against the pinned schema, and writes the merged artifact with its provenance. Only the manual upgrade touches the network; builds and tests read the committed file. The db corpus test converts each native case to Zotero's stored row shape, seeds one in-memory database through Drizzle, hydrates with getItemsByLibrary(), converts with itemToCsl(), and compares every item type with a labeled soft assertion. The comparison is exact after three explicit adaptations: `id`, `accessed`, and string date parts. Focused mapper tests now cover ZotLit-specific behavior only. Cases that the corpus exercises are removed; Extra cheater-syntax wiring and creator particle handling are added. Refs #613
Add CitationScanner, a Service holding the active document's Literature Note Citations in a zustand vanilla store. scanCitations is the pure core: it takes a link cache and a linkpath-to-Indexed-Key resolver and returns citations grouped by Indexed Key, so two links reaching one Literature Note share a reference number. Numbers follow first occurrence; occurrence positions come from LinkCache.position. Embeds never reach it, since Obsidian caches those apart from `links`. The scanner resolves each linkpath through getFirstLinkpathDest and reads zotero-key with note-index's itemKeyFromFrontmatter, so ordinary notes and dangling links drop out. Rescans run on active-leaf-change and on any metadata change: a linked note gaining or losing zotero-key also moves it in or out of the list. The first scan waits for metadataCache "resolved" when the cache is not yet initialized, matching NoteIndex. citationsEqual suppresses the store write when a rescan produces the same list, so an unrelated vault edit leaves subscribers untouched. parseLinktext is added to the Vitest obsidian mock. Refs #614
Vendor the official pandoc-wasm bridge over @bjorn3/browser_wasi_shim, narrowed to the conversion path: virtual files arrive as bytes, so a conversion runs synchronously against the in-memory filesystem. Expose it as a CitationEngine that hides Pandoc: CSL-JSON plus a CSL style render to HTML entries addressable by their ref id, and Markdown plus Lua filters, a resolve map, and a bibliography convert to docx or html bytes. Requests queue behind one another so concurrent callers never share the virtual filesystem, and disposal drains the queue and refuses later requests. Closes #622
A consumer that re-renders on every change queues one request per change behind a running conversion, and only the newest one still matters. A request now names a `supersedes` slot; a later request claiming that slot drops the one still waiting, which rejects with CitationRequestSupersededError. A running conversion keeps its place, and requests without a slot always run. Fold the paired style option and style file into one `styleInput` helper, and name Pandoc's `output-file` default in a PandocOptions type so the bridge reads it as a string instead of testing its type.
The plugin never ships the 58 MB Pandoc binary, so a build has to name the
download it trusts. Resolve that pin from the installed `pandoc-wasm`
package — the version it declares and the SHA-256 of the binary the engine
tests already run against — then prove the pin honest against upstream: pick
the `.wasm.zip` asset out of the matching `jgm/pandoc` release by suffix
(names carry the version and have changed shape between releases), require
it to hash to the digest GitHub reports for it, and require the binary
inside it to be the very binary the package holds. Any disagreement fails
the build.
Inline the result as `PINNED_PANDOC_ENGINE` so the resource manager reads
`{version, url, sha256}` as typed constants. A verified pin is remembered
under `node_modules/.cache`, so only a version bump pays the cross-check
download again.
Closes #623
Rendering references needs a style, and Zotero already manages a data directory full of them. Read that directory as the style source: index the `.csl` files by their `<info><id>` rather than their filename, so a renamed file keeps a selection valid, and resolve a dependent style through the `independent-parent` link it declares — including into `styles/hidden/`, where Zotero keeps parents that are not choices of their own. Store the choice as a synced CSL style ID on the Citations settings page, listing the visible styles by title. Leaving it unset, selecting a style Zotero has since removed, or pointing at a dependent whose parent is gone all resolve to no style XML, which the citation engine renders with its embedded default. ZotLit installs and updates nothing. Closes #624
The style list filled the dropdown only while the row was already attached, so a data-directory read that finished first left the picker showing nothing but "Default". Bind the read to the row's disposal instead, which a live row always outlives. The row also followed the Zotero data directory alone, so a setting written by vault sync while the tab is open kept the stale selection on screen. Subscribe to the settings service as the library row does. Move the option list into a pure function so the default entry, the installed styles, and the entry that keeps an uninstalled selection visible are testable without a DOM, and name the setting description after the CSL style it picks, the term the References Style vocabulary reserves.
…tekey Update the research notes for issues #605, #606, #609, and the packaging note to reflect spec #612: a third bibliography consumer (built-in export) that resolves data live through Better BibTeX or Zotero's local HTTP API instead of an external file, a WASM Pandoc citeproc replacing citeproc-js in the References Sidebar, bundling the Lua filter and defaults file inside the plugin instead of a separate Resource Release, a CLI/sandbox filter variant split, and the correction that Zotero's `citationKey` field is native since 7.0.31 rather than a Better-BibTeX-only field. Also add a new reference map cross-checking `itemToCsl()` against Zotero's own `itemToCSLJSON` implementation.
The citationKey field in fieldsCombined is Zotero's native field since 7.0.31; Better BibTeX only writes into this same field when present, it does not own a separate field. Update the comments that attributed the field to Better BibTeX so they match this behavior.
One zotlit-cite.lua source carries a cli region that calls `zotlit:resolve` against a live Obsidian process and a sandbox region that reads a pre-written resolve map. A Vite plugin splits the source at build time, so each shipped variant holds only its own resolution path. The filter converts wikilinks and Markdown links to Literature Notes into Cite nodes, parses Citation Fragments strictly, groups same-line Citation Runs, and reports every collected error before it stops the run. zotlit.yaml locates the filter by co-location and runs it before citeproc. A script test drives a native Pandoc over fixture Markdown for both variants — the sandbox one with a canned resolve map, the cli one with a stub `obsidian` CLI. Verified on Pandoc 3.1.1 (the supported floor) and 3.10.1. Closes #625
…and Plain The Lua filter walked Para and Plain blocks, so a Link inside a Header, a table cell, or an inline container stayed a Link and a `#cite:` fragment there reported no error at all. It now walks Inlines, which covers every inline list in the document. An empty `errors` array in the resolve map decoded to a truthy table, so a clean map aborted the run with "stopped on 0 error(s)". The guard now counts the entries. `fail-if-warnings: true` leaves zotlit.yaml: it made every Pandoc warning fatal, including ones with no citation involved. The documented command carries `--fail-if-warnings` as a flag instead. The script test drives the CLI variant through an error payload and a failed resolve call as well as the happy path, checks a heading, a table cell, and an emphasis run, and checks that an empty `errors` array converts as usual. Its fixtures move to a workspace tmp/ path and stay there when a check fails. Filenames come from one filter/names.ts, and the Vite plugin parses a variant id once. Refs #625
…e CLI handler
Registers `zotlit:resolve file=<absolute-path>` at plugin load. It scans the
file's link cache — links only, never embeds or frontmatter links — resolves
each target through Obsidian's source-aware link resolution, reads its
`zotero-key`, and looks the current citation key up under one database read
lease per invocation.
Success returns `{ "citations": { linkpath: citationKey } }` keyed by bare
percent-decoded linkpaths. Any failure returns `{ "errors": [...] }` instead,
carrying every discoverable problem under the six contract codes, so one
error stops all citations.
Closes #626
Better BibTeX answers first, through a liveness probe plus item.citationkey and item.export with the Better CSL JSON translator. Zotero's local HTTP API is the baseline, read once per library with the Zotero-Allowed-Request header. A local API that prefs report as disabled, and a closed Zotero, each name their own failure so the export UI can guide the user out of it. Results carry the CSL id their source gave them, re-indexed by Indexed Key. A missing item or citation key fails the whole request. Closes #627
Better BibTeX keeps a citation key unique within a library only, so a key two libraries share resolved a ref to the last-exported item. Address the export index by library ID and citation key. Also drop the unused Zotero HTTP port override, narrow the CSL reader to the JSON array text both sources send, and type the test's library-route capture with arkregex.
The plugin never ships the engine, so a vault has to fetch it — and only when the user says so. Put a resource manager behind the build-time pin: `install()` is the one door onto the network, the downloaded asset is verified against the pinned SHA-256 before those bytes become the cache, and they are stored uncompressed and content-addressed in OPFS, which Obsidian serves from one origin and therefore shares with every vault on the device. Writes go temp name → verify → rename, so two vaults downloading at once land on identical verified bytes and no reader ever opens a half-written binary. Once the new binary verifies, the ones earlier releases pinned are dropped; a `.part` entry is left alone because it belongs to a download still running elsewhere. Report one status union — absent, declined, installing, installed, and the download, checksum, and startup failures — so the shared fallback surface renders one arm instead of deriving its own combination of flags. The engine itself is instantiated on first use, which keeps startup to a cache listing and gives an init failure a place to land. Settings carry the install and uninstall controls; uninstall reaches the whole device. Closes #628
An install already running keeps its own schedule: whenever the download arrives it writes the binary and reports `installed`. Uninstall used to clear the cache underneath it, so the user asked for a removal and got a re-downloaded binary and an `installed` row. Wait the install out, then drop the cache, so the removal is the state that stands. Cover the cross-vault rename too: another vault can verify the same bytes and land them first, so the install renames onto a name that already exists and must still leave one verified entry. The fixture pin now comes from Node's own digest rather than a copy of the service's, and the setting tab takes the narrow engine port its siblings take.
Lists the active document's Literature Note Citations as a bibliography the Pandoc engine formats in the References style, keeping the scanner's document order and Reference Numbers. Item data comes from the database mapper, so the list renders while Zotero is closed. Without an engine the same list renders as a minimal reference list from the shared item-summary rendering, under a dismissible install hint; a download, checksum, or startup failure names itself in one callout above that list. Each entry opens its Literature Note, the Item in Zotero, or the Item's PDF, and a click walks the citation's occurrences in the editor. Closes #629
One command on the active Markdown file opens a modal for the output format (docx or html), the CSL style, and a save-dialog destination, then renders the note through the Pandoc engine and writes the bytes where the user chose. Citations resolve in-process from the document's own links; the resulting linkpath-to-CSL-id map reaches the bundled sandbox filter as a virtual file, which is how the WASM engine cites without a system command. Bibliography data comes from the Better BibTeX / local API source chain, so each wikilink cites whatever id its item carries and needs no citation key of its own. Every resolution and bibliography failure stops the export before Pandoc runs and names the notes, the setting, or the application behind it, so no output ever ships a silently incomplete bibliography. Closes #631
The zh-CN copy for a missing citation key now says 文献引用标识（citation key）, the term the vocabulary policy fixes for Chinese public copy. A refusal from the destination write reached the user as "Pandoc stopped the conversion", because one try wrapped the conversion and the write together. The write now reports itself, and both catches share the one describeError helper the export module already had. The docx half of the demo-note export now reads the document body back and asserts the citations and the bibliography, so both formats verify the same acceptance criterion.
The project contains code ported from Zotero (zotero/zotero, zotero/utilities, zotero/reader, zotero/note-editor, zotero/dataserver), all licensed under AGPL-3.0. MIT is incompatible with this derivation. - Replace LICENSE (MIT) with COPYING (AGPL-3.0) and symlink LICENSE → COPYING - Add "license": "AGPL-3.0-or-later" to root package.json - Add NOTICE with project copyright header and Zotero attribution
…tation keys The Citation Index owns a citekey resolution snapshot: in-memory maps from native citation key to Item and back, scoped to the configured citation library, rebuilt by one bulk database read on database change, on citation-library change, and once at startup. Every render surface reads it synchronously; a completed rebuild dispatches `resolution-changed`, so styling, the sidebar, and the render cache refresh through it. "Resolved" now means the citekey names a live Zotero Item. An Item without a Literature Note renders normally and creates the note on navigation; a key naming no live Item is the error state. The Citation Key Property retires: the setting and its settings-tab control go (stored settings migrate to v5), the missing-property notice goes with it, the Note Index citekey map goes, and wikilink display text comes from the snapshot's reverse map. The managed `citekey` frontmatter field stays as write-only template output. Closes #676
… service `whenResolved()` could stay pending forever. `#rebuildSnapshot()` parks on `await this.#db.ready` and returns early once `#stopped` flips, so a disposal that interrupts the first rebuild emitted no `resolution-changed` and left `#resolved` false. Every waiter hung. `whenIndexed()` has the identical shape over `#backfilled` and hangs the same way when disposal precedes the backfill, so both are fixed here: one `#waitFor` helper parks the caller and registers it in `#waiters`, and the disposal hook that flips `#stopped` flushes that set. A call made after disposal returns a resolved promise straight away.
Describe literal citation keys and Literature Note wikilinks across Live Preview, Source mode, and reading view. Add the related settings, command reference, and migration behavior.
Carry completed bibliography state and Entry Marker capability separately from visible entries so error-only lists follow the selected style. Keep omitted Items actionable, share marker columns across rows, and wrap long citation keys without hiding occurrence actions.
Follow Pandoc code-span delimiter, multiline, and escape semantics while preserving line-scoped citation clusters.\n\nFixes #677
Clarify that Resource Releases exclude Pandoc integration files. Define the version-matched integration pair and native workflow, and narrow the CLI guide to ZotLit-specific mechanics.
Complete the installed-plugin path for the Native Pandoc Workflow (closes #632): - Register `zotlit:pandoc-files` and `zotlit:pandoc-guide` CLI commands. The former returns the version-matched integration pair (`zotlit-cite.lua`, `zotlit.yaml`) as JSON; the latter prints the focused Pandoc CLI guide. - Add a "Save integration files" action to the Citations settings page. It picks a folder, confirms before replacing an existing pair, and restores the prior pair on a failed write. - Add the `zotlit-pandoc` model-invoked skill, covering setup, refresh, native export, and resolver diagnosis. - Extend the docs `.well-known/agent-skills` discovery index and archive routes to serve multiple skills (`zotlit-template` and `zotlit-pandoc`) instead of one hardcoded skill. - Add the new settings, notice, and CLI help strings to the Language Pack facade.
Document stable semantic classes across Obsidian editor surfaces and the DOM compatibility boundary. Add the package policy that keeps future hooks centralized, testable, and theme-overridable. Refs #678
Apply the documented citation and literature-note hooks across editor and Reading views, including unresolved citation states and passive literature-note links.
Use Obsidian link variables for literal citation keys in the editor and Reading view, so they match equivalent wikilink citations.
- Clarify the distinction between Citation and References Style in documentation. - Enhance the References Sidebar and In-text Citation Rendering to use a unified Document Citation Set with independent controls. - Introduce new settings for managing citation behavior, including the handling of Pandoc and Wikilink citations. - Supersede previous ADR-0020 with ADR-0022, streamlining citation key links and their management. - Document the implications of these changes on citation rendering and user experience.
Make the Document Citation Set the common membership and ordering source for in-text citation text and the References Sidebar. Preserve repeated occurrences for document-wide formatting while grouping cited works for the sidebar. Refs #683
Use one synced setting for formatted citekey and wikilink citations in Live Preview and Reading view. Keep citation source visible while formatting is pending or unavailable, and only publish complete document renders. Preserve resolved partial clusters in CSL numeric context. Migrate the legacy wikilink display setting and cover the unified behavior with focused and live-view tests.
Add the default-off Open Pandoc citations as links setting and migrate legacy Citation Key Links and development treatment values. Gate editor, reading, hover, and command behavior independently from formatted presentation, including immediate widget listener refreshes.
Keep unavailable CSL styles selected and surface one recovery notice. Return the references sidebar to its current minimal list when formatting is unavailable or fails, with a visible error for render failures.
Keep native, excluded, pending, and inactive Literature Note links outside the public citation hook contract. Align the default-on assertions with the landed Pandoc navigation setting.
Explain the Document Citation Set, citation settings, commands, theme hooks, and built-in export workflow. Document Open Pandoc citations as links as On by default.
Add introduced/updated semver frontmatter to every docs page and derive NEW/UPDATED badges from the current release line, shown in the sidebar, the page header, and the llms.txt Markdown edition. Each badge links to the matching changelog entry when one exists. - lib/docs-availability.ts resolves the release line and badge state - source.config.ts requires introduced/updated on every doc page - github-releases.ts filters draft releases and adds newestStableRelease() to determine the current Docs Release Line - gitConfig.branch now reads VERCEL_GIT_COMMIT_REF so preview deployments link to their own branch instead of main - CONTEXT.md/DESIGN.md/AGENTS.md document the new vocabulary and authoring rules for introduced/updated fields
# Conflicts: # apps/docs/content/docs/concepts/how-citekey-links-work.mdx # apps/docs/content/docs/concepts/index.mdx # apps/docs/content/docs/how-to/create-and-open-notes.mdx # apps/docs/content/docs/how-to/index.mdx # apps/docs/content/docs/reference/index.mdx
Target the 2.1.0-beta.0 release line: new pages (export-note-with- citations, theme-hooks) get introduced=updated=2.1.0-beta.0; existing pages with material content changes for the pandoc citation feature keep their original introduced and advance updated to 2.1.0-beta.0.
Docs pages no longer hand-author `introduced`/`updated` — both are optional now and default unset (no badge, no "Available since" line, a normal state for a page that hasn't shipped yet). release.ts gets a new phase, gated on obsidian being a release target: it diffs content/docs/**/*.mdx against the previous Stable Release Line tag (with rename detection to preserve `introduced` across a moved page), auto-accepts brand-new pages, and interactively reviews every changed/moved page with an inline diff before writing `updated`. That review step is what keeps "Updated Release" meaning a material change, not just any diff. DOCS_RELEASE_VERSION now reads a dedicated committed artifact, apps/docs/zotlit-release.json, written by the same phase, instead of reaching into apps/obsidian/package.json directly. Adds `pnpm --filter @zotlit/scripts preview-docs-availability` for a read-only dry run of the same scan. Reverts this session's forward-guessed 2.1.0-beta.0 frontmatter on the pandoc-ref pages — release.ts will assign it for real when that version actually ships. See ADR 0002 for the full rationale and rejected alternatives.
The scan treated the generated template-data reference page as an ordinary page and let a release pass hand-assign it introduced/ updated frontmatter. The generator never writes those fields, so the committed page went stale against generate:template-data and failed the Template data page check in CI. Add a GENERATED_DOCS_PAGES exclusion set so generated pages never surface as scan candidates, and drop the stray frontmatter from data.mdx to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.