Skip to content

Commit 0098761

Browse files
avivkelleravivkeller
committed
feat(react): support chunking pages
1 parent 18b9423 commit 0098761

45 files changed

Lines changed: 2200 additions & 221 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/chunked-generator.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@doc-kit/core': minor
3+
'@doc-kit/generator-react': minor
4+
---
5+
6+
feat: `dependent` generators and the `chunked` generator

‎.changeset/html-parallel-minify.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
'@doc-kit/generator-react': minor
3+
---
4+
5+
perf(html): one client entry chunk, HTML minification in the worker pool
6+
7+
- Every page now loads the same client entry (`getEntryId()` takes no
8+
argument), so the bundler emits one entry chunk for the whole site rather
9+
than a copy per page.
10+
- Rendered pages are minified across the worker pool instead of one after the
11+
other on the main thread; it was the single largest cost of the `html`
12+
generator, and grows with the page count.
13+
- Page titles are escaped in the document `<head>`, so a heading containing
14+
`"` no longer produces unparsable HTML.
15+
16+
Custom bundler adapters receive `build({ entry, virtualImports, pages,
17+
minifyPages, config })` in place of `entries`, and should call `minifyPages`
18+
on their final pages when `config.minify` is set.

‎docs/creating-generators.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,63 @@ export async function generate(input, worker) {
411411
}
412412
```
413413

414+
### Declaring a dependent
415+
416+
`dependsOn` pulls another generator's output _in_. The inverse, `dependent`,
417+
pushes a generator's output _into_ another generator's pipeline:
418+
419+
```mjs displayName="index.mjs"
420+
export default {
421+
name: 'chunked',
422+
423+
dependsOn: '@doc-kit/core/metadata',
424+
425+
// Deliver this generator's output through `html`
426+
dependent: '@doc-kit/generator-react/html',
427+
428+
generate,
429+
};
430+
```
431+
432+
The pipeline splices such a generator in front of the first generator on the
433+
way to its dependent that consumes the same `dependsOn`. Here `html` depends on
434+
`jsx-ast`, which depends on `metadata` — so `jsx-ast` is rewired to read from
435+
`chunked` instead:
436+
437+
```text
438+
ast → metadata → chunked → jsx-ast → html
439+
```
440+
441+
Requesting a generator that declares a dependent runs the dependent's whole
442+
pipeline, and the run's result is the dependent's output — `-t chunked`
443+
produces the `html` site. Several generators may splice in at the same point;
444+
they form a chain. A generator whose dependent pipeline never consumes its
445+
`dependsOn` is an error.
446+
447+
`dependent` also accepts an array. The generator is spliced into every listed
448+
pipeline; when requested, it is delivered through the dependents that are
449+
already part of the run, or through all of them when none is:
450+
451+
```mjs displayName="index.mjs"
452+
export default {
453+
name: 'chunked',
454+
dependsOn: '@doc-kit/core/metadata',
455+
dependent: [
456+
'@doc-kit/generator-react/html',
457+
'@doc-kit/generator-react/sitemap',
458+
],
459+
generate,
460+
};
461+
```
462+
463+
With this, `-t chunked` builds the site and the sitemap, while
464+
`-t html -t chunked` builds just the site.
465+
466+
Use a dependent when a generator transforms an intermediate representation
467+
(adding, filtering, or rewriting entries) rather than producing a new output
468+
format of its own. See the [`chunked`](./generators/chunked.md) generator for
469+
a worked example.
470+
414471
## File Output
415472

416473
### Writing Output Files

‎docs/generators.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ npx @doc-kit/cli generate -t html -t orama-db -t sitemap -i "docs/**/*.md" -o ou
1616
| [`orama-db`](./generators/orama-db.md) | The search index behind the `html` site's search box. |
1717
| [`llms-txt`](./generators/llms-txt.md) | An [`llms.txt`](https://llmstxt.org/) index for language models. |
1818
| [`sitemap`](./generators/sitemap.md) | A `sitemap.xml` for search engines. |
19+
| [`chunked`](./generators/chunked.md) | The `html` site and sitemap, plus one page per section of a module. |
1920

2021
### JSON ([`@doc-kit/core`](./packages/core.md))
2122

‎packages/core/src/__tests__/generators.test.mjs‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,33 @@ const syntheticGenerators = {
7272
return { all: input };
7373
},
7474
},
75+
// A generator delivered through another pipeline: it reads `metadata` and
76+
// declares `gen-d-all` as its dependent, so `gen-d` reads from it instead.
77+
'gen-splice': {
78+
name: 'gen-splice',
79+
dependsOn: 'metadata',
80+
dependent: 'gen-d-all',
81+
generate: async input => {
82+
record('gen-splice');
83+
return [...input, { spliced: true }];
84+
},
85+
},
86+
'gen-d': {
87+
name: 'gen-d',
88+
dependsOn: 'metadata',
89+
generate: async input => {
90+
record('gen-d');
91+
return { d: input };
92+
},
93+
},
94+
'gen-d-all': {
95+
name: 'gen-d-all',
96+
dependsOn: 'gen-d',
97+
generate: async input => {
98+
record('gen-d-all');
99+
return { all: input };
100+
},
101+
},
75102
};
76103

77104
mock.module('../generators/loader.mjs', {
@@ -92,8 +119,10 @@ mock.module('../generators/loader.mjs', {
92119
const generator = syntheticGenerators[specifier];
93120
generators.set(specifier, generator);
94121

95-
if (generator.dependsOn) {
96-
queue.push(generator.dependsOn);
122+
for (const related of [generator.dependsOn, generator.dependent]) {
123+
if (related) {
124+
queue.push(related);
125+
}
97126
}
98127
}
99128

@@ -162,4 +191,24 @@ describe('createGenerator orchestration', () => {
162191

163192
assert.deepStrictEqual(results, [[{ c: true }], { all: [{ c: true }] }]);
164193
});
194+
195+
it('delivers a generator through its dependent', async () => {
196+
const { runGenerators } = createGenerator();
197+
198+
const results = await runGenerators({
199+
target: ['gen-splice'],
200+
threads: 1,
201+
});
202+
203+
// Requesting only `gen-splice` runs its dependent's whole pipeline, with
204+
// `gen-d` reading the spliced output instead of `metadata` directly.
205+
assert.equal(runs.metadata, 1);
206+
assert.equal(runs['gen-splice'], 1);
207+
assert.equal(runs['gen-d'], 1);
208+
assert.equal(runs['gen-d-all'], 1);
209+
210+
assert.deepStrictEqual(results, [
211+
{ all: { d: [{ meta: 1 }, { spliced: true }] } },
212+
]);
213+
});
165214
});

‎packages/core/src/generators.mjs‎

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import {
55
loadGenerators,
66
resolveGeneratorSpecifier,
77
} from './generators/loader.mjs';
8+
import { resolvePipeline } from './generators/pipeline.mjs';
89
import logger from './logger/index.mjs';
910
import createWorkerPool from './threading/index.mjs';
1011
import createParallelWorker from './threading/parallel.mjs';
@@ -44,22 +45,24 @@ const createGenerator = () => {
4445
*
4546
* @param {string} specifier - Resolved generator specifier to schedule
4647
* @param {Map<string, GeneratorMetadata>} generators - Loaded generators
48+
* @param {Map<string, string | undefined>} inputOf - Each generator's
49+
* effective input generator (its dependency, unless another generator was
50+
* spliced in front of it via `dependent`)
4751
* @param {import('./utils/configuration/types').Configuration} configuration - Runtime options
4852
*/
49-
const scheduleGenerator = (specifier, generators, configuration) => {
53+
const scheduleGenerator = (specifier, generators, inputOf, configuration) => {
5054
if (cache.has(specifier)) {
5155
return;
5256
}
5357

5458
const generator = generators.get(specifier);
5559
const { name, generate, hasParallelProcessor } = generator;
5660

57-
const dependsOn =
58-
generator.dependsOn && resolveGeneratorSpecifier(generator.dependsOn);
61+
const dependsOn = inputOf.get(specifier);
5962

6063
// Schedule dependency first
6164
if (dependsOn && !cache.has(dependsOn)) {
62-
scheduleGenerator(dependsOn, generators, configuration);
65+
scheduleGenerator(dependsOn, generators, inputOf, configuration);
6366
}
6467

6568
generatorsLogger.debug(`Scheduling "${name}"`, {
@@ -104,8 +107,16 @@ const createGenerator = () => {
104107

105108
// Resolve shorthand names and load the full dependency closure up front,
106109
// so scheduling below is fully synchronous.
107-
const targets = target.map(resolveGeneratorSpecifier);
108-
const generators = await loadGenerators(targets);
110+
const generators = await loadGenerators(
111+
target.map(resolveGeneratorSpecifier)
112+
);
113+
114+
// Work out who reads from whom once generators declaring a `dependent`
115+
// have been spliced in, and which generators the run finally collects.
116+
const { targets, inputOf } = resolvePipeline(
117+
target.map(resolveGeneratorSpecifier),
118+
generators
119+
);
109120

110121
generatorsLogger.debug(`Starting pipeline`, {
111122
generators: targets.join(', '),
@@ -114,18 +125,14 @@ const createGenerator = () => {
114125

115126
// Compute consumer counts up front so dependencies can be evicted as soon
116127
// as their last consumer runs (must be ready before any generator starts).
117-
cache.populateConsumerCounts(targets, specifier => {
118-
const { dependsOn } = generators.get(specifier);
119-
120-
return dependsOn && resolveGeneratorSpecifier(dependsOn);
121-
});
128+
cache.populateConsumerCounts(targets, specifier => inputOf.get(specifier));
122129

123130
// Create worker pool
124131
pool = createWorkerPool(threads);
125132

126133
// Schedule all generators
127134
for (const specifier of targets) {
128-
scheduleGenerator(specifier, generators, configuration);
135+
scheduleGenerator(specifier, generators, inputOf, configuration);
129136
}
130137

131138
// Start all collections in parallel (don't await sequentially). Consuming

‎packages/core/src/generators/__tests__/index.test.mjs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import assert from 'node:assert/strict';
22
import { describe, it } from 'node:test';
33

4+
import { enforceArray } from '#utils/array.mjs';
5+
46
import {
57
allGenerators,
68
deprecatedGenerators,
@@ -62,6 +64,17 @@ describe('All Generators', () => {
6264
});
6365
});
6466

67+
it('should have valid dependent references', () => {
68+
loadedGenerators.forEach(([name, , generator]) => {
69+
for (const dependent of enforceArray(generator.dependent ?? [])) {
70+
assert.ok(
71+
validDependencies.includes(dependent),
72+
`Generator "${name}" declares dependent "${dependent}" which is not a valid generator specifier`
73+
);
74+
}
75+
});
76+
});
77+
6578
it('should resolve deprecated aliases to loadable generators', async () => {
6679
for (const [name, specifier] of Object.entries(deprecatedGenerators)) {
6780
assert.equal(resolveGeneratorSpecifier(name), specifier);

0 commit comments

Comments
 (0)