Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate llms.txt on docs build #3600

Merged
merged 4 commits into from
Apr 4, 2025
Merged

Generate llms.txt on docs build #3600

merged 4 commits into from
Apr 4, 2025

Conversation

Blargian
Copy link
Member

@Blargian Blargian commented Apr 3, 2025

Summary

Adds custom Docusaurus plugin to generate an llms.txt file on build.

Checklist

@Blargian Blargian requested a review from a team as a code owner April 3, 2025 09:44
@Blargian Blargian linked an issue Apr 3, 2025 that may be closed by this pull request
Copy link

vercel bot commented Apr 4, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
clickhouse-docs-ru ⬜️ Ignored (Inspect) Visit Preview Apr 4, 2025 4:07pm
clickhouse-docs-zh ⬜️ Ignored (Inspect) Visit Preview Apr 4, 2025 4:07pm


// List to track files with issues
let filesWithIssues = [];

// Exceptions list configuration
const EXCEPTIONS_FILE_PATH = path.join(process.cwd(), 'plugins/frontmatter-validation/frontmatter-exceptions.txt');
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added ability to have exceptions list for the frontmatter checker as yarn docusaurus build tries to build all locales. The translations docs don't yet have the slugs, we did the translations before we started enforcing this.


// Write llms.txt file
try {
fs.writeFileSync(path.join(outDir, "llms.txt"), llmsTxt);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llms.txt gets put in the same directory as our robots.txt. (build directory of docusaurus)

};
}

export default pluginLlmsTxt;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

@Blargian Blargian merged commit fa4d44b into main Apr 4, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add llms.txt file
2 participants