Repository navigation
feat: Add filesystem caching #93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
493f390
91d8cab
0303b3f
8a4179c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,3 +23,4 @@ uv.lock | |
| .mypy_cache/ | ||
| .ruff_cache/ | ||
| __pycache__/ | ||
| .markdown-exec-cache/ | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -111,6 +111,41 @@ grep extra_css README.md && exit 2 | |
| ``` | ||
| ```` | ||
|
|
||
| ### Caching | ||
|
|
||
| Speed up your builds by caching execution results: | ||
|
|
||
| ````md | ||
| ```python exec="yes" cache="yes" | ||
| # Expensive computation | ||
| import time | ||
| time.sleep(5) | ||
| print("Done!") | ||
| ``` | ||
| ```` | ||
|
|
||
| Use custom cache IDs for persistence across builds: | ||
|
|
||
| ````md | ||
| ```python exec="yes" cache="my-plot" | ||
| # Generate plot - will be cached | ||
| import matplotlib.pyplot as plt | ||
| # ... | ||
| ``` | ||
| ```` | ||
|
|
||
| Force cache refresh with `refresh="yes"`: | ||
|
Owner
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not convinced by the utility of this refresh feature. Having to update the code block, build docs, then modify the code block back seems tedious. I'd prefer a solution where an environment variable is used, as for global refreshes. This would require IDs of course, so, to be discussed. Generally speaking, this feature doesn't seem to come from the requirements identified in #4. It doesn't address what was said there.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have provide If user try to remove
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we could check every file under cache dir if it is currently mentioned in build system and and remove all legacy cache files. Thus we can remove the
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @pawamoy which approach do you prefer? Use
Owner
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The stale cache entries must be automatically deleted for sure π |
||
|
|
||
| ````md | ||
| ```python exec="yes" cache="my-plot" refresh="yes" | ||
| # This will always re-execute | ||
| ``` | ||
| ```` | ||
|
|
||
| See [caching documentation](https://pawamoy.github.io/markdown-exec/usage/caching/) for more details. | ||
|
|
||
| --- | ||
|
|
||
| See [usage](https://pawamoy.github.io/markdown-exec/usage/) for more details, | ||
| and the [gallery](https://pawamoy.github.io/markdown-exec/gallery/) for more examples! | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| # Caching | ||
|
Owner
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The page generally looks good, except for some parts which don't make a lot of sense. It's also a bit too verbose. This would have to be reworked (I will provide more details). |
||
|
|
||
| Markdown Exec supports filesystem-based caching of code execution results to speed up documentation builds and development workflows. | ||
|
|
||
| ## Overview | ||
|
|
||
| When generating images, charts, or running expensive computations in your documentation, re-executing the same code on every build can significantly slow down the rendering process. The caching feature allows you to: | ||
|
|
||
| - **Speed up builds**: Reuse previously computed results instead of re-executing code | ||
| - **Persist across builds**: All cache is stored on the filesystem for cross-build persistence | ||
| - **Global cache refresh**: Force refresh of all cached results with a single environment variable | ||
|
|
||
| ## Cache Storage | ||
|
|
||
| All cached results are stored in `.markdown-exec-cache/` in your project root directory: | ||
|
|
||
| ```sh | ||
| your-project/ | ||
| βββ docs/ | ||
| βββ mkdocs.yml | ||
| βββ .markdown-exec-cache/ | ||
| βββ abc123def456.cache # Hash-based cache files | ||
| ``` | ||
|
|
||
| Add this directory to your `.gitignore`: | ||
|
|
||
| ```gitignore | ||
| .markdown-exec-cache/ | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Hash-Based Caching | ||
|
|
||
| Enable caching by adding `cache="yes"` to your code block. A hash is computed from the code content and execution options: | ||
|
|
||
| ````md exec="1" source="tabbed-left" tabs="Markdown|Rendered" | ||
| ```python exec="yes" cache="yes" | ||
| import time | ||
| print(f"Executed at: {time.time()}") | ||
| ``` | ||
| ```` | ||
|
|
||
| The cache is automatically invalidated when the code or execution options change. | ||
|
|
||
| ### Cache Invalidation | ||
|
|
||
| The cache is automatically invalidated when the code content or execution options change (a new hash is computed). **Stale cache files** β from code blocks that have been removed or changed β are cleaned up automatically: | ||
|
|
||
| - **MkDocs builds**: at the end of each build (`on_post_build`), any `.cache` file not used during that build is deleted. | ||
| - **Standalone usage**: stale files are cleaned up when the Python process exits. | ||
|
|
||
| To force re-execution of **all** cached blocks (e.g. when an external dependency changes), use the `MARKDOWN_EXEC_CACHE_REFRESH` environment variable instead of touching the code: | ||
|
|
||
| ```bash | ||
| # Force refresh all caches during build | ||
| MARKDOWN_EXEC_CACHE_REFRESH=1 mkdocs build | ||
|
|
||
| # Or with other truthy values | ||
| MARKDOWN_EXEC_CACHE_REFRESH=yes mkdocs build | ||
| MARKDOWN_EXEC_CACHE_REFRESH=true mkdocs build | ||
| MARKDOWN_EXEC_CACHE_REFRESH=on mkdocs build | ||
| ``` | ||
|
|
||
| This is useful for: | ||
|
|
||
| - CI/CD pipelines where you want fresh builds | ||
| - Ensuring all documentation is up-to-date | ||
| - Debugging cache-related issues | ||
|
|
||
| ## Clearing Cache | ||
|
|
||
| Remove the entire cache directory: | ||
|
|
||
| ```bash | ||
| rm -rf .markdown-exec-cache/ | ||
| ``` | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Hash Computation**: For `cache="yes"`, a SHA-256 hash is computed from: | ||
|
|
||
| - The code content | ||
| - Execution options (language, HTML mode, working directory, etc.) | ||
|
|
||
| 1. **Cache Lookup**: Before execution, the filesystem cache is checked for a matching entry | ||
|
|
||
| 1. **Execution & Storage**: If no cached result is found: | ||
|
|
||
| - Code is executed | ||
| - Output is stored in the filesystem cache | ||
|
|
||
| 1. **Cache Retrieval**: Cached output is used instead of re-executing the code | ||
|
|
||
| ## Best Practices | ||
|
|
||
| ### When to Use Caching | ||
|
|
||
| β **Good use cases:** | ||
|
|
||
| - Generating plots, diagrams, or images | ||
| - Running expensive computations | ||
| - Calling external APIs or services | ||
| - Processing large datasets | ||
|
|
||
| β **Avoid caching for:** | ||
|
|
||
| - Simple print statements | ||
| - Code demonstrating output variations | ||
| - Time-sensitive or non-deterministic code | ||
|
|
||
| ### Choosing Cache Type | ||
|
|
||
| Use `cache="yes"` for all caching needs. The cache is automatically invalidated when the code or execution options change β no manual cache management needed. | ||
|
|
||
| ### Cache Invalidation Strategy | ||
|
|
||
| Cache is automatically invalidated when code or options change β no manual intervention needed. To force re-execution of all cached blocks, use: | ||
|
|
||
| ```bash | ||
| MARKDOWN_EXEC_CACHE_REFRESH=1 mkdocs build | ||
| ``` | ||
|
|
||
| Or [clear the cache directory](#clearing-cache) before the build. | ||
|
|
||
| ## Examples | ||
|
|
||
| ### Caching a Matplotlib Plot | ||
|
|
||
| ````markdown | ||
| ```python exec="yes" html="yes" cache="yes" | ||
| import matplotlib.pyplot as plt | ||
| import io | ||
| import base64 | ||
|
|
||
| # Expensive plot generation | ||
| fig, ax = plt.subplots() | ||
| ax.plot([1, 2, 3], [1, 4, 9]) | ||
| ax.set_title("Population Growth") | ||
|
|
||
| # Save to base64 | ||
| buffer = io.BytesIO() | ||
| plt.savefig(buffer, format='png') | ||
| buffer.seek(0) | ||
| img_str = base64.b64encode(buffer.read()).decode() | ||
| print(f'<img src="data:image/png;base64,{img_str}"/>') | ||
| plt.close() | ||
| ``` | ||
| ```` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Cache Not Working | ||
|
|
||
| 1. Ensure the cache directory is writable | ||
| 1. Check that you're using `cache="yes"` or a custom ID | ||
| 1. Verify the cache directory exists: `ls -la .markdown-exec-cache/` | ||
|
|
||
| ### Stale Cache Results | ||
|
|
||
| 1. Use `refresh="yes"` to force re-execution | ||
| 1. Delete the specific cache file | ||
| 1. Clear the entire cache directory | ||
|
|
||
| ### Large Cache Directory | ||
|
|
||
| Cache files accumulate over time. Periodically clean up: | ||
|
|
||
| ```bash | ||
| # See cache directory size | ||
| du -sh .markdown-exec-cache/ | ||
|
|
||
| # Remove all cache files | ||
| rm -rf .markdown-exec-cache/ | ||
| ``` | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Force some reason the AI advertises custom IDs as providing persistence across builds throughout the change, while this is of course provided by the caching feature globally. There will be some rewording to do.