Enforce working examples in docstrings and documentation

CONTRIBUTING.md

@@ -9,8 +9,9 @@ We'd love anyone interested to contribute to the Logfire SDK and documentation.
 3. [Install pre-commit](https://pre-commit.com/#install)
 4. Run `make install` to install dependencies
 5. Run `make test` to run unit tests
-6. Run `make format` to format code
-7. Run `make lint` to lint code
-8. run `make docs` to build docs and `make docs-serve` to serve docs locally
+6. Run `make test-update-examples` to format and update examples in the docs
+7. Run `make format` to format code
+8. Run `make lint` to lint code
+9. run `make docs` to build docs and `make docs-serve` to serve docs locally
 
 You're now set up to start contributing!

Makefile

@@ -32,6 +32,10 @@ typecheck:
 test:
 	uv run --no-sync coverage run -m pytest -n auto --dist=loadgroup
 
+.PHONY: test-update-examples  # Update the examples in the documentation
+test-update-examples:
+	uv run pytest --update-examples
+
 .PHONY: generate-stubs  # Generate stubs for logfire-api
 generate-stubs:
 	uv run stubgen -p logfire --include-docstrings --no-analysis

README.md

@@ -49,10 +49,11 @@ logfire auth
 
 Here's a simple manual tracing (aka logging) example:
 
-```python
-import logfire
+```python skip-run="true" skip-reason="blocking"
 from datetime import date
 
+import logfire
+
 logfire.configure()
 logfire.info('Hello, {name}!', name='world')
 
@@ -68,21 +69,24 @@ with logfire.span('Asking the user their {question}', question='age'):
 
 Or you can also avoid manual instrumentation and instead integrate with [lots of popular packages](https://logfire.pydantic.dev/docs/integrations/), here's an example of integrating with FastAPI:
 
-```py
-import logfire
-from pydantic import BaseModel
+```py skip-run="true" skip-reason="global-instrumentation"
 from fastapi import FastAPI
+from pydantic import BaseModel
+
+import logfire
 
 app = FastAPI()
 
 logfire.configure()
 logfire.instrument_fastapi(app)
 # next, instrument your database connector, http library etc. and add the logging handler
 
+
 class User(BaseModel):
     name: str
     country_code: str
 
+
 @app.post('/')
 async def add_user(user: User):
     # we would store the user here

docs/concepts.md

@@ -47,6 +47,7 @@ In this example:
 
 ```py
 from pathlib import Path
+
 import logfire
 
 logfire.configure()
@@ -75,7 +76,7 @@ In this example:
 2. The user input is captured in the terminal
 3. `dob` (date of birth) is displayed in the span. Logfire calculates the age from the `dob` and displays age in the debug message
 
-```py
+```py skip-run="true" skip-reason="non-deterministic"
 from datetime import date
 
 import logfire
@@ -120,6 +121,7 @@ low‑overhead signal about the overall health and performance of your services.
 
 ```python
 import time
+
 import logfire
 
 logfire.configure()

docs/guides/onboarding-checklist/add-auto-tracing.md

@@ -14,7 +14,7 @@ For example, suppose all your code lives in the `app` package, e.g. `app.main`,
 Instead of starting your application with `python app/main.py`,
 you could create another file outside of the `app` package, e.g:
 
-```py title="main.py"
+```py title="main.py" skip="true" skip-reason="intentional-error"
 import logfire
 
 logfire.configure()
@@ -56,7 +56,7 @@ This function will be called with an [`AutoTraceModule`][logfire.AutoTraceModule
 `filename` attributes. For example, this should trace all modules that aren't part of the standard library or
 third-party packages in a typical Python installation:
 
-```py
+```py skip="true" skip-reason="incomplete"
 import pathlib
 
 import logfire
@@ -76,7 +76,7 @@ logfire.install_auto_tracing(should_trace, min_duration=0)
 Once you've selected which modules to trace, you probably don't want to trace *every* function in those modules.
 To exclude a function from auto-tracing, add the [`no_auto_trace`][logfire.no_auto_trace] decorator to it:
 
-```py
+```py skip="true" skip-reason="incomplete"
 import logfire
 
 @logfire.no_auto_trace

docs/guides/onboarding-checklist/add-manual-tracing.md

@@ -6,7 +6,7 @@ description: "Intro to manual tracing with Logfire. Learn about recording attrib
 
 Here's a simple example of using Logfire:
 
-```python
+```python skip-run="true" skip-reason="blocking"
 import time
 
 import logfire
@@ -57,7 +57,7 @@ A _trace_ is a tree of spans/logs sharing the same root. Whenever you create a n
 
 Spans and logs can have structured data attached to them, e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 logfire.info('Hello', name='world')
 ```
 
@@ -71,7 +71,7 @@ Both spans and logs can have attributes containing arbitrary values which will b
 
 Sometimes it's useful to attach an attribute to a span after it's been created but before it's finished. You can do this by calling the `span.set_attribute` method:
 
-```python
+```python skip="true" skip-reason="incomplete"
 with logfire.span('Calculating...') as span:
     result = 1 + 2
     span.set_attribute('result', result)
@@ -99,7 +99,7 @@ Here you can see that:
 
 You can also set `span.message` after a span is started but before it's finished, e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 with logfire.span('Calculating...') as span:
     result = 1 + 2
     span.message = f'Calculated: {result}'
@@ -109,14 +109,14 @@ You could use `message` to filter for related records, e.g. `message like 'Hello
 
 To allow efficiently filtering for related records, span names should be _low cardinality_, meaning they shouldn't vary too much. For example, this would be bad:
 
-```python
+```python skip="true" skip-reason="incomplete"
 name = get_username()
 logfire.info('Hello ' + name, name=name)
 ```
 
 because now the `span_name` column will have a different value for every username. But this would be fine:
 
-```python
+```python skip="true" skip-reason="incomplete"
 word = 'Goodbye' if leaving else 'Hello'
 logfire.info(word + ' {name}', name=name)
 ```
@@ -125,7 +125,7 @@ because now the `span_name` column will only have two values (`'Goodbye {name}'`
 
 You can use the `_span_name` argument when you want the span name to be different from the message template, e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 logfire.info('Hello {name}', name='world', _span_name='Hello')
 ```
 
@@ -135,13 +135,13 @@ This will set the `span_name` to `'Hello'` and the `message` to `'Hello world'`.
 
 Instead of this:
 
-```python
+```python skip="true" skip-reason="incomplete"
 logfire.info('Hello {name}', name=name)
 ```
 
 it's much more convenient to use an f-string to avoid repeating `name` three times:
 
-```python
+```python skip="true" skip-reason="incomplete"
 logfire.info(f'Hello {name}')
 ```
 
@@ -160,7 +160,7 @@ Contrary to the previous section, this _will_ work well in Python 3.11+ because
 
 The `logfire.span` context manager will automatically record any exceptions that cause it to exit, e.g:
 
-```python
+```python skip-run="true" skip-reason="exception-demo"
 import logfire
 
 logfire.configure()
@@ -175,7 +175,7 @@ If you click on the span in the Live view, the panel on the right will have an '
 
 Exceptions which are caught and not re-raised will not be recorded, e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 with logfire.span('This is a span'):
     try:
         raise ValueError('This is an acceptable error not worth recording')
@@ -185,7 +185,7 @@ with logfire.span('This is a span'):
 
 If you want to record a handled exception, use the [`span.record_exception`][logfire.LogfireSpan.record_exception] method:
 
-```python
+```python skip="true" skip-reason="incomplete"
 with logfire.span('This is a span') as span:
     try:
         raise ValueError('Catch this error, but record it')
@@ -195,7 +195,7 @@ with logfire.span('This is a span') as span:
 
 Alternatively, if you only want to log exceptions without creating a span for the normal case, you can use [`logfire.exception`][logfire.Logfire.exception]:
 
-```python
+```python skip="true" skip-reason="incomplete"
 try:
     raise ValueError('This is an error')
 except ValueError:
@@ -208,15 +208,15 @@ except ValueError:
 
 Often you want to wrap a whole function in a span. Instead of doing this:
 
-```python
+```python skip="true" skip-reason="incomplete"
 def my_function(x, y):
     with logfire.span('my_function', x=x, y=y):
         ...
 ```
 
 you can use the [`@logfire.instrument`][logfire.Logfire.instrument] decorator:
 
-```python
+```python skip="true" skip-reason="incomplete"
 @logfire.instrument()
 def my_function(x, y):
     ...
@@ -229,7 +229,7 @@ The default span name will be something like `Calling module_name.my_function`.
 You can pass an alternative span name as the first argument to `instrument`, and it can even be a template
 into which arguments will be formatted, e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 @logfire.instrument('Applying my_function to {x=} and {y=}')
 def my_function(x, y):
     ...
@@ -259,7 +259,7 @@ To log a message with a variable level you can use `logfire.log`, e.g. `logfire.
 
 Spans are level `info` by default. You can change this with the `_level` argument, e.g. `with logfire.span('This is a debug span', _level='debug'):`. You can also change the level after the span has started but before it's finished with [`span.set_level`][logfire.LogfireSpan.set_level], e.g:
 
-```python
+```python skip="true" skip-reason="incomplete"
 with logfire.span('Doing a thing') as span:
     success = do_thing()
     if not success: