update to pydantic 2 (#625)

* update to pydantic 2

* update changelog

* typo

* add CI for Python 3.12

* drop support for python 3.8

* update python version for docs

* update python for docs docker container

* update python version in dockerfile

* handle post requests

* test wrapper

* pass through StacBaseModel

* keep py38

* change install order

* lint

* revert back to >=3.8 in setup.py

* add switch to use either TypeDict or StacPydantic Response

* lint and format with ruff

* remove comment

* update change log

* use Optional not | None

* use Optional not | None

* update dependencies

* hard code versions and address other comments

* remove response_model module, update openapi schema

* add responses to transactions

* do not wrap response into response_class

* fix tests

* update changelog, remove redundant variable

* lint bench

* reorder installs

* do not push benchmark if not in stac-utils/stac-fastapi repo

* Add text about response validation to readme.

* fix warning

* remove versions

* fix

* Update README.md

* update changelog

---------

Co-authored-by: vincentsarago <[email protected]>
Co-authored-by: Jonathan Healy <[email protected]>

Author: 3 people

Diff for: .github/workflows/cicd.yaml

@@ -81,6 +81,7 @@ jobs:
         run: python -m pytest stac_fastapi/api/tests/benchmarks.py --benchmark-only --benchmark-columns 'min, max, mean, median' --benchmark-json output.json
 
       - name: Store and benchmark result
+        if: github.repository == 'stac-utils/stac-fastapi'
         uses: benchmark-action/github-action-benchmark@v1
         with:
           name: STAC FastAPI Benchmarks

Diff for: .github/workflows/deploy_mkdocs.yml

@@ -20,10 +20,10 @@ jobs:
       - name: Checkout main
         uses: actions/checkout@v4
 
-      - name: Set up Python 3.8
+      - name: Set up Python 3.11
         uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.11
 
       - name: Install dependencies
         run: |

Diff for: .gitignore

@@ -129,6 +129,7 @@ docs/api/*
 
 # Virtualenv
 venv
+.venv/
 
 # IDE
 .vscode

Diff for: .pre-commit-config.yaml

@@ -1,10 +1,7 @@
 repos:
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: "v0.0.267"
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: "v0.2.2"
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]
-  - repo: https://github.com/psf/black
-    rev: 23.3.0
-    hooks:
-      - id: black
+      - id: ruff-format

Diff for: CHANGES.md

@@ -2,6 +2,15 @@
 
 ## [Unreleased]
 
+## Changes
+
+* Update to pydantic v2 and stac_pydantic v3 ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+* Removed internal Search and Operator Types in favor of stac_pydantic Types ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+* Fix response model validation ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+* Add Response Model to OpenAPI, even if model validation is turned off ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+* Use status code 201 for Item/Collection creation ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+* Replace Black with Ruff Format ([#625](https://github.com/stac-utils/stac-fastapi/pull/625))
+
 ## [2.5.5.post1] - 2024-04-25
 
 ### Fixed
@@ -48,6 +57,7 @@
 * Add `/queryables` link to the landing page ([#587](https://github.com/stac-utils/stac-fastapi/pull/587))
 - `id`, `title`, `description` and `api_version` fields can be customized via env variables
 * Add `DeprecationWarning` for the `ContextExtension`
+* Add support for Python 3.12
 
 ### Changed
 

Diff for: Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.8-slim as base
+FROM python:3.11-slim as base
 
 # Any python libraries that require system libraries to be installed will likely
 # need the following packages in order to build

Diff for: Dockerfile.docs

@@ -1,4 +1,4 @@
-FROM python:3.8-slim
+FROM python:3.11-slim
 
 # build-essential is required to build a wheel for ciso8601
 RUN apt update && apt install -y build-essential

Diff for: README.md

@@ -41,6 +41,18 @@ Backends are hosted in their own repositories:
 
 `stac-fastapi` was initially developed by [arturo-ai](https://github.com/arturo-ai).
 
+
+## Response Model Validation
+
+A common question when using this package is how request and response types are validated?
+
+This package uses [`stac-pydantic`](https://github.com/stac-utils/stac-pydantic) to validate and document STAC objects. However, by default, validation of response types is turned off and the API will simply forward responses without validating them against the Pydantic model first. This decision was made with the assumption that responses usually come from a (typed) database and can be considered safe. Extra validation would only increase latency, in particular for large payloads.
+
+To turn on response validation, set `ENABLE_RESPONSE_MODELS` to `True`. Either as an environment variable or directly in the `ApiSettings`.
+
+With the introduction of Pydantic 2, the extra [time it takes to validate models became negatable](https://github.com/stac-utils/stac-fastapi/pull/625#issuecomment-2045824578). While `ENABLE_RESPONSE_MODELS` still defaults to `False` there should be no penalty for users to turn on this feature but users discretion is advised.
+
+
 ## Installation
 
 ```bash

Diff for: pyproject.toml

@@ -1,5 +1,8 @@
 [tool.ruff]
+target-version = "py38" # minimum supported version
 line-length = 90
+
+[tool.ruff.lint]
 select = [
     "C9",
     "D1",
@@ -9,13 +12,13 @@ select = [
     "W",
 ]
 
-[tool.ruff.per-file-ignores]
+[tool.ruff.lint.per-file-ignores]
 "**/tests/**/*.py" = ["D1"]
 
-[tool.ruff.isort]
+[tool.ruff.lint.isort]
 known-first-party = ["stac_fastapi"]
 known-third-party = ["stac_pydantic", "fastapi"]
 section-order = ["future", "standard-library", "third-party", "first-party", "local-folder"]
 
-[tool.black]
-target-version = ["py38", "py39", "py310", "py311"]
+[tool.ruff.format]
+quote-style = "double"

Diff for: stac_fastapi/api/setup.py

@@ -6,9 +6,6 @@
     desc = f.read()
 
 install_requires = [
-    "attrs",
-    "pydantic[dotenv]<2",
-    "stac_pydantic==2.0.*",
     "brotli_asgi",
     "stac-fastapi.types",
 ]

Diff for: stac_fastapi/api/stac_fastapi/api/app.py

@@ -1,16 +1,17 @@
 """Fastapi app creation."""
 
+
 from typing import Any, Dict, List, Optional, Tuple, Type, Union
 
 import attr
 from brotli_asgi import BrotliMiddleware
 from fastapi import APIRouter, FastAPI
 from fastapi.openapi.utils import get_openapi
 from fastapi.params import Depends
-from stac_pydantic import Collection, Item, ItemCollection
-from stac_pydantic.api import ConformanceClasses, LandingPage
+from stac_pydantic import api
 from stac_pydantic.api.collections import Collections
-from stac_pydantic.version import STAC_VERSION
+from stac_pydantic.api.version import STAC_API_VERSION
+from stac_pydantic.shared import MimeTypes
 from starlette.responses import JSONResponse, Response
 
 from stac_fastapi.api.errors import DEFAULT_STATUS_CODES, add_exception_handlers
@@ -94,7 +95,7 @@ class StacApi:
             lambda self: self.settings.stac_fastapi_version, takes_self=True
         )
     )
-    stac_version: str = attr.ib(default=STAC_VERSION)
+    stac_version: str = attr.ib(default=STAC_API_VERSION)
     description: str = attr.ib(
         default=attr.Factory(
             lambda self: self.settings.stac_fastapi_description, takes_self=True
@@ -138,9 +139,17 @@ def register_landing_page(self):
         self.router.add_api_route(
             name="Landing Page",
             path="/",
-            response_model=LandingPage
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                api.LandingPage if self.settings.enable_response_models else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.json.value: {},
+                    },
+                    "model": api.LandingPage,
+                },
+            },
             response_class=self.response_class,
             response_model_exclude_unset=False,
             response_model_exclude_none=True,
@@ -157,9 +166,17 @@ def register_conformance_classes(self):
         self.router.add_api_route(
             name="Conformance Classes",
             path="/conformance",
-            response_model=ConformanceClasses
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                api.ConformanceClasses if self.settings.enable_response_models else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.json.value: {},
+                    },
+                    "model": api.ConformanceClasses,
+                },
+            },
             response_class=self.response_class,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -176,7 +193,15 @@ def register_get_item(self):
         self.router.add_api_route(
             name="Get Item",
             path="/collections/{collection_id}/items/{item_id}",
-            response_model=Item if self.settings.enable_response_models else None,
+            response_model=api.Item if self.settings.enable_response_models else None,
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.geojson.value: {},
+                    },
+                    "model": api.Item,
+                },
+            },
             response_class=GeoJSONResponse,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -194,9 +219,19 @@ def register_post_search(self):
         self.router.add_api_route(
             name="Search",
             path="/search",
-            response_model=(ItemCollection if not fields_ext else None)
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                (api.ItemCollection if not fields_ext else None)
+                if self.settings.enable_response_models
+                else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.geojson.value: {},
+                    },
+                    "model": api.ItemCollection,
+                },
+            },
             response_class=GeoJSONResponse,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -216,9 +251,19 @@ def register_get_search(self):
         self.router.add_api_route(
             name="Search",
             path="/search",
-            response_model=(ItemCollection if not fields_ext else None)
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                (api.ItemCollection if not fields_ext else None)
+                if self.settings.enable_response_models
+                else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.geojson.value: {},
+                    },
+                    "model": api.ItemCollection,
+                },
+            },
             response_class=GeoJSONResponse,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -237,9 +282,17 @@ def register_get_collections(self):
         self.router.add_api_route(
             name="Get Collections",
             path="/collections",
-            response_model=Collections
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                Collections if self.settings.enable_response_models else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.json.value: {},
+                    },
+                    "model": Collections,
+                },
+            },
             response_class=self.response_class,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -256,7 +309,17 @@ def register_get_collection(self):
         self.router.add_api_route(
             name="Get Collection",
             path="/collections/{collection_id}",
-            response_model=Collection if self.settings.enable_response_models else None,
+            response_model=api.Collection
+            if self.settings.enable_response_models
+            else None,
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.json.value: {},
+                    },
+                    "model": api.Collection,
+                },
+            },
             response_class=self.response_class,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,
@@ -283,9 +346,17 @@ def register_get_item_collection(self):
         self.router.add_api_route(
             name="Get ItemCollection",
             path="/collections/{collection_id}/items",
-            response_model=ItemCollection
-            if self.settings.enable_response_models
-            else None,
+            response_model=(
+                api.ItemCollection if self.settings.enable_response_models else None
+            ),
+            responses={
+                200: {
+                    "content": {
+                        MimeTypes.geojson.value: {},
+                    },
+                    "model": api.ItemCollection,
+                },
+            },
             response_class=GeoJSONResponse,
             response_model_exclude_unset=True,
             response_model_exclude_none=True,

Diff for: stac_fastapi/api/stac_fastapi/api/config.py

@@ -1,4 +1,5 @@
 """Application settings."""
+
 import enum
 
 

Diff for: stac_fastapi/api/stac_fastapi/api/errors.py

@@ -4,7 +4,7 @@
 from typing import Callable, Dict, Type, TypedDict
 
 from fastapi import FastAPI
-from fastapi.exceptions import RequestValidationError
+from fastapi.exceptions import RequestValidationError, ResponseValidationError
 from starlette import status
 from starlette.requests import Request
 from starlette.responses import JSONResponse
@@ -27,6 +27,7 @@
     DatabaseError: status.HTTP_424_FAILED_DEPENDENCY,
     Exception: status.HTTP_500_INTERNAL_SERVER_ERROR,
     InvalidQueryParameter: status.HTTP_400_BAD_REQUEST,
+    ResponseValidationError: status.HTTP_500_INTERNAL_SERVER_ERROR,
 }
 
 

Diff for: stac_fastapi/api/stac_fastapi/api/middleware.py

@@ -1,4 +1,5 @@
 """Api middleware."""
+
 import re
 import typing
 from http.client import HTTP_PORT, HTTPS_PORT