config-template-python

the uni-directional config template repo for all downstream repos

Use Cases

• config template: fork this repo to inherit the standardised configuration to start a new repo
• config sync: clone this template repo to the same parent folder of a project repo and run just config
• shared lib: clone this template repo to the same parent folder of a project repo and run just shared
• repo specific configs: explicitly annotated with comments and keep during merge
  • # * <-- repo specific config
  • # * repo specific config -->

Setup

To start using the base configs, you will need to install the following system packages:

MacOs:

packages: brew install uv just watchexec

Windows:

git bash: winget install --id Git.Git -e --source winget

packages: scoop install uv just watchexec

• uv - python version & dependency/virtualenv manager @1.8
• just - programmatic command runner @1.25
• watchexec - file watcher @1.25
• bash - the shell for running just commands (available on MacOS by default)

Other tooling:

• VSCode - IDE for sharing extensions with configs
• aws-connect - BP AWS Token Broker CLI (not used after migrating to Azure)

System Bin Dependencies:

• pyenv > python-build
  • [email protected] or openssl@3
  • readline
  • ncurses
  • zlib
• azure-blob-storage > requests > urllib3
  • [email protected]

Usage

Commands

To view the list of available commands, simply run just in your terminal. Commands that require setups:

Conventions

• /artefacts holds all large file assets, e.g. model, image, data
• /output holds all temporary outputs that would be ignored by git
• /notebook for any .ipynb notebooks (VSCode recommended for handling .env)
• /src/shared shared libraries across repos

Python Version

To update python version the following items would need to be updated to use the same version:

• .python-version - local development python version
• pyproject.toml - [project]requires-python version range for dependency resolution
• ruff.toml - target-version python legacy and future features check
• azure-pipelines.yml - variables:PYTHON_VERSION ci python version
• Dockerfile - FROM python: production python version

We use Python 3.10 instead of newer versions for the following reasons:

• use the same version of python in all our repos as much as possible
• could be updated to Python 3.11 now

Details

General Patterns

• pipeline(src)
  • __main__.py - the entrypoint file for just run or python -m src in Docker image
  • args.py - cli arguments definition for input parameters
  • process.py - the available processes of the pipeline
  • validators.py - pydantic validators
  • types.py - type definitions

Test Markers

check pytest.ini

Python Tools

• type checker - mypy
• notebook clean commit - nbstripout
• pre-commit hooks - pre-commit
• dotenv cli - python-dotenv
• linter & formatter - ruff
• code vulnerability scanner - bandit
• doc site generation - sphinx
• docker image scan - docker scout

Config Files

the files are folded by these categories in VSCode

IDE

• .vscode - shared VSCode extensions and settings

Local Development

• justfile - scripted command for development
• .editorconfig - editor config
• .pre-commit-config.yaml - pre-commit config
• .python-version - pyenv local python version
• cspell.config.yaml - code spell config for VSCode extension

Credentials

• .env-cache - local dev env vars that would go into .env
• .env-template - shared dev env vars that would go into .env

All Stages

• pyproject.toml - dependency specification
• .coveragerc - pytest coverage config
• .gitattributes - git attributes
• .gitignore - gitignore
• azure-pipeline.yml - azure pipeline config
• Dockerfile - Dockerfile for production builds
• mypy.ini - mypy type safety check config
• pytest.ini - pytest config
• ruff.toml - ruff linter, formatter config

Documentation

• README.md - documentation
• *.md - further documentations