Skip to content
BranchesTags

Repository files navigation

Ensemble Web NDB API

A serverless AWS Lambda-based API service for managing sheet music scores and related data for a brass ensemble. This service provides the backend API for the Notendatenbank (sheet music database) system.

Overview

This project implements a RESTful API using AWS Lambda and API Gateway (via Lambda Function URLs) to manage:

  • Sheet music scores metadata and files
  • Setlists for concerts and events with player allocations
  • Score samples and templates
  • Event-score-player associations
  • Integration with Google Drive for file storage
  • AI-powered score analysis using OpenAI

The API is deployed using AWS CDK (Cloud Development Kit) and runs as a containerized Lambda function. Automated deployments via GitHub Actions support separate dev and prod environments.

Key Features

  • Robust Error Handling: Standardized exception hierarchy with proper HTTP status codes
  • Automatic Retry Logic: Exponential backoff on transient Google Sheets API failures
  • Comprehensive Testing: 48 integration tests with mocked external dependencies (< 1s execution)
  • Type-Safe Field Definitions: Centralized field configurations eliminate duplicate code
  • Request Validation: Reusable validation helpers with structured error messages
  • Environment Caching: Global cache for expensive client initialization reduces cold starts
  • CI/CD Pipeline: Automated testing and deployments via GitHub Actions

Architecture

Components

  • AWS Lambda Function: Containerized Python function handling all API requests
  • S3 Bucket: Temporary storage for file uploads with automatic expiration
  • AWS Systems Manager Parameter Store: Secure storage for API credentials and configuration
  • Google Drive Integration: Primary storage for sheet music PDFs
  • Google Sheets Integration: Database for score metadata
  • OpenAI Integration: AI-powered analysis of uploaded scores

Technology Stack

  • Infrastructure: AWS CDK with Python
  • Runtime: Python 3.x on AWS Lambda (ARM64 architecture)
  • Container: Docker-based Lambda deployment
  • External Services: Google Drive, Google Sheets, OpenAI

API Endpoints

Scores Management

  • GET /v1/scores - Retrieve all scores with metadata
  • POST /v1/score - Create a new score entry
  • PUT /v1/score - Update an existing score

Setlists Management

  • GET /v1/setlists - List all setlists with full details (items and allocations)
  • POST /v1/setlist - Create new setlist with items and player allocations
  • PUT /v1/setlist - Update existing setlist (name and/or items)
  • POST /v1/download/setlist - Download ZIP with all score PDFs for a setlist

File Operations

  • POST /v1/download - Generate download URL for a score
  • GET /v1/upload - Get presigned URL for file upload
  • POST /v1/upload - Analyze uploaded score using AI

Score Information

  • GET /v1/scoreinfo/samples - Get sample scores and templates

Event Management

  • GET /v1/players - Get event-score-player associations
  • POST /v1/players - Create/update player assignments for events

Authentication

All API endpoints require Basic Authentication via the Authorization header.

Project Structure

ensemble-web-ndb/
├── app.py                        # CDK app entry point
├── cdk.json                      # CDK configuration
├── requirements.txt              # Python dependencies for CDK
├── requirements-dev.txt          # Development/testing dependencies
├── .github/
│   └── workflows/               # GitHub Actions CI/CD
│       ├── deploy-dev.yml       # Auto-deploy to dev environment
│       ├── deploy-prod.yml      # Auto-deploy to prod environment
│       └── test.yml             # Run tests on PRs and pushes
├── lambdaf/                      # Lambda function code
│   ├── handler.py               # Main request handler
│   ├── api_base.py             # Base API utilities
│   ├── methods.py              # Business logic implementation
│   ├── aws_env.py              # AWS environment utilities
│   ├── scores.py               # Score management client
│   ├── setlists.py             # Setlist management client
│   ├── gdfiles.py              # Google Drive integration
│   ├── gsclient.py             # Google Sheets base client
│   ├── genai.py                # OpenAI integration
│   ├── s3bucket.py             # S3 operations
│   ├── score_samples.py        # Sample scores management
│   ├── field_defs.py           # Centralized field definitions
│   ├── exceptions.py           # Exception hierarchy
│   └── validation.py           # Validation helpers
├── stacks/                       # CDK stack definitions
│   ├── lambdaf.py               # Lambda stack definition
│   ├── lambdaf.Dockerfile       # Lambda container definition
│   ├── lambdaf-requirements.txt # Lambda runtime dependencies
│   └── utils.py                 # CDK utilities (NameProvider)
├── tests/                        # Test suite
│   ├── conftest.py              # Pytest configuration
│   ├── integration/             # Integration tests
│   └── mocks/                   # Test fixtures and mocks
└── postman/                      # Postman API collection
    ├── NDB-API.postman_collection.json
    └── README.md

Prerequisites

  • Python 3.8 or later
  • AWS CLI configured with appropriate credentials
  • AWS CDK CLI installed (npm install -g aws-cdk)
  • Docker (for building Lambda container)

Setup

  1. Clone the repository

    git clone <repository-url>
cd ensemble-web-ndb

  2. Create virtual environment

    python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

  3. Install dependencies

    pip install -r requirements.txt

  4. Configure AWS Systems Manager parameters

    Create the following parameters in AWS Systems Manager Parameter Store.

    For production (ensemble-web-ndb-prod-*):

    • ensemble-web-ndb-prod-api-creds: JSON array with Basic Auth credentials 
      [{"user": "username", "password": "password"}]
    • ensemble-web-ndb-prod-google-creds: Google service account credentials JSON
    • ensemble-web-ndb-prod-config-dict: Configuration dictionary with: 
      {
  "google_drive_folder_id": "folder-id",
  "google_spreadsheet_key": "spreadsheet-id",
  "openai_key": "sk-...",
  "ui_auth_secret": "secret-for-ui"
}

    For development, create the same parameters with -dev- prefix.

  5. Setup Google Sheets Worksheets

    Create the following worksheets in your Google Sheets document:

    • scores - Score metadata (columns: id, title, composer, etc.)
    • setlists - Setlist data (columns: setlistId, setlistName, scoreId, orderIndex, allocations)
    • score_samples - Sample scores and templates

    See CLAUDE.md for detailed column definitions.

Deployment

Automated Deployment (Recommended)

The project uses GitHub Actions for automated deployments and testing:

  • Test Workflow: Automatically runs on all PRs and pushes to develop or main branches
    • Runs 48 integration tests covering all API endpoints
    • Generates coverage reports to ensure code quality
    • Validates changes before deployment
  • Dev Environment: Automatically deploys when code is pushed to develop branch
  • Prod Environment: Automatically deploys when code is pushed to main branch

Setup GitHub Actions

  1. Configure AWS Credentials in GitHub

    Option A - AWS OIDC (Recommended, no long-lived credentials):

    # Create OIDC identity provider in AWS IAM
# Create IAM role with deployment permissions
# Add role ARN to GitHub repository secrets as AWS_ROLE_ARN

    Option B - IAM User Access Keys:

    # Create deployment IAM user with CDK permissions
# Add to GitHub repository secrets:
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY
# - AWS_REGION

  2. Bootstrap Both Environments

    # Bootstrap dev environment
DEPLOY_ENV=dev cdk bootstrap

# Bootstrap prod environment
DEPLOY_ENV=prod cdk bootstrap

  3. Create AWS SSM Parameters for Both Environments

    For dev environment (ensemble-web-ndb-dev-*):

    • ensemble-web-ndb-dev-api-creds - API credentials JSON
    • ensemble-web-ndb-dev-google-creds - Google service account credentials
    • ensemble-web-ndb-dev-config-dict - Configuration dictionary

    For prod environment (ensemble-web-ndb-prod-*):

    • ensemble-web-ndb-prod-api-creds - API credentials JSON
    • ensemble-web-ndb-prod-google-creds - Google service account credentials
    • ensemble-web-ndb-prod-config-dict - Configuration dictionary

  4. Create develop Branch

    git checkout -b develop
git push -u origin develop

  5. Push to Deploy

    # Deploy to dev
git push origin develop

# Deploy to prod
git push origin main

Manual Deployment

For local development or testing, you can deploy manually:

  1. Set environment variable

    export DEPLOY_ENV=dev  # or 'prod'

  2. Bootstrap CDK (first time only)

    cdk bootstrap

  3. Deploy the stack

    cdk deploy

  4. Note the output After successful deployment, CDK will output the Lambda Function URL:

    Outputs:
ensemble-web-ndb-dev-LambdaF.ApiUrl = https://xxxxx.lambda-url.region.on.aws/

Development

Running Tests

The project includes 48 integration tests covering all API endpoints and core functionality:

  • Setlist Tests (22 tests): Create, update, list operations, validation, and helper methods
  • Score Tests (19 tests): CRUD operations, validation, field parsing, and edge cases
  • Score Samples Tests (8 tests): Sample retrieval, grouping, and validation

All tests use mocked Google Sheets API calls for fast, deterministic execution (< 1 second).

# Install dev dependencies
pip install -r requirements-dev.txt

# Run all tests
pytest tests/

# Run with coverage report
pytest tests/ --cov=lambdaf --cov-report=term-missing

# Run with HTML coverage report
pytest tests/ --cov=lambdaf --cov-report=html

# Run specific test file
pytest tests/integration/test_setlists_api.py -v

# Run specific test class
pytest tests/integration/test_scores_api.py::TestGetScores -v

Continuous Integration

The project uses GitHub Actions to automatically run tests on every pull request and push:

  • Tests run on Python 3.12
  • Coverage reports are generated and displayed in the workflow summary
  • Tests must pass before merging (recommended as a branch protection rule)

Testing API Endpoints

Use the Postman collection in postman/ directory:

  1. Import postman/NDB-API.postman_collection.json into Postman
  2. Set environment variables: baseUrl and authToken
  3. Test all endpoints with example requests

Viewing Logs

View Lambda logs for specific environment:

# Dev environment
aws logs tail /aws/lambda/ensemble-web-ndb-dev-lambdaf --follow

# Prod environment
aws logs tail /aws/lambda/ensemble-web-ndb-prod-lambdaf --follow

Configuration

Environment Variables (Lambda)

Set automatically by CDK based on DEPLOY_ENV:

  • API_CREDENTIALS_PARAM - SSM parameter name for API credentials
  • GOOGLE_CREDENTIALS_PARAM - SSM parameter name for Google credentials
  • CONFIG_DICT_PARAM - SSM parameter name for configuration
  • UPLOAD_BUCKET_PARAM - S3 bucket name for uploads

S3 Bucket Configuration

  • CORS enabled for browser uploads (PUT, GET methods)
  • Automatic object expiration after 1 day
  • Presigned URLs for secure uploads/downloads
  • Block public access enabled

Adding New Endpoints

  1. Add handler method in lambdaf/handler.py
  2. Implement business logic in lambdaf/methods.py
  3. Add field definitions to lambdaf/field_defs.py if needed
  4. Update path matching in handle_api_request()
  5. Add tests in tests/integration/
  6. Deploy changes (push to develop or main branch)

Architecture Notes

Request Flow

  1. API Gateway (Lambda Function URL) receives HTTP request
  2. lambdaf/handler.py::lambda_entry() - Entry point for all requests
  3. RequestHandler.handle_request() - Routes to appropriate handler based on path/method
  4. Handler methods call business logic in lambdaf/methods.py
  5. Business logic uses specialized clients (scores, setlists, gdfiles, etc.)

Caching Pattern

The EnvironmentMixin in aws_env.py uses a global PROPERTY_CACHE to cache expensive initialization operations (AWS clients, Google API credentials, OpenAI client). This ensures client instances are reused across invocations within the same Lambda container, reducing cold start overhead.

Error Handling

  • ValidationError - Client errors (400 Bad Request)
  • NotFoundError - Resource not found (404 Not Found)
  • ConflictError - Conflicts and operation errors (409 Conflict)
  • All exceptions inherit from NDBAPIException with structured error details
  • Automatic retry with exponential backoff on Google Sheets API errors

Contributing

Contributions are welcome! Please follow these guidelines:

  1. Create a feature branch from develop:

    git checkout develop
git pull origin develop
git checkout -b feature/your-feature-name

  2. Write tests for new functionality:

    • Add integration tests in tests/integration/
    • Ensure all tests pass: pytest tests/
    • Aim for high test coverage

  3. Follow code patterns:

    • Use centralized field definitions in field_defs.py
    • Raise appropriate exceptions from exceptions.py
    • Use validation helpers from validation.py
    • Add retry decorators to Google Sheets operations

  4. Create a pull request:

    • Target the develop branch
    • Provide clear description of changes
    • Ensure CI tests pass

  5. Deployment:

    • Merged PRs to develop automatically deploy to dev environment
    • Merged PRs to main automatically deploy to prod environment

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

The GPL v3 ensures that this software remains free and open source. Any modifications or distributions of this software must also be released under the GPL v3 license.

Related Documentation

Support

For issues, questions, or feature requests:

  • Create an issue in the GitHub repository
  • Include relevant logs and error messages
  • Tag with appropriate labels (bug, enhancement, question)

About

A serverless AWS Lambda-based API service for managing sheet music scores and related data for a brass ensemble. This service provides the backend API for the Notendatenbank (sheet music database) system.

Resources

Readme
GPL-3.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Used by

Contributors

Languages