Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@
.vscode/
tests/.work/
build/
gtfs_flex_to_gofs_lite/output
gtfs_flex_to_gofs/output
dist/
.claude/
75 changes: 59 additions & 16 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,34 +4,51 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

This is a Python tool that converts GTFS-Flex (General Transit Feed Specification - Flexible services) data to the GOFS-lite (General On-demand Feed Specification - lite) format. It processes on-demand/flexible transit services data and outputs standardized GOFS files.
This is a Python tool that converts GTFS-Flex (General Transit Feed Specification - Flexible services) data to the GOFS (General On-demand Feed Specification) format. It processes on-demand/flexible transit services data and outputs standardized GOFS files.

## Core Architecture

The conversion process follows this flow:
1. Load GTFS data using `gtfs_loader`
1. Load GTFS data using `gtfs_loader`
2. Extract relevant data through `operation_rules.py` which analyzes trip types and creates `GofsData`
3. Generate individual GOFS files via modules in `gtfs_flex_to_gofs_lite/files/`
4. Output structured GOFS-lite format files
3. Generate individual GOFS files via modules in `gtfs_flex_to_gofs/files/`
4. Output structured GOFS format files

### Key Components

- **`gofs_lite_converter.py`**: Main conversion orchestrator that coordinates all file generation
- **`gofs_converter.py`**: Main conversion orchestrator that coordinates all file generation
- **`gofs_data.py`**: Contains `GofsData` class that tracks extracted IDs and transfers from GTFS-Flex
- **`files/operation_rules.py`**: Core logic for classifying trip types (regular_service, deviated_service, pure_microtransit, other) and extracting zone-to-zone transfers
- **`files/` directory**: Individual modules for generating each GOFS file type (zones, calendars, fares, etc.)
- **`gofs_file.py`**: Base class for GOFS file objects with save functionality

### Trip Classification Logic

The system classifies GTFS trips into four types:
- `PURE_MICROTRANSIT`: Flexible zone-to-zone services
- `DEVIATED_SERVICE`: Routes with some flexible elements
- `REGULAR_SERVICE`: Fixed route services
- `OTHER`: Unclassified trips
The system classifies GTFS trips into four types based on stop time patterns in `operation_rules.py:get_type_of_trip()`:
- `PURE_MICROTRANSIT`: All stop times are flexible zones (have pickup/dropoff windows)
- `DEVIATED_SERVICE`: Mix of regular stops and flexible zones (3+ stop times required)
- `REGULAR_SERVICE`: All stop times are regular fixed stops
- `OTHER`: Invalid or unclassifiable trips (e.g., <2 stops)

Only `PURE_MICROTRANSIT` trips are converted to GOFS format.

### Conversion Workflow

1. **Trip Analysis** (`operation_rules.py:create()`):
- Iterates through all GTFS trips and their stop times
- Classifies each trip using `get_type_of_trip()`
- For `PURE_MICROTRANSIT` trips, extracts zone-to-zone transfers
- Populates `GofsData` with extracted IDs (routes, calendars, zones, booking rules)

2. **File Generation** (`gofs_converter.py:convert_to_gofs()`):
- Creates individual GOFS files using `GofsData` to filter relevant GTFS entities
- Files generated in order: operation_rules, zones, system_information, service_brands, vehicle_types, calendars, fares, wait_times, wait_time, booking_rules, gofs_versions, gofs
- Each file module has a `create()` function that returns a `GofsFile` object

3. **Output** (`gofs_converter.py:save_files()`):
- Single folder mode: All GOFS files in one directory
- Split-by-route mode: Creates one folder per route_id, with filtered operation_rules per folder

## Development Commands

### Installation
Expand All @@ -41,12 +58,20 @@ uv sync --extra dev

### Running Tests
```bash
python -m pytest .
uv run python -m pytest . # All tests
uv run python -m pytest tests/ -v # Verbose output
```

### Running the Tool
```bash
python -m gtfs_flex_to_gofs_lite --gtfs-dir <input_dir> --gofs-lite-dir <output_dir> --url <base_url>
python -m gtfs_flex_to_gofs --gtfs-dir <input_dir> --gofs-dir <output_dir> --url <base_url>

# With optional parameters:
--ttl <seconds> # Time-to-live for GOFS files (default: 86400)
--timestamp <unix_time> # Fixed timestamp for deterministic output
--split-by-route # Create separate GOFS folders per route
--out-gtfs-dir <dir> # Output directory for patched GTFS
--no-warning # Silence warnings
```

### Test Regeneration
Expand All @@ -63,15 +88,33 @@ python -m pytest tests/test_runner.py::test_default[test_simple_conversion]

Replace `test_simple_conversion` with any test case name from the `tests/` directory.

### Linting
```bash
uv run flake8
```

### Deployment

Deployment to PyPI is automated via GitHub Actions when the version in `pyproject.toml` is updated on the `main` branch. See `.github/workflows/release.yml` for details.

## Dependencies

- `py-gtfs-loader`: Custom GTFS loading library from TransitApp
- `shapely>=2.0.4`: Geometric operations
- `pytest`: Testing framework

## File Structure Notes
## Testing Structure

Test cases use a snapshot-based approach:
- Each test case lives in `tests/test_*/` directory
- `input/`: Contains GTFS files (including required `locations.geojson`)
- `expected_default/`: Contains expected GOFS output files
- `test_runner.py` uses pytest parametrization to automatically discover all test cases
- Tests compare generated output against expected files using `gtfs_loader.test_support`
- `createTests.sh` regenerates all expected output files (run after intentional changes)

### Important Testing Notes

- Test cases are in `tests/test_*/` with `input/` and `expected_default/` subdirectories
- Tests use `timestamp=0` for deterministic output
- The `locations.geojson` file in GTFS input is required for conversion - without it, conversion is skipped
- GOFS files are generated with specific headers including TTL, version, and timestamps
- Split-by-route functionality creates separate GOFS folders per route when enabled
- Test isolation: each test gets its own work directory created by `test_support.create_test_data()`
76 changes: 62 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,79 @@
Tool to convert GTFS-Flex data to the GOFS-lite format
# GTFS-flex-to-GOFS

To install:
* `uv sync --extra dev`
[![Build Status](https://github.com/TransitApp/GTFS-flex-to-GOFS/actions/workflows/pull-request.yml/badge.svg)](https://github.com/TransitApp/GTFS-flex-to-GOFS/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

To run test:
* `uv run python -m pytest .`
A Python tool to convert [GTFS-Flex](https://github.com/MobilityData/gtfs-flex) (General Transit Feed Specification - Flexible services) data to the [GOFS](https://gofs.org) (General On-demand Feed Specification) format.

You can use `createTests.sh` to regenerate the test.
This tool processes on-demand and flexible transit services data from GTFS-Flex feeds and outputs standardized GOFS files for consumption by trip planning applications and mobility platforms.

To deploy a new version, run:
## Installation

Install from PyPI:
```bash
pip install GTFS-flex-to-GOFS
```
rm -r dist/
uv build
TWINE_USERNAME=transit TWINE_REPOSITORY_URL=https://pypi.transitapp.com:443 TWINE_PASSWORD=[PASSWORD] twine upload dist/*

Or install from source:
```bash
git clone https://github.com/TransitApp/GTFS-flex-to-GOFS.git
cd GTFS-flex-to-GOFS
uv sync --extra dev
```

The twine password can be found in 1Password under the same `PyPI password`.
## Usage

### man
```bash
gtfs-flex-to-gofs --gtfs-dir <input_dir> --gofs-dir <output_dir> --url <base_url>
```
Convert GTFS-flex on-demand format to GOFS-lite

### Command Line Options

```
optional arguments:
-h, --help show this help message and exit
--gtfs-dir Dir input gtfs directory
--gofs-lite-dir Dir output gofs directory
--gofs-dir Dir output gofs directory
--url URL auto-discovery url. Base URL indicate for where each files will be uploaded (and downloadable)
--ttl TTL time to live of the generated gofs files in seconds (default: 86400)
--no-warning Silence warnings
```

## Development

### Running Tests

```bash
uv run python -m pytest .
# or for verbose output:
uv run python -m pytest tests/ -v
```

### Regenerating Test Fixtures

```bash
./createTests.sh
```

## Features

- Converts GTFS-Flex pure microtransit services to GOFS format
- Supports zone-based on-demand transit operations
- Generates all required GOFS files (zones, calendars, fares, booking rules, etc.)
- Optional split-by-route output for multi-service feeds
- Automated CI/CD with GitHub Actions

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Related Projects

- [GTFS-Flex Specification](https://github.com/MobilityData/gtfs-flex)
- [GOFS Specification](https://gofs.org)
- [py-gtfs-loader](https://github.com/TransitApp/py-gtfs-loader)
10 changes: 5 additions & 5 deletions createTests.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
python3 -m gtfs_flex_to_gofs_lite --gtfs-dir ./tests/test_simple_conversion/input --gofs-lite-dir ./tests/test_simple_conversion/expected_default --url "" --timestamp 0
python3 -m gtfs_flex_to_gofs_lite --gtfs-dir ./tests/test_non_pure_microtransit_route/input --gofs-lite-dir ./tests/test_non_pure_microtransit_route/expected_default --url "" --timestamp 0
python3 -m gtfs_flex_to_gofs_lite --gtfs-dir ./tests/test_ondemand_stops/input --gofs-lite-dir ./tests/test_ondemand_stops/expected_default --url "" --timestamp 0
python3 -m gtfs_flex_to_gofs_lite --gtfs-dir ./tests/test_added_calendar_dates/input --gofs-lite-dir ./tests/test_added_calendar_dates/expected_default --url "" --timestamp 0
python3 -m gtfs_flex_to_gofs_lite --gtfs-dir ./tests/test_removed_calendar_dates/input --gofs-lite-dir ./tests/test_removed_calendar_dates/expected_default --url "" --timestamp 0
uv run python -m gtfs_flex_to_gofs --gtfs-dir ./tests/test_simple_conversion/input --gofs-dir ./tests/test_simple_conversion/expected_default --url "" --timestamp 0
uv run python -m gtfs_flex_to_gofs --gtfs-dir ./tests/test_non_pure_microtransit_route/input --gofs-dir ./tests/test_non_pure_microtransit_route/expected_default --url "" --timestamp 0
uv run python -m gtfs_flex_to_gofs --gtfs-dir ./tests/test_ondemand_stops/input --gofs-dir ./tests/test_ondemand_stops/expected_default --url "" --timestamp 0
uv run python -m gtfs_flex_to_gofs --gtfs-dir ./tests/test_added_calendar_dates/input --gofs-dir ./tests/test_added_calendar_dates/expected_default --url "" --timestamp 0
uv run python -m gtfs_flex_to_gofs --gtfs-dir ./tests/test_removed_calendar_dates/input --gofs-dir ./tests/test_removed_calendar_dates/expected_default --url "" --timestamp 0
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
import gtfs_loader
from pathlib import Path

from .gofs_lite_converter import convert_to_gofs_lite
from .gofs_converter import convert_to_gofs
from .patch_gtfs import patch_gtfs
from .utils import yellow_text

Expand All @@ -11,11 +11,11 @@

def main(args):
gtfs = gtfs_loader.load(args.gtfs_dir, itineraries=args.itineraries)
gofs_lite_dir = Path(args.gofs_lite_dir)
gofs_dir = Path(args.gofs_dir)

gofs_lite_dir.mkdir(parents=True, exist_ok=True)
gofs_dir.mkdir(parents=True, exist_ok=True)

convert_to_gofs_lite(gtfs, gofs_lite_dir, args.ttl, args.url, args.split_by_route, args.timestamp, itineraries=args.itineraries)
convert_to_gofs(gtfs, gofs_dir, args.ttl, args.url, args.split_by_route, args.timestamp, itineraries=args.itineraries)

if args.out_gtfs_dir:
patch_gtfs(args, gtfs, itineraries=args.itineraries)
Expand All @@ -34,11 +34,11 @@ def print_args_warnings(args):

if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='Convert GTFS-flex on-demand format to GOFS-lite')
description='Convert GTFS-flex on-demand format to GOFS')
parser.add_argument(
'--gtfs-dir', help='input gtfs directory', metavar='Dir', required=True)
parser.add_argument(
'--gofs-lite-dir', help='output gofs directory', metavar='Dir', required=True)
'--gofs-dir', help='output gofs directory', metavar='Dir', required=True)
parser.add_argument(
'--out-gtfs-dir', help='output directory of patched gtfs', metavar='Dir', required=False)
parser.add_argument(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def extract_calendar_dates_only_calendar(gtfs, used_calendar_ids):

def create_calendar_with_list_dates(calendar_id, active_dates_calendar):
# take a list of dates, and create a calendar with all the dates with a list of exception for all the missing dates
# workaround for the lack of support for a list of supported dates in GOFS-lite
# workaround for the lack of support for a list of supported dates in GOFS
active_dates_calendar.sort()
start_date = active_dates_calendar[0]
end_date = active_dates_calendar[-1]
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from ..gofs_data import GofsTransfer
from gtfs_loader.schema import PickupType, DropOffType

from gtfs_flex_to_gofs_lite.utils import get_locations_group, get_zones
from gtfs_flex_to_gofs.utils import get_locations_group, get_zones
from enum import Enum

FILENAME = 'operating_rules'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
from dataclasses import dataclass
from typing import Any, List

from gtfs_flex_to_gofs_lite.gofs_data import GofsData
from gtfs_flex_to_gofs.gofs_data import GofsData

from ..gofs_file import GofsFile
from gtfs_flex_to_gofs_lite.utils import get_locations_group, get_zones
from gtfs_flex_to_gofs.utils import get_locations_group, get_zones
import math
import shapely.ops
import json
Expand Down Expand Up @@ -163,7 +163,7 @@ def _create_polygon_from_point_stop(self, stop_id):
return polygon_object

if stop_id not in self.gtfs.stops:
print(f"[GTFS-Flex-To-GOFS-Lite] - Missing {stop_id} from stops.txt")
print(f"[GTFS-Flex-To-GOFS] - Missing {stop_id} from stops.txt")
return None

stop = self.gtfs.stops[stop_id]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ def has_convertable_data(gtfs):
return gtfs.locations != {}


def convert_to_gofs_lite(gtfs, gofs_lite_dir, ttl, base_url, split_by_route=False, timestamp=None, itineraries=False):
def convert_to_gofs(gtfs, gofs_dir, ttl, base_url, split_by_route=False, timestamp=None, itineraries=False):
if not has_convertable_data(gtfs):
return GofsData()

Expand Down Expand Up @@ -125,4 +125,4 @@ def convert_to_gofs_lite(gtfs, gofs_lite_dir, ttl, base_url, split_by_route=Fals
file = gofs.create(gtfs, base_url, files_created)
register_created_file(files_created, file)

save_files(files_created, gofs_lite_dir, ttl, creation_timestamp, split_by_route)
save_files(files_created, gofs_dir, ttl, creation_timestamp, split_by_route)
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from gtfs_flex_to_gofs_lite.gofs_data import GofsData
from gtfs_flex_to_gofs.gofs_data import GofsData
import gtfs_loader
from .files import operation_rules

Expand Down
File renamed without changes.
12 changes: 6 additions & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "GTFS-flex-to-GOFS-lite"
name = "GTFS-flex-to-GOFS"
version = "0.5.0"
description = "Convert GTFS Flex to GOFS lite"
description = "Convert GTFS Flex to GOFS"
readme = "README.md"
license = {text = "MIT"}
authors = [
Expand All @@ -29,11 +29,11 @@ dev = [
]

[project.urls]
Homepage = "https://github.com/TransitApp/GTFS-flex-to-GOFS-lite"
Repository = "https://github.com/TransitApp/GTFS-flex-to-GOFS-lite"
Homepage = "https://github.com/TransitApp/GTFS-flex-to-GOFS"
Repository = "https://github.com/TransitApp/GTFS-flex-to-GOFS"

[tool.setuptools.packages.find]
include = ["gtfs_flex_to_gofs_lite*"]
include = ["gtfs_flex_to_gofs*"]

[project.scripts]
gtfs-flex-to-gofs-lite = "gtfs_flex_to_gofs_lite.__main__:main"
gtfs-flex-to-gofs = "gtfs_flex_to_gofs.__main__:main"
8 changes: 4 additions & 4 deletions tests/test_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from pathlib import Path
import gtfs_loader
from gtfs_loader import test_support
from gtfs_flex_to_gofs_lite import gofs_lite_converter
from gtfs_flex_to_gofs import gofs_converter

import time
time.time = lambda: 0
Expand All @@ -24,12 +24,12 @@ def do_test(feed_dir):
Path.mkdir(dest_dir)

gtfs = gtfs_loader.load(work_dir)
gofs_lite_converter.convert_to_gofs_lite(gtfs, dest_dir, 24 * 60 * 60, '', ("split_by_route" in str(feed_dir)))
gofs_converter.convert_to_gofs(gtfs, dest_dir, 24 * 60 * 60, '', ("split_by_route" in str(feed_dir)))
test_support.check_expected_output(feed_dir, dest_dir)

# Each folder contains both stop_times.txt and itinerary_cells.txt and should produce identical output
# converting either
Path.mkdir(dest_dir)
gtfs = gtfs_loader.load(work_dir, itineraries=True)
gofs_lite_converter.convert_to_gofs_lite(gtfs, dest_dir, 24 * 60 * 60, '', ("split_by_route" in str(feed_dir)), itineraries=True)
gofs_converter.convert_to_gofs(gtfs, dest_dir, 24 * 60 * 60, '', ("split_by_route" in str(feed_dir)), itineraries=True)
test_support.check_expected_output(feed_dir, dest_dir)
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,4 @@
}
]
}
}
}
Loading
Loading