Skip to content

Create a OpenReferral compliant API and data model - #1

Open
samau3 wants to merge 13 commits into
mainfrom
feat/hsds-data-models
Open

samau3 wants to merge 13 commits into
mainfrom
feat/hsds-data-models

Conversation

@samau3

@samau3 samau3 commented Aug 7, 2026

Copy link
Copy Markdown

This is a lossy public-directory conversion, not a 1:1 clone of the ShelterTech schema. Canonical HSDS rows live in public; the immutable source dump and conversion report are the provenance trail. Raw ShelterTech rows are not copied into the operational database.

  • DB: HSDS models and migration in the public schema alongside existing application tables
  • Import: npm run import:sheltertech converts 33 allowlisted tables, validates source data, requires empty canonical tables, and writes deterministic HSDS records in one transaction
  • API: Read-only HSDS 3.2.3 endpoints under /api; conversion profile linked from API metadata and the README
  • Report: JSON report includes source/canonical counts, ignored-table counts, field dispositions, report-only values, warnings, mappings, and inferred mappings

Conversion policy

ShelterTech data HSDS treatment / assumption
Core organizations, services, locations, contacts, phones, schedules, and documents Canonical HSDS entities
ShelterTech IDs Used for deterministic UUIDs and report mappings; original IDs are not exposed through canonical rows or APIs
Non-allowlisted public tables Contents ignored; table names and row counts reported
Unsupported allowlisted tables Import fails if populated: accessibilities, categories_keywords, categories_sites, keywords*, news_articles, synonym_groups, and synonyms
Orphan documents, funding, or languages Import fails before opening a transaction
Categories, eligibilities, sites, notes, instructions, and legacy flags Taxonomy terms and attributes
Multi-parent taxonomy First parent → parent_id; extras → additional_parent attributes
Explicit true / false flags Both create attributes; only null values are omitted
Services with explicit addresses_services Preserve exactly
Services without explicit address links Infer service-at-location rows from every organization address
Organization schedules Copy onto every organization location
Service schedules Keep on the service rather than copying to each service-at-location
Schedule timezone Treat as America/Los_Angeles local wall-clock time; no UTC conversion
Legacy 2400 Normalize to 00:00:00; closing gets a next-day note, opening moves to the following weekday
Overnight hours Preserve times and add a next-day/close_day note
Invalid or fractional schedule times Fail preflight rather than truncate or normalize silently
internal_note Dropped completely as private data
Audit fields, source_attribution, address_3/4, primary contact IDs, vocabulary, relationship priority, and open_day Report-only; not queryable through canonical APIs
resources.status Report-only
Service status Only '1' accepted; imported as active
Missing descriptions Use the entity name and record a name_as_description inferred mapping
Short + long descriptions Concatenate into one HSDS description
fee Write to both fees and fees_description
Phone ownership Preserve explicit organization/service/contact relationships; do not copy phones elsewhere
Phone extensions Store as strings to preserve leading zeroes
Unknown phone types Leave canonical type unset, preserve the ShelterTech label in description, and emit a warning
Invalid source URLs Preserve unchanged and emit a nonfatal warning; known dummy seed URLs are expected
top_only=true with parent_id Reject with HTTP 400

Reviewer feedback on assumptions made

  • Is inferring every organization address for services without explicit links safe?
  • Can organization schedules be assumed to apply to every organization location?
  • Should report-only fields remain outside the operational API?
  • Should unsupported tables and orphan rows block the entire import?
  • Is leaving unknown phone types unset preferable to assuming voice?
  • Are the local-time, 2400, and overnight schedule rules appropriate?
  • Are string extensions, local schedule times, and preserved invalid URLs acceptable HSDS 3.2.3 profile variations?

The checked-in seed is synthetic and does not exercise multi-address organizations, explicit service/address links, structured schedule times, private notes, or every orphan case. Production assumptions should be confirmed against a real dump and retained conversion report.

samau3 added 13 commits August 6, 2026 15:34
Introduce Open Referral HSDS 3.2.3 tables and immutable import audit storage alongside the existing public app schema.
Parse allowlisted seed tables into audit snapshots and upsert canonical HSDS records with deterministic UUIDs.
Add paginated JSON routes for services, organizations, taxonomies, taxonomy terms, and service-at-locations.
Describe the multi-schema layout, import workflow, and public HSDS API routes.
Drop the hsds/import_audit schema split and keep only canonical HSDS tables in public.
Require empty canonical tables, validate unsupported fields, and print a JSON report instead of persisting import audit rows.
Serialize linked attributes for taxonomy term detail endpoints.
Document the single-schema layout and saving the importer JSON report beside the source dump.
Keep a canonical parentId for multi-parent taxonomy terms, reject
invalid phones/addresses/days in preflight, and avoid rebuilding the
address index per service.
Load organization/service attributes in one query, include them on
nested full_service payloads and taxonomy-term lists, and stop
top_only from clobbering parent_id.
Point the importer at db/seed.sql by default and document optional
path overrides.
Integer storage dropped leading zeros from dump values like ext=0967.
Store the HSDS extension field as text through Prisma and import.
Document the conversion rules and align the importer and API metadata
with them, including ignored-table counts, warnings, and stricter
schedule and taxonomy query handling.
@samau3
samau3 marked this pull request as ready for review August 17, 2026 23:32
@samau3
samau3 requested a review from francisli August 17, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant