Mitwelten ML Backend

Pipeline Overview

graph TD
  ingest(ingest: file upload)-->db(db: metadata, datasets, results)
  ingest-->storage(storage: files)
  storage --> inference(inference: TensorFlow)
  db --> inference
  inference --> db

Loading

ingest

Import of assets, metadata to db (PostgreSQL), files to storage (minIO S3).

• extract metadata from files
• check for naming collisions with existing files
• upload metadata to db
• upload files to storage

The import of existing audio data is documented in the import-existing-audio report. The accompanying code is found in ingest/import_existing/. Further development will be based on this code.

audio uploader app

To upload audiofiles recorded by Audiomoths, you can use the mitwelten-audio-uploader app.

database

Retired database schema, for current setup check mitwelten-db-backend.

• files (metadata)
• datasets (collections of files)
• paramsets (inference configuration)
• results (inference)

For details check initialize_db.sql.

Set credentials in credentials.py (db)

storage

Bucket on minIO S3 backend, publicly accessible.

Set credentials in credentials.py (storage)