Implement prepbufr raobs obs builder 185

.github/workflows/ci.yml

@@ -23,10 +23,10 @@ jobs:
         run: pipx install poetry
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
+          python-version: "3.12"
       - name: Install dependencies
         run: |
-          poetry env use 3.11
+          poetry env use 3.12
           poetry install
       - name: Format with Ruff
         run: poetry run ruff format --check src tests
@@ -42,10 +42,10 @@ jobs:
         run: pipx install poetry
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
+          python-version: "3.12"
       - name: Install dependencies
         run: |
-          poetry env use 3.11
+          poetry env use 3.12
           poetry install
       - name: Unit test with pytest
         run: | 

docker/import/Dockerfile

@@ -27,7 +27,7 @@ RUN groupadd --gid ${ID} ${USERNAME} && \
 RUN apt-get update && apt-get upgrade -y && \
     # Install runtime deps for the script
     apt-get install -y curl jq && \
-    apt-get clean && rm -rf /var/lib/apt/lists/* 
+    apt-get clean && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 

docker/ingest/Dockerfile

@@ -8,7 +8,7 @@ FROM python:3.11-slim-bookworm AS builder
 RUN apt-get update && \
     # cftime build deps
     apt-get install -y build-essential && \
-    apt-get clean && rm -rf /var/lib/apt/lists/* 
+    apt-get clean && rm -rf /var/lib/apt/lists/*
 
 # Make sure the python tooling is up-to-date for when it's copied into the prod layer
 RUN pip install --no-cache-dir --upgrade pip setuptools wheel
@@ -40,7 +40,7 @@ RUN poetry install
 
 
 # The runtime image, used for running just the application with its dependencies
-FROM python:3.11-slim-bookworm AS prod
+FROM python:3.12-slim-bookworm AS prod
 
 ARG COMMITBRANCH=development
 ARG COMMITSHA=unspecified
@@ -76,7 +76,7 @@ RUN groupadd --gid ${ID} ${USERNAME} && \
 
 # Run OS updates
 RUN apt-get update && apt-get upgrade -y && \
-    apt-get clean && rm -rf /var/lib/apt/lists/* 
+    apt-get clean && rm -rf /var/lib/apt/lists/*
 
 # Copy just the vxingest app
 COPY ./src/ /app/

mats_metadata_and_indexes/metadata_files/README.md

@@ -1,13 +1,16 @@
 # MATS metadata and ingest documents
 
-The system relies on a couple of kinds of metadata. Ingest builders use ingest documents to control how an output document is created from its input source. The running system relies on various metadata documents to define what data is available and how it should be repesented in a GUI interface.
+The system relies on a couple of kinds of metadata. Ingest builders use ingest documents to control how an output document is created from its input source. The running system relies on various metadata documents to define what data is available and how it should be represented in a GUI interface.
 
 ## Ingest documents
 
-Ingest documents contain templates that define how a builder should assemble an out document.
+Ingest documents contain templates that define how a builder should assemble an output document.
 
 ### ingest document Example
 
+NOTE: prepbufr ingest documents have an extra element called "mnemonic_mapping" that maps prepbufr mnemonics to the template variables. This extra step is necessary because prepbufr fields can vary due to an associated program code.
+For example a program code of 1 is an initial value and a program code of 8 is a virtual value. The template.mnemonic_mapping section specifies whether the event program code is meaningful and which program code value is desired. Refer to the prepbufr [README.md](https://github.com/NOAA-GSL/VxIngest/blob/72793df75696ef711d79553a82be3b8a6c04653c/src/vxingest/prepbufr_to_cb/README.md) for a complete example and explanation of the prepbufr ingest template.
+
 This is the contents of "MD:V01:METAR:obs:ingest:netcdf". If
 you intend to use a metadata ingest document you must either
 be certain that it already exists or you must create it.
@@ -62,11 +65,15 @@ be certain that it already exists or you must create it.
 }
 ```
 
+### Ingest Template DSL
+
+Ingest templates implement a simple DSL (Domain specific Language) described here for a netcdf builder. The same syntax applies to all builder classes.
+
 The line
 ```"builder_type": "NetcdfObsBuilderV01"```
 defines a python class. These builder classes are defined
 in the [netcdf_to_cb/netcdf_builder.py](https://github.com/NOAA-GSL/VxIngest/blob/main/netcdf_to_cb/netcdf_builder.py) file. This class will interpret the
-load_spec and ingest data from a set of netcdf files retrieved from the path.
+ingest data from a set of netcdf files retrieved from the path.
 Whether the entire result set is combined into one document or multiple documents depends on the "builder_type".
 In this example the "NetcdfObsBuilderV01" combines all
 the data into one document with the data fields ingested as top level
@@ -113,8 +120,8 @@ def ceiling_transform(self, params_dict):
 The named function routine processes a named function entry from a template.
 The '_named_function_def' looks like "&named_function|*field1,*field2,*field3..."
 where "named_function" is the literal function name of a defined function.
-The name of the function and the function parameters are seperated by a "|" and
-the parameters are seperated by a ','.
+The name of the function and the function parameters are separated by a "|" and
+the parameters are separated by a ','.
 It is expected that field1, field2, and field3 etc are all valid variable names or constants.
 Each non constant field will be translated from the netcdf file into value1, value2 etc. Each
 constant will be positioned as is in the output document.
@@ -355,3 +362,4 @@ Running the script is pretty straightforward. cd to the root of the vxingest loc
 ```
 
 where credentials_file is the actual credentials file (full path). The script imports the document that you checked in. Typically this script is run automatically when the ingest is triggered.
+