Allow calling `met_db_load` as an imported module from other Python applications.

[joelheaps]

Describe the New Feature

Allow calling met_db_load as an imported module from other Python applications.

We have a use case for METdbLoad that involves installing METdataio as a package, importing METdbLoad.ush.met_db_load.main and calling it from our own Python module. (The details probably aren't relevant, but for context, we're subscribing to a message queue and loading new MET stats into the database as files arrive.)

pip installing METdataio worked well enough, but there were a few pain points when we tried to call METdbLoad.ush.met_db_load.main():

• met_db_load.main expects args to be passed directly from argparse (in dot notation), which makes it a little awkward to call.
• met_db_load.print_version looks up the package version in a way that seems to only work when invoking METdataio from a full copy of the Git repository. When METdataio is installed as a Python package, the docs/ folder containing the version info is no longer available at the expected relative path.
• A logger call in an except: block in met_db_load.py tries to reference self (though it's not part of a class/object).
• netcdf4 (or h5netcdf) and metcalcpy dependencies were missing from requirements.txt.
• Lastly, because dynamic inclusion of project requirements is disabled (commented) in pyproject.toml, metdataio dependencies aren't automatically installed when pip installing METdataio itself.

We could use subprocess to call met_db_load.py with typical CLI parameters from our app, but given that so many pieces seem to already be in place for a direct import to work, it seemed worthwhile to open an issue and submit a PR. It's somewhere between a bug report and feature request, so feel free to relabel it if need-be. Welcome any other feedback or critiques as well (this is my first time contributing here).

I've opened a draft PR with some relevant changes.

Acceptance Testing

Our desired workflow is essentially the following:
Install metdataio and its subpackages (this part already works, except for the runtime issues noted above).

pip install git+https://github.com/dtcenter/METdataio.git@main_v3.0
# or a 'git clone https://github.com/dtcenter/METdataio.git' followed by a 'cd METdataio && pip install .'

Import the DB loader and call it against a load spec XML.

from METdbLoad.ush.met_db_load import main as load_met_data

load_met_data(xmlfile, index=False)

Time Estimate

<1 day

Sub-Issues

• Implement keyword argument support in METdbLoad.met_db_load.main.
• Re-enable inclusion of dependencies from requirements.txt in pyproject.toml

Relevant Deadlines

None

Funding Source

None

Define the Metadata

I'm not quite sure how best to fill out this section.

Assignee

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as a METdataio-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a METdataio-X.Y.Z version, select the METdataio-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: METdataio-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[bikegeek]

Could you please provide us with your affiliation (e.g. NOAA, NASA, METoffice, etc.)?

[joelheaps]

Air Force Weather