merge in main updates#216
Merged
kiihne-noaa merged 49 commits intoanomaly_tools_changesfrom Apr 2, 2026
Merged
Conversation
Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
… checkout installation
…enience file to mirror new lack of cruft
- Configure handlers on getLogger('epmt') instead of root logger
- Remove third-party logger suppression (matplotlib, numba, parso)
- Fix inconsistent logger names in epmt_concat.py, epmt_daemon.py,
epmtlib.py, epmt_convert_csv.py to use __name__
- Add NullHandler to epmt/__init__.py per Python best practices
- Fix test_logfn to use epmt logger instead of root logger
Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
…ean stale comments - Remove 16 stale '# you can use other name' comments across 5 files - Remove 7 redundant function-local loggers in epmt_job.py - Remove 8 redundant function-local loggers in epmt_stat.py - Remove 2 redundant function-local loggers in epmt_query.py - Remove 1 redundant function-local logger + stale comment in epmt_cmds.py - Remove 10 redundant function-local loggers in epmtlib.py - Add module-level logger to epmtlib.py, epmt_convert_csv.py, orm/op.py - Remove function-local from logging import getLogger in orm/op.py Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
… comments
- epmt_daemon.py: fix logging.root.handlers → getLogger('epmt').handlers
(handlers moved to epmt logger in previous commit, but daemon fork code
was still iterating empty root logger handlers)
- epmt_cmd_dbcare.py, epmt_daemon.py: normalize to 'from logging import
getLogger' style matching all other modules
- epmt_outliers.py, epmt_query.py: remove stale import-ordering comments
that referenced the old root-logger approach
- epmt_cmds.py, epmt_query.py: remove commented-out debug code that
bypassed the logging infrastructure
Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Convert 17 logging calls from eager string formatting (.format() and + concatenation) to lazy % formatting with comma-separated arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert 17 logging calls from .format() and string concatenation to lazy % formatting with comma-separated arguments (W1201, W1202). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert logging calls from .format(), f-strings, and % operator to lazy % formatting with comma-separated arguments (W1202, W1203). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert logging calls from eager string formatting (.format(), f-strings, % operator, string concatenation) to lazy % formatting with comma-separated arguments. This defers string interpolation until the message is actually emitted, improving performance when log levels are filtered out. Fixes W1201 (logging-not-lazy), W1202 (logging-format-interpolation), and W1203 (logging-fstring-interpolation) violations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert logging calls from .format(), % operator, and f-string interpolation to lazy % formatting with comma-separated arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert .format(), % operator, and string concatenation in logging calls to use lazy % formatting with comma-separated arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…liers.py Convert all logging calls from .format() string interpolation to lazy % formatting with comma-separated arguments, as recommended by pylint W1202. This defers string formatting until the message is actually emitted, improving performance when log messages are filtered out. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert all logging calls from .format(), f-string, %, and string concatenation to lazy % formatting per pylint rules: - logging-format-interpolation (W1202): 277 occurrences - logging-not-lazy (W1201): 37 occurrences - logging-fstring-interpolation (W1203): 15 occurrences Files fixed: epmt_outliers.py (68), epmt_stat.py (53), epmt_query.py (49), epmt_cmds.py (43), epmt_job.py (31), epmt_convert_csv.py (17), epmt_concat.py (17), epmt_daemon.py (13), general.py (12), epmtlib.py (12), epmt_exp_explore.py (6), orm/__init__.py (3), op.py (2), epmt_cmd_show.py (2), epmt_cmd_retire.py (1) Bump pylint --fail-under from 7.1 to 7.4 in build_and_test_epmt.yml. Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Code review identified 5 logging calls using %d format specifiers where the codebase convention is %s. Normalized for consistency. Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
… and fix inverted warn parameter - Remove deprecated spurious warning in delete_jobs() that fired when warn=False - Add fltr parameter to delete_jobs() to pass through to get_jobs() - In retire_jobs(), use ~Job.ref_models.any() filter to exclude model-associated jobs from both the count and each chunk query - Remove deprecated warn=False from internal retire_jobs calls Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
* pylint: enable consider-using-f-string check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_convert_csv.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_submit.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_exp_explore.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_query.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_cmds.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_daemon.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_cmds.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in orm/sqlalchemy/general.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in orm/sqlalchemy/models.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_cmd_help.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_cmd_show.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in orm/__init__.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in orm/op.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_explore.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmtlib.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_db_migration.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_rootcause.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_default_settings.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_outliers.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in test/test_lib.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_concat.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_stat.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_job.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_outliers.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix consider-using-f-string in epmt_query.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_job.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_cmds.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_exp_explore.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_convert_csv.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in orm/sqlalchemy/models.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmtlib.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_stat.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in test/test_lib.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in test/test_anysh.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_outliers.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_daemon.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in test/test_query.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in epmt_query.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in test/test_submit.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: fix line-too-long in test/test_outliers.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * pylint: bump fail-under threshold to 7.2, update README badge Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: repair indentation/syntax errors introduced by f-string conversion Fix IndentationError in orm/op.py line 74 (logger.warning under if block) and SyntaxError in orm/sqlalchemy/general.py line 612 (ternary expression parenthesization inside text() call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: wrap numpy floats with int() for :d format specifiers in epmt_exp_explore.py The f-string conversion changed '%12d' to ':12d' format specs, but numpy operations (np.sum, np.min, np.max) return float64 values. Unlike %-formatting, f-string ':d' requires actual integers. Wrap with int() to fix ValueError on float values. Fixes test_integration_explore CI failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: repair indentation of logger.warning in orm/__init__.py except block The logger.warning at line 92 was outside the except block, causing UnboundLocalError when the try block succeeded (variable 'e' only assigned in the except branch). Re-indented under except. Fixes test_dbsize_json CI failure on postgres. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase pylint fail-under threshold for epmt module Updated pylint minimum score requirement for epmt module. * Update pylint badge version in README Updated pylint badge to reflect new version. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a cleanup DELETE to remove any previously-inserted process rows for the job before re-inserting from staging. This handles re-processing after a partial failure (e.g., SLURM time limit kill) where some rows were committed but staging was not cleaned up. The cleanup runs in the same transaction as the INSERT so both are rolled back together on failure. Guarded by nprocs > 0 to prevent accidental deletion when there are no staging rows to re-insert. Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Replace j.hosts = list(hosts) with logic that only appends hosts not already associated with the job. This prevents duplicate INSERT attempts that cause psycopg2.errors.UniqueViolation when post_process_job is called on a job whose host associations were partially persisted by a prior run. Add test_reprocess_job_no_duplicate_hosts to verify the fix. Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
Add test_dbcare.py with 13 tests covering: - epmt_dbcare orchestration (skip-all, retire noop, age threshold) - retire_jobs efficiency and model pre-filter (PR #189) - retire_refmodels noop behavior - delete_jobs fltr parameter and warn spam removal (PR #189) - epmt_retire command ordering (models before jobs) - sig_handler(signo, frame) signature (PR #193) Uses in-memory SQLite with a JSONB->JSON compiler shim for fast, isolated execution (~2 seconds). Tests that depend on unmerged PRs auto-skip gracefully. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace commented-out vacuum stub in epmt_cmd_dbcare.py with working implementation using raw DBAPI autocommit connection (VACUUM cannot run inside a transaction) - Query pg_stat_user_tables for dead row stats before and after vacuum - Set max_parallel_maintenance_workers=0 for serial execution - VACUUM (VERBOSE) processes_staging, processes, jobs individually - Add 4 vacuum unit tests to test_dbcare.py (dispatch, no-engine, table list) - Register test_dbcare in meta.yaml, build_and_test_epmt.yml, docker_build_test.yml, and create_test_conda_env.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…sor already closed error In orm_raw_sql(), the connection was closed before returning the CursorResult proxy. When callers iterated over results (e.g., 20K+ staged processes), this caused psycopg2.InterfaceError: cursor already closed. Fix by calling res.fetchall() to materialize results into memory before connection.close(). Also close connection in the commit=True path which previously leaked connections. Update epmt_cmd_dbcare.py callers to work with materialized list of rows instead of CursorResult (.scalars().all() replaced with direct index/list access). Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com>
…rm_raw_sql Since orm_raw_sql() now returns a plain list via fetchall(), callers in orm/__init__.py that used .fetchall() or .fetchone() on the result fail with AttributeError. Replace: - orm_sql(cmd).fetchall()[0][0] → orm_sql(cmd)[0][0] - orm_sql(cmd).fetchone()[0] → orm_sql(cmd)[0][0] Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com> Agent-Logs-Url: https://github.com/NOAA-GFDL/epmt/sessions/ba1a0164-a880-4fc7-9d9f-da0627ee6a42
Add support for Docker builds on tag pushes.
* Initial plan * Add installation instructions to README and move detailed content to DEVELOPER.md Closes #205 - adds conda/pip/source installation instructions Closes #11 - simplifies README to a concise quick-start guide Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com> Agent-Logs-Url: https://github.com/NOAA-GFDL/epmt/sessions/d7c05a3a-fca7-4e0e-a6e6-12ece98dc840 * Rename EPMT to Experiment Process / Metadata Tool Updated README to reflect new project name and features. * complete overhaul of advancedd/developer style information * remove no longer existing sections * last tweaks --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ilaflott <6273252+ilaflott@users.noreply.github.com> Co-authored-by: Ian Laflotte <ilaflott@gmail.com>
* Update pylint thresholds in runtests script * Update pylint threshold to 8.2 * Update pylint thresholds for epmt module and ui * Refactor settings import to remove error handling Simplified the import of user-specific settings by removing the try-except block. * Remove find_diffs_in_envs function Removed the find_diffs_in_envs function from epmt_cmds.py. it's not covered, and it's not called, and everything still works! so it must not be needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Issue ticket number, link (if applicable)
Checklist
A loose guide to provide structure for contributions