Nexus: add error screening functionality#6056
Conversation
|
Aside: What are your thoughts on handling errors such as when a submitted slurm job has simply disappeared and we don't have any error or output files? e.g. Sysadmin flushes all jobs from slurm queue. What parts of Nexus could catch that / should already catch that? |
brockdyer03
left a comment
brockdyer03
left a comment
There was a problem hiding this comment.
This is an incomplete review, but I figured I'd give a few bits of feedback while I had time.
| return pattern | ||
|
|
||
|
|
||
| @lru_cache(maxsize=None) |
There was a problem hiding this comment.
This should get replaced with functools.cache which provides the same functionality and is more common.
| ... ) | ||
| True | ||
| >>> find_error_keys( | ||
| ... "step 1\\nLAPACK computational failure: ZHEEV did not converge", |
There was a problem hiding this comment.
Would be more readable as
'''
step 1
LAPACK computational failure: ZHEEV did not converge
'''| if not return_lines: | ||
| return pattern.search(text) is not None | ||
|
|
||
| lines = [line for line in text.splitlines() if pattern.search(line)] |
There was a problem hiding this comment.
Explicit is better than implicit
lines = [
line for line in text.splitlines() if pattern.search(line) is not None
]| return pattern.search(text) is not None | ||
|
|
||
| lines = [line for line in text.splitlines() if pattern.search(line)] | ||
| return bool(lines), lines |
There was a problem hiding this comment.
Also here:
return (
len(lines) > 0,
lines
)
|
@prckent Nexus queries the Slurm queue each poll. When a process drops out of the Slurm queue, the job is marked as finished and Simulation.check_sim_status is executed next. That function should check if all files exist, etc. If not, it should be marked as a failure (mostly is, I believe). For simulations exercising patience, a timeout limit should be added. This is on my todo list. |
|
I recently authored a PR that added error checking to the PySCF If the intention is to replace it, you'll need to take the error types that I pulled out of PySCF and put them in here. |
3 participants
This PR adds precursor functionality to screen for simulation run failures. Failure types come from various sources: operating system, filesystem, mpi, fortran/c++/cuda/hip runtime, compiled libraries, python, python modules, and error patterns specific to individual simulation codes.
In
error_keys.pythefind_error_keysfunction can be used to screen for errors from a target (.out/.err) file or text block. The types of errors being screened for can be selected flexibly. Regex is used extensively for efficiency. Can also output individual lines that contain error matches, but this should be much slower.The eventual intention is to incorporate this function into the
check_sim_statusmember function for each simulation code. The failure detection is meant to be a backup if success is not directly determined by inspecting e.g. files produced by simulation runs. For codes like pyscf, this will likely be the first line of defense since success is nearly impossible to gauge directly due to the complete flexibility allowed by the input script.This PR was made with assistance from GPT-5.6 Sol.