Skip to content

Provenance is incomplete when using --parallel and warning for subprocesses #1348

Open
@pagecp

Description

@pagecp

Expected Behavior

When running with --parallel, the provenance information should be complete, as it is when not using --parallel.

Actual Behavior

No information is generated on subprocesses.
Warnings are generated:
INFO [workflow ] start
INFO [workflow ] starting step create_environment
INFO [step create_environment] start
INFO [job create_environment] Output of job will be cached in /home/mpiuser/sfs/d4p/7/.cache/f976e771472a74d2ed9ade9537ed2de2
INFO [step create_environment] start
INFO [job create_environment] /home/mpiuser/sfs/d4p/7/.cache/f976e771472a74d2ed9ade9537ed2de2$ sh
/tmp/tmp5f8v3e01/stga713749b-586f-42fe-a7a1-b7c95a144fe7/env_preparation.sh
/tmp/tmp5f8v3e01/stga713749b-586f-42fe-a7a1-b7c95a144fe7/cyclone_workflow_git/input_files.txt
WARNING research_obj set but one of process_run_id or prov_obj is missing from runtimeContext: <cwltool.context.RuntimeContext object at 0x7f1d6cac9510>
INFO [job create_environment_2] Output of job will be cached in /home/mpiuser/sfs/d4p/7/.cache/a9163b5e721c8920a6259c698cf9c328
/tmp/tmp5f8v3e01/stga713749b-586f-42fe-a7a1-b7c95a144fe7
/home/mpiuser/sfs/d4p/7/.cache/f976e771472a74d2ed9ade9537ed2de2
INFO [step create_environment] start
INFO [job create_environment_3] Output of job will be cached in /home/mpiuser/sfs/d4p/7/.cache/3a905e249cd7ce79270656bd425419a6
INFO [job create_environment] completed success
INFO [job create_environment_2] /home/mpiuser/sfs/d4p/7/.cache/a9163b5e721c8920a6259c698cf9c328$ sh
/tmp/tmpz1f0q5ar/stg3e01e411-b03b-4a65-983e-98f78454f395/env_preparation.sh
/tmp/tmpz1f0q5ar/stg3e01e411-b03b-4a65-983e-98f78454f395/cyclone_workflow_git/input_files2.txt
WARNING research_obj set but one of process_run_id or prov_obj is missing from runtimeContext: <cwltool.context.RuntimeContext object at 0x7f1d6cac9510>
/tmp/tmpz1f0q5ar/stg3e01e411-b03b-4a65-983e-98f78454f395
/home/mpiuser/sfs/d4p/7/.cache/a9163b5e721c8920a6259c698cf9c328
INFO [job create_environment_3] /home/mpiuser/sfs/d4p/7/.cache/3a905e249cd7ce79270656bd425419a6$ sh
/tmp/tmpm7zyftu7/stgd32cbf80-56b8-47bb-ae4b-38a0901ea738/env_preparation.sh
/tmp/tmpm7zyftu7/stgd32cbf80-56b8-47bb-ae4b-38a0901ea738/cyclone_workflow_git/input_files3.txt
WARNING research_obj set but one of process_run_id or prov_obj is missing from runtimeContext: <cwltool.context.RuntimeContext object at 0x7f1d6cac9510>
INFO [job create_environment_2] completed success
/tmp/tmpm7zyftu7/stgd32cbf80-56b8-47bb-ae4b-38a0901ea738
/home/mpiuser/sfs/d4p/7/.cache/3a905e249cd7ce79270656bd425419a6
INFO [job create_environment_3] completed success
INFO [step create_environment] completed success
(...)

Full Traceback

see attached log file (too large to paste)
logs.txt

Your Environment

  • cwltool version:
    /usr/local/bin/cwltool 3.0.20200807132242

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions