Problems with subworkflows run iteratively

[agnieszkapiecuch]

I tried this:

I tried running main workflow with subworkflow.
I have prepared some mocks to reproduce issues.
My main workflow(mock-create-assessment-main.yaml) triggers subworkflow (mock-create-assessment-sub.yaml)
And it does it 200 times (there is lop "for each")
I have set: SYNAPSE_RUNNER_LIFECYCLE_EVENTS: 'false'

Additionally, I created mock "mock-empty-sub" (subworkflow) when this subworkflow does nothing and produces no data to see if in that scenario problems appears as well (it does)

document:
  dsl: 1.0.0
  namespace: default
  name: mock-empty-sub
  version: 0.1.0
do:
- doNothing:
    wait:
      seconds: 2
    output:
      as: empty

document:
  dsl: 1.0.0
  namespace: default
  name: mock-create-assessment-main
  version: 0.1.0
do:
- setContext:
    set:
      currentDateTime: ${if ($input.data.currentDateTime|length > 0) then $input.data.currentDateTime else now | strftime("%Y-%m-%d_%H-%M-%S") end}
    export:
      as:
        errors: []
        destinationPathClaims: ${"claims/"+ .currentDateTime +"/claims.json"}
        destinationPathErrors: ${"claims/"+ .currentDateTime +"/errors.json"}
        files:
        - ES-3008640857
        - PL-812436D4-E51D-B1C7-976E-6867AECEC275
        - ES-3016111707
        - ES-3013601732
        - GB-D0189929UKAENG1
        - ES-3015274267
        - GB-PW13641VBR004SN
        - GB-R0368684RSARCV2
        - ES-3018126980
        - GB-AA037155AAUICL
        - PL-296CC0BC-F956-F3B2-54BF-904DB9A7573A
        - ES-3012614806
        - GB-LC001017VBR050PO
        - ES-3012493454
        - GB-CL19187VBR016AB
        - DE-F2DEBDCE-40F1-D9DC-87A4-50E52390ABF5
        - PL-15A02684-425B-585F-23D8-42FD2A952733
        - CH-421827D6-C958-2926-E7F8-411AAD8B88A1
        - GB-V0192235VIZION
        - ES-3017224527
        - PL-75C0DFF9-B369-0D36-43F9-34A2F2581741
        - GB-AD186755ADMIRECV
        - GB-W0020549AVIINS
        - PL-9184F0CC-7295-9A7F-EFBB-B9B3197FB956
        - PL-1DAE26A3-D2E0-4BE2-51DD-4E8FC5B7579D
        - PL-526CAA9D-9B72-8C8F-9A75-EB9D4F81A146
        - PL-006D3418-C0E7-9E02-8C9B-8E347A540A58
        - DE-32928CD0-95A1-90F3-FDC3-355595306F6C
        - GB-V0430699VIZION
        - GB-B0030727AVIINS
        - ES-3014968405
        - ES-3016055159
        - GB-N0097832AVIINS
        - ES-3012597589
        - ES-3015746786
        - GB-J0149215AVIINS
        - GB-LP383054LVRCV
        - DE-FD6CF7AB-9B2B-FB57-72E7-B04ECF6EC4DD
        - PL-9BAF05B7-2179-41FF-B947-F5C49CBD23BC
        - PL-16C99823-D04B-A843-8D5C-177A0C4C55EF
        - ES-3014800470
        - ES-3015043315
        - GB-EU418894ESURE3
        - ES-3016222784
        - GB-V0461733VIZION
        - DE-321020DE-4653-E1F4-63C3-D8D0B453F4FE
        - CH-B6AEAF29-9F1C-F250-0F2D-EED736E33312
        - ES-3013834393
        - ES-3016505101
        - GB-MS282587LINKZEN
        - PL-22D35859-F769-7C8F-7B1A-75ABB452ABCB
        - GB-N0087628AVIINS
        - PL-764FDC7F-B947-EA0E-26AD-0E7159544B8E
        - ES-3012525781
        - GB-ARC16171VBRA33BT
        - GB-V0415959VIZION
        - GB-AD155131ADMIRECV
        - PL-1FA1E913-EE5D-5851-B4A7-07E0AC2583A4
        - GB-R0348435RSARCV2
        - PL-00916888-FCDA-1765-58E1-140CD77904C7
        - PL-48F88844-2994-C123-948B-F475B76013A4
        - GB-B0712269AMCRE01
        - ES-3017831250
        - GB-PV025509PARKRCV
        - CH-10554373-3441-D974-1E20-1F470F4AAD0F
        - ES-3016150800
        - ES-3018304676
        - ES-3012726273
        - GB-EU435718ESURE3
        - PL-6CFD9C01-70E3-3387-F8FD-12B39A9213C4
        - ES-3015626409
        - PL-FC595639-FD39-1202-F72D-6AAC947A8A31
        - GB-IM338477INCIDENT
        - ES-3016536469
        - ES-3016134094
        - ES-3016180074
        - GB-SG009052SANDGRCV
        - ES-3017087313
        - ES-3017845285
        - DE-6AFCA8FD-1E94-C700-7557-3B39AAC4FDD9
        - PL-B944C4C8-D973-5953-EA37-2E1F5053F0E8
        - GB-M0143555AVIINS
        - PL-E1B1304D-8E4E-53AB-C6C1-9608BCD7292A
        - GB-B0785672AMCRE01
        - GB-AD068674ADMIRECV
        - GB-V0431504VIZION
        - ES-3012454471
        - GB-IM451133INCIDENT
        - GB-B1008178AMCRE01
        - ES-3012496413
        - GB-CL18523VBR012CT
        - ES-3018104269
        - GB-D0164335UKAENG1
        - GB-JM004849VBR020BT
        - GB-AUK09323VBRA38UB
        - ES-3018724840
        - ES-3016302167
        - PL-4FF7341F-24E9-17A9-00E8-440AE5BF455A
        - GB-B0815565AMCRE01
        - GB-MS246165LINKZEN
        - ES-3016303257
        - GB-PB029551VBR051RG
        - ES-3017991909
        - ES-3016898069
        - GB-MAP00168VBR008AL
        - GB-LP903789LVRCV
        - PL-F11D7DF9-EF17-BC61-99DD-6193F80A84F8
        - ES-3016013222
        - ES-3016094581
        - PL-5518C711-C241-E538-7C77-A5D4D3650C4C
        - GB-RT001910VBR014TS
        - GB-HU000037VBR001EW
        - PL-35871396-7DB3-DEDF-55D0-B7AE2198CB95
        - GB-IM434540INCIDENT
        - PL-3A96A6E0-43F7-A9E4-535C-3D397B773A25
        - GB-B0717952AMCRE01
        - DE-C7F5DD65-1073-686C-69E6-88D33C834498
        - PL-9468925C-4538-C547-2864-654A6DB64455
        - ES-3012387013
        - GB-JC000164VBR045SN
        - PL-A0BA1D1C-878A-3997-2108-D6BB4B8E80BB
        - ES-3015886771
        - ES-3017698342
        - ES-3017163495
        - GB-SG006264VBRA56RG
        - GB-J0148796AVIINS
        - ES-3014822194
        - PL-BE1482CA-74D5-185F-71FE-2478C061614F
        - GB-LP913266LVRCV
        - GB-RXAB1002VBR024DA
        - GB-MS230903LINKZEN
        - ES-3016304253
        - PL-4A783A5B-1E1C-75D0-C3DF-764A189FA5E7
        - ES-3016203600
        - ES-3014666649
        - GB-LP280962LVRCV
        - GB-MJA08479VBR013ME
        - PL-6B046E61-CCE6-29CA-3505-9931E8E18A7C
        - GB-CV143251ADMIRECV
        - GB-LP862822LVRCV
        - ES-3017156768
        - GB-A0062813AVIINS
        - ES-3017460301
        - DE-C34B3989-4589-FCDB-F652-9696D8567897
        - GB-MJR11611VBRA40NP
        - GB-MS281167LINKZEN
        - GB-V0389544VIZION
        - GB-V0176621VIZION
        - ES-3016173879
        - ES-3013546956
        - GB-LP877290LVRCV
        - PL-85485365-0FE3-D77F-BA1D-975B7005A009
        - ES-3011672463
        - CH-226400E6-00FF-D95F-7511-41FF85588CD8
        - ES-3008298055
        - GB-CV181177ADMIRECV
        - PL-2FC6C293-AB8E-1996-559B-33F54F2AD577
        - PL-31255118-F8CF-886A-C202-2BCAB56D193C
        - PL-176553DD-26D9-5FCC-1510-C745B9A6C799
        - PL-F4D9E338-1EE4-A9BE-B46D-0BE275EBB889
        - PL-040AAD64-67AA-1868-363D-B6077BF24053
        - GB-D0197046UKAENG1
        - GB-V0307516VIZION
        - ES-3013085286
        - GB-ASL00462VBR040LS
        - PL-A05EFB85-9282-5FF0-9EE9-EDE8A8785E4D
        - ES-3017579940
        - ES-3018207569
        - GB-IA001451VBR062DN
        - DE-73021808-D6F4-DCD9-606E-AB2560FECC97
        - ES-3013526572
        - GB-EL27535VBR005AL
        - ES-3015686898
        - GB-LP720243LVRCV
        - PL-2D24CD99-8844-CBA5-E582-60A545F48645
        - ES-3014922568
        - GB-B0898813AMCRE01
        - ES-3016536721
        - ES-3017773387
        - GB-LC000136VBRA01CM
        - PL-6C76F4F5-7D50-017D-BB17-179FE9FCF418
        - GB-SG005480SGINSTRN
        - PL-63EDD42B-FAB0-00DF-0D77-2D886EE6FF0B
        - ES-3016905871
        - PL-D96E0638-02FA-2534-261F-6AEF12B1FEBB
        - FR-FR84122646638
        - GB-TP041927ADMIRECV
        - PL-899F2DCE-5215-F6F2-25D5-F5CF9CC7E1A2
        - ES-3017507295
        - DE-263866B3-F283-8734-138C-4CB4BBA774E8
        - ES-3016812475
        - PL-C130195D-C3E3-D2A2-1CC9-AAF7A6B9D836
        - GB-AD000011PAYG1096
        - ES-3014735988
        - DE-DFEFBC22-CE4A-451D-9F1F-910AF9011859
        - ES-3015700204
        - PL-E5048881-8B8F-9E58-EEA8-3A5D8252EA61
        - ES-3018709330
        - GB-R0244837RSARCV2
        - GB-GM007979VBR039NG
        - ES-3013646971
- createClaims:
    for:
      each: file
      in: ${$context.files}
    do:
    - refreshAzureToken:
        if: ($index % 40 == 0 or $index == 0)
        set:
          not_im: not_im
        output:
          as: empty
        export:
          as: '$context + {"azureToken": "azuretoken111111111111111111111111111111111111111"}'
    - tryCreateClaimSubWorkflow:
        try:
        - createClaimSubWorkflow:
            run:
              workflow:
                namespace: default
                name: mock-create-assessment-sub
                version: 0.1.0
                input:
                  access:
                    username: username
                    password: pass
                    orgId: orgId
                    language: ln
                    countryCode: AA
                    createAssessmentEndpoint: createAssessmentEndpointcreateAssessmentEndpoint
                    generateImageRefsEndpoint: generateImageRefsEndpointgenerateImageRefsEndpoint
                    createTaskEndpoint: createTaskEndpointcreateTaskEndpoint
                    addImageToAssessEndpoint: addImageToAssessEndpointaddImageToAssessEndpoint
                    gatewayToken: gatewayTokengatewayTokengatewayTokengatewayTokengatewayTokengatewayToken
                  data:
                    bucket: myBucket
                    projectClaimData: projectClaim
                    project: myProject
                  azureToken: azureTokenazureTokenazureToken
                  file_path: ${"claimsDataSummaryPathclaimsDataSummaryPath/"+$file}
                  destinationPathErrors: ${$context.destinationPathErrors}
                  destinationPathClaims: ${$context.destinationPathClaims}
                  index: ${$index}
        catch:
          errors: {}
          as: error
          do:
          - addErrorFromWorkflow:
              set:
                error: ${$error}
              output:
                as:
                  errors: ${$context.errors + [.error]}
              export:
                as: '$context + {errors: .errors}'
- addToFileError:
    if: $context.errors| length > 0
    wait:
      seconds: 1
    output:
      as: empty
- setOutput:
    set:
      claimsFile: ${$context.destinationPathClaims}

document:
  dsl: 1.0.0
  namespace: default
  name: mock-create-assessment-sub
  version: 0.1.0
do:
- downloadClaimData:
    set:
      not_im: not_im
    output:
      as: empty
    export:
      as:
        claimData:
          internal_case_number: GB-AD186755ADMIRECV
          country: AA
          images:
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-0F037F505667344F83D4AC2C05D81171-7.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-0FA0012500B7154AB254C52B3CDEE2F4-19.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-1B67B92317298440AC69C1D47F0583FA-12.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-2DAB0AC37EFFD14CB011FA7CF60DDDBB-22.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-4015198C55D9DA499967B89C98474A35-5.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-498466B428FBF7418D23BADACF125E38-13.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-595D3B468089894C90DBD18AC013E1BD-8.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-5B50B28D9D946C48BCFF99FF6D16DD5A-21.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-5CE4BAE2197C464181A2112689103080-6.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-6361C26F008AA04290386697EA3E4354-20.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-7928C2E45F71CB46864173D79772318C-15.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-7ABCFE87D1773840AC746F363CD60E98-3.jpg
          - ingestion/GB/GB-AD186755ADMIRECV/GB-AD186755ADMIRECV-BFFCFBE6B0A4494DAB463598E4612F0B-7DA39B46A0BC934FBBF1EC58CB48CC5B-10.jpg
          bucket: some_bucket
          vehicle:
            manufacturerCodeAX: "00"
            modelCodeAX: "66"
            submodelCodeAX: "21"
            manufacturerName: myCar
            modelName: myModel
            submodelName: mySubmodel
            buildYear: "2015"
            modelOptions:
            - C2
            - C3
            - C5
            - C6
            - C7
            - C8
            - D2
            - D8
            - E2
            - E9
            - F2
            - F5
            - F6
            - F8
            - G1
            - G6
            - G7
            - G8
            - H1
            - H3
            - H4
            - H8
            - J6
            - J7
            - K1
            - K8
            - L2
            - L3
            - L7
            - M3
            - M4
            - N7
            - P6
            - P8
            - Q1
            - Q2
            - Q9
            - R4
            - T9
            - U1
            - U3
            - U8
            - V4
- createClaim:
    try:
    - createTask:
        set:
          taskId: ${$workflow.input.index|tostring+"3FDEAF52-481E-7AD2-E47E-696D6E6C4AF8"}
          caseId: ${$workflow.input.index|tostring+"77DEAF52-481E-7AD2-E47E-696D6E6C4AF8"}
    - postCreateTask:
        set:
          gatewayToken: ${$workflow.input.access.gatewayToken}
          createAssessmentEndpoint: ${$workflow.input.access.createAssessmentEndpoint +"/" + $input.taskId }
        export:
          as: '$context  +  { "claim": {"taskId": $input.taskId, "caseId":  $input.caseId} }  '
    - tryCreateAssessment:
        try:
        - createAssessment:
            wait:
              seconds: 1
            output:
              as: empty
        catch:
          errors:
            with:
              status: 503
    - preGenerateImageRefs:
        set:
          azureToken: ${$workflow.input.azureToken}
    - generateImageRefs:
        set:
          not_im: not_im
        output:
          as:
            refs:
            - id: 954e02ac-bfa8-4f57-9120-2af0124e3d1f
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.320530591Z
              status: CREATED
            - id: 5ff2801c-e1e2-4e5a-862d-9849c30e5f80
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.323678870Z
              status: CREATED
            - id: b8445a37-6cbe-42e8-a517-dbcd02538bf4
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.321384269Z
              status: CREATED
            - id: 5529e4b5-f24e-4db5-8654-487361d68d54
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.321853378Z
              status: CREATED
            - id: e05a772d-51ba-4bac-a8c7-3be48d9c3dcb
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.323266790Z
              status: CREATED
            - id: d048cae9-9a15-4df5-8acc-1deeeda228ad
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.324247749Z
              status: CREATED
            - id: a2c8dc50-fa96-4b94-bbb4-028c3e7cc1d0
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.325268809Z
              status: CREATED
            - id: 2f650f77-eb7a-4a84-8822-360976f41e38
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.325757984Z
              status: CREATED
            - id: 99db4dc2-7f98-4c06-b94b-f6c155409f1d
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.326373623Z
              status: CREATED
            - id: f6a70893-3eb8-4ce5-8043-22cdcecc03a0
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.326715156Z
              status: CREATED
            - id: dd69dbc8-8324-4f9c-9b85-92ecbe9d9a77
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.328155830Z
              status: CREATED
            - id: fbcad077-375f-4df7-94a9-a148b4f3462f
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.328653624Z
              status: CREATED
            - id: 7d229a1c-6710-45ef-a55f-274da3836838
              imageUploadUrl:
                url: "22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222"
                expiresAt: 2025-09-08T09:32:15.329134396Z
              status: CREATED
    - preUpload:
        set:
          gatewayToken: ${$workflow.input.access.gatewayToken}
          addImageToAssessEndpoint: ${$workflow.input.access.addImageToAssessEndpoint + "/" + $context.claim.taskId + "/images/globalId"}
          refs: ${$input.refs}
    - uploadImages:
        for:
          each: imageRef
          in: ${$input.refs}
          at: index
        do:
        - uploadImage:
            set:
              not_im: not_im
            output:
              as: ${{"gatewayToken":$input.gatewayToken} + {"addImageToAssessEndpoint":$input.addImageToAssessEndpoint}}
        - tryAddImageToAssessment:
            try:
            - addImageToAssessment:
                wait:
                  seconds: 1
                output:
                  as: ${{"gatewayToken":$input.gatewayToken} + {"addImageToAssessEndpoint":$input.addImageToAssessEndpoint}}
            catch:
              errors:
                with:
                  status: 503
    - addToFileCreatedClaim:
        wait:
          seconds: 1
    catch:
      errors: {}
      as: error
      do:
      - addToFileError:
          wait:
            seconds: 1

This happened:

1. Cancellation error
   This happens in different situations, I tried to associate it with memory consumption, but it happened with different sizes, here example with quite big NET I/O but it is not the rule

      qapter-batch-operator-1    | Unhandled exception. System.ObjectDisposedException: The CancellationTokenSource has been disposed.
      qapter-batch-operator-1    |    at Synapse.Operator.Services.WorkflowInstanceHandler.OnPersistLogBatchAsync()
      qapter-batch-operator-1    |    at Synapse.Operator.Services.WorkflowInstanceHandler.<StartProcessAsync>b__46_0(Object _) in /src/src/operator/Synapse.Operator/Services/WorkflowInstanceHandler.cs:line 140
      qapter-batch-operator-1    |    at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
      qapter-batch-operator-1    |    at System.Threading.ThreadPoolWorkQueue.Dispatch()
      qapter-batch-operator-1    |    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
      qapter-batch-operator-1 exited with code 139
   
    Container stats:       
    
              CONTAINER ID   NAME                                                                 CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
              c0d9c449a986   local-qapter-batch-operator-1                                        3.49%     158.9MiB / 7.666GiB   2.02%     3.7GB / 139MB     1.81MB / 0B       26
              85275f5aed42   local-api-1                                                          0.18%     298.3MiB / 7.666GiB   3.80%     2.32GB / 4.74GB   9.03MB / 12.3kB   41
              9b4b2bcb069c   local-redis-1                                                        0.73%     68.9MiB / 7.666GiB    0.88%     1.89GB / 7.46GB   0B / 130MB        5
              4bea3b5c9edf   mock-create-assessment-main-56f639911303.default-ab2f6076d872        0.10%     772.8MiB / 7.666GiB   9.84%     1.58GB / 10.2MB   12.2MB / 0B       500
   

2. Timeout awaiting response

   A.
   In this case subworkflow ended OK(after 5 s), however in the main flow it was reported as timeout and main workflow got stuck as shown on the picture1

    type: https://serverlessworkflow.io/dsl/errors/types/runtime
                title: Runtime Error
                status: 500
                detail: '[500 - Runtime Error] Timeout awaiting response (outbound=0KiB, inbound=0KiB, 6590ms elapsed, timeout is 5000ms), command=HEXISTS, next: HEXISTS workflow-instances.synapse.io, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 4, in-pipe: 0, out-pipe: 0, last-in: 258507, cur-in: 0, sync-ops: 1, async-ops: 24377, serverEndpoint: redis:6379, conn-sec: 1616.24, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: cfe2dbd60e4f(SE.Redis-v2.8.16.12844), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=12,Free=32755,Min=8,Max=32767), POOL: (Threads=23,QueuedItems=5,CompletedItems=95562,Timers=6), v: 2.8.16.12844 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)'
                instance: /do/1/createClaims/for/95/do/1/tryCreateClaimSubWorkflow/try/0/createClaimSubWorkflow

Container stats:

  CONTAINER ID   NAME                                                             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
  c78b556a4c81   local-qapter-batch-operator-1                                    4.65%     114.9MiB / 7.666GiB   1.46%     225MB / 15.4MB   0B / 0B          27
   cfe2dbd60e4f   local-api-1                                                     0.23%     223MiB / 7.666GiB     2.84%     547MB / 900MB    238kB / 12.3kB   41
  1726e4bff277   local-redis-1                                                    0.92%     15.12MiB / 7.666GiB   0.19%     405MB / 1.13GB   0B / 4.85MB      5
  d0d4dbafc1b0   mock-main-workflow-empty-sub-157c98c3f3c9.default-adc19620fb57   0.19%     806.3MiB / 7.666GiB   10.27%    516MB / 6.02MB   0B / 0B          493

B. In this case main workflow failed with error

      api-1                      |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
      api-1                      | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      api-1                      |       An unhandled exception has occurred while executing the request.
      api-1                      |       System.Exception: An error occurred while committing pending transactions. Changes have been rolled back.
      api-1                      |        ---> StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 9325ms elapsed, timeout is 5000ms), command=HSET, next: HSET default, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 4, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 1, async-ops: 25582, serverEndpoint: redis:6379, conn-sec: 2046.36, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: 8cf3f0dcaa3a(SE.Redis-v2.8.16.12844), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=12,Free=32755,Min=8,Max=32767), POOL: (Threads=21,QueuedItems=3,CompletedItems=113640,Timers=6), v: 2.8.16.12844 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
      api-1                      |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.RedisDatabase.WriteResourceAsync(String group, String version, String plural, Resource resource, Boolean specHasChanged, ResourceWatchEventType operationType, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Redis/Services/RedisDatabase.cs:line 446
      api-1                      |          --- End of inner exception stack trace ---
      api-1                      |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.RedisDatabase.WriteResourceAsync(String group, String version, String plural, Resource resource, Boolean specHasChanged, ResourceWatchEventType operationType, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Redis/Services/RedisDatabase.cs:line 454
      api-1                      |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.RedisDatabase.PatchSubResourceAsync(Patch patch, String group, String version, String plural, String name, String subResource, String namespace, String resourceVersion, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Redis/Services/RedisDatabase.cs:line 259
      api-1                      |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.ResourceRepository.PatchSubResourceAsync(Patch patch, String group, String version, String plural, String name, String subResource, String namespace, String resourceVersion, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented/Services/ResourceRepository.cs:line 236
      api-1                      |          at Neuroglia.Data.Infrastructure.ResourceOriented.IResourceRepositoryExtensions.PatchStatusAsync[TResource](IResourceRepository repository, Patch patch, String name, String namespace, String resourceVersion, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Abstractions/Extensions/IResourceRepositoryExtensions.cs:line 265
      
      

  Stats before crash:
  Wed Sep 10 09:36:20 CEST 2025
   total        used        free      shared  buff/cache   available
  Mem:           7.7Gi       3.5Gi       2.5Gi       182Mi       2.1Gi       4.2Gi
  Swap:          2.0Gi          0B       2.0Gi


      CONTAINER ID   NAME                                                             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O     PIDS
      7480ae3e876b   local-qapter-batch-operator-1                                  178.27%   125.6MiB / 7.666GiB   1.60%     226MB / 15.1MB   0B / 0B       30
      8cf3f0dcaa3a   local-api-1                                                    409.53%   423.4MiB / 7.666GiB   5.39%     602MB / 1.13GB   0B / 12.3kB   63
      8f6d0356a111   local-redis-1                                                    1.11%     16.04MiB / 7.666GiB   0.20%     434MB / 1.21GB   0B / 4.8MB    5
      1cd67a5de42f   mock-main-workflow-empty-sub-14afaffdb924.default-95cee0cc5d0b   0.21%     823.3MiB / 7.666GiB   10.49%    568MB / 6.45MB   0B / 0B       518



 Stats after crash:
 Wed Sep 10 09:38:27 CEST 2025
   total        used        free      shared  buff/cache   available
  Mem:           7.7Gi       2.6Gi       3.4Gi       147Mi       2.1Gi       5.1Gi
  Swap:          2.0Gi          0B       2.0Gi


   
    CONTAINER ID   NAME                                                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O     PIDS
    7480ae3e876b   local-qapter-batch-operator-1                                      3.57%     126MiB / 7.666GiB     1.60%     237MB / 15.6MB   0B / 0B       30
    8cf3f0dcaa3a   local-api-1                                                      549.90%   330.9MiB / 7.666GiB   4.22%     631MB / 1.17GB   0B / 12.3kB   58
    8f6d0356a111   local-redis-1                                                      1.24%     16.12MiB / 7.666GiB   0.21%     454MB / 1.27GB   0B / 4.8MB    5
    1cd67a5de42f   mock-main-workflow-empty-sub-14afaffdb924.default-95cee0cc5d0b     0.00%     0B / 0B               0.00%     0B / 0B          0B / 0B       0

3. With the properly working operator, anyway from time-to-time subworkflow got stuck with status “pending” and freezes main workflow.

4. During processing run time of each subworkflow increases which makes it no efficient. (I guess it is due to increasing NET I/O (both main workflow and operator))
   Here some data

   Wed Sep 10 10:58:30 CEST 2025

   total used free shared buff/cache available
   Mem: 7.7Gi 2.5Gi 3.4Gi 189Mi 2.2Gi 5.1Gi
   Swap: 2.0Gi 0B 2.0Gi

    CONTAINER ID   NAME                                                            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
    2babd9348656   local-qapter-batch-operator-1                                   2.44%     84.88MiB / 7.666GiB   1.08%     4.59MB / 1.03MB   0B / 0B           27
    4d17f71942fd   local-api-1                                                    57.52%    145.1MiB / 7.666GiB   1.85%     13.6MB / 29.4MB   12.3kB / 12.3kB   40
    5c1b4983fd39   local-redis-1                                                   2.71%     9.84MiB / 7.666GiB    0.13%     12.5MB / 27MB     0B / 0B           5
    d3ec856f72c2   mock-create-assessment-main-09ff4ef16d10.default-4de93a62b11c   2.24%     64.18MiB / 7.666GiB   0.82%     7.79MB / 99.7kB   0B / 0B           19
   

   Wed Sep 10 13:58:03 CEST 2025

   total used free shared buff/cache available
   Mem: 7.7Gi 4.5Gi 1.1Gi 211Mi 2.6Gi 3.2Gi
   Swap: 2.0Gi 0B 2.0Gi

   CONTAINER ID   NAME                                                            CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
   2babd9348656   local-qapter-batch-operator-1                                   65.82%    262.3MiB / 7.666GiB   3.34%     2.64GB / 167MB    0B / 0B           31
   4d17f71942fd   local-api-1                                                     459.59%   520MiB / 7.666GiB     6.62%     3.45GB / 6.88GB   12.3kB / 12.3kB   75
   5c1b4983fd39   local-redis-1                                                   1.10%     88.04MiB / 7.666GiB   1.12%     2.56GB / 8.19GB   0B / 313MB        5
   d3ec856f72c2   mock-create-assessment-main-09ff4ef16d10.default-4de93a62b11c   0.94%     1.223GiB / 7.666GiB   15.96%    2.43GB / 15.2MB   0B / 0B           681
   ### I expected this:
   

No response

Is there a workaround?

No response

Anything else?

Additionally, with ever - also properly finished main (as an example: mock-main-workflow-OK-empty-sub.yaml - only two iterations) and subworkflows - I'm getting in logs errors; however, it seems not to harm:

api-1                      | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
api-1                      |       An unhandled exception has occurred while executing the request.
api-1                      |       Neuroglia.ProblemDetailsException: [404 - Not Found] Failed to find the specified resource 'synapse.io/v1/namespace/default/workflow-instances/mock-empty-sub-be7679ba593243f8a0b5011f7038f972'.
api-1                      |          at Synapse.Api.Application.Queries.Resources.Generic.GetResourceQueryHandler`1.HandleAsync(GetResourceQuery`1 query, CancellationToken cancellationToken) in /src/src/api/Synapse.Api.Application/Queries/Resources/Generic/GetResourceQuery.cs:line 62
api-1                      |          at Neuroglia.Mediation.RequestPipeline`2.HandleAsync(IRequest`1 request, IServiceProvider serviceProvider, CancellationToken cancellationToken)
api-1                      |          at Neuroglia.Mediation.Mediator.ExecuteAsync[TResult](IRequest`1 request, CancellationToken cancellationToken)
api-1                      |          at Synapse.Api.Http.NamespacedResourceController`1.GetResource(String name, String namespace, CancellationToken cancellationToken)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
api-1                      |          at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
api-1                      |          at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
api-1                      |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
api-1                      | fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
api-1                      |       An unhandled exception has occurred while executing the request.
api-1                      |       Neuroglia.ProblemDetailsException: [404 - Not Found] Failed to find the specified resource 'synapse.io/v1/namespace/default/workflow-instances/mock-empty-sub-675e6658262c50a68ca7b20e42d76798'.
api-1                      |          at Synapse.Api.Application.Queries.Resources.Generic.GetResourceQueryHandler`1.HandleAsync(GetResourceQuery`1 query, CancellationToken cancellationToken) in /src/src/api/Synapse.Api.Application/Queries/Resources/Generic/GetResourceQuery.cs:line 62
api-1                      |          at Neuroglia.Mediation.RequestPipeline`2.HandleAsync(IRequest`1 request, IServiceProvider serviceProvider, CancellationToken cancellationToken)
api-1                      |          at Neuroglia.Mediation.Mediator.ExecuteAsync[TResult](IRequest`1 request, CancellationToken cancellationToken)
api-1                      |          at Synapse.Api.Http.NamespacedResourceController`1.GetResource(String name, String namespace, CancellationToken cancellationToken)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
api-1                      |          at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
api-1                      |          at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
api-1                      |          at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
api-1                      |          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

document:
  dsl: 1.0.0
  namespace: default
  name: mock-main-workflow-OK-empty-sub
  version: 0.1.0
do:
- setContext:
    set:
      currentDateTime: ${if ($input.data.currentDateTime|length > 0) then $input.data.currentDateTime else now | strftime("%Y-%m-%d_%H-%M-%S") end}
    export:
      as:
        errors: []
        destinationPathClaims: ${"claims/"+ .currentDateTime +"/claims.json"}
        destinationPathErrors: ${"claims/"+ .currentDateTime +"/errors.json"}
        files:
        - ES-3008640857
        - PL-812436D4-E51D-B1C7-976E-6867AECEC275
- createClaims:
    for:
      each: file
      in: ${$context.files}
    do:
    - refreshAzureToken:
        if: ($index % 40 == 0 or $index == 0)
        set:
          not_im: not_im
        output:
          as: empty
        export:
          as: '$context + {"azureToken": "azuretoken111111111111111111111111111111111111111"}'
    - tryCreateClaimSubWorkflow:
        try:
        - createClaimSubWorkflow:
            run:
              workflow:
                namespace: default
                name: mock-empty-sub
                version: 0.1.0
                input:
                  access:
                    username: username
                    password: pass
                    orgId: orgId
                    language: ln
                    countryCode: AA
                    createAssessmentEndpoint: createAssessmentEndpointcreateAssessmentEndpoint
                    generateImageRefsEndpoint: generateImageRefsEndpointgenerateImageRefsEndpoint
                    createTaskEndpoint: createTaskEndpointcreateTaskEndpoint
                    addImageToAssessEndpoint: addImageToAssessEndpointaddImageToAssessEndpoint
                    gatewayToken: gatewayTokengatewayTokengatewayTokengatewayTokengatewayTokengatewayToken
                  data:
                    bucket: myBucket
                    projectClaimData: projectClaim
                    project: myProject
                  azureToken: azureTokenazureTokenazureToken
                  file_path: ${"claimsDataSummaryPathclaimsDataSummaryPath/"+$file}
                  destinationPathErrors: ${$context.destinationPathErrors}
                  destinationPathClaims: ${$context.destinationPathClaims}
                  index: ${$index}
        catch:
          errors: {}
          as: error
          do:
          - addErrorFromWorkflow:
              set:
                error: ${$error}
              output:
                as:
                  errors: ${$context.errors + [.error]}
              export:
                as: '$context + {errors: .errors}'
- addToFileError:
    if: $context.errors| length > 0
    wait:
      seconds: 1
    output:
      as: empty
- setOutput:
    set:
      claimsFile: ${$context.destinationPathClaims}

Platform(s)

No response

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.name: Bug Report 🐞

[ericlacher]

Hi all

@agnieszkapiecuch's issue is currently a blocker for us, and I'd like to get some insights into synapse's functionality on scheduling to be able to judge whether this is an issue of us misusing the system, or a bug.

Our goal is to execute a large number of sub-workflows, initiated by a higher-level workflow.

We have a large set of items we want to process, using a for-loop and sub-workflows spawned.
What we see is an increased memory usage, as well as high processing times.

So, the questions we'd like to get answered are

1. is this usecase supported by synapse (or are we expected to trigger the individual sub-workflows by individual events without the "orchestrating" main workflow?)
2. how is scheduling working in synapse in general for subworkflows? E.g. when having a single operator, I'd assume it is "occupied" with the main workflow with no operators left to pick up the subworkflows. Does this hold? Or what is the mechanism in such cases?
   2.1 if it does not hold, how is throttling supposed to work? E.g. how can I control the parallel workflows being executed?

What we wanted to achieve is sunsetting a custom batch script and instead rely on sth like synapse to define the flows in a descriptive way and reduce boilerplate code (e.g. for http requests).

Thank y'all,
Eric

[cdavernas]

@agnieszkapiecuch @ericlacher I'm currently investigating, but I'm guessing it is related to the bloating of the main workflow's operational data, which exponentially grows with the data returned by each subflow.

This would explain both the Redis timeouts and the linear increase in processing time for each executed subflow.

This is a complex issue though, and my hands are kind of full for the time being, so any help would be more than welcome!