Some questions about in external_petsc_problem of calling FormIFunction and FormIJacobian?

[Wang-Yihu]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

How are the function PETSC_EXTERN PetscErrorCode FormIFunction(TS, PetscReal, Vec, Vec, Vec, void *) and PETSC_EXTERN PetscErrorCode FormIJacobian(TS, PetscReal, Vec, Vec, PetscReal, Mat, Mat, void *) called in the example of \modules\external_petsc_solver\test\tests\external_petsc_problem\moose_as_parent.i?

It seems that only the function PETSC_EXTERN PetscErrorCode PETScExternalSolverCreate(MPI_Comm, TS *) calls these two functions, but it also seems that this function is called only once by TS & ExternalPetscSolverApp::getPetscTS(). It's impossible! Because I find these two functions are called multiple times: in the petsc iteration step, for SNES Function norm, FormIFunction is called, and for KSP Residual norm, FormIJacobian is called.
So how does this paradox happen?

[GiudGiud]

hello

in PETScExternalSolverCreate(MPI_Comm comm, TS * ts)
we have this call:

  ierr = TSSetIFunction(*ts, NULL, FormIFunction, nullptr);
  CHKERRQ(ierr);

which tells the petsc time stepper what function it can use to compute Udot - RHS

then the time stepper will call this function on every time step

[Wang-Yihu]

OK, I see it. It seems that:
If FormIFunction and FormIJacobian are called, PETScExternalSolverCreate must be called before. And if PETScExternalSolverCreate is called, getPetscTS() must be called before.
But it seems that both PETScExternalSolverCreate and getPetscTS() are not called by every time step?

[GiudGiud]

But it seems that both PETScExternalSolverCreate and getPetscTS() are not called by every time step?

For the first one that's expected, the solver does not need to be re-created every time we solve, it can be re-used
For the second one, same reason, and the time stepper must be kept in an attribute or something because it is called every time step (as is the solver)

[Wang-Yihu]

OK. I've found the calling routine: PETScDMDAMesh::PETScDMDAMesh calls TS & ExternalPetscSolverApp::getPetscTS, and TS & ExternalPetscSolverApp::getPetscTS calls PETScExternalSolverCreate. All of them are called only once.
But I still do not know how is it be re-used as you said when MOOSE only calls that once? And i do not know what is the time stepper must be kept in an attribute or something...
Or do I make the mistake? In fact getPetscTS() is called for multiple times with time steps, not for only once, but i do not find this?

[Wang-Yihu]

Haha, I find the answer. They are called by functions in /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019 in conda environment maybe generated by PETScDiffusionFDM.C which do not appear in \modules\external_petsc_solver\.

We can use gdb tool and backtrace to find this.

1. Make the dbg file. In projects/moose/modules/external_petsc_solver directory:

conda activate moose
METHOD=dbg make -j8

2. In projects/moose/modules/external_petsc_solver/test/tests/external_petsc_problem

gdb --args ../../../external_petsc_solver_dbg -i moose_as_parent.i
break FormIFunction

y

run

backtrace

c

It will return:

#0  FormIFunction (ts=0x555555caba00, U=0x555555fac860, Udot=0x555555fbd520, 
    F=0x555555f724c0)
    at /home/sjtu/projects/moose/modules/external_petsc_solver/src/petscsolver/PETScDiffusionFDM.C:202
#1  0x00007fffed381396 in TSComputeIFunction ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#2  0x00007fffed3beace in SNESTSFormFunction_Theta ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#3  0x00007fffed37e37a in SNESTSFormFunction ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#4  0x00007fffed32db39 in SNESComputeFunction ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#5  0x00007fffed376b57 in SNESSolve_NEWTONLS ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#6  0x00007fffed334a7d in SNESSolve ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#7  0x00007fffed3bf0a8 in TSStep_Theta ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#8  0x00007fffed383198 in TSStep ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#9  0x00007fffed385060 in TSSolve ()
   from /home/sjtu/mambaforge3/envs/moose/lib/libpetsc.so.3.019
#10 0x00007ffff7f62d1a in externalPETScDiffusionFDMSolve (ts=0x555555caba00, 
--Type <RET> for more, q to quit, c to continue without paging--c
    0x555555d12d20, u=0x555555d123f0, dt=0.20000000000000001, time=0.20000000000000001, converged=0x555555cc8a08) at /home/sjtu/projects/moose/modules/external_petsc_solver/src/petscsolver/PETScDiffusionFDM.C:186
#11 0x00007ffff7f68936 in ExternalPETScProblem::externalSolve (this=0x555555cc5c80) at /home/sjtu/projects/moose/modules/external_petsc_solver/src/problems/ExternalPETScProblem.C:75
#12 0x00007ffff6085b34 in ExternalProblem::solve (this=0x555555cc5c80) at /home/sjtu/projects/moose/framework/src/problems/ExternalProblem.C:62
#13 0x00007ffff531b4bc in FEProblemSolve::solve (this=0x555555d2ecb0) at /home/sjtu/projects/moose/framework/src/executioners/FEProblemSolve.C:284
#14 0x00007ffff5320384 in FixedPointSolve::solveStep (this=0x555555cb1280, begin_norm=@0x555555f82090: 0, end_norm=@0x555555f07ab0: 0, transformed_dofs=...) at /home/sjtu/projects/moose/framework/src/executioners/FixedPointSolve.C:426
#15 0x00007ffff531f24a in FixedPointSolve::solve (this=0x555555cb1280) at /home/sjtu/projects/moose/framework/src/executioners/FixedPointSolve.C:278
#16 0x00007ffff65f060c in TimeStepper::step (this=0x555555d626c0) at /home/sjtu/projects/moose/framework/src/timesteppers/TimeStepper.C:168
#17 0x00007ffff532ee32 in Transient::takeStep (this=0x555555d2e970, input_dt=0.20000000000000001) at /home/sjtu/projects/moose/framework/src/executioners/Transient.C:432
#18 0x00007ffff5bba95f in TransientMultiApp::solveStep (this=0x555555a9f570, dt=0.20000000000000001, target_time=0.20000000000000001, auto_advance=false) at /home/sjtu/projects/moose/framework/src/multiapps/TransientMultiApp.C:414
#19 0x00007ffff60cd2e6 in FEProblemBase::execMultiApps (this=0x5555559fcc00, type=..., auto_advance=false) at /home/sjtu/projects/moose/framework/src/problems/FEProblemBase.C:4800
#20 0x00007ffff531fe79 in FixedPointSolve::solveStep (this=0x5555557bc7e0, begin_norm=@0x555555eff870: 0, end_norm=@0x555555efe1e0: 0, transformed_dofs=...) at /home/sjtu/projects/moose/framework/src/executioners/FixedPointSolve.C:387
#21 0x00007ffff531f24a in FixedPointSolve::solve (this=0x5555557bc7e0) at /home/sjtu/projects/moose/framework/src/executioners/FixedPointSolve.C:278
#22 0x00007ffff65f060c in TimeStepper::step (this=0x5555559cceb0) at /home/sjtu/projects/moose/framework/src/timesteppers/TimeStepper.C:168
#23 0x00007ffff532ee32 in Transient::takeStep (this=0x555555a591c0, input_dt=-1) at /home/sjtu/projects/moose/framework/src/executioners/Transient.C:432
#24 0x00007ffff532d924 in Transient::execute (this=0x555555a591c0) at /home/sjtu/projects/moose/framework/src/executioners/Transient.C:307
#25 0x00007ffff4fd2902 in MooseApp::executeExecutioner (this=0x5555557a8ad0) at /home/sjtu/projects/moose/framework/src/base/MooseApp.C:1138
#26 0x00007ffff4fd977c in MooseApp::run (this=0x5555557a8ad0) at /home/sjtu/projects/moose/framework/src/base/MooseApp.C:1513
#27 0x00005555555576a9 in main (argc=3, argv=0x7fffffffdab8) at /home/sjtu/projects/moose/modules/external_petsc_solver/src/main.C:35

Oh! The route is ExternalPETScProblem::externalSolve calls externalPETScDiffusionFDMSolve, externalPETScDiffusionFDMSolve generates lots of libpetsc.so.3.019, and functions in libpetsc.so.3.019 calls FormIFunction