Restricted Additive Schwarz with multiple layers of overlap #2369
Conversation
Remove confusing, unused and unnecessary ParallelRestrictedAdditiveSchwarz.hpp file. Enable multiple overlap layers and restricted additive schwarz.
CMakeLists.txt
Outdated
| EXE_NAME flow_onephase | ||
| DEPENDS "opmsimulators" | ||
| LIBRARIES "opmsimulators") | ||
| if (FALSE) |
ebos/eclcpgridvanguard.hh
Outdated
| grid().globalCell().data(), grid().size(0)); | ||
| this->schedule().filterConnections(activeCells); | ||
| } | ||
| } |
| typedef Dune::CollectiveCommunication< int > communication_type; | ||
| #endif | ||
|
|
||
| #if DUNE_VERSION_NEWER(DUNE_ISTL, 2, 6) |
There was a problem hiding this comment.
2.6 is minimum, no need for the ifdefery
|
Please also report timings. |
There was a problem hiding this comment.
The diff of this PR is much shorter than I expected from the description.
Where do I choose the RAS preconditioner? I could not find the option.
What preconditioner is used internally when solving?
I seems like all the logic is implemented in opm-grid which should not care about linear solver stuff. Is that right?
| // If overlapLayers_ > 1 we are using Restricted Additive Schwarz and therefore need | ||
| // the residual at overlap DoFs. | ||
| if (overlapLayers_ > 1) | ||
| parallelInformation_arg.copyOwnerToAll(istlb, istlb); |
There was a problem hiding this comment.
Please explain why this is only needed for more than one overlap layers. Shouldn't the right hand side always be consistent (all procs have the same values?)
There was a problem hiding this comment.
The right hand side is normally unique, which is correct with only one overlap since this will give Dirichlet condition for the copy (?). I think the problem is that of some reason the residual is only calculated for the owner cells since this is the only needed for the original solvers. With this copying and the additional setting of zero for the copy attribute (i.e. "useless overlap") the right hand side is correct. The correct thing is to assemble for all owner and overlap (I would have prefered all) and then set it to zero (at the same time as the matrix elements is fixed to get correct Dirichlet condition). If amg solver is used I think a change in the redistribution function is needed to get correct behavoir if the coarse scale communicator has overlap attributes.
|
|
||
| // Add just a single element to ghost rows | ||
| if (elem.partitionType() != Dune::InteriorEntity) | ||
| if (elem.partitionType() == Dune::GhostEntity) |
There was a problem hiding this comment.
This is totally unrelated to this PR but it just struck me when reading the diff:
Should we move the for-loop over the wellConnectionsGraph_ to else branch and explicitly use noGhostMat_->setrowsize(1);. That seems clearer and might prevent surprises if we distribute the wells.
|
Thanks for the efforts. I will try to test how it works with the model 2 variants. |
|
Please ignoring my previous message about the parallel running failure with model 2. I did not replace |
|
FYI, my comment of OPM/opm-grid#449 apply here to. I think if we want to support RAS then we should use an explicit option for choosing it and not |
|
preliminary testings with with two realizations of model2, with Maybe we need more tests. I think it is something good to have for later testing and study, or something we can keep improving. |
|
If the only reasonable solver with none empty "overlap attributes" why do we need an extra option for it? |
5 participants
Add option to use Restricted Additive Schwarz (RAS) as parallel preconditioner instead of Block-Jacobi (BJ). This is made possible by adding multiple layers of overlap between subdomains after partitioning the grid, and PR depend on PR OPM/opm-grid#449. The overlap layers are separated into two types: The Overlap layer(s) and the outer Ghost layer. The matrix entries in the overlap rows are then valuable and can be used in the preconditioner. There is no need for a RAS preconditioner class, since the index set, the comm.project function and comm.copyOwnerToAll function handles everything for us.
Norne with transmissibility edge-weights
To demonstrate the impact of RAS on the convergence, consider Norne with default partitioning options using 1 and 2 layers of overlap.
We observe a slight reduction in iteration count when using RAS over BJ for N=2,16,32 and 64, but higher for n=4 and 8. The main reason for these unimpressive results are that parallel flow on Norne with transmissibility weights produce almost no increase in iterations compared to sequential flow. For different or larger cases running on more processors the benefit could be greater.
Norne with uniform edge-weights
To see an example where RAS has a more significant impact, consider Norne again, but now use uniform edge-weights to partition the grid.
Here we see a lower iteration count for all N except 2. We also observe that the benefit of RAS increase with N.
Note that the tables show total number of iterations and not total time. The lower iteration count of RAS comes at the cost of increased parallel overhead.