handle CGDims properly in serial and MPI

[TomMelt]

Handle CGDims properly in serial and MPI

TODO

• rebase on latest develop to fix failing mac ci once PR ci: fix mac ci workflow to detect venv #970 is merged

---

Fix failing tests for PR #879

• Modify ModelArray::setDimensions to support MPI local and global dims
• add support for new setDimensions function into the tests and code

---

Background

The underlying problem is that the x_cg and y_cg dimensions are wrong in the initial model file. For example, when I run the config_june23.cfg it uses input file init_25km_NH.nc. In there the dimensions are listed as follows:

netcdf init_25km_NH {
dimensions:
	ydim = 121 ;
	xdim = 154 ;
	yvertex = 122 ;
	xvertex = 155 ;
	y_cg = 122 ;
	x_cg = 155 ;
	dg_comp = 1 ;
	dgstress_comp = 3 ;
	ncoords = 2 ;

They should be y_cg = 243 and x_cg = 309, for CGDegree=2.

These get read in here in ParaGridIO::getModelState and then set on line 137.

nextsimdg/core/src/ParaGridIO.cpp

Lines 81 to 137 in ca3dddb

	ModelState ParaGridIO::getModelState(const std::string& filePath)
	#endif
	{
	ModelState state;
	try {
	#ifdef USE_MPI
	netCDF::NcFilePar ncFile(filePath, netCDF::NcFile::read, metadata.mpiComm);
	#else
	netCDF::NcFile ncFile(filePath, netCDF::NcFile::read);
	#endif
	// Dimensions and DG components
	std::multimap<std::string, netCDF::NcDim> dimMap = ncFile.getDims();
	for (auto entry : ModelArray::definedDimensions) {
	auto dimType = entry.first;
	if (dimCompMap.count(dimType) > 0)
	// TODO Assertions that DG in the file equals the compile time DG in the model. See
	// #205
	continue;
	ModelArray::DimensionSpec& dimensionSpec = entry.second;
	// Find dimensions in the netCDF file by their name in the ModelArray details
	netCDF::NcDim dim = ncFile.getDim(dimensionSpec.name);
	// Also check the old name
	if (dim.isNull()) {
	dim = ncFile.getDim(dimensionSpec.altName);
	}
	// If we didn't find a dimension with the dimensions name or altName, throw.
	if (dim.isNull()) {
	throw std::out_of_range(
	std::string("No netCDF dimension found corresponding to the dimension named ")
	+ dimensionSpec.name + std::string(" or ") + dimensionSpec.altName);
	}
	#ifdef USE_MPI
	auto dimName = dim.getName();
	size_t localLength = 0;
	size_t start = 0;
	if (dimType == ModelArray::Dimension::X) {
	localLength = metadata.localExtentX;
	start = metadata.localCornerX;
	} else if (dimType == ModelArray::Dimension::Y) {
	localLength = metadata.localExtentY;
	start = metadata.localCornerY;
	} else if (dimType == ModelArray::Dimension::XVERTEX) {
	localLength = metadata.localExtentX + 1;
	start = metadata.localCornerX;
	} else if (dimType == ModelArray::Dimension::YVERTEX) {
	localLength = metadata.localExtentY + 1;
	start = metadata.localCornerY;
	} else {
	localLength = dim.getSize();
	start = 0;
	}
	ModelArray::setDimension(dimType, dim.getSize(), localLength, start);
	#else
	ModelArray::setDimension(dimType, dim.getSize());

The localLength is then overridden later in BBMDynamics.cpp on line 125:

nextsimdg/core/src/modules/DynamicsModule/BBMDynamics.cpp

Line 125 in ca3dddb

ModelArray::setDimensions(ModelArray::Type::CG, cgDims);

Note the function ModelArray::setDimensions only sets the localLength and not the globalLength and that's why you needed to make the changes here:

nextsimdg/core/src/ParaGridIO.cpp

Lines 266 to 267 in ca3dddb

	size_t dimSz = (dimCompMap.count(dim)) ? ModelArray::nComponents(dimCompMap.at(dim))
	: dimSz = entry.second.localLength;

I think this logic needs to be re-written to use the ModelMetadata singleton so that it will work for MPI and serial code. Then we need to also modify function ModelArray::setDimensions or create a new one that will also set the globalLength.

[timspainNERSC]

I think a smarter structure for the dimensions might be useful, rather than changing the function signature of ModelArray::dimensions depending on how the model is compiled.

[timspainNERSC]

I must admit that I don't like the idea of having a different function signature serial and MPI versions of the code. I wonder if a better solution might be changing the result of ModelArray::dimensions from a plain MultiDim to a vector of not scalars but local-global pairs in the MPI case.

[timspainNERSC]

Also, the metadata class could have a function to generate the correct form of the combined local/global dims.

[timspainNERSC]

This would become ```
ModelArray::setDimensions(ModelArray::Type::H, { { 1, 1 }, { 1, 1 } });