Provide a way to reduce memory pressure due to data allocations held by ESMF Field and Array objects

[theurich]

There are use cases where it would be convenient if ESMF could offer a way to easily reduce the memory pressure caused by the memory allocation held by ESMF Field and Array objects for user data.

One such use case came up under the NEPTUNE async output work, where multiple output streams (each implemented as an NUOPC component) can share the same resources (compute nodes) using the same petList. These output stream components run sequentially. It turns out that memory pressure is often the limiting factors of how many output streams can be placed on the same nodes. The memory allocations held by ESMF Field/Array objects, for this specific example these are all in the ImportState of the component. Once filled with data (often during the first time a NUOPC_Connector transfers data into these objects), the associated memory is paged in, and RSS goes up. This memory is held in physical pages until the component is finalized, destroying the Fields in the ImportState. This is problematic for sequentially running components using the same compute node resources. It would be convenient if physical pages associated with ESMF Field/Array objects' data allocations could be released when the component returns from its Advance() method, as to allow the next component to utilize the physical memory (when the associated Connector transfers data, and then the component uses the data during its Advance).

There are two options I think that could be explored to provide this feature:

1. Utilize posix_madvice() to give the POSIX_MADV_DONTNEED advice for data allocations held inside the LocalArray. Then provide ESMF_ArrayMemAdvice() and ESMF_FieldMemAdvice() methods that can be used at the end of the user Advance() implementation on any Arrays/Fields whose data is no longer needed. (Notice that I already implemented this on the Array side under https://github.com/esmf-org/esmf/tree/feature/MemAdvice, however my testing has not shown any direct RSS reduction, and also memory pressure leading to OOM issues for the specific test case remained unchanged. I am wondering if this has to do with the fact that memory for data allocations in the use case were allocated via Fortran allocate() statements?)

2. Provide ESMF_Field/ArrayDeallocate() and ESMF_Field/ArrayAllocate() methods to explicitly control data allocations for a persistent ESMF Field/Array object (i.e. outside of Destroy()/Create()). I think the implementation of this might be more straight forward, especially if allocations go through the Fortran allocator implementation. The LocalArray level will need to support this by using the appropriate de-/allocator. Utilizing this for the NUOPC user case requires that checks are added to the NUOPC_Connector implementation to see if Fields/Arrays are data allocated before using as destinations inside SMM execution. That is where the Field/ArrayAllocate() would be called. The associated Field/ArrayDeallocate() would be at the end of the user Advance() method.