HPX 2

[msimberg]

I'm opening this as a placeholder for discussion on long term plans for HPX (which we may call 2). See this as a wishlist of things that should be finally deprecated, cleaned up, or implemented.

I'll start (please extend this list):

• Follow semver
• Complete modularization
• CMake modernization
• Namespace and header location cleanup (clear policy on what's part of the API, what's experimental, what's private) (Formalize public API, namespace, and release policy  #4364)
• Executors; there should be a clear story about what executor to use in which situations
• Libfabric improvements in master: this is one of the strongest selling points of distributed HPX to existing applications using MPI (features have to be amazing for them to care, performance is a much better selling point)
• ROCm support (minimal future support, not full parallel algorithms)

Many of these things are already in progress but moving to 2 would allow us to "officially" clean up and break compatibility without feeling too bad about ourselves. If needed we can keep 1.X in lifesupport mode for users who can't take all the breaking changes yet.

The namespace cleanup is partly motivated by having e.g. an experimental namespace (this has been discussed before, but should be formalized) so that we can introduce new features and if needed break or remove them without deprecation warnings as long as they are experimental.

The alternative to this is of course keep incrementally adding and removing features as we've been doing now, but I think going for a new major release would be good for both visibility and for allowing us to clean up things properly.

[McKillroy]

I want to suggest to add the Doxygen documentation to the build, along with other documentation improvements.
E.g. the dependency graphs created by Doxygen are incredibly useful, also when I need to find out which construct requires which #include I refer to the Doxygen docs.

[msimberg]

@McKillroy just to be clear, what you mean is to use the "normal" HTML documentation generated by Doxygen, as opposed to the one we include using Breathe in Sphinx (because the latter is not particularly good...)?

[McKillroy]

@msimberg Yes. Breathe is an attempt to feed Doxygen into Sphinx, which in theory is a nice idea, but practically doesn't work that well.

My idea of documentation would be to have the high level docs in Sphinx, because that's what it's good for, and low level stuff and API details in the code, so a programmer can directly supply it while coding.

Doxygen - despite it's shortcomings still imho is the best ad hoc documentation for C++ programmers available, and the graphs are incredibly useful.

E.g. I started refactoring our code to use the concepts of components, packages and package groups, where we put components ( hpp/cpp pairs + test drivers as suggested by John Lakos https://www.youtube.com/watch?v=QjFpKJ8Xx78 ) into package subdirectories and Doxygen immediately shows us package dependencies by using the directory dependency graph. I wouldn't know how to do that and have a graphical feedback without Doxygen. Also I use Doxygen and its graphs a lot while trying to understand the structure of the HPX codebase.

And after all - the best documentation tool helps nothing if people don't document their code - Doxygen imo is the tool with the lowest threshold to get something done documentation-wise.
It is painful for me to see a great tool (HPX) in front of me with all the amount of work put into it by numerous talented people and having a ton of issues because of lack of documentation. The lacking documentation is the single most problem for me when working with HPX.

If Ste||ar-Group really wants HPX to become THE library to work with when parallelizing code documentation has to improve significantly and that cannot be done by just a single dedicated person or a student polishing the grammar. It's something every single programmer has to do, and Doxygen is the way to go here. Sphinx should serve as the Manual or Cookbook, the "HPX Bible" with occasional references to Doxygen via Breathe or otherwise.

[msimberg]

We had a brief discussion (@aurianer, @biddisco, @hkaiser, and myself) yesterday on how to proceed towards 2.0.0. I've tried to write down here the rough consensus from that discussion. 1.6.0 and further 1.X releases do not have to come 6 months apart. We can do them as quickly as a significant change is there. There are a few things from the above list (libfabric cleanup, ROCm) that I didn't put in a specific release. We can squeeze them in to 1.6.0 or any later 1.X release as things are ready. Please add the major things you think are missing.

1.5.0

• (already there: CMake targets, local runtime)
• All local modules in place (Local runtime module etc. #4606, Futures module #4592)
• API headers and namespaces for local functionality in place (without this users will have to first change their headers to the replacements we have now, and then again to the final headers we define in HEP 2: Public API for HPX 2.0.0 #4626)

1.6.0

• Split up main library into three parts (according to HEP 1: build configurations #4593)
• C++20 conformance for parallel algorithms (Adapt parallel algorithms to C++20 #4822)

1.7.0

• If needed

2.0.0

• All modules in place, including distributed
• API headers and namespaces for distributed functionality in place

Edit: Added C++20 conformance for parallel algorithms.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.