[ntuple] Add partioning to `RNTupleJoinTable` #17919

enirolf · 2025-03-07T16:32:00Z

With this addition, the RNTupleJoinTable can be split up into several mappings from join values to entry numbers, according to some (numeric) partition key. Using a partition key is optional; a default partition key is used when none has been specified by the user.

The RNTupleJoinTable now has an inner class, REntryMapping, which in practice now represents what the RNTupleJoinTable represented previously, i.e., a mapping from join field values to entry numbers:

entryMapping
|- {x_0, y_0} -> 42
|- {x_1, y_1} -> 99
|- {x_2, y_2} -> 12
|- ...

The join table itself now instead provides a mapping from partition keys to (a collection of) these entry mappings:

joinTable
|- 0
   |- entryMapping_0
      |- {x_0, y_0} -> 42
      |- ...
|- 4
   |- entryMapping_0
      |- {x_0, y_0} -> 99
      |- ...
   |- entryMapping_1
      |- ...
|- ...
|- kDefaultPartitionKey
   |- entryMapping_0
      |- {x_0, y_0} -> 12
      |- ...
   |- ...

The reason one partition can contain multiple entry mappings is twofold:

This is needed when only the default partition key is used (or in other words, no partitions are used
Less relevant for now, but we could foresee cases where the partition keys are based on some (meta-data) attribute that is shared across more than one page source.

The most immediate use case of the partitioning approach added in this PR, is that this way, the RNTupleJoinTable itself is not restricted to one page source anymore (this restriction is now in REntryMapping instead). This is useful for the integration into the RNTupleProcessor, where we want to be able to create joins of chains of RNTuples and have to deal with more than one page source (see also #17132).

As a side-effect, the state management of the join table and the notion of lazy building has changed. There is no single isBuilt state anymore, and the Add function eagerly builds the mapping for the provided page source and adds it to the join table. As such, the responsibility of deciding whether to eagerly or lazily build the join table is moved to the application using the join table (i.e. by strategically calling Add).

github-actions · 2025-03-07T18:49:51Z

Test Results

20 files 20 suites 5d 7h 5m 26s ⏱️
2 726 tests 2 726 ✅ 0 💤 0 ❌
52 591 runs 52 591 ✅ 0 💤 0 ❌

Results for commit 130714e.

♻️ This comment has been updated with latest results.

vepadulano

Nice improvement! Some minor comments for now

vepadulano · 2025-03-10T14:02:31Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+         RCombinedJoinFieldValue(const std::vector<NTupleJoinFieldValue_t> &fieldValues)
+         {
+            fFieldValues.reserve(fieldValues.size());
+            fFieldValues = fieldValues;
+         }


Shouldn't this be equivalent?

Suggested change

RCombinedJoinFieldValue(const std::vector<NTupleJoinFieldValue_t> &fieldValues)

{

fFieldValues.reserve(fieldValues.size());

fFieldValues = fieldValues;

}

RCombinedJoinFieldValue(const std::vector<NTupleJoinFieldValue_t> &fieldValues): fFieldValues(fieldValues)

{

}

vepadulano · 2025-03-10T14:02:59Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+      class RCombinedJoinFieldValue {
+      public:


Maybe struct better represents the class layout in this case?

vepadulano · 2025-03-10T14:04:03Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

-   /// https://www.boost.org/doc/libs/1_55_0/doc/html/hash/reference.html#boost.hash_combine).
-   struct RCombinedJoinFieldValueHash {
-      inline std::size_t operator()(const RCombinedJoinFieldValue &joinFieldValue) const
+      /// The mapping itself. Maps field values (or combinations thereof in case the join key is a composed of multiple


Suggested change

/// The mapping itself. Maps field values (or combinations thereof in case the join key is a composed of multiple

/// The mapping itself. Maps field values (or combinations thereof in case the join key is composed of multiple

vepadulano · 2025-03-10T14:19:21Z

tree/ntuple/v7/test/ntuple_join_table.cxx

+      switch (*run) {
+      // For run == 0, no entry indexes are expected because they were added using the default partition key
+      case 0: EXPECT_EQ(entryIdxs.size(), 0ul); break;
+      // For run == 0, we expect exactly one entry index


Suggested change

// For run == 0, we expect exactly one entry index

// For run == 1, we expect exactly one entry index

silverweed

A couple of minor comments

silverweed · 2025-03-11T10:33:53Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+      //////////////////////////////////////////////////////////////////////////
+      /// Container for the combined hashes of join field values.
+      struct RCombinedJoinFieldValue {
+      public:


This public is superfluous

silverweed · 2025-03-11T10:35:50Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

-   /// \throws RException If the join table has not been built, and can therefore not be used yet.
-   void EnsureBuilt() const;
+   /// \brief Get all entry mappings in the join table.
+   const std::vector<REntryMapping *> GetAllMappings() const;


Since you're returning a value, it's better to return it non-const (same thing for the next function)

hahnjo

I left many comments. The answer to some of them may well be that it's motivated by future changes. In that case please feel free to simply resolve the thread(s).

tree/ntuple/v7/src/RNTupleJoinTable.cxx

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

tree/ntuple/v7/test/ntuple_join_table.cxx

hahnjo · 2025-03-17T17:31:30Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+   /// \return The entry numbers that correspond to `valuePtrs`. When there are no corresponding entries, an empty
+   /// vector is returned.
+   std::vector<ROOT::NTupleSize_t> GetEntryIndexes(const std::vector<void *> &valuePtrs) const
+   {
+      return GetEntryIndexes(valuePtrs, fPartitionKeys);
+   }


Is this useful in case of multiple partitions since the return value loses the information which mapping the indexes are relative to? Maybe what we want instead is GetEntryIndexes(const std::vector<void *> &valuePtrs, PartitionKey_t partitionKey = kDefaultPartitionKey)?

The issue I see with that interface is that it may give false negatives when there are mappings in partitions other than the default. I think it's okay to lose information here, if one needs this information they should instead use (GetEntryIndexes(const std::vector<void *> &valuePtrs, PartitionKey_t partitionKey) to be able to track the partitions.

Yes, I now realize that the information is equally lost for GetEntryIndexes(const std::vector<void *> &valuePtrs, const std::vector<PartitionKey_t> &partitionKeys). I'm still not sure how useful this is (at the moment) since a mapping spans an entire page source, but I guess this will change in the future.

hahnjo · 2025-03-18T15:47:09Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

@@ -35,57 +35,98 @@ namespace Internal {
 // clang-format on
 class RNTupleJoinTable {
 public:
-   using NTupleJoinValue_t = std::uint64_t;
+   using JoinValue_T = std::uint64_t;


Suggested change

using JoinValue_T = std::uint64_t;

using JoinValue_t = std::uint64_t;

(that's what the commit message says...)

hahnjo · 2025-03-18T15:55:23Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+   /// \return The entry numbers that correspond to `valuePtrs`. When there are no corresponding entries, an empty
+   /// vector is returned.
+   std::vector<ROOT::NTupleSize_t> GetEntryIndexes(const std::vector<void *> &valuePtrs) const
+   {
+      return GetEntryIndexes(valuePtrs, fPartitionKeys);
+   }


Yes, I now realize that the information is equally lost for GetEntryIndexes(const std::vector<void *> &valuePtrs, const std::vector<PartitionKey_t> &partitionKeys). I'm still not sure how useful this is (at the moment) since a mapping spans an entire page source, but I guess this will change in the future.

hahnjo · 2025-03-18T16:03:07Z

tree/ntuple/v7/inc/ROOT/RNTupleJoinTable.hxx

+   /// partition.
+   ///
+   /// \return A reference to the updated join table.
+   RNTupleJoinTable &Add(RPageSource &pageSource, PartitionKey_t partitionKey = kDefaultPartitionKey);


Two more options:
4. (variation of 2.) continue to have a single RNTupleJoinTable::Build, but pass it all page sources at once. This still requires the RNTupleJoinTable to figure out which entry mapping a page source belonged to - not sure I see how to do this.
5. (variation of 3.) have RNTupleJoinTable::Add also build the mapping, but lazily construct the entire join table in RNTupleJoinProcessor. This may also simplify the code a bit since a join table is always built, but it's still deferred from the user perspective.

I currently favor 5. but that's because I thought of it last, so I probably haven't discovered its shortcomings yet 😃

With this addition, the join table can be split up into several mappings from join values to entry numbers, according to some (numeric) partition key. It has several use cases, but the immediate one is that with this approach, the join table is not restricted to one page source anymore. This is useful for the integration into the `RNTupleProcessor`, where we want to be able to create joins of chains of RNTuples and have to deal with more than one page source. As a side-effect, the state management of the join table and the notion of lazy building has changed. There is no single `isBuilt` state anymore, and the `Add` function eagerly builds the mapping for the provided page source and adds it to the join table. As such, the responsibility of deciding whether to eagerly or lazily build the join table is moved to the application using the join table (i.e. by strategically calling `Add`).

Since it is declared within `RNTupleJoinTable`, it is clear that this type belongs to RNTuple from the context.

enirolf added the in:RNTuple label Mar 7, 2025

enirolf requested review from hahnjo, pcanal, silverweed and vepadulano March 7, 2025 16:32

enirolf self-assigned this Mar 7, 2025

enirolf force-pushed the ntuple-join-table-partitions branch 2 times, most recently from 6834e5a to 7ebbde6 Compare March 7, 2025 16:41

enirolf force-pushed the ntuple-join-table-partitions branch from 7ebbde6 to 4cc1604 Compare March 10, 2025 14:05

vepadulano requested changes Mar 10, 2025

View reviewed changes

enirolf force-pushed the ntuple-join-table-partitions branch from 4cc1604 to 8e2228d Compare March 10, 2025 14:58

enirolf marked this pull request as ready for review March 10, 2025 15:47

enirolf requested a review from jblomer as a code owner March 10, 2025 15:47

enirolf requested a review from vepadulano March 10, 2025 15:48

silverweed reviewed Mar 11, 2025

View reviewed changes

enirolf force-pushed the ntuple-join-table-partitions branch from 8e2228d to cdcfb92 Compare March 11, 2025 14:06

enirolf requested a review from silverweed March 11, 2025 14:06

hahnjo reviewed Mar 17, 2025

View reviewed changes

enirolf force-pushed the ntuple-join-table-partitions branch from cdcfb92 to 130714e Compare March 18, 2025 14:21

hahnjo reviewed Mar 18, 2025

View reviewed changes

enirolf added 4 commits March 19, 2025 09:06

[ntuple][NFC] Rename fieldNames to joinFieldNames

d8b458f

[ntuple][NFC] Rename joinFieldValues to castJoinValues

3e6b6ac

[ntuple][NFC] Rename NTupleJoinValue_t to JoinValue_t

1e7f395

Since it is declared within `RNTupleJoinTable`, it is clear that this type belongs to RNTuple from the context.

enirolf force-pushed the ntuple-join-table-partitions branch from 130714e to 1e7f395 Compare March 19, 2025 08:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ntuple] Add partioning to `RNTupleJoinTable` #17919

[ntuple] Add partioning to `RNTupleJoinTable` #17919

enirolf commented Mar 7, 2025 •

edited

Loading

github-actions bot commented Mar 7, 2025 •

edited

Loading

vepadulano left a comment

vepadulano Mar 10, 2025

vepadulano Mar 10, 2025

vepadulano Mar 10, 2025

vepadulano Mar 10, 2025

silverweed left a comment

silverweed Mar 11, 2025

silverweed Mar 11, 2025

hahnjo left a comment

hahnjo Mar 17, 2025

enirolf Mar 18, 2025

hahnjo Mar 18, 2025

hahnjo Mar 18, 2025

hahnjo Mar 18, 2025

hahnjo Mar 18, 2025

	/// The mapping itself. Maps field values (or combinations thereof in case the join key is a composed of multiple
	/// The mapping itself. Maps field values (or combinations thereof in case the join key is composed of multiple

	// For run == 0, we expect exactly one entry index
	// For run == 1, we expect exactly one entry index

	using JoinValue_T = std::uint64_t;
	using JoinValue_t = std::uint64_t;

[ntuple] Add partioning to RNTupleJoinTable #17919

Are you sure you want to change the base?

[ntuple] Add partioning to RNTupleJoinTable #17919

Conversation

enirolf commented Mar 7, 2025 • edited Loading

github-actions bot commented Mar 7, 2025 • edited Loading

Test Results

vepadulano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silverweed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hahnjo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[ntuple] Add partioning to `RNTupleJoinTable` #17919

[ntuple] Add partioning to `RNTupleJoinTable` #17919

enirolf commented Mar 7, 2025 •

edited

Loading

github-actions bot commented Mar 7, 2025 •

edited

Loading