Immutable user object

[nibix]

Description

For now, this is only a draft PR. Its purpose is to demonstrate the necessary changes and to trigger a discussion whether we can continue this approach - possibly with certain precautions. It is not complete yet, especially the LDAP code is still requiring adaptions.

Fixes #5168

This PR makes the User object immutable. This brings a number of significant advantages:

• An immutable object means that the binary data created by serialization does not change as well. That means, the serialized data can be computed once per user object and then re-used again and again. The existing authentication cache even allows us to re-use the serialized data across different requests by the same user.

• Additionally, a cache similar to the cache maintained by BackendRegistry can be utilized to also cut down the number of de-serialization operations to a single one per user per node. Such a cache would map the serialized binary data to the actual user object.

• Immutable objects are inherently thread-safe. That makes any synchronization or locking mechanisms on the user object unnecessary.

• Category:
  • Enhancement
• Why these changes are required?
  • The serialization of the user object takes significant time, reducing the throughput. The changes help to minimize serialization operations. Additionally, they make the code more mature.
• What is the old behavior before changes and new behavior after changes?
  • No behavioral changes

Issues Resolved

[List any issues this PR will resolve]

Is this a backport? If so, please add backport PR # and/or commits #, and remove backport-failed label from the original PR.

Do these changes introduce new permission(s) to be displayed in the static dropdown on the front-end? If so, please open a draft PR in the security dashboards plugin and link the draft PR here

Testing

[Please provide details of testing done: unit testing, integration testing and manual testing]

Check List

• New functionality includes testing
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

Now that main targets JDK 21, we should start to look for opportunities in this repo to use Text Blocks. It would particularly be useful in some test classes where we define JSON payloads over multiple lines.

[nibix]

FYI: The synchronization here can be removed again because user.getSecurityRoles() cannot be any longer concurrently modified as observed in #4441

[nibix]

FYI: The class InjectedUser is no longer necessary, as it was only used to transport the value transportAddress back to its direct caller, which would set it into the thread context. Afterwares, the additional fields in InjectedUser would not be used any more. Additionally, the InjectdUser was serialized to a normal User object.

Thus, we have replaced Injected user by the Result class below which is evaluated by the direct caller and then discarded.

[nibix]

This is not particularly nice, but it is necessary to keep backwards compatibility with nodes on older OpenSearch versions. To a certain extent, I am inclined again to use a different serialization format on clusters with only new nodes in order to get rid of these synchronizedSet() calls.

[nibix]

Benchmark Results

We did some initial benchmarks on the new implementation, here are the results.

Disclaimer

The usual benchmarking disclaimer: Of course, these figures can only represent very specific scenarios. Other scenarios can look quite different, as there are so many variables which can be different. Yet, especially the "slow parts" of the benchmark results can give one an impression where real performance issues are.

Test setup

• We used OpenSearch 3.0.0-alpha1 for all tests
• All plugins were removed, except the security plugin
• Audit logging was disabled
• Authentication: Only the internal backend auth was active. All other authentication types were disabled.

K8s resources per node

• Java Heap: 32 GB
• RAM: 64 GB
• CPU: 8 cores
• Kubernetes version: v1.32.2

Tested dimensions

Users

We compared four different users:

• User with 10 internal attributes
• User with 100 internal attributes
• User with 1000 internal attributes
• User authenticated by admin certificate

The number of internal attributes influences the size of the user object, both unserialized and serialized. Thus, it influences the resources needed for serialization, deserialization and also the network traffic for inter-node communication. It would have been also possible to test with different numbers of roles. Both variables would influence the serialization performance in similar ways.

Users with 1000 attributes might be an extreme case. Still, especially users authenticated by Active Directory/LDAP might get (sometimes unknown to administrators) quite a lot of attributes from the LDAP backend.

The user authenticated by admin certificate bypasses most parts of the security plugin code (although not the serialization code) and can be thus seen as a upper threshold for what can be achieved.

Operations

We tested four different operations:

• Bulk indexing with 10 items per request
• Bulk indexing with 100 items per request
• Searching on single index
• Searching on 20 indices (using index pattern)

Test results

A commented summary of the results follows in the upcoming sections.

The raw test results can be also reviewed at https://docs.google.com/spreadsheets/d/1HeJ8HFOqF5S4FEDiLXZ4d8eniU9UCp_-FJ3XBrojVME/edit?usp=sharing

Indexing throughput

Bulk size 10

In this and the following charts, the dashed lines represent OpenSearch with the standard security plugin. The solid lines represent OpenSearch with the security plugin with the optimized privilege evaluation code.

The blue line represents requests authenticated by the super admin certificate, which by-passes most of the security plugin. Thus, the blue line forms a kind of "hull curve", it can be seen as a rough theoretical maximum from a security plugin point of view.

We see that users with 10 attributes experience a throughput improvement of about 6% with the new implementation. Users with 100 attributes get an 18% improvement. The extreme case where users have 1000 attributes sees a 46% improvement.

While the new implementation shows significant improvements, we still see a declining throughput with a growing number of user attributes. This is expected, as we only optimize the process of serialization/deserialization. The increased network load stays unchanged compared to the old implementation.

Bulk size 100

Bulk requests with more items can be processed more efficiently. Thus, as we are looking at a docs/s throughput, the improvements are a bit smaller in this case.

Users with 10 attributes experience a 2.5% improvement with the new implementation. Users with 100 attributes see an 8% improvement. For 1000 attributes, we get a 32% improvement.

Search throughput

Single index

In this benchmark, we compare the operation throughput for search requests performed on a single index.

For users with 10 attributes, we only see a small throughput improvement of 1.2%. Users with 100 attributes get an improvement of 13%. Users with 1000 attributes get an improvement of 83%.

20 indices

This benchmark tests searches operations that search through 20 indices at once (specified by an index pattern).

Compared to the single index case, we see much more pronounced improvements. This is likely due to the increased inter-node communication required by searching on several indices with several shards distributed over a cluster.

Users with 10 attributes see a 21% improvement. Users with 100 attributes get a throughput that is 39% higher than the on the old version. Users with 1000 attributes see an improvement of 384%.

@cwperks @kkhatua