Refactor RestApiComplianceAuditlogTest to use TestAuditLogImpl.doThenWaitForMessages

[cwperks]

Description

Refactors RestApiComplianceAuditlogTest to make this test suite more stable and re-enable an ignored test.

The primary problem in this test suite is that messages that are auto populated on startup get polluted with other audit log messages generated when a user calls REST APIs. To make this more stable, a sleep was added after setup to ignore the auto init messages and only isolate messages generated on the API calls in each test.

testAutoInit is also updated to make it stable. There are 4 automatic messages generated on cluster startup that have to do with the seeding of the security index and audit log index.

• Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Test Fix

Issues Resolved

#3730

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

@peternied This PR refactors the whole RestApiComplianceAuditLog test suite to make it more stable.

This keeps the testAutoInit test. That test verifies that when the security index and audit log index are created that there are audit log entries that capture those operations. Its not the greatest test, but I think we should have some assertions around that behavior.

[codecov]

Codecov Report

Merging #3733 (1f498fb) into main (5114bb7) will decrease coverage by 0.03%.
Report is 3 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3733      +/-   ##
==========================================
- Coverage   65.24%   65.21%   -0.03%     
==========================================
  Files         297      297              
  Lines       21129    21129              
  Branches     3451     3451              
==========================================
- Hits        13786    13780       -6     
- Misses       5643     5652       +9     
+ Partials     1700     1697       -3     

see 5 files with indirect coverage changes

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-2.x
# Create a new branch
git switch --create backport/backport-3733-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6b8a3e494c7933b93f6ee4f1a7956b728ed48a43
# Push it to GitHub
git push --set-upstream origin backport/backport-3733-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3733-to-2.x.