8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes #28363

jianglizhou · 2025-11-17T22:34:14Z

Please review the fix in StubGenerator::aesgcm_avx512 and StubGenerator::aesgcm_avx2 to handle some edge cases with input sizes that are not multiple of the block size.

Thanks to Thomas Holenstein and Lukas Zobernig for analyzing the issue and providing the test case!

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes (Bug - P3)

Contributors

Thomas Holenstein <[email protected]>
Lukas Zobernig <[email protected]>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28363/head:pull/28363
$ git checkout pull/28363

Update a local copy of the PR:
$ git checkout pull/28363
$ git pull https://git.openjdk.org/jdk.git pull/28363/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28363

View PR using the GUI difftool:
$ git pr show -t 28363

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28363.diff

Using Webrev

Link to Webrev Comment

…tubs cause AES-GCM encryption failure for certain payload sizes

….com and [email protected].

bridgekeeper · 2025-11-17T22:36:07Z

👋 Welcome back jiangli! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

jianglizhou · 2025-11-17T22:36:30Z

/contributor [email protected]
/contributor [email protected]

openjdk · 2025-11-17T22:37:50Z

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

openjdk · 2025-11-17T22:38:22Z

@jianglizhou Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

openjdk · 2025-11-17T22:38:30Z

@jianglizhou Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

openjdk · 2025-11-17T22:38:48Z

@jianglizhou The following labels will be automatically applied to this pull request:

hotspot-compiler
security

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

jianglizhou · 2025-11-17T22:40:09Z

/contributor Thomas Holenstein [email protected]
/contributor Lukas Zobernig [email protected]

openjdk · 2025-11-17T22:41:46Z

@jianglizhou Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

openjdk · 2025-11-17T22:41:53Z

@jianglizhou Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <[email protected]>

User names can only be used for users in the census associated with this repository. For other contributors you need to supply the full name and email address.

jianglizhou · 2025-11-17T22:45:47Z

/contributor add Thomas Holenstein [email protected]
/contributor add Lukas Zobernig [email protected]

openjdk · 2025-11-17T22:47:10Z

@jianglizhou
Contributor Thomas Holenstein <[email protected]> successfully added.

openjdk · 2025-11-17T22:48:08Z

@jianglizhou
Contributor Lukas Zobernig <[email protected]> successfully added.

mlbridge · 2025-11-17T22:52:24Z

Webrevs

TobiHartmann · 2025-11-18T06:17:30Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+
+public class TestAesGcmIntrinsic {
+
+  static final SecureRandom SECURE_RANDOM = newDefaultSecureRandom();


Drive-by comment: Java code should use 4x whitespace indentation.

@TobiHartmann, thanks! Fixed.

shipilev

Good catch! Some stylistic comments for the product fix, and suggestions for the test.

shipilev · 2025-11-18T08:33:32Z

src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp


  __ bind(MESG_BELOW_32_BLKS);
  __ subl(len, 16 * 16);
+  __ cmpl(len, 256);


From the stylistic logic, this should be written as 16 * 16, to match the surrounding subl and addl.

Thanks for the detailed review, @shipilev! Fixed.

shipilev · 2025-11-18T08:48:31Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+  public static void main(String[] args) throws Exception {
+    TestAesGcmIntrinsic test = new TestAesGcmIntrinsic();
+    long startTime = System.currentTimeMillis();
+    while (System.currentTimeMillis() - startTime < 60 * 1000) {


I get that you want a stress test. But time-limiting puts the test into weird condition: it can have different number of iterations, depending on auxiliary load on the machine. These tests are running in parallel with lots of other tests, so it is not uncommon. Do you even need to repeat jitFunc() call multiple times? Looks like it traverses the interesting configurations in one go?

I did some testing today. For 200 runs, removing the time-limited loop, there is 89 runs out of 200 fail. So I changed to use an iteration of three runs, all 200 runs fail without the fix.

shipilev · 2025-11-18T08:50:51Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+    for (int messageSize = SPLIT_LEN; messageSize < SPLIT_LEN + 300; messageSize++) {
+      byte[] message = randBytes(messageSize);
+      try {
+        byte[] ciphertext = gcmEncrypt(key, message, aad);


I believe it makes sense to check that round-trip is successful, e.g. that decrypt(encrypt(message)) == message. Currently we implicitly rely on exceptions being thrown from the incorrectly executing code, which is IMO too weak -- in the boundary conditions like these, there might be bugs that do not manifest in visible exceptions, and just the encryption is subtly broken.

That's a good idea. I added decrypt part and the check as suggested.

With the changes, there were more common parts in the test. I moved common code into helper methods.

shipilev · 2025-11-18T08:53:05Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+import javax.crypto.spec.GCMParameterSpec;
+import javax.crypto.spec.SecretKeySpec;
+
+public class TestAesGcmIntrinsic {


This sounds like TestGCMSplitBound or some such; it is not a generic test for AES/GCM intrinsic.

I renamed to TestAesGcmIntrinsic name, when converting the original test into the jtreg test. TestGCMSplitBound SGTM. Changed.

shipilev · 2025-11-18T08:53:26Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+        throw new RuntimeException("ciphertext is null");
+      }
+    }
+    for (int messageSize = SPLIT_LEN; messageSize < SPLIT_LEN + 300; messageSize++) {


[SPLIT_LEN - 300; SPLIT_LEN + 300] for completeness, perhaps?

shipilev · 2025-11-18T08:57:02Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestAesGcmIntrinsic.java

+
+public class TestAesGcmIntrinsic {
+
+  static final SecureRandom SECURE_RANDOM = newDefaultSecureRandom();


Do you really need a SecureRandom here? Random RANDOM = Utils.getRandomInstance(); gets you the pre-seeded random instance, which can be used to repeatably reproduce failures.

I kept the SecureRandom without changing. I think that could be more related to what the original reproducible.

- Rename test to TestGCMSplitBound.java - Change test range to [SPLIT_LEN - 300; SPLIT_LEN + 300].

- Replace time-bound loop with an iteration of three runs. - Add encrypt part and check to make sure the encrypted message is the same as the original.

shipilev · 2025-11-20T18:08:40Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java

+      byte[] nonce = randBytes(IV_SIZE_IN_BYTES);
+      System.arraycopy(ciphertext, 0, nonce, 0, IV_SIZE_IN_BYTES);
+      Cipher cipher = getCipher(key, aad, nonce);
+      return cipher.doFinal(ciphertext, IV_SIZE_IN_BYTES, ciphertext.length - IV_SIZE_IN_BYTES);


Indenting is still 2-space here.

Indeed. Fixed, thanks.

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java

shipilev · 2025-11-20T19:37:11Z

test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.java

+        AlgorithmParameterSpec params =
+            new GCMParameterSpec(8 * TAG_SIZE_IN_BYTES, nonce, 0, nonce.length);
+        Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
+        cipher.init(Cipher.ENCRYPT_MODE, keySpec, params);


Er. This is used from gcmDecrypt? How does it work without Cipher.DECRYPT_MODE?

Good catch. Interestingly the test passed for me on my local machine. Fixed to use Cipher.DECRYPT_MODE when doing gcmDecrypt.

Also an interesting new finding, with the decrypted message verification, I see there are 2 failures out of 200 runs with AVX512. I'm filing a new issue on the specifically, so it can be investigated.

Filed https://bugs.openjdk.org/browse/JDK-8372364.

…java Applied, thanks. Co-authored-by: Aleksey Shipilëv <[email protected]>

…DK-8371864

openjdk · 2025-11-21T00:11:18Z

⚠️ @jianglizhou This pull request contains merges that bring in commits not present in the target repository. Since this is not a "merge style" pull request, these changes will be squashed when this pull request in integrated. If this is your intention, then please ignore this message. If you want to preserve the commit structure, you must change the title of this pull request to Merge <project>:<branch> where <project> is the name of another project in the OpenJDK organization (for example Merge jdk:master).

…dom data in gcmDecrypt. Suggested by AI.

sviswa7 · 2025-11-22T00:17:51Z

src/hotspot/cpu/x86/stubGenerator_x86_64_aes.cpp


  __ bind(MESG_BELOW_32_BLKS);
  __ subl(len, 16 * 16);
+  __ cmpl(len, 16 * 16);
+  __ jcc(Assembler::lessEqual, ENC_DEC_DONE);


I think the fix should instead be to just move the addl to pos before the MESG_BELOW_32_BLKS, as below:

+ __ addl(pos, 16 * 16); __ bind(MESG_BELOW_32_BLKS); __ subl(len, 16 * 16); - __ addl(pos, 16 * 16);

This is because on fall through path addl is needed but not while coming from line 3479 via jcc. For the latter, the addl has already been done on line 3477.

jianglizhou added 2 commits November 13, 2025 15:04

JDK-8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics s…

518dad9

…tubs cause AES-GCM encryption failure for certain payload sizes

Add TestAesGcmIntrinsic.java. The test is authored by tholenst@google…

fb6a1ec

….com and [email protected].

openjdk bot added security [email protected] hotspot-compiler [email protected] labels Nov 17, 2025

Fix Whitespace errors

338a99d

openjdk bot added the rfr Pull request is ready for review label Nov 17, 2025

TobiHartmann reviewed Nov 18, 2025

View reviewed changes

shipilev reviewed Nov 18, 2025

View reviewed changes

jianglizhou added 4 commits November 19, 2025 16:42

Fix indentation.

eb1a13d

Stylistic change: '256' to '16 * 16'.

4ef42ff

Address shipilev's comments:

54e619a

- Rename test to TestGCMSplitBound.java - Change test range to [SPLIT_LEN - 300; SPLIT_LEN + 300].

Address shipilev coments:

f1e7291

- Replace time-bound loop with an iteration of three runs. - Add encrypt part and check to make sure the encrypted message is the same as the original.

openjdk bot removed the rfr Pull request is ready for review label Nov 20, 2025

Fix Whitespace error.

528b1b4

openjdk bot added the rfr Pull request is ready for review label Nov 20, 2025

shipilev reviewed Nov 20, 2025

View reviewed changes

Fix indentation

e99a441

jianglizhou and others added 3 commits November 20, 2025 12:07

Update test/jdk/com/sun/crypto/provider/Cipher/AES/TestGCMSplitBound.…

54fbf2b

…java Applied, thanks. Co-authored-by: Aleksey Shipilëv <[email protected]>

Use Cipher.DECRYPT_MODE for gcmDecrypt.

ee58323

Merge branch 'JDK-8371864' of ssh://github.com/jianglizhou/jdk into J…

8617ab4

…DK-8371864

Change to just create a byte array for 'nonce' without generating ran…

d26d0ee

…dom data in gcmDecrypt. Suggested by AI.

sviswa7 reviewed Nov 22, 2025

View reviewed changes


		public class TestAesGcmIntrinsic {

		static final SecureRandom SECURE_RANDOM = newDefaultSecureRandom();

8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes #28363

Are you sure you want to change the base?

8371864: GaloisCounterMode.implGCMCrypt0 AVX512/AVX2 intrinsics stubs cause AES-GCM encryption failure for certain payload sizes #28363

Conversation

jianglizhou commented Nov 17, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Contributors

Reviewing

Uh oh!

bridgekeeper bot commented Nov 17, 2025

Uh oh!

jianglizhou commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jianglizhou commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

jianglizhou commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

openjdk bot commented Nov 17, 2025

Uh oh!

mlbridge bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shipilev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianglizhou commented Nov 17, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Nov 17, 2025 •

edited

Loading

mlbridge bot commented Nov 17, 2025 •

edited

Loading