[X86] Manage atomic load of fp -> int promotion in DAG

[jofrn]

When lowering atomic <1 x T> vector types with floats, selection can fail since
this pattern is unsupported. To support this, floats can be casted to
an integer type of the same size.

---

Stack:

• [AtomicExpand] Add bitcasts when expanding load atomic vector #120716
• [SelectionDAG][X86] Remove unused elements from atomic vector. #125432
• [SelectionDAG][X86] Split via Concat <n x T> vector types for atomic load #120640
• [SelectionDAG][X86] Widen <2 x T> vector types for atomic load #120598
• [X86] Add atomic vector tests for unaligned >1 sizes. #120387
• [X86] Manage atomic load of fp -> int promotion in DAG #120386 ⬅
• [SelectionDAG] Legalize <1 x T> vector types for atomic load #120385
• IR/Verifier: Allow vector type in atomic load and store #120384

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

[llvmbot]

@llvm/pr-subscribers-backend-x86

Author: None (jofrn)

Changes

When lowering atomic <1 x T> vector types with floats, selection can fail since
this pattern is unsupported. To support this, floats can be casted to
an integer type of the same size.

---

Stack:

• #120387
• #120386 ⬅
• #120385
• #120384

⚠️ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

---

Full diff: https://github.com/llvm/llvm-project/pull/120386.diff

2 Files Affected:

• (modified) llvm/lib/Target/X86/X86ISelLowering.cpp (+4)
• (modified) llvm/test/CodeGen/X86/atomic-load-store.ll (+74-1)

diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 2571873dba8483..8006d32d077a65 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -2595,6 +2595,10 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
         setOperationAction(Op, MVT::f32, Promote);
   }
 
+  setOperationPromotedToType(ISD::ATOMIC_LOAD, MVT::f16, MVT::i16);
+  setOperationPromotedToType(ISD::ATOMIC_LOAD, MVT::f32, MVT::i32);
+  setOperationPromotedToType(ISD::ATOMIC_LOAD, MVT::f64, MVT::i64);
+
   // We have target-specific dag combine patterns for the following nodes:
   setTargetDAGCombine({ISD::VECTOR_SHUFFLE,
                        ISD::SCALAR_TO_VECTOR,
diff --git a/llvm/test/CodeGen/X86/atomic-load-store.ll b/llvm/test/CodeGen/X86/atomic-load-store.ll
index 9cac8167542d8b..2bde0d2ffd06ad 100644
--- a/llvm/test/CodeGen/X86/atomic-load-store.ll
+++ b/llvm/test/CodeGen/X86/atomic-load-store.ll
@@ -1,12 +1,17 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-apple-macosx10.7.0 -verify-machineinstrs | FileCheck %s
-; RUN: llc < %s -mtriple=x86_64-apple-macosx10.7.0 -verify-machineinstrs -O0 | FileCheck %s
+; RUN: llc < %s -mtriple=x86_64-apple-macosx10.7.0 -verify-machineinstrs -O0 | FileCheck %s --check-prefix=CHECK0
 
 define void @test1(ptr %ptr, i32 %val1) {
 ; CHECK-LABEL: test1:
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    xchgl %esi, (%rdi)
 ; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: test1:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    xchgl %esi, (%rdi)
+; CHECK0-NEXT:    retq
   store atomic i32 %val1, ptr %ptr seq_cst, align 4
   ret void
 }
@@ -16,6 +21,11 @@ define void @test2(ptr %ptr, i32 %val1) {
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl %esi, (%rdi)
 ; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: test2:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movl %esi, (%rdi)
+; CHECK0-NEXT:    retq
   store atomic i32 %val1, ptr %ptr release, align 4
   ret void
 }
@@ -25,6 +35,11 @@ define i32 @test3(ptr %ptr) {
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl (%rdi), %eax
 ; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: test3:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movl (%rdi), %eax
+; CHECK0-NEXT:    retq
   %val = load atomic i32, ptr %ptr seq_cst, align 4
   ret i32 %val
 }
@@ -34,6 +49,64 @@ define <1 x i32> @atomic_vec1_i32(ptr %x) {
 ; CHECK:       ## %bb.0:
 ; CHECK-NEXT:    movl (%rdi), %eax
 ; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: atomic_vec1_i32:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movl (%rdi), %eax
+; CHECK0-NEXT:    retq
   %ret = load atomic <1 x i32>, ptr %x acquire, align 4
   ret <1 x i32> %ret
 }
+
+define <1 x half> @atomic_vec1_half(ptr %x) {
+; CHECK-LABEL: atomic_vec1_half:
+; CHECK:       ## %bb.0:
+; CHECK-NEXT:    movzwl (%rdi), %eax
+; CHECK-NEXT:    pinsrw $0, %eax, %xmm0
+; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: atomic_vec1_half:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movw (%rdi), %cx
+; CHECK0-NEXT:    ## implicit-def: $eax
+; CHECK0-NEXT:    movw %cx, %ax
+; CHECK0-NEXT:    ## implicit-def: $xmm0
+; CHECK0-NEXT:    pinsrw $0, %eax, %xmm0
+; CHECK0-NEXT:    retq
+  %ret = load atomic <1 x half>, ptr %x acquire, align 4
+  ret <1 x half> %ret
+}
+
+define <1 x float> @atomic_vec1_float(ptr %x) {
+; CHECK-LABEL: atomic_vec1_float:
+; CHECK:       ## %bb.0:
+; CHECK-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: atomic_vec1_float:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; CHECK0-NEXT:    retq
+  %ret = load atomic <1 x float>, ptr %x acquire, align 4
+  ret <1 x float> %ret
+}
+
+define <1 x bfloat> @atomic_vec1_bfloat(ptr %x) {
+; CHECK-LABEL: atomic_vec1_bfloat:
+; CHECK:       ## %bb.0:
+; CHECK-NEXT:    movzwl (%rdi), %eax
+; CHECK-NEXT:    pinsrw $0, %eax, %xmm0
+; CHECK-NEXT:    retq
+;
+; CHECK0-LABEL: atomic_vec1_bfloat:
+; CHECK0:       ## %bb.0:
+; CHECK0-NEXT:    movw (%rdi), %cx
+; CHECK0-NEXT:    ## implicit-def: $eax
+; CHECK0-NEXT:    movw %cx, %ax
+; CHECK0-NEXT:    ## implicit-def: $xmm0
+; CHECK0-NEXT:    pinsrw $0, %eax, %xmm0
+; CHECK0-NEXT:    retq
+  %ret = load atomic <1 x bfloat>, ptr %x acquire, align 4
+  ret <1 x bfloat> %ret
+}
+

[jyknight]

Presumably similar changes to other backends are also required?

[RKSimon]

Handle bf16 as well?

[jofrn]

bf16 is already lowered properly without promotion.

And yes, other backends would either have to promote these here or implement them explicitly.

[arsenm]

SelectionDAG making anything legal by default was a terrible mistake but we're not going to fix that here

[arsenm]

This won't do anything without also overriding shouldCastAtomicLoadInIR. AtomicExpand will cast these, and these FP typed ATOMIC_LOADS will never form

[jofrn]

These will still form in from v1f16, v1f32, and v1f64, which don't get translated in AtomicExpand. It is reachable during DAG to DAG translation. After scalarization, we promote:

1. v1f32,ch = AtomicLoad<(load acquire (s32) from %ir.x)> t0, t2
2. f32,ch = AtomicLoad<(load acquire (s32) from %ir.x)> t0, t2 // scalarize
3. i32,ch = AtomicLoad<(load acquire (s32) from %ir.x)> t0, t2 // cast

You mentioned earlier that a follow-up should have AtomicExpand's shouldCastAtomicLoadInIR return none for these; this doesn't affect these cases here though.

[arsenm]

Oh right, I forgot about this ugly detail