New bare metal machine not producing attestations #314

fwoodruff-ab · 2025-01-16T11:25:10Z

Describe the support request

Having previously got guests and attestations working, I am now on a new machine on a new bare metal provider. I am able to launch a TD guest but I can't generate an attestation report (it's empty).

I ran the attestation setup instructions and I noticed that cat /var/log/mpa_registration.log was giving bad logs.
I tried setting SGX Factory Reset and SGX Auto MP Registration to 'enable'. This has resolved the registration logs but I am still not able to generate attestations from guests.

How do I debug this and get attestations working on this machine?

System report

Git ref

c0d28c1131c66dd25f72f966e6701ceea506d50b

Operating system details

Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.1 LTS
Release:	24.04
Codename:	noble

Kernel version

6.8.0-1015-intel #22-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 12 14:47:29 UTC 2024 x86_64 x86_64 GNU/Linux

TDX kernel logs

[    4.236539] virt/tdx: BIOS enabled: private KeyID range [32, 64)
[    4.237535] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
[   30.175830] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20240407, build_num 744
[   30.175833] virt/tdx: CMR: [0x100000, 0x77800000)
[   30.175836] virt/tdx: CMR: [0x100000000, 0x107a000000)
[   30.175837] virt/tdx: CMR: [0x1080000000, 0x207c000000)
[   30.175838] virt/tdx: CMR: [0x2080000000, 0x307c000000)
[   30.175839] virt/tdx: CMR: [0x3080000000, 0x407c000000)
[   30.909476] virt/tdx: 1050644 KB allocated for PAMT
[   30.909487] virt/tdx: module initialized
...
[    4.236539] virt/tdx: BIOS enabled: private KeyID range [32, 64)
[    4.237535] virt/tdx: Disable ACPI S3. Turn off TDX in the BIOS to use ACPI S3.
[   30.175830] virt/tdx: TDX module: attributes 0x0, vendor_id 0x8086, major_version 1, minor_version 5, build_date 20240407, build_num 744
[   30.175833] virt/tdx: CMR: [0x100000, 0x77800000)
[   30.175836] virt/tdx: CMR: [0x100000000, 0x107a000000)
[   30.175837] virt/tdx: CMR: [0x1080000000, 0x207c000000)
[   30.175838] virt/tdx: CMR: [0x2080000000, 0x307c000000)
[   30.175839] virt/tdx: CMR: [0x3080000000, 0x407c000000)
[   30.909476] virt/tdx: 1050644 KB allocated for PAMT
[   30.909487] virt/tdx: module initialized

TDX CPU instruction support

CPU supports TDX according to /proc/cpuinfo

Model specific registers (MSRs)

MK_TME_ENABLED bit: 1 (expected value: 1)
SEAM_RR bit: 1 (expected value: 1)
NUM_TDX_PRIV_KEYS: 20
SGX_AND_MCHECK_STATUS: 0 (expected value: 0)
Production platform: Production (expected value: Production)

CPU details

 INTEL(R) XEON(R) GOLD 6526Y

QEMU package details

Status: Installed
Package: qemu-system-x86
Version: 2:8.2.2+ds-0ubuntu1.4+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

Libvirt package details

Status: Installed
Package: libvirt-clients
Version: 10.0.0-2ubuntu8.3+tdx1.2
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

OVMF package details

Status: Installed
Package: ovmf
Version: 2024.02-3+tdx1.0
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-release/ubuntu noble/main amd64 Packages

sgx-dcap-pccs package details

Status: Installed
Package: sgx-dcap-pccs
Version: 1.21-0ubuntu1
APT-Sources: https://ppa.launchpadcontent.net/kobuk-team/tdx-attestation-release/ubuntu noble/main amd64 Packages

tdx-qgs package details

Status: Installed
Package: tdx-qgs
Version: 1.22.100.3-noble1
APT-Sources: https://download.01.org/intel-sgx/sgx_repo/ubuntu noble/main amd64 Packages

sgx-ra-service package details

Status: Installed
Package: sgx-ra-service
Version: 1.22.100.3-noble1
APT-Sources: https://download.01.org/intel-sgx/sgx_repo/ubuntu noble/main amd64 Packages
Description: Intel(R) Software Guard Extensions Multi-Package Registration Agent Service: Enables SGX Remote Attestation for Multi-Package platforms

sgx-pck-id-retrieval-tool package details

Status: Installed
Package: sgx-pck-id-retrieval-tool
Version: 1.22.100.3-noble1
APT-Sources: https://download.01.org/intel-sgx/sgx_repo/ubuntu noble/main amd64 Packages

QGSD service status

● qgsd.service - Intel(R) TD Quoting Generation Service
     Loaded: loaded (/usr/lib/systemd/system/qgsd.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-01-16 18:17:19 UTC; 7min ago
    Process: 2152 ExecStartPre=/bin/chown -R qgsd:qgsd /var/opt/qgsd/ (code=exited, status=0/SUCCESS)
    Process: 2214 ExecStartPre=/bin/chmod 0750 /var/opt/qgsd/ (code=exited, status=0/SUCCESS)
    Process: 2237 ExecStartPre=/opt/intel/tdx-qgs/linksgx.sh (code=exited, status=0/SUCCESS)
    Process: 2286 ExecStart=/opt/intel/tdx-qgs/qgs (code=exited, status=0/SUCCESS)
   Main PID: 2292 (qgs)
      Tasks: 5 (limit: 308631)
     Memory: 17.2M (peak: 17.8M)
        CPU: 499ms
     CGroup: /system.slice/qgsd.service
             └─2292 /opt/intel/tdx-qgs/qgs

Jan 16 18:22:12 ns31694252 qgsd[2292]: tee_att_get_quote_size return 0x1100f
Jan 16 18:22:12 ns31694252 qgsd[2292]: call tee_att_init_quote
Jan 16 18:22:12 ns31694252 qgsd[2292]: [QCNL] Encountered CURL error: (60) SSL peer certificate or SSH remote key was not OK
Jan 16 18:22:12 ns31694252 qgsd[2292]: [QPL] Failed to get quote config. Error code is 0xb033
Jan 16 18:22:12 ns31694252 qgsd[2292]: [get_platform_quote_cert_data ../td_ql_logic.cpp:302] Error returned from the p_sgx_get_quote_config API. 0xe065
Jan 16 18:22:12 ns31694252 qgsd[2292]: tee_att_init_quote return 0x11001
Jan 16 18:22:12 ns31694252 qgsd[2292]: tee_att_get_quote_size return 0x1100f
Jan 16 18:22:12 ns31694252 qgsd[2292]: resp_size is 0
Jan 16 18:22:12 ns31694252 qgsd[2292]: About to shutdown and close socket
Jan 16 18:22:12 ns31694252 qgsd[2292]: erased a connection, now [0]

PCCS service status

● pccs.service - Provisioning Certificate Caching Service (PCCS)
     Loaded: loaded (/usr/lib/systemd/system/pccs.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-01-16 18:21:10 UTC; 3min 22s ago
       Docs: https://github.com/intel/SGXDataCenterAttestationPrimitives/blob/master/QuoteGeneration/pccs/README.md
   Main PID: 2849 (node)
      Tasks: 15 (limit: 308631)
     Memory: 42.4M (peak: 57.7M)
        CPU: 1.329s
     CGroup: /system.slice/pccs.service
             └─2849 /usr/bin/node /opt/intel/sgx-dcap-pccs/pccs_server.js

Jan 16 18:21:10 ns31694252 systemd[1]: Started pccs.service - Provisioning Certificate Caching Service (PCCS).
Jan 16 18:21:11 ns31694252 node[2849]: 2025-01-16 18:21:11.036 [info]: HTTPS Server is running on: https://localhost:8081

MPA registration logs (last 30 lines)

[16-01-2025 06:01:50] INFO: SGX Registration Agent version: 1.21.100.3
[16-01-2025 06:01:50] INFO: Starts Registration Agent Flow.
[16-01-2025 06:01:51] INFO: Registration Flow - PLATFORM_ESTABLISHMENT or TCB_RECOVERY passed successfully.
[16-01-2025 06:01:51] INFO: Finished Registration Agent Flow.
[16-01-2025 06:17:18] INFO: SGX Registration Agent version: 1.21.100.3
[16-01-2025 06:17:18] INFO: Starts Registration Agent Flow.
[16-01-2025 06:17:18] INFO: Registration Flow - Registration status indicates registration is completed successfully. MPA has nothing to do.
[16-01-2025 06:17:18] INFO: Finished Registration Agent Flow.

The text was updated successfully, but these errors were encountered:

syncronize-issues-to-jira · 2025-01-16T11:25:19Z

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/PEK-1603.

This message was autogenerated

jorgeantonio21 · 2025-01-16T12:06:49Z

I had this same issue, I did set SGX Factory Reset, but was able to enable SGX Auto MP Registration and the mpa registration worked (it seems). I had previously a different machine without the SGX Auto MP Registration and I couldn't make it work, so had to change. In the meantime, I am still having problems with quote generation even after successful MPA registration (see #313)

hector-cao · 2025-01-16T12:21:21Z

@fwoodruff-ab Could you please run system-report.sh as sudo ? I see permission issues in the attached output

fwoodruff-ab · 2025-01-16T18:29:01Z

Hi, I was wrong about the machine not having SGX Auto MP Registration in the BIOS. I have updated the text. I have also run the system report with sudo. This is essentially an entirely different question after these changes, apologies

hector-cao · 2025-01-16T18:37:48Z

@fwoodruff-ab did you run the pccs-configure script ?

jorgeantonio21 · 2025-01-16T21:39:54Z

@fwoodruff-ab, @hector-cao I think this becomes a very similar problem to the one I am having

fwoodruff-ab · 2025-01-17T10:42:19Z

@hector-cao I ran /usr/bin/pccs-configure and sudo systemctl restart pccs yes :)
Happy to give you SSH access to the machine if that helps

hector-cao · 2025-01-17T10:43:26Z

Ýes, that would be great

fwoodruff-ab · 2025-01-17T12:17:47Z

@hector-cao , I've sent you an email. I hope that's ok

hector-cao · 2025-01-17T12:49:55Z

@fwoodruff-ab the qgsd output indicates to me that the certificate used to communicate with PCCS service is not good, the only thing i would think of is to make sure to run pccs-configure with the appropriate selected options (see README). You can try to run it again

fwoodruff-ab · 2025-01-17T14:46:21Z

That solved it, thank you!
Yesterday I tried the above with the right certificates initially, before trying again with the self-signed certificates to debug this. If I figure out what changed from the first time I did this, I'll update here.

hector-cao · 2025-01-17T14:50:43Z

Great ! Closing

fwoodruff-ab changed the title ~~Platform registration behaving differently on new bare metal machine~~ New bare metal machine cannot produce attestations Jan 16, 2025

fwoodruff-ab changed the title ~~New bare metal machine cannot produce attestations~~ New bare metal machine not producing attestations Jan 16, 2025

hector-cao closed this as completed Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New bare metal machine not producing attestations #314

New bare metal machine not producing attestations #314

fwoodruff-ab commented Jan 16, 2025 •

edited

Loading

syncronize-issues-to-jira bot commented Jan 16, 2025

jorgeantonio21 commented Jan 16, 2025

hector-cao commented Jan 16, 2025 •

edited

Loading

fwoodruff-ab commented Jan 16, 2025

hector-cao commented Jan 16, 2025

jorgeantonio21 commented Jan 16, 2025

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025 •

edited

Loading

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025

New bare metal machine not producing attestations #314

New bare metal machine not producing attestations #314

Comments

fwoodruff-ab commented Jan 16, 2025 • edited Loading

Git ref

Operating system details

Kernel version

TDX kernel logs

TDX CPU instruction support

Model specific registers (MSRs)

CPU details

QEMU package details

Libvirt package details

OVMF package details

sgx-dcap-pccs package details

tdx-qgs package details

sgx-ra-service package details

sgx-pck-id-retrieval-tool package details

QGSD service status

PCCS service status

MPA registration logs (last 30 lines)

syncronize-issues-to-jira bot commented Jan 16, 2025

jorgeantonio21 commented Jan 16, 2025

hector-cao commented Jan 16, 2025 • edited Loading

fwoodruff-ab commented Jan 16, 2025

hector-cao commented Jan 16, 2025

jorgeantonio21 commented Jan 16, 2025

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025 • edited Loading

fwoodruff-ab commented Jan 17, 2025

hector-cao commented Jan 17, 2025

fwoodruff-ab commented Jan 16, 2025 •

edited

Loading

hector-cao commented Jan 16, 2025 •

edited

Loading

hector-cao commented Jan 17, 2025 •

edited

Loading