[BUG] udev path mappings for data disks broken in Rocky Linux 9.1 image

[arvindshmicrosoft]

Describe the bug: A clear and concise description of what the bug is.

Firstly, I'm not sure if this is an image issue, or an issue with WALinuxAgent. Trying here first.

Summary: On image erockyenterprisesoftwarefoundationinc1653071250513:rockylinux-9:rockylinux-9:9.1.20230215 (from the Azure marketplace), the mapping of data disks to paths under /dev/disk/azure/ is not happening. In the example below, /dev/sda is a 128GiB data disk which is not mapped under /dev/disk/azure/scsi1. Instead, it seems to map the disk directly to /dev/disk/azure. If there are multiple data disks, only one of them gets mapped, the others are not linked at all.

[myuser@svr000000 ~]$ ls -l /dev/disk/azure
lrwxrwxrwx. 1 root root 6 Nov 18 05:54 /dev/disk/azure -> ../sdc

[myuser@svr000000 ~]$ sudo lsblk -a
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   48G  0 disk
├─sda1   8:1    0  100M  0 part /boot/efi
├─sda2   8:2    0 1000M  0 part /boot
├─sda3   8:3    0  7.8G  0 part /
├─sda4   8:4    0    4M  0 part
└─sda5   8:5    0    1M  0 part
sdb      8:16   0  150G  0 disk
└─sdb1   8:17   0  150G  0 part /mnt
sdc      8:32   0  128G  0 disk
sr0     11:0    1  634K  0 rom

[myuser@svr000000 ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="myuser-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

Comparatively, here's the equivalent setup but with Rocky 8.7 (image urn: erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:8.7.20230215)

[myuser@svr000000 ~]$ sudo ls -l /dev/disk/azure/scsi1/
total 0
lrwxrwxrwx. 1 root root 12 Nov 18 06:10 lun0 -> ../../../sdc

[myuser@svr000000 ~]$ sudo lsblk -a
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   48G  0 disk
├─sda1   8:1    0  100M  0 part /boot/efi
├─sda2   8:2    0 1000M  0 part /boot
├─sda3   8:3    0    1M  0 part
└─sda4   8:4    0 46.9G  0 part /
sdb      8:16   0  150G  0 disk
└─sdb1   8:17   0  150G  0 part /mnt
sdc      8:32   0  128G  0 disk
sr0     11:0    1  628K  0 rom

[myuser@svr000000 ~]$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="myuser-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

Distro and WALinuxAgent details (please complete the following information):

• Distro and Version: Rocky Linux 9.1
• WALinuxAgent version:

WALinuxAgent-2.7.0.6 running on rocky 9.1
Python: 3.9.14
Goal state agent: 2.9.1.1

Additional context
Updating to Rocky Linux 9.2 (via. sudo yum update && sudo yum upgrade) does not help. Updating the WALinuxAgent to 2.9.1.1 (via. sudo python setup.py install --force) also does not help.

Log file attached
N/A

[narrieta]

@anhvoms Provisioning issue

[arvindshmicrosoft]

I'm seeing a similar issue on another Fedora 39 image created from the Azure specific cloud image, which was used to provision a VMSS with 2 data disks. 2 of the 3 VMSS instances are fine. But the 3rd one has an issue with the device mappings. So maybe this is not just Rocky Linux specific?

[myuser@mysvr ~]$ waagent -version
WALinuxAgent-2.9.0.4 running on Fedora 39
Python: 3.12.0
Goal state agent: 2.9.0.4

[myuser@mysvr ~]$ sudo lsblk -a
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  150G  0 disk
└─sda1   8:1    0  150G  0 part /mnt
sdb      8:16   0   48G  0 disk
├─sdb1   8:17   0  100M  0 part /boot/efi
├─sdb2   8:18   0 1000M  0 part /boot
├─sdb3   8:19   0    4M  0 part
├─sdb4   8:20   0    1M  0 part
└─sdb5   8:21   0 46.9G  0 part /home
                                /
sdc      8:32   0  128G  0 disk
sdd      8:48   0  128G  0 disk
sr0     11:0    1  628K  0 rom
zram0  252:0    0    8G  0 disk [SWAP]

[myuser@mysvr ~]$ sudo ls -l /dev/disk/azure
lrwxrwxrwx. 1 root root 6 Nov 21 02:31 /dev/disk/azure -> ../sdc

[twinter-amosfivesix]

I have the same issue on RHEL 9.0 and 9.1 images I created myself that use two data disks.

[twinter@RHEL-9-0 ~]$ waagent -version
WALinuxAgent-2.3.0.2 running on rhel 9.0
Python: 3.9.10
Goal state agent: 2.12.0.2

[twinter@RHEL-9-0 ~]$  sudo lsblk -a
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  200G  0 disk
sdb      8:16   0   12G  0 disk
sdc      8:32   0   40G  0 disk
├─sdc1   8:33   0  600M  0 part /boot/efi
├─sdc2   8:34   0    1G  0 part /boot
└─sdc3   8:35   0 38.4G  0 part /

[twinter@RHEL-9-0 ~]$  sudo ls -l /dev/disk/azure
lrwxrwxrwx. 1 root root 6 Mar  5 13:48 /dev/disk/azure -> ../sdb
[twinter@RHEL-9-0 ~]$

[twinter-amosfivesix]

I've found that using sudo lsscsi --brief and parsing the results is a reasonable workaround.