-
Notifications
You must be signed in to change notification settings - Fork 9
Description
The phenomenon is that the process crashes almost immediately after invoking the rvm
command. It is very easy to reproduce when using rr
. (Note that you need to compile rr
from source in order to make rr
support the 32-bit x86 architecture on 64-bit OS. Compiling the git head using the default configuration should just work, but the rr
from your Linux distro may not have 32-bit support.)
Example 1
In one particular execution, the executable JikesRVM
was mapped near address 0x61b4????
.
(rr) info proc mappings
process 128175
Mapped address spaces:
Start Addr End Addr Size Offset Perms objfile
0x61b43000 0x61b44000 0x1000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b44000 0x61b46000 0x2000 0x1000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b46000 0x61b48000 0x2000 0x3000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b48000 0x61b49000 0x1000 0x4000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x61b49000 0x61b4a000 0x1000 0x5000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x62287000 0x622a9000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
0x6fffd000 0x70001000 0x4000 0x0 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpage_32.so
0x70001000 0x70002000 0x1000 0x0 rw-s /tmp/rr-shared-preload_thread_locals-131561-25
0x70003000 0x70004000 0x1000 0x0 r-xp
0x70004000 0x70104000 0x100000 0x0 rw-s /tmp/rr-shared-syscallbuf.128175-131561-28
0xe8400000 0xe841e000 0x1e000 0x0 r--p /usr/lib32/libc.so.6
0xe841e000 0xe85a0000 0x182000 0x1e000 r-xp /usr/lib32/libc.so.6
0xe85a0000 0xe861b000 0x7b000 0x1a0000 r--p /usr/lib32/libc.so.6
0xe861b000 0xe861d000 0x2000 0x21a000 r--p /usr/lib32/libc.so.6
0xe861d000 0xe861e000 0x1000 0x21c000 rw-p /usr/lib32/libc.so.6
0xe861e000 0xe8623000 0x5000 0x0 rw-p
0xe8800000 0xe881b000 0x1b000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe881b000 0xe89ce000 0x1b3000 0x1b000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe89ce000 0xe8a9c000 0xce000 0x1ce000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8a9c000 0xe8a9d000 0x1000 0x29c000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8b2c000 0xe8b2e000 0x2000 0x0 rw-p
0xe8b2e000 0xe8b31000 0x3000 0x0 r--p /usr/lib32/libgcc_s.so.1
0xe8b31000 0xe8b50000 0x1f000 0x3000 r-xp /usr/lib32/libgcc_s.so.1
0xe8b50000 0xe8b56000 0x6000 0x22000 r--p /usr/lib32/libgcc_s.so.1
0xe8b56000 0xe8b57000 0x1000 0x28000 rw-p /usr/lib32/libgcc_s.so.1
0xe8b57000 0xe8b62000 0xb000 0x0 r--p /usr/lib32/libm.so.6
0xe8b62000 0xe8bf0000 0x8e000 0xb000 r-xp /usr/lib32/libm.so.6
0xe8bf0000 0xe8c27000 0x37000 0x99000 r--p /usr/lib32/libm.so.6
0xe8c27000 0xe8c28000 0x1000 0xcf000 r--p /usr/lib32/libm.so.6
0xe8c28000 0xe8c29000 0x1000 0xd0000 rw-p /usr/lib32/libm.so.6
0xe8c29000 0xe8c2c000 0x3000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c2c000 0xe8c39000 0xd000 0x3000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c39000 0xe8c40000 0x7000 0x10000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c40000 0xe8c41000 0x1000 0x16000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c41000 0xe8c42000 0x1000 0x17000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c42000 0xe8c43000 0x1000 0x0 r--p /usr/lib32/libdl.so.2
0xe8c43000 0xe8c44000 0x1000 0x1000 r-xp /usr/lib32/libdl.so.2
0xe8c44000 0xe8c45000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c45000 0xe8c46000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c46000 0xe8c47000 0x1000 0x3000 rw-p /usr/lib32/libdl.so.2
0xe8c8c000 0xe8c8d000 0x1000 0x0 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c8d000 0xe8c94000 0x7000 0x1000 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c94000 0xe8c98000 0x4000 0x8000 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c98000 0xe8c99000 0x1000 0xc000 rw-p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c99000 0xe8ca4000 0xb000 0x0 rw-p
0xe8ca4000 0xe8ca6000 0x2000 0x0 r-xp [vdso]
0xe8ca6000 0xe8ca7000 0x1000 0x0 r--p /usr/lib32/ld-linux.so.2
0xe8ca7000 0xe8cca000 0x23000 0x1000 r-xp /usr/lib32/ld-linux.so.2
0xe8cca000 0xe8cd8000 0xe000 0x24000 r--p /usr/lib32/ld-linux.so.2
0xe8cd8000 0xe8cda000 0x2000 0x31000 r--p /usr/lib32/ld-linux.so.2
0xe8cda000 0xe8cdb000 0x1000 0x33000 rw-p /usr/lib32/ld-linux.so.2
0xff98f000 0xff9b2000 0x23000 0x0 rw-p [stack]
Then JikesRVM executed the mmap
syscall in mapImageFile
to map RVM.data.image
. The stack trace is:
(rr) bt
#0 0x70000002 in syscall_traced ()
#1 0xe8c929ad in _raw_syscall () at /home/wks/tocompile/rr/build-release/32/preload/raw_syscall.S:34
#2 0xe8c8dcd3 in traced_raw_syscall (call=call@entry=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:377
#3 0xe8c905e7 in sys_quotactl (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:3710
#4 syscall_hook_internal (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4229
#5 syscall_hook (call=0x681fffd8) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4355
#6 syscall_hook (call=<optimized out>) at /home/wks/tocompile/rr/build-release/32/preload/syscallbuf.c:4339
#7 0xe8c8d341 in _syscall_hook_trampoline () at /home/wks/tocompile/rr/build-release/32/preload/syscall_hook.S:131
#8 0xe8c8d3b2 in _syscall_hook_trampoline_90_90_90 () at /home/wks/tocompile/rr/build-release/32/preload/syscall_hook.S:211
#9 0x6ffff005 in __kernel_vsyscall ()
#10 0xe851b366 in ?? () from /usr/lib32/libc.so.6
#11 0xe852216c in mmap64 () from /usr/lib32/libc.so.6
#12 0xe8c2d7a1 in mapImageFile (fileName=0xff9b0d0f "/home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image", targetAddress=0x60000000, executable=1 '\001',
writable=1 '\001', roundedImageSize=0xff9ae8d4) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:319
#13 0xe8c2d8cf in createVM (vmInSeparateThread=0) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:359
#14 0xe8c2dffa in JNI_CreateJavaVM (mainJavaVM=0xff9ae95c, mainJNIEnv=0xff9ae960, initArgs=0xff9ae96c) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:497
#15 0x61b459a0 in main (argc=19, argv=0xff9aea68) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/main.c:594
After that, the mapping becomes:
(rr) info proc mappings
process 128175
Mapped address spaces:
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x62287000 0x622a9000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
0x6fffd000 0x70001000 0x4000 0x0 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpage_32.so
0x70001000 0x70002000 0x1000 0x0 rw-s /tmp/rr-shared-preload_thread_locals-131561-25
0x70003000 0x70004000 0x1000 0x0 r-xp
0x70004000 0x70104000 0x100000 0x0 rw-s /tmp/rr-shared-syscallbuf.128175-131561-28
0xe8400000 0xe841e000 0x1e000 0x0 r--p /usr/lib32/libc.so.6
0xe841e000 0xe85a0000 0x182000 0x1e000 r-xp /usr/lib32/libc.so.6
0xe85a0000 0xe861b000 0x7b000 0x1a0000 r--p /usr/lib32/libc.so.6
0xe861b000 0xe861d000 0x2000 0x21a000 r--p /usr/lib32/libc.so.6
0xe861d000 0xe861e000 0x1000 0x21c000 rw-p /usr/lib32/libc.so.6
0xe861e000 0xe8623000 0x5000 0x0 rw-p
0xe8800000 0xe881b000 0x1b000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe881b000 0xe89ce000 0x1b3000 0x1b000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe89ce000 0xe8a9c000 0xce000 0x1ce000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8a9c000 0xe8a9d000 0x1000 0x29c000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
0xe8b2c000 0xe8b2e000 0x2000 0x0 rw-p
0xe8b2e000 0xe8b31000 0x3000 0x0 r--p /usr/lib32/libgcc_s.so.1
0xe8b31000 0xe8b50000 0x1f000 0x3000 r-xp /usr/lib32/libgcc_s.so.1
0xe8b50000 0xe8b56000 0x6000 0x22000 r--p /usr/lib32/libgcc_s.so.1
0xe8b56000 0xe8b57000 0x1000 0x28000 rw-p /usr/lib32/libgcc_s.so.1
0xe8b57000 0xe8b62000 0xb000 0x0 r--p /usr/lib32/libm.so.6
0xe8b62000 0xe8bf0000 0x8e000 0xb000 r-xp /usr/lib32/libm.so.6
0xe8bf0000 0xe8c27000 0x37000 0x99000 r--p /usr/lib32/libm.so.6
0xe8c27000 0xe8c28000 0x1000 0xcf000 r--p /usr/lib32/libm.so.6
0xe8c28000 0xe8c29000 0x1000 0xd0000 rw-p /usr/lib32/libm.so.6
0xe8c29000 0xe8c2c000 0x3000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c2c000 0xe8c39000 0xd000 0x3000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c39000 0xe8c40000 0x7000 0x10000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c40000 0xe8c41000 0x1000 0x16000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c41000 0xe8c42000 0x1000 0x17000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
0xe8c42000 0xe8c43000 0x1000 0x0 r--p /usr/lib32/libdl.so.2
0xe8c43000 0xe8c44000 0x1000 0x1000 r-xp /usr/lib32/libdl.so.2
0xe8c44000 0xe8c45000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c45000 0xe8c46000 0x1000 0x2000 r--p /usr/lib32/libdl.so.2
0xe8c46000 0xe8c47000 0x1000 0x3000 rw-p /usr/lib32/libdl.so.2
0xe8c8c000 0xe8c8d000 0x1000 0x0 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c8d000 0xe8c94000 0x7000 0x1000 r-xp /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c94000 0xe8c98000 0x4000 0x8000 r--p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c98000 0xe8c99000 0x1000 0xc000 rw-p /home/wks/tocompile/rr/build-release/install/lib/rr/librrpreload_32.so
0xe8c99000 0xe8ca4000 0xb000 0x0 rw-p
0xe8ca4000 0xe8ca6000 0x2000 0x0 r-xp [vdso]
0xe8ca6000 0xe8ca7000 0x1000 0x0 r--p /usr/lib32/ld-linux.so.2
0xe8ca7000 0xe8cca000 0x23000 0x1000 r-xp /usr/lib32/ld-linux.so.2
0xe8cca000 0xe8cd8000 0xe000 0x24000 r--p /usr/lib32/ld-linux.so.2
0xe8cd8000 0xe8cda000 0x2000 0x31000 r--p /usr/lib32/ld-linux.so.2
0xe8cda000 0xe8cdb000 0x1000 0x33000 rw-p /usr/lib32/ld-linux.so.2
0xff98f000 0xff9b2000 0x23000 0x0 rw-p [stack]
Note that the memory mappings for all sections in the JikesRVM
executable are overwritten by RVM.data.image
which spans from 0x60000000
to 0x6218a000
.
It is also interesting that the mmap succeeded silently without warning about existing mmaps at the same location.
The execution went on. It eventually crashed when calling jikesrvm_alloc
. That function is defined in glue.asm
which is statically lined into the JikesRVM
executable. The address that was supposed to hold the function body of jikesrvm_alloc
is now mapped to RVM.data.image
and the address contains a sequence of 0x00
bytes which may be interpreted as add %al,(%eax)
. The program crashed with SIGSEGV, with the message
Failing instruction offset: 0xfd11f1fe in method sysAlloc with descriptor (Lorg/vmmagic/unboxed/Address;IIII)Lorg/vmmagic/unboxed/Address; declared by class with descriptor Lorg/jikesrvm/runtime/SysCall;
Example 2
The error is non-deterministic. (I left ASLR enabled by default.) In another execution, the JikesRVM
was mapped to another address:
Start Addr End Addr Size Offset Perms objfile
0x64533000 0x64534000 0x1000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64534000 0x64536000 0x2000 0x1000 r-xp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64536000 0x64538000 0x2000 0x3000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64538000 0x64539000 0x1000 0x4000 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x64539000 0x6453a000 0x1000 0x5000 rw-p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
0x655c5000 0x655e7000 0x22000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
Then it was RVM.code.image
that overwrote the mappings for JikesRVM
.
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x64000000 0x6521e000 0x121e000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.code.image
0x655c5000 0x655e6000 0x21000 0x0 rw-p [heap]
0x68000000 0x68200000 0x200000 0x0 rwxp
And the process managed to execute the third mmap for RVM.rmap.image
Start Addr End Addr Size Offset Perms objfile
0x60000000 0x6218a000 0x218a000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
0x64000000 0x6521e000 0x121e000 0x0 rwxp /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.code.image
0x655c5000 0x655e6000 0x21000 0x0 rw-p [heap]
0x67000000 0x67074000 0x74000 0x0 r--p /home/wks/projects/mmtk-github/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.rmap.image
0x68000000 0x68200000 0x200000 0x0 rwxp
and then it crashed due to SIGSEGV.
430 if (((*hasharr ^ new_hash) >> 1) == 0)
(rr) bt
#0 do_lookup_x (undef_name=undef_name@entry=0xe8e229a2 "checkMalloc", new_hash=new_hash@entry=1550431963, old_hash=old_hash@entry=0xff955b90, ref=0xe8e2221c, result=0xff955b94, scope=0xe8ed2b9c,
i=<optimized out>, version=0xe8e9bd10, flags=1, skip=0x0, type_class=1, undef_map=0xe8e9ab30) at dl-lookup.c:430
#1 0xe8ea852e in _dl_lookup_symbol_x (undef_name=0xe8e229a2 "checkMalloc", undef_map=0xe8e9ab30, ref=0xff955c10, symbol_scope=<optimized out>, version=0xe8e9bd10, type_class=1, flags=1, skip_map=0x0)
at dl-lookup.c:793
#2 0xe8eae1d6 in _dl_fixup (l=0xe8e9ab30, reloc_arg=768) at dl-runtime.c:95
#3 0xe8eb02d4 in _dl_runtime_resolve () at ../sysdeps/i386/dl-trampoline.S:38
#4 0xe8e2f40d in sysMonitorCreate () at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/sysThread.c:671
#5 0xe8e2e540 in sysInitialize () at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/sysThread.c:125
#6 0xe8e25c1d in createVM (vmInSeparateThread=0) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:459
#7 0xe8e25ffa in JNI_CreateJavaVM (mainJavaVM=0xff955d2c, mainJNIEnv=0xff955d30, initArgs=0xff955d3c) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/jvm.c:497
#8 0x645359a0 in main (argc=19, argv=0xff955e38) at /home/wks/projects/mmtk-github/jikesrvm/tools/bootloader/main.c:594
Line 671 in sysThread.c
attempts to call checkMalloc
, but the stack trace shows it didn't enter checkMalloc
, but it ended up crashing in the system linker/loader when trying to resolve the symbol checkMalloc
. It is likely because the linking metadata was overwritten by the mmap of RVM.*.image
.
How is this related to other "failing instruction" errors?
I am not sure, but it is quite likely that #108 is caused by some code sections overwritten by irrelevant mmaps.