Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: "fatal error: found pointer to free object" on arm64 #71932

Closed
tianon opened this issue Feb 24, 2025 · 10 comments
Closed

cmd/compile: "fatal error: found pointer to free object" on arm64 #71932

tianon opened this issue Feb 24, 2025 · 10 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Critical A critical problem that affects the availability or correctness of production systems built using Go NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@tianon
Copy link
Contributor

tianon commented Feb 24, 2025

Go version

go version go1.24.0 linux/arm64

Output of go env in your module/workspace:

Details:
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/go/.cache'
GOCACHEPROG=''
GODEBUG=''
GOENV='/tmp/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3828578984=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/tmp/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I wish I had a smaller reproducer, but even this one only reproduces sometimes so a bisect to 98e719f is the best I've been able to do (notaryproject/notary#1708 (comment)).

With a checkout of https://github.com/notaryproject/notary, running GOARCH=arm64 go test -count 100 -run TestPublishDelegations ./client (on native arm64 hardware, emulated hardware, or even QEMU user-space emulation) should fail pretty reliably. I have not been able to reproduce a failure on any other architecture.

What did you see happen?

notaryproject/notary#1708

runtime: marked free object in span 0xffff5ec83ac0, elemsize=208 freeindex=3 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
...
fatal error: found pointer to free object

runtime stack:
...

(where most of the entries in the runtime stack are not related to the code being tested, and the ones that are don't have any obvious issues I see, and don't fail on any other architectures)

Full Log:
runtime: marked free object in span 0xffff5ec83ac0, elemsize=208 freeindex=3 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0x4000824000 alloc marked  
0x40008240d0 alloc marked  
0x40008241a0 alloc marked  
0x4000824270 free  marked   zombie
0x0000004000824270:  0x0000000000000040  0x0000000000dc5b68 
0x0000004000824280:  0x0000004000320a50  0x00000040006ab140 
0x0000004000824290:  0x0000000000000040  0x0000000000dc6078 
0x00000040008242a0:  0x0000004000320aa0  0x00000040006ab180 
0x00000040008242b0:  0x0000000000000040  0x0000000000dc6078 
0x00000040008242c0:  0x0000004000320af0  0x00000040006ab080 
0x00000040008242d0:  0x0000000000000040  0x0000000000dc5b68 
0x00000040008242e0:  0x0000004000320b90  0x0000000000000000 
0x00000040008242f0:  0x0000000000000000  0x0000000000000000 
0x0000004000824300:  0x0000000000000000  0x0000000000000000 
0x0000004000824310:  0x0000000000000000  0x0000000000000000 
0x0000004000824320:  0x0000000000000000  0x0000000000000000 
0x0000004000824330:  0x0000000000000000  0x0000000000000000 
0x4000824340 free  unmarked
0x4000824410 free  unmarked
0x40008244e0 free  unmarked
0x40008245b0 free  unmarked
0x4000824680 free  unmarked
0x4000824750 free  unmarked
0x4000824820 free  unmarked
0x40008248f0 free  unmarked
0x40008249c0 free  unmarked
0x4000824a90 free  unmarked
0x4000824b60 free  unmarked
0x4000824c30 free  unmarked
0x4000824d00 free  unmarked
0x4000824dd0 free  unmarked
0x4000824ea0 free  unmarked
0x4000824f70 free  unmarked
0x4000825040 free  unmarked
0x4000825110 free  unmarked
0x40008251e0 free  unmarked
0x40008252b0 free  unmarked
0x4000825380 free  unmarked
0x4000825450 free  unmarked
0x4000825520 free  unmarked
0x40008255f0 free  unmarked
0x40008256c0 free  unmarked
0x4000825790 free  unmarked
0x4000825860 free  unmarked
0x4000825930 free  unmarked
0x4000825a00 free  unmarked
0x4000825ad0 free  unmarked
0x4000825ba0 free  unmarked
0x4000825c70 free  unmarked
0x4000825d40 free  unmarked
0x4000825e10 free  unmarked
fatal error: found pointer to free object

runtime stack:
runtime.throw({0xca144f?, 0x4000824340?})
	/usr/lib/go-1.24/src/runtime/panic.go:1096 +0x38 fp=0xffff5fe1e4c0 sp=0xffff5fe1e490 pc=0x47a5b8
runtime.(*mspan).reportZombies(0xffff5ec83ac0)
	/usr/lib/go-1.24/src/runtime/mgcsweep.go:877 +0x2f0 fp=0xffff5fe1e540 sp=0xffff5fe1e4c0 pc=0x42de50
runtime.(*sweepLocked).sweep(0x0?, 0x0)
	/usr/lib/go-1.24/src/runtime/mgcsweep.go:652 +0x414 fp=0xffff5fe1e660 sp=0xffff5fe1e540 pc=0x42ce04
runtime.(*mcentral).uncacheSpan(0xffff5fe1e6f8?, 0x47552c?)
	/usr/lib/go-1.24/src/runtime/mcentral.go:236 +0xac fp=0xffff5fe1e690 sp=0xffff5fe1e660 pc=0x41c3dc
runtime.(*mcache).releaseAll(0xffffb4b30f30)
	/usr/lib/go-1.24/src/runtime/mcache.go:292 +0x190 fp=0xffff5fe1e700 sp=0xffff5fe1e690 pc=0x41bc30
runtime.(*mcache).prepareForSweep(0xffffb4b30f30)
	/usr/lib/go-1.24/src/runtime/mcache.go:329 +0x4c fp=0xffff5fe1e730 sp=0xffff5fe1e700 pc=0x41bd4c
runtime.gcMarkTermination.func4(0x4000064f08)
	/usr/lib/go-1.24/src/runtime/mgc.go:1214 +0x24 fp=0xffff5fe1e760 sp=0xffff5fe1e730 pc=0x475474
runtime.forEachPInternal(0xcd7940)
	/usr/lib/go-1.24/src/runtime/proc.go:2073 +0x184 fp=0xffff5fe1e7e0 sp=0xffff5fe1e760 pc=0x447dc4
runtime.gcMarkTermination.forEachP.func6()
	/usr/lib/go-1.24/src/runtime/proc.go:2032 +0x40 fp=0xffff5fe1e810 sp=0xffff5fe1e7e0 pc=0x422230
runtime.systemstack(0x0)
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:244 +0x6c fp=0xffff5fe1e820 sp=0xffff5fe1e810 pc=0x47fd5c

goroutine 7 gp=0x4000106e00 m=6 mp=0x4000480008 [flushing proc caches]:
runtime.systemstack_switch()
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:201 +0x8 fp=0x400006cc40 sp=0x400006cc30 pc=0x47fcd8
runtime.forEachP(...)
	/usr/lib/go-1.24/src/runtime/proc.go:2021
runtime.gcMarkTermination({0x60?, 0x2a7ac62a15523?, 0x4?, 0x0?})
	/usr/lib/go-1.24/src/runtime/mgc.go:1213 +0x5e0 fp=0x400006ce70 sp=0x400006cc40 pc=0x421ac0
runtime.gcMarkDone()
	/usr/lib/go-1.24/src/runtime/mgc.go:1014 +0x340 fp=0x400006cf10 sp=0x400006ce70 pc=0x421270
runtime.gcBgMarkWorker(0x400003ed20)
	/usr/lib/go-1.24/src/runtime/mgc.go:1559 +0x384 fp=0x400006cfb0 sp=0x400006cf10 pc=0x422954
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x28 fp=0x400006cfd0 sp=0x400006cfb0 pc=0x422598
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x400006cfd0 sp=0x400006cfd0 pc=0x482204
created by runtime.gcBgMarkStartWorkers in goroutine 6
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x140

goroutine 1 gp=0x40000021c0 m=nil [chan receive]:
runtime.gopark(0x40000f5a18?, 0x548530?, 0x78?, 0xf8?, 0xffff5ecca278?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40000f5970 sp=0x40000f5950 pc=0x47a6d8
runtime.chanrecv(0x40003a60e0, 0x40000f5a77, 0x1)
	/usr/lib/go-1.24/src/runtime/chan.go:664 +0x42c fp=0x40000f59f0 sp=0x40000f5970 pc=0x41152c
runtime.chanrecv1(0x1358300?, 0xdda080?)
	/usr/lib/go-1.24/src/runtime/chan.go:506 +0x14 fp=0x40000f5a20 sp=0x40000f59f0 pc=0x4110c4
testing.(*T).Run(0x4000106a80, {0xc9acb5?, 0x400025bb38?}, 0xcd7540)
	/usr/lib/go-1.24/src/testing/testing.go:1859 +0x388 fp=0x40000f5b00 sp=0x40000f5a20 pc=0x548608
testing.runTests.func1(0x4000106a80)
	/usr/lib/go-1.24/src/testing/testing.go:2279 +0x40 fp=0x40000f5b40 sp=0x40000f5b00 pc=0x54a590
testing.tRunner(0x4000106a80, 0x400025bc68)
	/usr/lib/go-1.24/src/testing/testing.go:1792 +0xe4 fp=0x40000f5b90 sp=0x40000f5b40 pc=0x5478a4
testing.runTests(0x400000ea38, {0x1304000, 0x8c, 0x8c}, {0x4000212620?, 0x7?, 0x13599c0?})
	/usr/lib/go-1.24/src/testing/testing.go:2277 +0x3ec fp=0x40000f5c90 sp=0x40000f5b90 pc=0x54a4ac
testing.(*M).Run(0x400022e6e0)
	/usr/lib/go-1.24/src/testing/testing.go:2142 +0x588 fp=0x40000f5ec0 sp=0x40000f5c90 pc=0x549208
main.main()
	_testmain.go:323 +0x90 fp=0x40000f5f40 sp=0x40000f5ec0 pc=0xaa12d0
runtime.main()
	/usr/lib/go-1.24/src/runtime/proc.go:283 +0x284 fp=0x40000f5fd0 sp=0x40000f5f40 pc=0x4441d4
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000f5fd0 sp=0x40000f5fd0 pc=0x482204

goroutine 2 gp=0x4000002c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000070f90 sp=0x4000070f70 pc=0x47a6d8
runtime.goparkunlock(...)
	/usr/lib/go-1.24/src/runtime/proc.go:441
runtime.forcegchelper()
	/usr/lib/go-1.24/src/runtime/proc.go:348 +0xb8 fp=0x4000070fd0 sp=0x4000070f90 pc=0x444528
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000070fd0 sp=0x4000070fd0 pc=0x482204
created by runtime.init.7 in goroutine 1
	/usr/lib/go-1.24/src/runtime/proc.go:336 +0x24

goroutine 3 gp=0x4000003180 m=nil [runnable]:
runtime.gopark(0x1357e01?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000071760 sp=0x4000071740 pc=0x47a6d8
runtime.goparkunlock(...)
	/usr/lib/go-1.24/src/runtime/proc.go:441
runtime.bgsweep(0x400004e080)
	/usr/lib/go-1.24/src/runtime/mgcsweep.go:316 +0x108 fp=0x40000717b0 sp=0x4000071760 pc=0x42c3d8
runtime.gcenable.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:204 +0x28 fp=0x40000717d0 sp=0x40000717b0 pc=0x420218
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000717d0 sp=0x40000717d0 pc=0x482204
created by runtime.gcenable in goroutine 1
	/usr/lib/go-1.24/src/runtime/mgc.go:204 +0x6c

goroutine 4 gp=0x4000003340 m=nil [sleep]:
runtime.gopark(0x400009c060?, 0x2a7ac63fa8ac0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000071f20 sp=0x4000071f00 pc=0x47a6d8
runtime.goparkunlock(...)
	/usr/lib/go-1.24/src/runtime/proc.go:441
runtime.(*scavengerState).sleep(0x1359dc0, 0x4115360c00000000)
	/usr/lib/go-1.24/src/runtime/mgcscavenge.go:504 +0xf8 fp=0x4000071f90 sp=0x4000071f20 pc=0x42a078
runtime.bgscavenge(0x400004e080)
	/usr/lib/go-1.24/src/runtime/mgcscavenge.go:662 +0x9c fp=0x4000071fb0 sp=0x4000071f90 pc=0x42a41c
runtime.gcenable.gowrap2()
	/usr/lib/go-1.24/src/runtime/mgc.go:205 +0x28 fp=0x4000071fd0 sp=0x4000071fb0 pc=0x4201b8
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000071fd0 sp=0x4000071fd0 pc=0x482204
created by runtime.gcenable in goroutine 1
	/usr/lib/go-1.24/src/runtime/mgc.go:205 +0xac

goroutine 5 gp=0x4000003c00 m=nil [finalizer wait]:
runtime.gopark(0x18000001b8?, 0x1000000000000?, 0xf8?, 0x5?, 0x9a940c?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000070590 sp=0x4000070570 pc=0x47a6d8
runtime.runfinq()
	/usr/lib/go-1.24/src/runtime/mfinal.go:196 +0x108 fp=0x40000707d0 sp=0x4000070590 pc=0x41f218
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000707d0 sp=0x40000707d0 pc=0x482204
created by runtime.createfing in goroutine 1
	/usr/lib/go-1.24/src/runtime/mfinal.go:166 +0x80

goroutine 18 gp=0x400038a540 m=nil [chan receive]:
runtime.gopark(0x400030b7c0?, 0x400000f260?, 0x48?, 0x4f?, 0x624d68?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40006c4ef0 sp=0x40006c4ed0 pc=0x47a6d8
runtime.chanrecv(0x40003a61c0, 0x0, 0x1)
	/usr/lib/go-1.24/src/runtime/chan.go:664 +0x42c fp=0x40006c4f70 sp=0x40006c4ef0 pc=0x41152c
runtime.chanrecv1(0x0?, 0x0?)
	/usr/lib/go-1.24/src/runtime/chan.go:506 +0x14 fp=0x40006c4fa0 sp=0x40006c4f70 pc=0x4110c4
runtime.unique_runtime_registerUniqueMapCleanup.func2(...)
	/usr/lib/go-1.24/src/runtime/mgc.go:1796
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:1799 +0x3c fp=0x40006c4fd0 sp=0x40006c4fa0 pc=0x42343c
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40006c4fd0 sp=0x40006c4fd0 pc=0x482204
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
	/usr/lib/go-1.24/src/runtime/mgc.go:1794 +0x78

goroutine 4926 gp=0x400038a8c0 m=nil [runnable]:
reflect.(*rtype).Field(0xc1e320?, 0x0?)
	/usr/lib/go-1.24/src/reflect/type.go:740 +0x12c fp=0x4000054570 sp=0x4000054570 pc=0x504dcc
github.com/docker/go/canonical/json.typeFields({0xdd5000?, 0xc1e320?})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/encode.go:1059 +0x84c fp=0x4000054bd0 sp=0x4000054570 pc=0x7964dc
github.com/docker/go/canonical/json.cachedTypeFields({0xdd5000, 0xc1e320}, 0x0)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/encode.go:1227 +0x12c fp=0x4000054c50 sp=0x4000054bd0 pc=0x796e5c
github.com/docker/go/canonical/json.(*decodeState).object(0x4000738c30, {0xb39f80?, 0x40007640a0?, 0xcd82f0?})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/decode.go:640 +0x5e8 fp=0x4000054e20 sp=0x4000054c50 pc=0x78e0b8
github.com/docker/go/canonical/json.(*decodeState).value(0x4000738c30, {0xb39f80?, 0x40007640a0?, 0x799734?})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/decode.go:368 +0x80 fp=0x4000054e70 sp=0x4000054e20 pc=0x78cc30
github.com/docker/go/canonical/json.(*decodeState).unmarshal(0x4000738c30, {0xb39f80?, 0x40007640a0?})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/decode.go:168 +0x124 fp=0x4000054f00 sp=0x4000054e70 pc=0x78c464
github.com/docker/go/canonical/json.Unmarshal({0x4000759000, 0xf27, 0x1000}, {0xb39f80, 0x40007640a0})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/docker/go/canonical/json/decode.go:96 +0xb4 fp=0x4000054f40 sp=0x4000054f00 pc=0x78c184
github.com/theupdateframework/notary/tuf/data.canonicalJSON.Unmarshal(...)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/tuf/data/serializer.go:28
github.com/theupdateframework/notary/tuf/data.(*canonicalJSON).Unmarshal(0x10000200054f98?, {0x4000759000?, 0xffffb4b30108?, 0x10?}, {0xb39f80?, 0x40007640a0?})
	<autogenerated>:1 +0x48 fp=0x4000054f80 sp=0x4000054f40 pc=0x7affc8
github.com/theupdateframework/notary/tuf/data.RootFromSigned(0x4000213300)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/tuf/data/root.go:159 +0x74 fp=0x4000055020 sp=0x4000054f80 pc=0x7ab8f4
github.com/theupdateframework/notary/trustpinning.ValidateRoot(0x0, 0x4000213300, {0xc9563c, 0x11}, {0x0?, 0x0?, 0x0?})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/trustpinning/certs.go:89 +0x98 fp=0x4000055470 sp=0x4000055020 pc=0x7c81a8
github.com/theupdateframework/notary/tuf.(*repoBuilder).loadRoot(0x400071a2a0, {0x40004f5500?, 0x40000555d8?, 0x417c68?}, 0x1, 0x0, 0x5a?)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/tuf/builder.go:442 +0x78 fp=0x4000055590 sp=0x4000055470 pc=0x7cccd8
github.com/theupdateframework/notary/tuf.(*repoBuilder).loadOptions(0x400071a2a0, {0xc88440, 0x4}, {0x40004f5500, 0x10bf, 0x1500}, 0x1, 0x0, 0x0, 0x0)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/tuf/builder.go:288 +0x42c fp=0x4000055650 sp=0x4000055590 pc=0x7cbccc
github.com/theupdateframework/notary/tuf.(*repoBuilder).Load(0x4000055601?, {0xc88440?, 0x40000556e8?}, {0x40004f5500?, 0x40006a0bf0?, 0x1?}, 0x1?, 0xd0?)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/tuf/builder.go:249 +0x2c fp=0x40000556a0 sp=0x4000055650 pc=0x7cb77c
github.com/theupdateframework/notary/tuf.(*repoBuilderWrapper).Load(0x4000055738?, {0xc88440?, 0x4?}, {0x40004f5500?, 0x40000556d0?, 0x511a94?}, 0x0?, 0x0?)
	<autogenerated>:1 +0x38 fp=0x40000556f0 sp=0x40000556a0 pc=0x7d63f8
github.com/theupdateframework/notary/client.bootstrapClient({{0xc9563c, 0x11}, {0x0, 0x0, 0x0}, {0xdcb9d0, 0x40002ffba8}, {0xdcac08, 0x40003168c0}, {0xdcc7d8, ...}, ...})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/tufclient.go:371 +0x5d0 fp=0x4000055870 sp=0x40000556f0 pc=0xa61a30
github.com/theupdateframework/notary/client.LoadTUFRepo({{0xc9563c, 0x11}, {0x0, 0x0, 0x0}, {0xdcb9d0, 0x40002ffba8}, {0xdcac08, 0x40003168c0}, {0xdcc7d8, ...}, ...})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/tufclient.go:436 +0xe0 fp=0x4000055930 sp=0x4000055870 pc=0xa61e40
github.com/theupdateframework/notary/client.(*repository).updateTUF(0x4000600870, 0x0?)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/client.go:128 +0xcc fp=0x4000055a00 sp=0x4000055930 pc=0xa55dcc
github.com/theupdateframework/notary/client.(*repository).publish(0x4000600870, {0xdcb930, 0x4000206d80})
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/client.go:568 +0x34 fp=0x4000055d80 sp=0x4000055a00 pc=0xa58234
github.com/theupdateframework/notary/client.(*repository).Publish(0x4000600870)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/client.go:551 +0x28 fp=0x4000055de0 sp=0x4000055d80 pc=0xa58138
github.com/theupdateframework/notary/client.TestRotateRootMultiple(0x4000649a40)
	/build/reproducible-path/notary-0.7.0+git20240416.9d2b3b3+ds1/_build/src/github.com/theupdateframework/notary/client/client_test.go:3093 +0x53c fp=0x4000055f60 sp=0x4000055de0 pc=0xa7efcc
testing.tRunner(0x4000649a40, 0xcd7540)
	/usr/lib/go-1.24/src/testing/testing.go:1792 +0xe4 fp=0x4000055fb0 sp=0x4000055f60 pc=0x5478a4
testing.(*T).Run.gowrap1()
	/usr/lib/go-1.24/src/testing/testing.go:1851 +0x2c fp=0x4000055fd0 sp=0x4000055fb0 pc=0x54872c
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000055fd0 sp=0x4000055fd0 pc=0x482204
created by testing.(*T).Run in goroutine 1
	/usr/lib/go-1.24/src/testing/testing.go:1851 +0x374

goroutine 8 gp=0x4000106fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x2a7ac5ead906b?, 0x4000280ae0?, 0x1b?, 0xa?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000072710 sp=0x40000726f0 pc=0x47a6d8
runtime.gcBgMarkWorker(0x400003ed20)
	/usr/lib/go-1.24/src/runtime/mgc.go:1423 +0xdc fp=0x40000727b0 sp=0x4000072710 pc=0x4226ac
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x28 fp=0x40000727d0 sp=0x40000727b0 pc=0x422598
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000727d0 sp=0x40000727d0 pc=0x482204
created by runtime.gcBgMarkStartWorkers in goroutine 6
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x140

goroutine 9 gp=0x4000107180 m=nil [GC worker (idle)]:
runtime.gopark(0x2a7ac62869dbb?, 0x3?, 0x41?, 0x97?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000072f10 sp=0x4000072ef0 pc=0x47a6d8
runtime.gcBgMarkWorker(0x400003ed20)
	/usr/lib/go-1.24/src/runtime/mgc.go:1423 +0xdc fp=0x4000072fb0 sp=0x4000072f10 pc=0x4226ac
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x28 fp=0x4000072fd0 sp=0x4000072fb0 pc=0x422598
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000072fd0 sp=0x4000072fd0 pc=0x482204
created by runtime.gcBgMarkStartWorkers in goroutine 6
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x140

goroutine 10 gp=0x4000107340 m=nil [GC worker (idle)]:
runtime.gopark(0x2a7ac49d08e10?, 0x3?, 0x1f?, 0x2f?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x4000073710 sp=0x40000736f0 pc=0x47a6d8
runtime.gcBgMarkWorker(0x400003ed20)
	/usr/lib/go-1.24/src/runtime/mgc.go:1423 +0xdc fp=0x40000737b0 sp=0x4000073710 pc=0x4226ac
runtime.gcBgMarkStartWorkers.gowrap1()
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x28 fp=0x40000737d0 sp=0x40000737b0 pc=0x422598
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000737d0 sp=0x40000737d0 pc=0x482204
created by runtime.gcBgMarkStartWorkers in goroutine 6
	/usr/lib/go-1.24/src/runtime/mgc.go:1339 +0x140

goroutine 4875 gp=0x40007cfc00 m=nil [IO wait]:
runtime.gopark(0x40006c5af8?, 0x47e470?, 0x38?, 0x5b?, 0x77bcf4?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40006c5ac0 sp=0x40006c5aa0 pc=0x47a6d8
runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
	/usr/lib/go-1.24/src/runtime/netpoll.go:575 +0x158 fp=0x40006c5b00 sp=0x40006c5ac0 pc=0x43ce28
internal/poll.runtime_pollWait(0xffff6d8a7be8, 0x72)
	/usr/lib/go-1.24/src/runtime/netpoll.go:351 +0xa0 fp=0x40006c5b30 sp=0x40006c5b00 pc=0x479890
internal/poll.(*pollDesc).wait(0x4000118d00?, 0x4000523000?, 0x0)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40006c5b60 sp=0x40006c5b30 pc=0x4bf568
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x4000118d00, {0x4000523000, 0x1000, 0x1000})
	/usr/lib/go-1.24/src/internal/poll/fd_unix.go:165 +0x1fc fp=0x40006c5c00 sp=0x40006c5b60 pc=0x4c081c
net.(*netFD).Read(0x4000118d00, {0x4000523000?, 0x80?, 0x8?})
	/usr/lib/go-1.24/src/net/fd_posix.go:55 +0x28 fp=0x40006c5c50 sp=0x40006c5c00 pc=0x653548
net.(*conn).Read(0x4000074908, {0x4000523000?, 0x0?, 0x40006c5d28?})
	/usr/lib/go-1.24/src/net/net.go:194 +0x34 fp=0x40006c5ca0 sp=0x40006c5c50 pc=0x661dd4
net/http.(*persistConn).Read(0x400051e5a0, {0x4000523000?, 0xdbf8b8?, 0x12e1ec0?})
	/usr/lib/go-1.24/src/net/http/transport.go:2122 +0x4c fp=0x40006c5d00 sp=0x40006c5ca0 pc=0x77759c
bufio.(*Reader).fill(0x400071af60)
	/usr/lib/go-1.24/src/bufio/bufio.go:113 +0xf8 fp=0x40006c5d40 sp=0x40006c5d00 pc=0x52f3b8
bufio.(*Reader).Peek(0x400071af60, 0x1)
	/usr/lib/go-1.24/src/bufio/bufio.go:152 +0x60 fp=0x40006c5d60 sp=0x40006c5d40 pc=0x52f520
net/http.(*persistConn).readLoop(0x400051e5a0)
	/usr/lib/go-1.24/src/net/http/transport.go:2275 +0x12c fp=0x40006c5fb0 sp=0x40006c5d60 pc=0x77824c
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/go-1.24/src/net/http/transport.go:1944 +0x28 fp=0x40006c5fd0 sp=0x40006c5fb0 pc=0x776b78
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40006c5fd0 sp=0x40006c5fd0 pc=0x482204
created by net/http.(*Transport).dialConn in goroutine 4873
	/usr/lib/go-1.24/src/net/http/transport.go:1944 +0x11c4

goroutine 4987 gp=0x4000740e00 m=nil [IO wait]:
runtime.gopark(0x400046d908?, 0x4c24c8?, 0x40?, 0x0?, 0x8?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40000f98e0 sp=0x40000f98c0 pc=0x47a6d8
runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
	/usr/lib/go-1.24/src/runtime/netpoll.go:575 +0x158 fp=0x40000f9920 sp=0x40000f98e0 pc=0x43ce28
internal/poll.runtime_pollWait(0xffff6d8a7ad0, 0x72)
	/usr/lib/go-1.24/src/runtime/netpoll.go:351 +0xa0 fp=0x40000f9950 sp=0x40000f9920 pc=0x479890
internal/poll.(*pollDesc).wait(0x400020c180?, 0x4000224000?, 0x0)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40000f9980 sp=0x40000f9950 pc=0x4bf568
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x400020c180, {0x4000224000, 0x1000, 0x1000})
	/usr/lib/go-1.24/src/internal/poll/fd_unix.go:165 +0x1fc fp=0x40000f9a20 sp=0x40000f9980 pc=0x4c081c
net.(*netFD).Read(0x400020c180, {0x4000224000?, 0x400046dab8?, 0x759e5c?})
	/usr/lib/go-1.24/src/net/fd_posix.go:55 +0x28 fp=0x40000f9a70 sp=0x40000f9a20 pc=0x653548
net.(*conn).Read(0x4000494090, {0x4000224000?, 0x72?, 0x4000816090?})
	/usr/lib/go-1.24/src/net/net.go:194 +0x34 fp=0x40000f9ac0 sp=0x40000f9a70 pc=0x661dd4
net/http.(*connReader).Read(0x4000816f00, {0x4000224000, 0x1000, 0x1000})
	/usr/lib/go-1.24/src/net/http/server.go:798 +0x234 fp=0x40000f9b20 sp=0x40000f9ac0 pc=0x75a064
bufio.(*Reader).fill(0x40002a2600)
	/usr/lib/go-1.24/src/bufio/bufio.go:113 +0xf8 fp=0x40000f9b60 sp=0x40000f9b20 pc=0x52f3b8
bufio.(*Reader).Peek(0x40002a2600, 0x4)
	/usr/lib/go-1.24/src/bufio/bufio.go:152 +0x60 fp=0x40000f9b80 sp=0x40000f9b60 pc=0x52f520
net/http.(*conn).serve(0x40003b4480, {0xdc79a0, 0x4000817170})
	/usr/lib/go-1.24/src/net/http/server.go:2137 +0x664 fp=0x40000f9fa0 sp=0x40000f9b80 pc=0x75f204
net/http.(*Server).Serve.gowrap3()
	/usr/lib/go-1.24/src/net/http/server.go:3454 +0x30 fp=0x40000f9fd0 sp=0x40000f9fa0 pc=0x7642a0
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40000f9fd0 sp=0x40000f9fd0 pc=0x482204
created by net/http.(*Server).Serve in goroutine 4983
	/usr/lib/go-1.24/src/net/http/server.go:3454 +0x3d8

goroutine 4983 gp=0x4000648a80 m=nil [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40006cbb90 sp=0x40006cbb70 pc=0x47a6d8
runtime.netpollblock(0x7000000000?, 0x6?, 0x0?)
	/usr/lib/go-1.24/src/runtime/netpoll.go:575 +0x158 fp=0x40006cbbd0 sp=0x40006cbb90 pc=0x43ce28
internal/poll.runtime_pollWait(0xffff6d8a7d00, 0x72)
	/usr/lib/go-1.24/src/runtime/netpoll.go:351 +0xa0 fp=0x40006cbc00 sp=0x40006cbbd0 pc=0x479890
internal/poll.(*pollDesc).wait(0x40004dba80?, 0x406c88?, 0x0)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40006cbc30 sp=0x40006cbc00 pc=0x4bf568
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/go-1.24/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x40004dba80)
	/usr/lib/go-1.24/src/internal/poll/fd_unix.go:620 +0x24c fp=0x40006cbce0 sp=0x40006cbc30 pc=0x4c3e3c
net.(*netFD).accept(0x40004dba80)
	/usr/lib/go-1.24/src/net/fd_unix.go:172 +0x28 fp=0x40006cbda0 sp=0x40006cbce0 pc=0x654f78
net.(*TCPListener).accept(0x400028e400)
	/usr/lib/go-1.24/src/net/tcpsock_posix.go:159 +0x24 fp=0x40006cbdf0 sp=0x40006cbda0 pc=0x66a9e4
net.(*TCPListener).Accept(0x400028e400)
	/usr/lib/go-1.24/src/net/tcpsock.go:380 +0x2c fp=0x40006cbe30 sp=0x40006cbdf0 pc=0x669c7c
net/http.(*onceCloseListener).Accept(0x40003b4480?)
	<autogenerated>:1 +0x30 fp=0x40006cbe50 sp=0x40006cbe30 pc=0x78a170
net/http.(*Server).Serve(0x400010d500, {0xdc6168, 0x400028e400})
	/usr/lib/go-1.24/src/net/http/server.go:3424 +0x290 fp=0x40006cbf80 sp=0x40006cbe50 pc=0x763f10
net/http/httptest.(*Server).goServe.func1()
	/usr/lib/go-1.24/src/net/http/httptest/server.go:311 +0x5c fp=0x40006cbfd0 sp=0x40006cbf80 pc=0x80c82c
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40006cbfd0 sp=0x40006cbfd0 pc=0x482204
created by net/http/httptest.(*Server).goServe in goroutine 4926
	/usr/lib/go-1.24/src/net/http/httptest/server.go:309 +0x6c

goroutine 4876 gp=0x4000649340 m=nil [select]:
runtime.gopark(0x40006c7f38?, 0x2?, 0xa8?, 0x7d?, 0x40006c7ee4?)
	/usr/lib/go-1.24/src/runtime/proc.go:435 +0xc8 fp=0x40006c7d70 sp=0x40006c7d50 pc=0x47a6d8
runtime.selectgo(0x40006c7f38, 0x40006c7ee0, 0x4000573380?, 0x0, 0x40008016e0?, 0x1)
	/usr/lib/go-1.24/src/runtime/select.go:351 +0x6c4 fp=0x40006c7ea0 sp=0x40006c7d70 pc=0x457cd4
net/http.(*persistConn).writeLoop(0x400051e5a0)
	/usr/lib/go-1.24/src/net/http/transport.go:2590 +0x9c fp=0x40006c7fb0 sp=0x40006c7ea0 pc=0x77995c
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/go-1.24/src/net/http/transport.go:1945 +0x28 fp=0x40006c7fd0 sp=0x40006c7fb0 pc=0x776b18
runtime.goexit({})
	/usr/lib/go-1.24/src/runtime/asm_arm64.s:1223 +0x4 fp=0x40006c7fd0 sp=0x40006c7fd0 pc=0x482204
created by net/http.(*Transport).dialConn in goroutine 4873
	/usr/lib/go-1.24/src/net/http/transport.go:1945 +0x120c
FAIL	github.com/theupdateframework/notary/client	10.905s

What did you expect to see?

The test passing. 😅

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Feb 24, 2025
@tianon
Copy link
Contributor Author

tianon commented Feb 24, 2025

For added context, the test does pass on Go 1.23, and as noted above, I did a bisect down to 98e719f, but unfortunately that's where things went over my head. 🙇 ❤️

cc @randall77 ❤️

@randall77
Copy link
Contributor

Thanks for the detailed report.
I suspect those new rules, when applied to pointer types, might be making an out-of-range temp pointer.
I will take a look.

@randall77 randall77 self-assigned this Feb 24, 2025
@randall77
Copy link
Contributor

@gopherbot please open a backport issue for 1.24.

@gopherbot
Copy link
Contributor

Backport issue(s) opened: #71938 (for 1.24).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/652078 mentions this issue: cmd/compile: don't pull constant offsets out of pointer arithmetic

@randall77 randall77 added the Critical A critical problem that affects the availability or correctness of production systems built using Go label Feb 24, 2025
@randall77 randall77 modified the milestones: Go1.24.1, Go1.25 Feb 24, 2025
@thaJeztah
Copy link
Contributor

Thanks @tianon ! And thanks @randall77 for picking this up ❤️

🙈 not sure if it's just noise, or "related"; we got a report in docker/docker from a packager trying to use go1.24, and they ran into a SEGFAULT. We have not yet bisected that (or tried to reproduce), as we're currently on go1.23 ourselves, but could there be any relation? From moby/moby#49513 (comment) (but I think there's some other traces in the ticket)

(lldb) target create "/usr/bin/dockerd" --core "/var/tmp/coredump-qxcSvq"
Core file '/var/tmp/coredump-qxcSvq' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'dockerd', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x0000563381cd1c58 dockerd`runtime.mallocgcSmallNoscan + 216
    frame #1: 0x0000563381d30fd9 dockerd`runtime.mallocgc + 185
    frame #2: 0x0000563381d36429 dockerd`runtime.growslice + 1481
    frame #3: 0x0000563381d2d916 dockerd`runtime.vgetrandomPutState + 86
    frame #4: 0x0000563381d011e5 dockerd`runtime.mexit + 453

@mvdan
Copy link
Member

mvdan commented Feb 25, 2025

We hit this too: #71451 (comment)

Thanks @tianon for reporting this! I attempted to do the same, but clearly I went to the wrong issue :)

@randall77 although I wonder if that earlier issue could be related? The platform is different, but the panic looks very similar.

@randall77
Copy link
Contributor

randall77 commented Feb 25, 2025

Directly related, no. This issue is arm64 only.
However, this kind of bug, making an invalid pointer mid-pointer-arithmetic, has happened before (e.g. CL 225798), and I would not be terribly surprised if there's a similar bug lurking in amd64 (or other archs) somewhere.

@dmitshur dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 26, 2025
@tianon
Copy link
Contributor Author

tianon commented Feb 26, 2025

Sorry, I meant to test this before it merged, but FWIW I've confirmed that the fix works in the codebase I was testing against. ❤️

Thank you!

@thaJeztah
Copy link
Contributor

Directly related, no. This issue is arm64 only. However, this kind of bug, making an invalid pointer mid-pointer-arithmetic, has happened before (e.g. CL 225798), and I would not be terribly surprised if there's a similar bug lurking in amd64 (or other archs) somewhere.

Thanks! Yes, fully understandable. It was really a bit of a drive-by comment from me, in hopes that possibly it would ring a bell (from the little information available) or if not, to ever so slightly spread awareness.

We have not yet opened a ticket (and I don't think the original reporter has), as we currently don't have a small and consistent reproducer, and "build this giant project with obscure bash scripts (for <reasons>)" unlikely would be very acceptable for the Go maintainers. We might still do so though if we don't find a different option (and if we have clear steps to reproduce).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. Critical A critical problem that affects the availability or correctness of production systems built using Go NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

6 participants