Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/gopls: "internal error reading (shared cache|typerefs data)" (ENOSPC?) bug report (via telemetry) #67433

Open
adonovan opened this issue May 16, 2024 · 10 comments
Open

x/tools/gopls: "internal error reading (shared cache|typerefs data)" (ENOSPC?) bug report (via telemetry) #67433

adonovan opened this issue May 16, 2024 · 10 comments
Assignees
@adonovan
Labels
gopls/telemetry-wins gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
gopls/v0.19.0

Comments

@adonovan
Copy link
Member

adonovan commented May 16, 2024 •
edited
Loading 

#!stacks
("bug.Reportf" || "bug.Errorf") && 
  ("runCached:21" || "runCached:+24" || "runCached:+18" || /* analysis */
   "typerefData:+5")  /* typerefs */

This stack -iVK7w was reported by telemetry:

	// Access the cache.
	var summary *analyzeSummary
	const cacheKind = "analysis"
	if data, err := filecache.Get(cacheKind, key); err == nil {
		// cache hit
		analyzeSummaryCodec.Decode(data, &summary)
		if summary == nil { // debugging #66732
			bug.Reportf("analyzeSummaryCodec.Decode yielded nil *analyzeSummary")
		}
	} else if err != filecache.ErrNotFound {
		return nil, bug.Errorf("internal error reading shared cache: %v", err) // <--- here
	} else {
gopls/bug
golang.org/x/tools/gopls/internal/util/bug.report:+35
golang.org/x/tools/gopls/internal/util/bug.Errorf:+2
golang.org/x/tools/gopls/internal/cache.(*analysisNode).runCached:+24
golang.org/x/tools/gopls/internal/cache.(*Snapshot).Analyze.func6.1:+4
golang.org/x/sync/errgroup.(*Group).Go.func1:+3
runtime.goexit:+0

golang.org/x/tools/[email protected] go1.20.4 windows/amd64 vscode (1)

Issue created by golang.org/x/tools/gopls/internal/telemetry/cmd/stacks.

Dups: m4MTKQ UFNCpw e960MA T1ADGg HYjhzQ
@adonovan adonovan added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. gopls Issues related to the Go language server, gopls. Tools This label describes issues relating to any tools in the x/tools repository. gopls/telemetry-wins labels May 16, 2024
@gopherbot gopherbot added this to the Unreleased milestone May 16, 2024
@adonovan
Copy link
Member Author

The possible causes of this failure are:

  • can't create cache (e.g. disk full?)
  • can't hash executable (e.g. gopls unlinked, or chmodded?)
  • I/O error reading index file (e.g. EACCES, EISDIR, ENOENT?)

The first seems the most likely. The others can only happen if someone is meddling with the executable or the cache directory.

@adonovan
Copy link
Member Author

Very different stack, but same cause. (Probably the same user given the rarity, timing, and OS/ARCH.)

func (s *Snapshot) typerefData(ctx context.Context, id PackageID, imports map[ImportPath]*metadata.Package, cgfs []file.Handle) ([]byte, error) {
	key := typerefsKey(id, imports, cgfs)
	if data, err := filecache.Get(typerefsKind, key); err == nil {
		return data, nil
	} else if err != filecache.ErrNotFound {
		bug.Reportf("internal error reading typerefs data: %v", err) // <-- here
	}

This stack m4MTKQ was reported by telemetry:

gopls/bug
golang.org/x/tools/gopls/internal/util/bug.report:+35
golang.org/x/tools/gopls/internal/util/bug.Reportf:+1
golang.org/x/tools/gopls/internal/cache.(*Snapshot).typerefData:+5
golang.org/x/tools/gopls/internal/cache.(*Snapshot).typerefs:+8
golang.org/x/tools/gopls/internal/cache.(*packageHandleBuilder).buildPackageHandle:+19
golang.org/x/tools/gopls/internal/cache.(*Snapshot).getPackageHandles.func2.1:+8
golang.org/x/sync/errgroup.(*Group).Go.func1:+3
runtime.goexit:+0

golang.org/x/tools/[email protected] go1.20.4 windows/amd64 vscode (1)

Issue created by golang.org/x/tools/gopls/internal/telemetry/cmd/stacks.

@adonovan adonovan changed the title x/tools/gopls: "internal error reading shared cache" bug report (via telemetry) x/tools/gopls: "internal error reading (shared cache|typerefs data)" bug report (via telemetry) May 16, 2024
@hyangah hyangah modified the milestones: Unreleased, gopls/v0.15.4 May 20, 2024
@gabyhelp gabyhelp mentioned this issue Jun 21, 2024
Open
@adonovan adonovan changed the title x/tools/gopls: "internal error reading (shared cache|typerefs data)" bug report (via telemetry) x/tools/gopls: "internal error reading (shared cache|typerefs data)" (ENOSPC?) bug report (via telemetry) Aug 14, 2024
@findleyr
Copy link
Member

Looked at this briefly. We should probably not have a bug that can be reasonably triggered by an OS error, but should also do a better job of surfacing this error to the user. Bumping to v0.18.

@findleyr findleyr modified the milestones: gopls/v0.17.0, gopls/v0.18.0 Oct 22, 2024
@adonovan
Copy link
Member Author

adonovan commented Oct 22, 2024 •
edited
Loading

Looked at this briefly. We should probably not have a bug that can be reasonably triggered by an OS error, but should also do a better job of surfacing this error to the user. Bumping to v0.18.

I think all we need to do here is treat ENOSPC as a cache miss. As a bonus we might also want to showMessage an error when the space in the cache volume is tight; currently we report just an internal event when filecache.Set fails.

@adonovan
Copy link
Member Author

This stack UFNCpw was reported by telemetry:

golang.org/x/tools/[email protected] go1.21.4 darwin/arm64 other (6)

@adonovan
Copy link
Member Author

This stack e960MA was reported by telemetry:

golang.org/x/tools/[email protected] go1.21.4 darwin/arm64 other (6)

@adonovan
Copy link
Member Author

This stack e960MA was reported by telemetry:

golang.org/x/tools/[email protected] go1.21.4 darwin/arm64 other (6)

@adonovan
Copy link
Member Author

This stack T1ADGg was reported by telemetry:

golang.org/x/tools/[email protected] go1.23.4 darwin/arm64 other,vscode (3)

@adonovan adonovan self-assigned this Dec 30, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/639395 mentions this issue: gopls/internal/filecache: Get: mitigate failure due to ENOSPC

gopherbot pushed a commit to golang/tools that referenced this issue Jan 2, 2025
@adonovan
gopls/internal/filecache: Get: mitigate failure due to ENOSPC
b0dc9b1 
filecache.Get operations sometimes fail. This CL enumerates
a number of causes, and mitigates the likely most common
one--failure to create the cache due to ENOSPC--by forcing
cache creation during early startup.
This also minimizes the time window during which deletion
of the gopls executable is a possible cause.

If we continue to observe failures, the mostly likely remaining
cause is deletion of the cache while gopls is running.
This CL details a possible mitigation.

Updates golang/go#67433

Change-Id: I3545e56f7af308afba3527a418757b3cf4573569
Reviewed-on: https://go-review.googlesource.com/c/tools/+/639395
Reviewed-by: Robert Findley <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
@adonovan
Copy link
Member Author

adonovan commented Jan 3, 2025

This stack HYjhzQ was reported by telemetry:

golang.org/x/tools/[email protected] go1.23.4 darwin/arm64 other,vscode (3)

@findleyr findleyr modified the milestones: gopls/v0.18.0, gopls/v0.19.0 Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adonovan adonovan

Labels
gopls/telemetry-wins gopls Issues related to the Go language server, gopls. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
gopls/v0.19.0
Development

No branches or pull requests
4 participants
@hyangah @adonovan @gopherbot @findleyr