x/tools/gopls: poor performance with very large packages in import graph #61207

sff2578 · 2023-06-23T18:25:51Z

What version of Go, VS Code & VS Code Go extension are you using?

Version Information

Run go version to get version of Go from the VS Code integrated terminal.
- go version go1.18.10 linux/amd64
Run gopls -v version to get version of Gopls from the VS Code integrated terminal.
- golang.org/x/tools/gopls v0.12.3
  golang.org/x/tools/[email protected] h1:u0wCI9uvt7mnmri6bFBIaWw1XCN6PN8hKv55Zwd+GbE=
Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.
Check your installed extensions to get the version of the VS Code Go extension
- v0.39.0
Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > Go: Locate Configured Go Tools command.
- go: /usr/local/go1.18.10/bin/go: go version go1.18.10 linux/amd64
gotests: /usr/local/gopkgs/bin/gotests (version: v1.6.0 built with go: go1.18.10)
gomodifytags: /usr/local/gopkgs/bin/gomodifytags (version: v1.16.0 built with go: go1.18.10)
impl: /usr/local/gopkgs/bin/impl (version: v1.1.0 built with go: go1.18.10)
goplay: /usr/local/gopkgs/bin/goplay (version: v1.0.0 built with go: go1.18.10)
dlv: /usr/local/gopkgs/bin/dlv (version: v1.20.2 built with go: go1.18.10)
staticcheck: /usr/local/gopkgs/bin/staticcheck (version: v0.3.3 built with go: go1.18.10)
gopls: /usr/local/gopkgs/bin/gopls (version: v0.12.3 built with go: go1.18.10)

Share the Go related settings you have added/edited

Run Preferences: Open Settings (JSON) command to open your settings.json file.
Share all the settings with the go. or ["go"] or gopls prefixes.

Describe the bug

i manually installed gopls with version v0.11.0, but after some time(less than an hour), go extension will automatically update it to the latest version. I'm having issue with latest gopls(comsuming too much CPU and slow) so i need to stay with older version

Steps to reproduce the behavior:

manually install gopls v0.11.0
connect vscode to server, work for a while
observer gopls is upgraded, in the "go output" of vscode, saw logs:

Installing 1 tool at /usr/local/gopkgs/bin in module mode.
  [email protected]

Installing golang.org/x/tools/[email protected] (/usr/local/gopkgs/bin/gopls) SUCCEEDED

Screenshots or recordings

If applicable, add screenshots or recordings to help explain your problem.

The text was updated successfully, but these errors were encountered:

findleyr · 2023-06-23T18:56:34Z

I'm having issue with latest gopls(comsuming too much CPU and slow) so i need to stay with older version

Can you please explain more? The latest gopls should not be appreciably slower than v0.11.0. If you are observing this, it would really help us if you could share more of what you see. Are you working in an open-source project? What do you perceive as being slow? When do you notice the CPU?

sff2578 · 2023-06-24T00:32:03Z

I'm not working on an open source project, i rely on gopls to do auto formatting and auto complete, with version v0.12.3, when i save, it take about 10-15 seconds to complete the save. Up on save, a notice pop up saying: Saving xxx.go:Getting code actions from "Go"
also when i use an interface, waiting for the suggestion dropdown list, it takes even longer. Meantime i have a top running to see the usage, gopls sometimes take 500% of CPU or even go up to 1000%.
with v0.11.0, this won't happen, everything happens within a secode and CPU is not that high

findleyr · 2023-06-26T14:21:01Z

@sff2578 what you describe is a bug, plain and simple. We want [email protected]+ to perform approximately the same as v0.11.0, using much less memory (and in the future, we want it to perform better than v0.11.0 across the board).

I know of certain areas where we could reduce CPU in v0.12.x (#60926), but those would be say a 50% reduction in CPU while typing. I wasn't aware of any order-of-magnitude regressions. We would greatly appreciate it if you would work with us to help us understand the problems you are facing. This could help many other users.

To prevent auto-upgrade, you can use the following setting, but as mentioned above we hope that you will work with us to resolve your problems with v0.12.x:
https://github.com/golang/vscode-go/wiki/settings#gotoolsmanagementautoupdate

Can you please run the gopls stats -anon command using gopls@latest, and share your results? That will give us a sense of the size of your repo. The -anon flag should remove any output that could contain file paths, but please do inspect the json blob to make sure it is something you are comfortable sharing.

findleyr · 2023-06-26T14:23:19Z

CC @adonovan, who can also help track down this regression.

sff2578 · 2023-06-26T21:15:59Z

Thanks for the pointer, it now works fine without upgrading gopls, here's the output of the gopls stats -anon command:

{
  "DirStats": {
    "Files": 706,
    "TestdataFiles": 0,
    "GoFiles": 606,
    "ModFiles": 1,
    "Dirs": 23
  },
  "GOARCH": "amd64",
  "GOOS": "linux",
  "GoVersion": "go1.18.10",
  "GoplsVersion": "v0.12.4",
  "InitialWorkspaceLoadDuration": "10.497067387s",
  "MemStats": {
    "HeapAlloc": 1640464000,
    "HeapInUse": 1990836224,
    "TotalAlloc": 4751304536
  },
  "WorkspaceStats": {
    "Files": {
      "Total": 7815,
      "Largest": 4950165,
      "Errs": 0
    },
    "Views": [
      {
        "GoCommandVersion": "go1.18.10",
        "AllPackages": {
          "Packages": 1073,
          "LargestPackage": 514,
          "CompiledGoFiles": 7880,
          "Modules": 176
        },
        "WorkspacePackages": {
          "Packages": 9,
          "LargestPackage": 505,
          "CompiledGoFiles": 673,
          "Modules": 1
        },
        "Diagnostics": 56
      }
    ]
  }
}

findleyr · 2023-06-26T21:21:02Z

Thanks @sff2578, that's really useful.

I notice something about your workspace, which may contribute tot he problem. According to that stats command, you have a package with 505 files in it. There are certain aspects of our new implementation that may perform very poorly on such huge packages.

Are you often working in that package? Did you notice the performance problems more significantly when working in that package?

sff2578 · 2023-06-27T21:08:59Z

Thanks for the reply. i do heavily rely on this package, i use the interfaces and functions in this package a lot. i tried manually created a simple package, it is not hitting this issue

adonovan · 2023-06-29T14:56:00Z

I just ran an ecosystem metrics query for large packages, and quickly found several with >500 files. For example, this one has 672: https://github.com/oracle/oci-go-sdk/blob/master/dataintegration/. I ran gopls v0.12 on it and observed that it was indeed slow, especially to save a file, though still quite usable. A CPU profile showed time spent in parsing, type-checking, and analysis, but not typerefs. (It also showed the usual exaggerated hotspot of filecache.gc because of macOS setitimer syscall skew. I need to try it again on Linux)

findleyr · 2023-07-06T19:02:02Z

I'm going to repurpose this issue to track fixing performance with large packages.

crclz · 2023-07-12T09:41:55Z

I and my colleagues is experiencing the same problem. gopls v0.12.x is very slow when working on any large golang package. I and my colleagues switched to go pls v0.11.0 and solved this problem.

We have tested on MacOs and linux. gopls produces same behavior.

crclz · 2023-07-12T09:59:34Z

@findleyr here is my output of gopls stats -anon

{
  "DirStats": {
    "Files": 3668,
    "TestdataFiles": 0,
    "GoFiles": 491,
    "ModFiles": 1,
    "Dirs": 386
  },
  "GOARCH": "amd64",
  "GOOS": "linux",
  "GoVersion": "go1.20.5",
  "GoplsVersion": "v0.12.4",
  "InitialWorkspaceLoadDuration": "5.689082703s",
  "MemStats": {
    "HeapAlloc": 285930864,
    "HeapInUse": 391864320,
    "TotalAlloc": 5906972808
  },
  "WorkspaceStats": {
    "Files": {
      "Total": 7270,
      "Largest": 7677237,
      "Errs": 0
    },
    "Views": [
      {
        "GoCommandVersion": "go1.20.5",
        "AllPackages": {
          "Packages": 1415,
          "LargestPackage": 291,
          "CompiledGoFiles": 7535,
          "Modules": 269
        },
        "WorkspacePackages": {
          "Packages": 79,
          "LargestPackage": 67,
          "CompiledGoFiles": 624,
          "Modules": 1
        },
        "Diagnostics": 0
      }
    ]
  }
}

gopherbot · 2023-07-14T17:59:15Z

Change https://go.dev/cl/509558 mentions this issue: gopls/internal/regtest/bench: add an 'oracle' benchmark repo

This repo has a package with a large number of files, which has interesting performance characteristics. For golang/go#61207 Change-Id: Id313762019ca85dc2c03c7dc23b005a81d900419 Reviewed-on: https://go-review.googlesource.com/c/tools/+/509558 TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]> Reviewed-by: Alan Donovan <[email protected]>

gopherbot · 2023-07-21T13:16:11Z

Change https://go.dev/cl/511337 mentions this issue: gopls/internal/lsp/cache: cache recently parsed files

gopherbot · 2023-07-21T14:31:38Z

Change https://go.dev/cl/503440 mentions this issue: gopls/internal/lsp/cache: move purgeFuncBodies into the parse cache

Our arbitrary choice of caching 200 recent files has proven to be problematic when editing very large packages (golang/go#61207). Fix this by switching to time-based eviction, but with a minimum cache size to optimize the case where editing is resumed after a while. This caused an increase in high-water mark during initial workspace load, but that is mitigated by avoiding the parse cache when type checking for import (i.e. non-workspace files): such parsed files are almost never cache hits as they are only ever parsed once, and would instead benefit more from avoiding ParseComments. Move the ownership of the parseCache to the cache.Session (and pass it to each View) to make its lifecycle clearer and avoid passing it around to each snapshot. For golang/go#61207 Change-Id: I357d8b1fa36eabb516dbb7147266df0e5153ac11 Reviewed-on: https://go-review.googlesource.com/c/tools/+/511337 TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Alan Donovan <[email protected]> Run-TryBot: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]>

When working in a package we must repeatedly re-build package handles, which requires parsing with purged func bodies. Although purging func bodies leads to faster parsing, it is still expensive enough to warrant caching. Move the 'purgeFuncBodies' field into the parse cache. For golang/go#61207 Change-Id: I90575e5b6be7181743e8376c24312115a1029188 Reviewed-on: https://go-review.googlesource.com/c/tools/+/503440 gopls-CI: kokoro <[email protected]> Reviewed-by: Alan Donovan <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Robert Findley <[email protected]>

findleyr · 2023-07-26T23:34:21Z

@sff2578 @crclz we'd love it if you'd try the v0.13.0 prerelease, and let us know if it improves things for you.

go install golang.org/x/tools/[email protected]

Also, do any of you have "staticcheck" enabled? We have observed that this can cause significant slowdown on v0.12.x vs v0.11.0.

Thanks.

findleyr · 2023-07-31T13:43:27Z

We believe this is largely mitigated, based on our own benchmarking. If this is not the case for your repos, please comment with more information.

crclz · 2023-08-06T06:41:56Z

@findleyr Thank you for the new version of gopls! Comparing with v0.12.2, the speed of code formatting of v0.13.0 is largely optimized.
However, when editing a large go package, the intellisense speed of v0.13.0 is still slower than v0.11.0 (roughly ~500ms vs <100ms), which is quite significant (at least for me). I hope the intellisense speed also could be optimized.

findleyr · 2023-08-06T15:17:54Z

@crclz that's a surprising difference, which is unexpected. Out of curiosity, does this reproduce when only a single file is open (there are reasons why intellisense may be slower with many files open).

It would be helpful if we could dig into this more, grabbing a profile.

crclz · 2023-08-06T15:59:08Z

@findleyr Only a single file is open. I have recorded my screen, wish it could help.

v0.13.0:

v0.11.0:

adonovan · 2023-08-07T08:58:27Z

Re-opening due to above report of slow completion in large packages.

findleyr · 2023-08-07T13:11:32Z

I suspect this may be a result of our change to the meaning of "completionBudget": previously, we would start the timer before type checking, now we start it after (and we ensure that the depth=0 pass completes).

In that case, I think the fix here is to support streaming completion results, which will significantly reduce the perceived latency. Then we don't have to care so much about e.g. how many unimported completion items we compute.

gopherbot · 2023-08-07T22:12:42Z

Change https://go.dev/cl/516678 mentions this issue: gopls/internal/regtest/bench: add an oracle completion benchmark

gopherbot · 2023-08-07T22:12:43Z

Change https://go.dev/cl/511435 mentions this issue: gopls/internal/lsp: avoid duplicate type checking following invalidation

findleyr · 2023-08-07T22:23:21Z

@sff2578 I'm having trouble reproducing the behavior you observe, even in very large packages. However, I have theories.

I'm curious if the CL above improves completion performance for you. If you don't mind, could you test it out?

git clone https://go.googlesource.com/tools
cd tools/gopls
git fetch https://go.googlesource.com/tools refs/changes/35/511435/2 && git checkout FETCH_HEAD
go install

That should significantly reduce CPU during autocompletion in large packages, but doesn't have a large impact on latency for me. It may be different on your computer, if you have less processing available.

Could you also try reducing your completion budget, setting "completionBudget": "10ms", to see if that has an effect?

Thanks.

sff2578 · 2023-08-07T22:28:40Z

@findleyr sorry was busy recently, i experienced the same thing as @crclz mentioned, i'll try the steps you provided and share the result when i have time

findleyr · 2023-08-08T12:50:13Z

In that case, I think the fix here is to support streaming completion results, which will significantly reduce the perceived latency.

BTW, unfortunately streaming support is not available in VS Code: microsoft/vscode#105870. We could perhaps add it for other clients, but it's harder to justify the engineering effort if not available in VS Code.

The enormous dataintegration package in oracle demonstrates a clear regression from [email protected] in completion CPU utilization during autocompletion. On my machine latency appears about the same, but on more resource-constrained machines this may not be the case. For golang/go#61207 Change-Id: I59631f34fe0d8d5d3329c9444d4e485840ad85ed Reviewed-on: https://go-review.googlesource.com/c/tools/+/516678 TryBot-Result: Gopher Robot <[email protected]> gopls-CI: kokoro <[email protected]> Run-TryBot: Robert Findley <[email protected]> Reviewed-by: Hyang-Ah Hana Kim <[email protected]>

Following a keystroke, it is common to compute both diagnostics and completion results. For small packages, this sometimes results in redundant work, but not enough to significantly affect benchmarks. However, for very large packages where type checking takes >100ms, these two operations always run in parallel recomputing the same shared state. This is made clear in the oracle completion benchmark. Fix this by guarding type checking with a mutex, and slightly delaying initial diagnostics to yield to other operations (though because diagnostics will also recompute shared, it doesn't matter too much which operation acquires the mutex first). For golang/go#61207 Change-Id: I761aef9c66ebdd54fab8c61605c42d82a8f412cc Reviewed-on: https://go-review.googlesource.com/c/tools/+/511435 gopls-CI: kokoro <[email protected]> Run-TryBot: Robert Findley <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Hyang-Ah Hana Kim <[email protected]>

findleyr · 2023-08-11T12:49:56Z

@sff2578 and @crclz, have you had any luck testing that patch? I'm very curious if it helped.

crclz · 2023-08-11T13:30:58Z

@findleyr I was a little busy on weekdays, and I will test it on weekends. And I realized that optimizing the performance requires a code repo that can reproduce the high completion time. Sharing code is forbidden by my company, so I am also trying to create a sharable repo from my company's repo.

findleyr · 2023-08-11T13:58:23Z

No worries. I'm trying to decide whether to land this CL in [email protected]. If you can determine whether it helps your use-case, it will help me decide. If you could test it this weekend, that would be great.

I don't think it's necessary to create a sharable repo. We have other repos to test on, and this CL helps there. If it doesn't help you, we'll figure out why and find a public repo that has the same behavior.

Thanks.

crclz · 2023-08-12T06:26:23Z

@findleyr I did some test just now and had some conclusion:

v0.11.0 has the best completion speed
Patch could slightly increase the speed
Adding ui.completion.completionBudget: 10ms could significantly increase the speed, but the speed is still slower than v0.11.0

Here is the experiment record:

More information is written in this document: https://bytedance.feishu.cn/docx/H8X3dQpfYovJdbxXkZLcQnOQnlf

findleyr added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 26, 2023

findleyr changed the title ~~gopls: always update my gopls to latest version~~ x/tools/gopls: poor performance with very large packages in import graph Jul 6, 2023

findleyr removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jul 6, 2023

findleyr transferred this issue from golang/vscode-go Jul 6, 2023

findleyr added this to the gopls/v0.12.5 milestone Jul 6, 2023

findleyr added the gopls Issues related to the Go language server, gopls. label Jul 6, 2023

findleyr mentioned this issue Jul 16, 2023

x/tools/gopls: Is there a way to reduce the memory usage? #61352

Closed

findleyr closed this as completed Jul 31, 2023

adonovan reopened this Aug 7, 2023

findleyr modified the milestones: gopls/v0.13.0, gopls/v0.13.2 Aug 7, 2023

findleyr modified the milestones: gopls/v0.13.2, gopls/v0.14.0 Aug 14, 2023

findleyr modified the milestones: gopls/v0.14.0, gopls/v0.15.0 Oct 9, 2023

findleyr modified the milestones: gopls/v0.15.0, gopls/v0.16.0 Jan 10, 2024

findleyr added the gopls/incremental label Mar 13, 2024

findleyr modified the milestones: gopls/v0.16.0, gopls/v0.17.0 Mar 13, 2024

findleyr removed the gopls/incremental label Mar 13, 2024

findleyr modified the milestones: gopls/v0.17.0, gopls/v0.18.0 Oct 22, 2024

findleyr modified the milestones: gopls/v0.18.0, gopls/v0.19.0 Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/gopls: poor performance with very large packages in import graph #61207

x/tools/gopls: poor performance with very large packages in import graph #61207

sff2578 commented Jun 23, 2023

Run `code -v` or `code-insiders -v` to get version of VS Code or VS Code Insiders.

findleyr commented Jun 23, 2023

sff2578 commented Jun 24, 2023

findleyr commented Jun 26, 2023

findleyr commented Jun 26, 2023

sff2578 commented Jun 26, 2023

findleyr commented Jun 26, 2023

sff2578 commented Jun 27, 2023

adonovan commented Jun 29, 2023 •

edited

Loading

findleyr commented Jul 6, 2023

crclz commented Jul 12, 2023

crclz commented Jul 12, 2023

gopherbot commented Jul 14, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 21, 2023

findleyr commented Jul 26, 2023

findleyr commented Jul 31, 2023

crclz commented Aug 6, 2023

findleyr commented Aug 6, 2023

crclz commented Aug 6, 2023

adonovan commented Aug 7, 2023 •

edited

Loading

findleyr commented Aug 7, 2023

gopherbot commented Aug 7, 2023

gopherbot commented Aug 7, 2023

findleyr commented Aug 7, 2023

sff2578 commented Aug 7, 2023

findleyr commented Aug 8, 2023

findleyr commented Aug 11, 2023

crclz commented Aug 11, 2023

findleyr commented Aug 11, 2023

crclz commented Aug 12, 2023

x/tools/gopls: poor performance with very large packages in import graph #61207

x/tools/gopls: poor performance with very large packages in import graph #61207

Comments

sff2578 commented Jun 23, 2023

What version of Go, VS Code & VS Code Go extension are you using?

Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.

Share the Go related settings you have added/edited

Describe the bug

Steps to reproduce the behavior:

Screenshots or recordings

findleyr commented Jun 23, 2023

sff2578 commented Jun 24, 2023

findleyr commented Jun 26, 2023

findleyr commented Jun 26, 2023

sff2578 commented Jun 26, 2023

findleyr commented Jun 26, 2023

sff2578 commented Jun 27, 2023

adonovan commented Jun 29, 2023 • edited Loading

findleyr commented Jul 6, 2023

crclz commented Jul 12, 2023

crclz commented Jul 12, 2023

gopherbot commented Jul 14, 2023

gopherbot commented Jul 21, 2023

gopherbot commented Jul 21, 2023

findleyr commented Jul 26, 2023

findleyr commented Jul 31, 2023

crclz commented Aug 6, 2023

findleyr commented Aug 6, 2023

crclz commented Aug 6, 2023

adonovan commented Aug 7, 2023 • edited Loading

findleyr commented Aug 7, 2023

gopherbot commented Aug 7, 2023

gopherbot commented Aug 7, 2023

findleyr commented Aug 7, 2023

sff2578 commented Aug 7, 2023

findleyr commented Aug 8, 2023

findleyr commented Aug 11, 2023

crclz commented Aug 11, 2023

findleyr commented Aug 11, 2023

crclz commented Aug 12, 2023

Run `code -v` or `code-insiders -v` to get version of VS Code or VS Code Insiders.

adonovan commented Jun 29, 2023 •

edited

Loading

adonovan commented Aug 7, 2023 •

edited

Loading