Dart LSP slow startup/first response on large codebase

[RobertBrunhage]

Dart SDK version: 3.0.6 (stable) (Tue Jul 11 18:49:07 2023 +0000) on "macos_arm64"

LSP takes a long time before LSP actions work

Not sure how to debug this so if you have tips/approaches, please let me know!

In short, when I open a project with Neovim the dartls LSP starts and in a large project takes towards half a minute for anything related to the LSP to work (hover docs, code completion, code actions etc).

Below I added duration for dart analyze with time dart analyze with a small project and large project to see the difference. Not sure if this command is even helpful in debugging but at least it shows the difference I experience.

Small project

real    0.65s
user    0.48s
sys     0.35s
cpu     127%

Large project

real    31.19s
user    33.99s
sys     8.12s
cpu     135%

---

The large project contains, internal packages and is in general very large (>690k LOC counted with find . -name '*.dart' | xargs wc -l), but I would still expect it to be faster than 30 seconds for anything to work, and the subsequent action takes ~half the time of that (18 seconds). After those two (first 30 sec wait for a LSP action, and then 18 sec, it becomes almost instant).

Again, if there are better ways to debug this, please let me know.

I will add this here as well as that is where I got the config from (mine is a bit leaner as I don't use outline etc.
https://github.com/neovim/nvim-lspconfig/blob/master/lua/lspconfig/server_configurations/dartls.lua

---

Tweet referencing this issue -> https://twitter.com/RobertBrunhage/status/1680840204309913600

[bwilkerson]

@DanTup

[DanTup]

Below I added duration for dart analyze

Do you mean it takes the same time (~30s) for both dart analyze and via the LSP server, or that the LSP server takes ~30s but dart analyze is faster?

If you start the server with --instrumentation-log-file=/path/to/log.txt it'll produce a log with timestamps that might help understand what's happening during that period (note: the log will likely contain parts of your source code but if you can't share that might be able to use the timestamps at the start of the lines to see where the time is going).

[RobertBrunhage]

Below I added duration for dart analyze

Do you mean it takes the same time (~30s) for both dart analyze and via the LSP server, or that the LSP server takes ~30s but dart analyze is faster?

dart analyze was just there to have some kind of benchmark for something, and it did showcase about the difference in the perceived LSP speed. In this case I just counted the seconds when I was trying to do code actions with the LSP and I just happened to use the dart analyze to do some kind of benchmark for the speed in general (might have been unrelated)

If you start the server with --instrumentation-log-file=/path/to/log.txt it'll produce a log with timestamps that might help understand what's happening during that period (note: the log will likely contain parts of your source code but if you can't share that might be able to use the timestamps at the start of the lines to see where the time is going).

Got a log now and the first thing I see is the initial request starts at 1689575376779 and the remaining files starts to analyze at 1689575389867, which is 13 seconds

1689575376763:Ver:1638536083352700528870:unknown.client.lsp:unknown:1.34.0:3.0.0
1689575376779:
1689575376831:
1689575376834:Req:{} // this requested the whole file it seems when I tried to check the hover docs of a class
1689575376880:Res:{"id"::4,"jsonrpc"::"2.0","method"::"client/registerCapability","params"::{"registrations"::[{"id"::"0","method"::"textDocument/hover","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}]}},{"id"::"1","method"::"textDocument/formatting","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}]}},{"id"::"2","method"::"textDocument/rangeFormatting","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}]}},{"id"::"3","method"::"textDocument/definition","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}]}},{"id"::"4","method"::"textDocument/codeAction","registerOptions"::{"codeActionKinds"::["source","source.organizeImports","source.fixAll","source.sortMembers","quickfix","refactor"],"documentSelector"::[{"language"::"dart","scheme"::"file"}]}},{"id"::"5","method"::"textDocument/rename","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}],"prepareProvider"::true}},{"id"::"6","method"::"textDocument/inlayHint","registerOptions"::{"documentSelector"::[{"language"::"dart","scheme"::"file"}],"resolveProvider"::false}}]}}
1689575376880:Req:{"result"::[{"analysisExcludedFolders"::["\/Users\/robertbrunhage\/AppData\/Local\/Pub\/Cache","\/Users\/robertbrunhage\/.pub-cache","\/opt\/homebrew\/","\/Users\/robertbrunhage\/tools\/flutter\/"],"updateImportsOnRename"::true,"completeFunctionCalls"::true,"showTodos"::true}],"jsonrpc"::"2.0","id"::3}
1689575376881:Req:{"result"::null,"jsonrpc"::"2.0","id"::4}
1689575389867:Res:{"jsonrpc"::"2.0","method"::"textDocument/publishDiagnostics","params"::{"diagnostics"::[],"uri"::"file::///Users/robertbrunhage/work/something/android/app/src/profile/AndroidManifest.xml"}}

This one was 3 seconds (does it really need to analyze the xml files?)

1689575390506:Res:{"jsonrpc"::"2.0","method"::"textDocument/publishDiagnostics","params"::{"diagnostics"::[],"uri"::"file::///Users/robertbrunhage/work/something/build/firebase_analytics/intermediates/aapt_friendly_merged_manifests/debug/aapt/AndroidManifest.xml"}}
1689575393482:Res:{"jsonrpc"::"2.0","method"::"textDocument/publishDiagnostics","params"::{"diagnostics"::[],"uri"::"file::///Users/robertbrunhage/work/something/build/some_plugin/intermediates/merged_manifest/debug/AndroidManifest.xml"}}

Another one that took a couple of seconds

1689575394397:Res:{"jsonrpc"::"2.0","method"::"textDocument/publishDiagnostics","params"::{"diagnostics"::[],"uri"::"file::///Users/robertbrunhage/work/something/packages/something_package/pubspec.yaml"}}
1689575396449:Res:{"id"::5,"jsonrpc"::"2.0","method"::"window/workDoneProgress/create","params"::{"token"::"ANALYZING"}}
1689575397093:Res:{"jsonrpc"::"2.0","method"::"textDocument/publishDiagnostics","params"::{"diagnostics"::[],"uri"::"file::///Users/robertbrunhage/work/something/lib/main.dart"}}
1689575397096:Res:{//removed}
1689575397097:Req:{//removed}
1689575397172:Req:{"result"::null,"jsonrpc"::"2.0","id"::5}
1689575397173:Res:{"jsonrpc"::"2.0","method"::"$/progress","params"::{"token"::"ANALYZING","value"::{"kind"::"begin","title"::"Analyzing…"}}}

In the end I can see the following which is also >10 sec (note that above is a similar line but that happened in the middle of the log)

1689575399907:Res:{"jsonrpc"::"2.0","method"::"$/progress","params"::{"token"::"ANALYZING","value"::{"kind"::"end"}}}
1689575412971:Req:{"id"::4,"jsonrpc"::"2.0","method"::"shutdown"}
1689575412972:Res:{"id"::4,"jsonrpc"::"2.0","result"::null}
1689575412995:Req:{"jsonrpc"::"2.0","method"::"exit"}

[DanTup]

dart analyze was just there to have some kind of benchmark for something, and it did showcase about the difference in the perceived LSP speed

Sorry, it's still not clear to me whether dart analyze is also taking around the same amount of time, or whether the issue is specific to going through the LSP server. It's expected that the LSP server will be slower to respond during startup and initial analysis, so if the delay you see roughly matches the time taken to run dart analyze in the same folder, that's probably expected.

Got a log now and the first thing I see is the initial request starts at 1689575376779 and the remaining files starts to analyze at 1689575389867, which is 13 seconds

The time prior to the first publishDiagnostics includes discovery of all of the projects (which includes walking down the whole open directory looking for pubspec.yamls). Can you confirm whether the folder that is being set as the workspace root is the folder you expect, or if it's something much larger (such as your home directory)? The directory will be in the line that contains "method"::"initialize" and could be either rootPath, rootUri or workspaceFolders.

One thing that would probably speed things up at the expensive of only analyzing a subset of projects is setting onlyAnalyzeProjectsWithOpenFiles. Usually the server will analyze the entire folder that the client gives to it, but with this option it will only look for projects related to open files.

[RobertBrunhage]

Sorry, it's still not clear to me whether dart analyze is also taking around the same amount of time, or whether the issue is specific to going through the LSP server. It's expected that the LSP server will be slower to respond during startup and initial analysis, so if the delay you see roughly matches the time taken to run dart analyze in the same folder, that's probably expected.

I would say it roughly matches the time of dart analyze. So in a small/new project the LSP and dart analyze is pretty much instant <1 sec. On large projects it's >30s for both LSP commands (the first one) and dart analyze. Does that clarify?

Can you confirm whether the folder that is being set as the workspace root is the folder you expect

This matches the project's folder path (copy pasted and removing the project name: "rootPath"::"\/Users\/robertbrunhage\/work\/something)

One thing that would probably speed things up at the expensive of only analyzing a subset of projects is setting onlyAnalyzeProjectsWithOpenFiles. Usually the server will analyze the entire folder that the client gives to it, but with this option it will only look for projects related to open files.

I can already see (and did this to try and speed things up) that I am using onlyAnalyzeProjectsWithOpenFiles

"onlyAnalyzeProjectsWithOpenFiles"::true,"outline"::false,"flutterOutline"::false

---

I also noticed that after this initial LSP analyzing and the first LSP action takes place I would expect all the subsequent requests to be fast, but I find that sometimes the 2nd and 3rd requests are also slow (not 30 sec slow but more around 10 sec slow), might be another topic though but wanted to bring it up anyway :)

[DanTup]

Thanks for the clarification. It sounds like the issue is just that initial analysis takes longer than you'd expect. Without a concrete project to test with, it's hard to say whether the times you're seeing are normal or not.

I'm surprised that onlyAnalyzerProjectsWithOpenFiles didn't make a difference though (unless you had files open that touched every project). If you run the server with --port=8123 and then open http://localhost:8123/ so you see a different number of contexts created with this setting? You should see 0 contexts with no open files, or 1 for each project you have an open file from.

On the left here I had an open workspace with 6 projects, then when I enabled onlyAnalyzeProjectsWithOpenFiles and restarted, only the project for the open file produced a context.

If your workspace has a large number of projects and you're not opening files from many of them, I would expect this to make a noticeable difference.

[RobertBrunhage]

@DanTup I did the same and is stating 14 contexts. All those are local packages, I always just open the entire project and not some folder containing other projects.

here was also another look at the timings (learned something new)

[DanTup]

I did the same and is stating 14 contexts

What files did you have open? You shouldn't see lots of contexts when using onlyAnalyzeProjectsWithOpenFiles if you don't have files open from 14 projects.

For example, if I open the pkg folder in the SDK with onlyAnalyzeProjectsWithOpenFiles=false then I see 5contexts. If I setonlyAnalyzeProjectsWithOpenFiles=trueand open the same folder, I see 0 contexts. If I open a file (for exampleanalysis_server\lib\src\lsp\lsp_analysis_server.dart`) then I see a single context.

Can you confirm whether you're seeing 14 contexts with no/one open file, or if you had many files open? If you had zero/one files open, are you able to verify in the log that onlyAnalyzeProjectsWithOpenFiles is still true, and how many textDocument.didOpen lines are logged (to ensure the open files the server is told about match what you're expecting)?

Thanks!

[joeldegerman99]

@RobertBrunhage @DanTup I have the same issue where I work. We have around 2.3k files and the analyze and lsp is super slow. I do not have any statistics but if requested I could get some if you want some more data to look at?

[DanTup]

@joeldegerman99 it would be interesting to know:

• how many contexts you have (see above)
• how long is "super slow"
• whether this is mainly after you've pulled significant changes to the project or upgraded SDK, or is it also slow if you just run Developer: Reload Window in VS Code (eg. everything is fully cached)
• whether you're using onlyAnalyzeProjectsWithOpenFiles (I don't really recommend using this setting - it trades some performance at startup for performance when opening/closing files, but it would be useful to know whether you enabled it)

[Reprevise]

@DanTup I have a pretty large Flutter project. It takes ~30 seconds (as seen in the timings) to actually start writing Dart with an active LSP. I only have 1 context, and I use whatever the default is for onlyAnalyzeProjectsWithOpenFiles.

Also is it a known issue that the "Analyzer Additional Args" setting doesn't get applied when the Reload Window command is triggered?

[DanTup]

@Reprevise 30s does sound pretty slow to me - does this reproduce after a restart (eg. with no code changes or SDK change)? For comparison, when I open my analysis server workspace, I see:

• 9 contexts
• 3,000 Dart files
• 2.5s for the initialized notification handling

Can you reproduce this on a public project? An instrumentation log may help, but it can contain parts of your source files so is best collected with a public project.

Also is it a known issue that the "Analyzer Additional Args" setting doesn't get applied when the Reload Window command is triggered?

Nope, it's definitely not :) That command actually tears down the whole extension host and restarts it, so it's not clear to me how Reload Window would behave differently to something like just restarting VS Code. If you enable the analysis server log it should record exactly what arguments it's starting the server with. If you think it's not working right, please file an issue at https://github.com/Dart-Code/Dart-Code/issues/new/choose with details. Thanks!

[Reprevise]

Can you reproduce this on a public project? An instrumentation log may help, but it can contain parts of your source files so is best collected with a public project.

I cannot, I tried it with the flutter project and the LSP was pretty speedy, even though its lib folder takes up 3x more space than ours. Perhaps it has something to do with the number of dependencies we use? We have 76 normal dependencies, and 11 dev dependencies.

[DanTup]

@Reprevise if you copy your list of dependencies/dev_dependencies into a new project (and add imports for them), does it repro?

[joeldegerman99]

@DanTup We have 5 private packages, can it be a performance bottleneck and about the same amount of dependencies as @Reprevise ? I will fix something for you to look at tomorrow or start of next week.

[Reprevise]

@Reprevise if you copy your list of dependencies/dev_dependencies into a new project (and add imports for them), does it repro?

Do the imports have to be used?

[DanTup]

@joeldegerman99 I wouldn't expect private packages to be any different to public packages. Are they just path packages pointing to other folders? You could try commenting them out of your pubspec (and running pub get) to see if whether that changes the timings disproportionately.

@Reprevise I'm not certain, but I don't think so - I think the analyzer just needs to "discover" the files to load them. However if it seems to not work let me know and I'll do some testing.

Some other numbers that could be useful to know.. if you run the Dart: Open Analyzer Diagnostics from VS Code and then click on the "Contexts" page, for each context under "Context Files" there will be a count of "added files" and "implicit files". Those counts may be useful (for each context) to get an idea of how many files there are including dependencies.

[Reprevise]

Something else to add is that I've been iterating on this same Flutter project for a few years now. I thought I'd try cloning it from the source repository and the LSP is pretty damn fast now.

The clone from the source repository has 1 less "added files" (688) and 5 less "implicit files" (3517) and it starts in 5 seconds compared to ~25 seconds on the "old" project.

[DanTup]

@Reprevise that's odd - are there any differences in things that might not be in the source folder (for example VS Code settings that might be gitignored) that could be different between them?

[Reprevise]

@Reprevise that's odd - are there any differences in things that might not be in the source folder (for example VS Code settings that might be gitignored) that could be different between them?

Yeah I think the issue was the .dart_tool folder which ballooned to ~700mb over the years. Running a flutter clean fixed the issue in the "old" project.

[DanTup]

Hmm, that's interesting to know - it's probably too late now if you've already cleaned it, but it'd be interesting to know why it got so big. @joeldegerman99 it might be interesting for you to try the same - rename the .dart_tool folder (so we can poke around in it if it helps), re-run pub get for all the projects, then time it again.

[DanTup]

@Reprevise out of interest, does your project have any code-gen? If you run your app a few times (maybe making some trivial changes) , do any files start building up in that directory?

[Reprevise]

@Reprevise out of interest, does your project have any code-gen? If you run your app a few times (maybe making some trivial changes) , do any files start building up in that directory?

Yes, our project uses code-gen pretty heavily. On the first run of build_runner, the .dart_tool folder went from 89.1 MB (I did run the app after running flutter clean) to 148 MB. After adding and then removing fields of a freezed class, it went to ~151 MB.

[DanTup]

@Reprevise thanks, I'll do some testing!

[DanTup]

@Reprevise I did some testing with the Freezed example, and although the .dart_tool folder grow initially, it didn't continue to grow as I kept making changes (it stuck at 50.6MB on my machine for the example app). I tried different freezed versions and changing SDK constraints, but it didn't seem like anything caused orphaned files or the folder to keep growing.

If it does keep growing for you, please let me know the steps.

[DanTup]

@Reprevise @joeldegerman99 also - could you confirm which OSes you're on? We have some thoughts about the time spent setting up watchers, and the watcher implementations are platform-specific. Thanks!

[Reprevise]

@Reprevise @joeldegerman99 also - could you confirm which OSes you're on? We have some thoughts about the time spent setting up watchers, and the watcher implementations are platform-specific. Thanks!

I'm on Windows 11. I'll get back to you on your previous comment tonight.

[Reprevise]

I'm seeing 15s long initialization times w/o any analyzer plugins. I had custom_lint as a plugin to analyze riverpod stuff which made initialization take ~45s so I've removed that for now.

Still though, taking out my ~5.8GB build folder and ~500MB .dart_tool folder made it so much faster (~4 seconds instead of ~15).

[DanTup]

@Reprevise

I'm seeing 15s long initialization times w/o any analyzer plugins. I had custom_lint as a plugin to analyze riverpod stuff which made initialization take ~45s so I've removed that for now.

Are you able to capture an instrumentation log with the plugin enabled? It adding 30s sounds suspicious, but the fix for that should've been included in 3.22.

What's the timing (without plugins) if you take out only the build folder and not .dart_tool (eg., if build was somehow excluded by default for Flutter apps, where would that leave you)? And does adding "dart.analysisExcludedFolders": ["build"] to your settings also get you to that same time?

[Reprevise]

I'll send the instrumentation log w/ custom_lint to the logs email shortly. Done.

Having the bloated .dart_tool folder makes no noticeable impact to the timing.

With regards to deleting vs excluding the build folder, deleting the folder feels slightly faster than excluding it though I have no solid evidence to back that up.

[DanTup]

@Reprevise thanks! Looking at the log you sent me, it appears that the server started at 15:49:04.926 (GMT) and the first analysis ended at 15:49:21.133 which looks like around 15 seconds (and the initialization appears to complete very quickly). I can also see the plugin generating diagnostics from 15:49:37.494 to 15:49:48.198 which is approx 11 seconds. What is it you're measuring when you're seeing 45s? (as far as I can tell, you're not hitting the 30s issue I noted should've been fixed in this version).

With regards to deleting vs excluding the build folder, deleting the folder feels slightly faster than excluding it though I have no solid evidence to back that up.

If you're able to capture a log of each of these (without the plugin) I'd be interested in comparing them. I hoped there would be no difference, but I don't know if excluding prevents us walking the folder entirely.

[Reprevise]

@DanTup I'm going to send you another log but it'll be unrelated to your request about the build folder. I've captured the long startup time and what I believe to possibly be another issue where after not doing anything in VSCode then coming back, the analysis server starts re-analyzing the whole project again.

About the long startup time, would it be correct to measure the time between the first entry in the log, and the entry right before the first publishDiagnostics?

[DanTup]

where after not doing anything in VSCode then coming back, the analysis server starts re-analyzing the whole project again.

That does sound odd - it's probably worth filing a specific issue about that with details (and I'll review the log you send). If you can include approx timestamps of when you first opened the project, went away, and came back, that may be useful too.

About the long startup time, would it be correct to measure the time between the first entry in the log, and the entry right before the first publishDiagnostics?

When trying to understand if something is a startup/initial analysis issue, I'll look for when this appears in the log:

Res:{"jsonrpc"::"2.0","method"::"$/progress","params"::{"token"::"ANALYZING","value"::{"kind"::"end"}}}

This is when the server believes it has completed analysis. However, the specifics of what is startup/initial analysis is probably not so important here if something appears to be taking much longer when you open a project. If it seems to you that something at startup is taking ~45s then I'm interested in understanding what that might be (but that requires understanding what you're seeing to reach that conclusion - what are you seeing/measuring?).

[Reprevise]

@DanTup I've opened a new issue and included the line in the log when I believe I came back.

To summarize (because there are so many comments here):

• Large build folder adds a lot of time (+~10s) to the LSP (this is the main issue, which is solved by either periodically deleting the build folder, or excluding it from analysis in VSCode settings, either of which bring the time of LSP startup down to ~4s)
• Plugins like custom_lint add ~30s to LSP "initialization" (but its still usable before this initialization w/ features like autocomplete, etc)

This slow startup time (~15s) is my main concern here, I personally don't really care about the ~30s of custom_lint startup because the LSP is still usable.

[DanTup]

@Reprevise

Plugins like custom_lint add ~30s to LSP "initialization"

Can you confirm what you're measuring for this? Do you mean it takes an extra 30s for custom_lint to report its own errors, or something else? (I know it's not your priority, but if plugins are causing any non-plugin functionality from the server to be delayed by anything like 30s, I would like to understand what's happening there).

Will think about how we might be able to handle build without just globally excluding it (there's no reason you couldn't have a build folder in a non-Flutter project, or nested inside your project that you might expect to be analyzed).

[Reprevise]

@DanTup

Can you confirm what you're measuring for this? Do you mean it takes an extra 30s for custom_lint to report its own errors, or something else? (I know it's not your priority, but if plugins are causing any non-plugin functionality from the server to be delayed by anything like 30s, I would like to understand what's happening there).

Sure, so what I mean is that the LSP starts up, and reports the 18 diagnostics in my project, and autocomplete and the rest of the LSP features come online. Then, custom_lint takes ~30s to report all its diagnostics and that's when I'm guessing the "initialized" event gets sent.

[DanTup]

Then, custom_lint takes ~30s to report all its diagnostics and that's when I'm guessing the "initialized" event gets sent.

The initialized event is actually sent by the client to the server. The timing shown in the diagnostic page is how long the server took to execute all code in the handler for that event. At a glance, I can't see that it should wait on anything like analysis or plugins - is it possible you can send me a matching log file and a screenshot of the Timing page? If possible, close all open files and restart VS Code then open the project but don't open any files until > 30s when you think it's finished (this will keep the log clearer of unrelated noise).

Could you also confirm the exact version of custom_lint, analyzer, analyzer_plugin you're using? (from your pubspec.lock or .dart_tool/package_config.json`) for me to see if I can reproduce locally.

Thanks!

[Reprevise]

@DanTup

I have emailed you a clean instr log of the analyzer, w/o me opening any files.

The versions are as follows:

• analyzer: 6.4.1
• analyzer_plugin: 0.11.3
• custom_lint: 0.6.4

At a glance, I can't see that it should wait on anything like analysis or plugins

To be clear, the Dart LSP is perfectly usable (with autocomplete, etc) while custom_lint is doing it's own analysis so that 30s I'm waiting I can still write code using the full features of the LSP.

[DanTup]

@Reprevise sorry for the delay - I just looked at this but you didn't include a copy of the timing page showing exactly how long the initialized event was recorded as. Could you capture a log file and include a matching screenshot showing the exact timing recording for the initialized event when the plugin is enabled (but, without the large build folder).

(I could understand the large build folder making initialized take longer because the server would walk that folder, but I'm trying to track down why plugins would be doing it - in my testing I'm getting quite fast initialized times).

[Reprevise]

@DanTup I have emailed you the logs and the screenshot of the timings page. The log is w/o the large build folder but does have custom_lint enabled.

Incase you can't see the screenshot in the email, here it is:

[PaulGrandperrin]

Hi,
I just want to share my experience, since it seems that it's a little bit on the extreme side.

I'm using vscode on a small/medium sized flutter app.

The dart language server takes about half an hour to one hour before it can offer any completion or type information to the IDE.

While I'm waiting for it to start working, it uses about 65% of 1 CPU core (when I have 4 cores / 8 threads) and basically does almost no IOs.

Since an hour is a long time to wait every time I launch vscode, I tried many things to understand what it is doing and why it's not even 100%ing my CPU or my SSD.

sysdig -c topfiles_times.lua didn't show anything useful, nor strace -f -e trace=open,openat -p $DART_PID.

Finally, I found a lead with ls -l /proc/$DART_PID/fd/. Here I can see that many files are open in random places in my /nix/store/ (I'm using NixOS).

So it seems that it is scanning my whole OS, which is huge, and doing so very slowly to avoid saturating the CPU or the SSD.

Looking more at the opened file descriptors, I see that the .direnv folder from my project folder. direnv is a small utility to have per-project dev environments. My direnv points to files in my /nix/store/ where my dev env is stored. I guess that's how it then looses itself scanning my whole /nix/store/.

So I'm trying to exclude .direnv from the dart analysis, but it seems this parameter is not working, or I am missing something because nothing changes:

 "dart.analysisExcludedFolders": [
    ".direnv",
  ],

EDIT: the language server also takes about 11GB of resident memory which also seems very excessive.

[DanTup]

@PaulGrandperrin could you file a specific issue for the problem you're seeing (that is, excluded folders are still enumerated during startup) and CC me?

[bwilkerson]

Do you, by any chance, have any symbolic links under your project directory? The analyzer will blindly follow symbolic links while trying to figure out what files are part of your project. But it doesn't (normally) look at anything outside the project directory, except that it will walk up the parent directories if it hasn't found either an analysis options file or a .dart_tool/package_config.json file.

(I'm assuming that by "project directory" you mean the directory you opened in the IDE.)

[PaulGrandperrin]

@DanTup Ok I'll do that when I go back to this problem and investigate a little more.

@bwilkerson Yes, the way direnv and nix work together is by creating a tree of symlinks in the .direnv folder that points to packages in /nix/store and this nix store is also a huge tree of symlinks!

So my guess is that Dart follows those symlinks, which makes sense. I'll investigate a little more to try to understand better how things unfold.

Is there a way to make the language server extra extra verbose and record everything it does in a file?

[PaulGrandperrin]

One important thing is that, dart, my flutter sdk and all my dev env comes from this .devenv directory so I'm thinking that I can't really just exclude this directory.

This might also explain why the language server ignore the exclusion, because its SDK files are in there. @DanTup

(I'm assuming that by "project directory" you mean the directory you opened in the IDE.)

@bwilkerson Yes, that's what I meant

I'll try to give you more info later, thanks for your replies.

[DanTup]

This might also explain why the language server ignore the exclusion, because its SDK files are in there. @DanTup

I'm not sure there won't be issues by having your SDK inside an ignored folder, but I'm surprised that we walk all of that folder if it's excluded (in a quick test, I couldn't repro the server trying to ready anything nested inside a folder excluded with dart.analysisExcludedFolders but I don't know if my repro is similar enough to yours).

[PaulGrandperrin]

@DanTup I spent a few more hours trying to understand this issue and here's what I found so far:

My project also uses rust and actually rust-analyzer had the exact same issue! I didn't notice before because it was a few orders of magnitude faster and didn't use any more memory than usual. I've been able to solve the issue with rust-analyzer but unfortunately it was very specific to it and didn't help me with understanding the issue with dart.

I understood something about my project's file tree is making the bug appear or disappear.
My project tree is like this:

/subProjectRust/.direnv
/subProjectRust/Cargo.toml
/subProjectFlutter/.direnv
/subProjectFlutter/pubspec.yaml

When I open the project with vscode at the /, dart will go into /subProjectRust/.direnv , follow the symlinks, end up in /nix/store/ and spend a hour scanning everything. I tried absolutely everything to exclude this folder, nothing works:

  "dart.analysisExcludedFolders": [
    "common_lib",
    "common_lib/direnv",
    "/common_lib",
    "/common_lib/.direnv",
    "common_lib/**",
    "common_lib/direnv/**",
    "/common_lib/**",
    "/common_lib/.direnv/**",
    "../common_lib",
    "../common_lib/direnv",
    "../common_lib",
    "../common_lib/.direnv",
    "../common_lib/**",
    "../common_lib/direnv/**",
    "../common_lib/**",
    "../common_lib/.direnv/**",
  ],
  "files.exclude": {
   "**/.direnv": true,
   "**/.direnv/**": true,
   "common_lib": true,
   "/common_lib": true,
   "common_lib/": true,
   "/common_lib/": true,
   "common_lib/**": true,
   "/common_lib/**": true,
  },
  "files.watcherExclude": {
   "**/.direnv": true,
   "**/.direnv/**": true,
   "common_lib": true,
   "/common_lib": true,
   "common_lib/": true,
   "/common_lib/": true,
   "common_lib/**": true,
   "/common_lib/**": true,
  },

But if I open with vscode the subproject /subProjectFlutter, then no issues, and it doesn't go into .direnv.

So I think I'm going to give up here, and just have multiple IDEs open at the same time, with the flutter/dart extensions only enabled for the flutter sub project.

[PaulGrandperrin]

My bad, actually, dart also scans everything when only opening the flutter subproject but is somehow much faster, similarly to rust-analyzer.

I really don't know why, and I now need to do other things..

[DanTup]

@PaulGrandperrin are you able to reproduce this in an isolated example (eg. by creating a new project and putting a symlink inside it to somewhere further up the tree)? I wasn't able to repro the behaviour you see last time I tested, so having instructions to produce a project that triggers it would really help. Thanks!

[bwilkerson]

@PaulGrandperrin

All of the attempts you posted above appear to be VC Code settings. Curious to know whether you've attempted to use an analysis options file to disable analysis of everything under subProductRust? If you're not familiar with analysis options files, or using them in this way, you can read more at https://dart.dev/tools/analysis#excluding-code-from-analysis. Id be interested to know whether adding an analysis options file at / allows you to open / without analyzing all of the files under subProductRust or any of the .devenv directories.

One important thing is that, dart, my flutter sdk and all my dev env comes from this .devenv directory so I'm thinking that I can't really just exclude this directory.

Excluding files and directories in the analysis options file doesn't prevent the analyzer from looking in them when there's a direct import that it needs to follow, it just prevents it from traversing those directories to find files to analyze. It should be fine to exclude them.