Dynamic completion for Nushell

[chklauser]

Please complete the following tasks

• I have searched the discussions
• I have searched the open and rejected issues

Clap Version

master

Describe your use case

Bash, zsh, fish, etc. have very cool support for dynamic completion. Dynamic completion is a huge boost to the usability of a tool.

The clap_complete_nushell crate doesn't support dynamic completion yet. While the "external" completion story in Nushell is still a bit shaky, it's nonetheless possible for Nushell to get completions from external programs. We just need to implement it.

Describe the solution you'd like

The source <(COMPLETE=nushell some-clap-binary --) pattern fundamentally won't work in Nushell. Nushell requires all code to be statically available in the file system before configuration gets loaded. This makes it a bit tricky to make installation of the dynamic completion support both convenient and "auto-updating" (in case a new version of a clap-based CLI gets installed).

Additionally, and unlike every other shell that we support, Nushell only supports a single, globally registered "external completer". For users, who want to have more than one external completer, the suggestion is to have a "meta-completer" that looks at the command line (first arg, which is the binary name) and dispatches the completion request to an appropriate completer.

A solution could look like this:
(1) The user asks the clap-based CLI tool to generate a "completion generator" source file

COMPLETE=nushell some-clap-tool -- | save --raw ~/.config/nushell/generate-some-clap-tool-completions.nu

This "completion generator" needs to be as stable and minimal as possible. It could look something like this:

COMPLETE=nushell COMPLETE_nu=module some-clap-tool -- | save --raw ($env.NU_LIB_DIRS.0 | path join some-clap-tool-completer.nu)

Each time env.nu is loaded, this writes an updated version of the actual completion hook. The additional COMPLETE_nu=module environment variable signals that the clap-based tool should generate the completion hook and not the "completion generator" file.

(2) The user (manually) includes this source file in their env.nu

# env.nu
source ./generate-some-clap-tool-completions.nu

(3) The user (manually) uses the regularly re-generated module in their config.nu

# config.nu
use some-clap-tool-completer
# and then EITHER
some-clap-tool-completer install
# OR
$env.config.completions.external.completer = { |spans| 
  if (some-clap-tool-completer handles $spans) {
    some-clap-tool-completer complete $spans
  } else {
    # other completers that the user wants to dispatch to
  }
}

Alternatives, if applicable

Autoload Directories
A hidden feature of Nushell. It will be officially documented in version 0.101. Files in those autoload directories get loaded after config.nu. They sound like a good target for completions except for the issue that there can only be one global completer. If we just plonk the clap-generated file into an autoload directory, we take away the user's control over how external completion works in their shell.

There are some upcoming changes to Nushell configuration.

Additional Context

Nushell docs

• External Completions
• Suggestion for the case of multiple completers
• Configuration (env.nu vs config.nu)

[epage]

Nushell requires all code to be statically available in the file system before configuration gets loaded. This makes it a bit tricky to make installation of the dynamic completion support both convenient and "auto-updating" (in case a new version of a clap-based CLI gets installed).

In #5668 (comment) there is discussion of lazy loading. Unsure if that would be sufficient for nushell.

Additionally, and unlike every other shell that we support, Nushell only supports a single, globally registered "external completer". For users, who want to have more than one external completer, the suggestion is to have a "meta-completer" that looks at the command line (first arg, which is the binary name) and dispatches the completion request to an appropriate completer.

Ouch, this sounds like a big limitation.

If I'm understanding correctly, we basically require users to hook us into their hand-written meta-completer? Not too ideal. Is there any talk about fixing this so custom completers can better coordinate and have a good out-of-the-box experience?

[Hofer-Julian]

Thanks for looking into this @chklauser!

In your PR you added install, which

`globally registers a completer that falls back to whatever completer was previously installed if handles rejects completing a command line.

I think that is a good way of working around nushell's current limitations.

There are some upcoming changes to Nushell configuration.

Also, this is nowadays released. Users now only have things in their config that they explicitly changed, without all the defaults as it was before.

Autoload Directories
A hidden feature of Nushell. It will be officially documented in version 0.10

This is now official documented: https://www.nushell.sh/book/configuration.html#configuration-overview
Looking at the docs ($nu.data-dir)/vendor/autoloads would be a good place to add an autoloading script: https://www.nushell.sh/book/configuration.html#startup-variables

Personally, I think it's fine if the first iteration of dynamic completion support for nushell cannot conveniently auto-update.

Ideally, one only needs to run a command like this to set up completions or to update them:

COMPLETE=nushell some-clap-tool -- | save --raw `($nu.data-dir)/vendor/autoloads/generate-some-clap-tool-completions.nu

That script would set up the completion as well as wrap existing completers similar to how install in #5841 works

[chklauser]

There is a trade off here. Nushell differentiates between "modules" and "scripts". That means a file either exports definitions (when used together with use) or it executes code (when sourceed). The autoloads directory is the latter kind. If this is the only script we generate, we can essentially only offer the "install" version (not the "handles" + "complete" version). We cannot generate a file that can be used as a module and a script at the same time.

Users who would like to write their own meta-completer will not be able to use the script we generate directly. That's probably not the end of the world, though. Because nushell and clap both understand JSON, the integration is probably very stable.

The carapace-bin completer makes the same trade off. With one (IMO weird) difference: their completer only installs itself if there is no existing completer installed. I think, as a user, I would prefer the chaining approach in #5841.

[Hofer-Julian]

The carapace-bin completer makes the same trade off. With one (IMO weird) difference: their completer only installs itself if there is no existing completer installed. I think, as a user, I would prefer the chaining approach in #5841.

Agreed, I was considering linking to carapace's approach in my reply, but its behaviour doesn't make too much sense IMO.

@epage is there anything else you'd like to discuss before moving on to the actual implementation?

[epage]

Could you clarify for me how a user who has two binaries using this feature would register both of them?

Also, should we block on lazy loading (#5668)?

[Hofer-Julian]

Could you clarify for me how a user who has two binaries using this feature would register both of them?

You mean two different applications, right?
That should just work, since the first one wraps the original completer, and the second one wraps that again. If the binary name matches, the completer will be used, if not the next one will be tried and so on.

Also, should we block on lazy loading (#5668)?

As useful as that would be, I don't see a reason to block on it.

[chklauser]

One more idea from the nushell discord: combine custom completers (for a single external command decl) with rest parameters

def "nu-complete foo" [ctx: string, pos: int] { }
def foo [...rest: any@"nu-complete foo"] { }

I initially discounted custom completions because I didn't want nushell to handle the completion of --args since clap has more detailed knowledge about what is permitted in a particular command line. But if we can let the clap completer handle the entire command line, that's not an issue.

Custom completions on an external command has the advantage that we don't mess with globally installed external completers.

I have not tested it though. This approach is not documented explicitly.

[chklauser]

Note for the implementation: we need to check that we handle some pipeline | some-clap-tool arg1 correctly. Nushell includes the | as a first argument. I suppose some tools might exhibit different behavior as part of a pipeline and thus be interested in this distinction. For clap, we just need to filter that out.

[Hofer-Julian]

One more idea from the nushell discord: combine custom completers (for a single external command decl) with rest parameters

Oh, that is interesting. This approach would indeed be pretty clean, assuming that it works as expected

[chklauser]

(@Hofer-Julian if you are itching to tackle this issue, please feel free. I won't find time in the next couple of weeks myself)

[Hofer-Julian]

(@Hofer-Julian if you are itching to tackle this issue, please feel free. I won't find time in the next couple of weeks myself)

Thanks for the notice!
Might give it a go this weekend.

[Hofer-Julian]

@chklauser could you please link your jj fork where you implemented dynamic completions for nushell?

[chklauser]

@Hofer-Julian that would be jj-vcs/jj@main...chklauser:jj:push-rppypvvytuqv

[Hofer-Julian]

One more idea from the nushell discord: combine custom completers (for a single external command decl) with rest parameters

def "nu-complete foo" [ctx: string, pos: int] { }
def foo [...rest: any@"nu-complete foo"] { }

I've tried out this approach, but nushell needs to adapt to allow this workflow, I've opened an issue for that: nushell/nushell#14806

For now, I consider the implementation in #5841 as the best solution.
I tested it locally and everything works fine.
Also, the usage is the same as with the non-dynamic completions: people add the generation of the completion in env.nu and call it then in config.nu.

[epage]

Could you clarify for me how a user who has two binaries using this feature would register both of them?

You mean two different applications, right? That should just work, since the first one wraps the original completer, and the second one wraps that again. If the binary name matches, the completer will be used, if not the next one will be tried and so on.

I'm referring to the following from the Issue

Additionally, and unlike every other shell that we support, Nushell only supports a single, globally registered "external completer". For users, who want to have more than one external completer, the suggestion is to have a "meta-completer" that looks at the command line (first arg, which is the binary name) and dispatches the completion request to an appropriate completer.

---

Also, should we block on lazy loading (#5668)?

As useful as that would be, I don't see a reason to block on it.

I'm referring to the following from the Issue

The source <(COMPLETE=nushell some-clap-binary --) pattern fundamentally won't work in Nushell. Nushell requires all code to be statically available in the file system before configuration gets loaded. This makes it a bit tricky to make installation of the dynamic completion support both convenient and "auto-updating" (in case a new version of a clap-based CLI gets installed).

Auto-updating is a hard requirement imo.

[chklauser]

Auto-updating is a hard requirement imo.

Shouldn't that be the responsibility of the distribution/packager? I understand that auto update is convenient for tools that are cargo installed but that distribution method should be the exception, not the norm.

[Hofer-Julian]

Auto-updating is a hard requirement imo.

Shouldn't that be the responsibility of the distribution/packager? I understand that auto update is convenient for tools that are cargo installed but that distribution method should be the exception, not the norm.

I tend to agree, but I think it's not that relevant for this discussion. Auto-updating will work exactly the same as it did with static completions. 🙂

[epage]

That is a discussion for #5668. Unless the direction of that is changed, auto-updating is a requirement.

[Hofer-Julian]

auto-updating is a requirement

@epage I will try my best to summarize the situation, in order to make sure we talk about the same thing.

What I mean when I talk about auto-updating of nushell completions with clap-complete as is (no dynamic completions), is the following:
With rattler-build it works like this:

Add the following env.nu:

mkdir ~/.cache/rattler-build
rattler-build completion --shell nushell | save -f ~/.cache/rattler-build/completions.nu

Add the following to config.nu

use ~/.cache/rattler-build/completions.nu *

During shell start, env.nu will be called first and generates the completions, then config.nu is called and imports them.
The split is necessary since nushell needs all included code available during parsing time. I assume that's what @chklauser meant with:

The source <(COMPLETE=nushell some-clap-binary --) pattern fundamentally won't work in Nushell. Nushell requires all code to be statically available in the file system before configuration gets loaded.

The result is the same as with other shells: if you update your CLI programs, your completions will also be updated automatically.

Now back to dynamic completions and #5841:

That will work exactly the same. You generate completions in env.nu and import them in config.nu.