Shell doesn't support Cygwin zsh

[edemaine]

A possible addition to #4568:

1. When using Cygwin's zsh as my shell, the extension sends the PowerShell command & "C:/Program Files/Python39/python.exe" c:/Users/edemaine/.../filename.py. zsh doesn't understand the leading &. Relevant configuration:

    "terminal.integrated.defaultProfile.windows": "Cygwin ZSH",
    "terminal.integrated.automationShell.windows": "C:\\cygwin\\bin\\zsh.exe",
    "terminal.integrated.profiles.windows": {
        "PowerShell": {
                "source": "PowerShell",
                "icon": "terminal-powershell"
        },
        "Command Prompt": {
                "path": [
                        "${env:windir}\\Sysnative\\cmd.exe",
                        "${env:windir}\\System32\\cmd.exe"
                ],
                "args": [],
                "icon": "terminal-cmd"
        },
        "Git Bash": {
                "source": "Git Bash"
        },
        "Cygwin ZSH": {
                "path": [
                        "c:\\cygwin\\bin\\zsh.exe",
                        "c:\\cygwin64\\bin\\zsh.exe"
                ],
                "args": [
                        "-i"
                ]
        }
    }

2. When using Cygwin's zsh as my shell and Windows cmd as my automation shell, the extension runs Python in zsh (and then runs the PowerShell command):

    "terminal.integrated.defaultProfile.windows": "Cygwin ZSH",
    "terminal.integrated.automationShell.windows": "C:\\Windows\\System32\\cmd.exe",

   I'm confused by this one because I thought it was fixed in a recent update (last week). But now it seems to be broken again.

Environment data

• VS Code version: 1.60.1
• Extension version (available under the Extensions sidebar): v2021.9.1230869389
• OS and version: Windows 10.0.19042
• Python version (& distribution if applicable, e.g. Anaconda): 3.9.7
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Value of the python.languageServer setting: Default

Logs

Output for Python in the Output panel (in the second scenario above)

User belongs to experiment group 'pythonaacf'
User belongs to experiment group 'pythonJediLSP'
User belongs to experiment group 'pythonDiscoveryModuleWithoutWatcher'
User belongs to experiment group 'pythonTensorboardExperiment'
User belongs to experiment group 'PythonPyTorchProfiler'
User belongs to experiment group 'pythonDeprecatePythonPath'
User belongs to experiment group 'pythonSortEnvs'
User belongs to experiment group 'pythonRunFailedTestsButtonDisplayedcf'
User belongs to experiment group 'pythonRefreshTestsButtonDisplayed'
User belongs to experiment group 'pythonRememberDebugConfigcf'
Info 2021-09-15 15:59:43: Testing: Setting up watcher for c:\Users\edemaine\...
Info 2021-09-15 15:59:43: Searching for conda.
Info 2021-09-15 15:59:43: Probing conda binary: conda
Info 2021-09-15 15:59:43: Error: spawn conda ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
    at onErrorNT (internal/child_process.js:465:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -4058,
  code: 'ENOENT',
  syscall: 'spawn conda',
  path: 'conda',
  spawnargs: [ 'info', '--json' ]
}
Info 2021-09-15 15:59:44: Watcher disabled
Info 2021-09-15 15:59:44: Watcher disabled
Info 2021-09-15 15:59:44: Searching for workspace virtual envs in: c:\Users\edemaine\...
Info 2021-09-15 15:59:44: Getting roots
Info 2021-09-15 15:59:44: Found roots
Info 2021-09-15 15:59:44: Watcher disabled
Info 2021-09-15 15:59:44: Getting roots
Info 2021-09-15 15:59:44: Found interpreter for C:\Program Files\Python39\python.exe,c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\pythonFiles\interpreterInfo.py
Info 2021-09-15 15:59:44: Found roots
Info 2021-09-15 15:59:44: Start watching root c:\Users\edemaine\... for globs ["python.exe","*/python.exe","*/Scripts/python.exe"]
Info 2021-09-15 15:59:44: Start watching: c:\Users\edemaine\... with pattern python.exe using VSCode API
Info 2021-09-15 15:59:44: Start watching: c:\Users\edemaine\... with pattern */python.exe using VSCode API
Info 2021-09-15 15:59:44: Start watching: c:\Users\edemaine\... with pattern */Scripts/python.exe using VSCode API
Info 2021-09-15 15:59:44: Found interpreter for C:\Program Files\Inkscape\python.exe,c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\pythonFiles\interpreterInfo.py
Info 2021-09-15 15:59:45: Couldn't locate the conda binary.
Info 2021-09-15 15:59:45: Environments added to cache [{"name":"","location":"","kind":"global-system","executable":{"filename":"C:\\Program Files\\Inkscape\\python.exe","sysPrefix":"C:/Program Files/Inkscape","ctime":-1,"mtime":-1},"display":"Python 2.7.15 64-bit (system)","version":{"major":2,"minor":7,"micro":15,"release":{"level":"final","serial":0},"sysVersion":"2.7.15 (default, Oct 11 2018, 12:09:51)  [GCC 8.2.0 64 bit (AMD64)]"},"arch":3,"distro":{"org":""},"source":["path env var"]},{"name":"","location":"","kind":"global-other","executable":{"filename":"C:\\Program Files\\Python39\\python.exe","sysPrefix":"C:\\Program Files\\Python39","ctime":-1,"mtime":-1},"display":"Python 3.9.7 64-bit","version":{"major":3,"minor":9,"micro":7,"release":{"level":"final","serial":0},"sysVersion":"3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]"},"arch":3,"distro":{"org":"PythonCore","defaultDisplayName":"Python 3.9 (64-bit)"},"source":["path env var","windows registry"]}]
Info 2021-09-15 15:59:45: Display locator refreshing progress, Class name = g, completed in 1ms, has a falsy return value, , Return Value: undefined
Info 2021-09-15 15:59:45: Hide locator refreshing progress, Class name = g, completed in 0ms, has a falsy return value, , Return Value: undefined
Python interpreter path: C:\Program Files\Python39\python.exe
Info 2021-09-15 15:59:45: Display locator refreshing progress, Class name = g, completed in 1ms, has a falsy return value, , Return Value: undefined
Info 2021-09-15 15:59:45: Hide locator refreshing progress, Class name = g, completed in 0ms, has a falsy return value, , Return Value: undefined
Starting Pylance language server.
Error 2021-09-15 15:59:45: Failed to check if file needs to be fixed EntryNotFound (FileSystemError): Unable to read file 'c:\Users\edemaine\...\.vscode\settings.json' (Error: Unable to resolve non-existing file 'c:\Users\edemaine\...\.vscode\settings.json')
    at _handleError (c:\Users\edemaine\AppData\Local\Programs\Microsoft VS Code\resources\app\out\vs\workbench\services\extensions\node\extensionHostProcess.js:94:160087)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at y.readText (c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\out\client\extension.js:9:314351)
    at p.doesFileNeedToBeFixed (c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\out\client\extension.js:59:823969)
    at c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\out\client\extension.js:59:823096
    at async Promise.all (index 1)
    at p.getFilesToBeFixed (c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\out\client\extension.js:59:823042)
    at p.updateTestSettings (c:\Users\edemaine\.vscode\extensions\ms-python.python-2021.9.1230869389\out\client\extension.js:59:822669) {
  code: 'FileNotFound'
}
Info 2021-09-15 15:59:46: Starting language server, Class name = r, completed in 685ms, has a falsy return value, , Return Value: undefined
Info 2021-09-15 15:59:46: Cached data exists getEnvironmentVariables, c:\Users\edemaine\...\filename.py
Info 2021-09-15 15:59:46: [object Object]. Shell identified as undefined 
Info 2021-09-15 15:59:46: Shell path 'c:\cygwin\bin\zsh.exe'
Info 2021-09-15 15:59:46: Shell path identified as shell 'other'
Info 2021-09-15 15:59:46: Terminal shell path 'c:\cygwin\bin\zsh.exe' identified as shell 'other'
Info 2021-09-15 15:59:46: [object Object]. Shell identified as other 
Info 2021-09-15 15:59:46: Shell path from user settings 'null'
Info 2021-09-15 15:59:46: [object Object]. Shell identified as other 
Info 2021-09-15 15:59:46: Shell path 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'
Info 2021-09-15 15:59:46: Shell path identified as shell 'powershell'
Info 2021-09-15 15:59:46: Shell path from user env 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'
Info 2021-09-15 15:59:46: [object Object]. Shell identified as powershell 
Info 2021-09-15 15:59:46: Shell identified as 'powershell'
Info 2021-09-15 15:59:46: Searching for conda.
Info 2021-09-15 15:59:46: Searching for conda.
Info 2021-09-15 15:59:46: > conda --version
> conda --version
Info 2021-09-15 15:59:46: Display locator refreshing progress, Class name = g, completed in 1ms, has a falsy return value, , Return Value: undefined
Info 2021-09-15 15:59:46: Hide locator refreshing progress, Class name = g, completed in 0ms, has a falsy return value, , Return Value: undefined
Info 2021-09-15 15:59:46: Cached data exists getEnvironmentVariables, <No Resource>
Info 2021-09-15 15:59:46: Cached data exists getEnvironmentVariables, c:\Users\edemaine\...

Notably, I don't notice any shell identification logs from baseShellDetector. Does this code not get run on Windows?

[karthiknadig]

@edemaine You have to set logging level to debug to see the logs from this line:

vscode-python/src/client/common/terminal/shellDetectors/baseShellDetector.ts

Line 69 in 3698950

traceVerbose(`Shell path '${shellPath}'`);

[edemaine]

Thanks for the tip! I tried code --log debug . and code --log trace . but I still get the same log. Indeed, no additional logs get generated in Output / Python when runnning F5 (python.execInTerminal). (In Output / Log (Window) I can see python.execInTerminal getting executed, but nothing else.) Perhaps a consequence of "Failed to check if file needs to be fixed / EntryNotFound / FileNotFound" error?

[karthiknadig]

I meant this setting for python extension logs:

[edemaine]

Oops, thanks again. Now there are relevant log messages (added to original post).

[karthiknadig]

Looks like we might need to add zsh.eze to the pattern detection here:

vscode-python/src/client/common/terminal/shellDetectors/baseShellDetector.ts

Line 27 in 3698950

const IS_ZSH = /(zsh$)/i;

[edemaine]

Nice spot. I'm happy to make a PR to expand the shell detection, and that should fix issue 1.

Any idea why issue 2 arises? Why does it run one shell (the unrecognized zsh), and not the shell given by automationShell, and then send the input for a different shell?

[karthiknadig]

@karrtikr Is issue 2 due to python extension or VS Code?

[karrtikr]

It seems like it's us. Looking at the logs VSCode correctly identifies the shell

Info 2021-09-15 15:59:46: Shell path 'c:\cygwin\bin\zsh.exe'
Info 2021-09-15 15:59:46: Shell path identified as shell 'other'
Info 2021-09-15 15:59:46: Terminal shell path 'c:\cygwin\bin\zsh.exe' identified as shell 'other'

And we add the & here based on whether it's powershell:

vscode-python/src/client/common/terminal/helper.ts

Lines 62 to 70 in 31ce50c

	public buildCommandForTerminal(terminalShellType: TerminalShellType, command: string, args: string[]) {
	const isPowershell =
	terminalShellType === TerminalShellType.powershell ||
	terminalShellType === TerminalShellType.powershellCore;
	const commandPrefix = isPowershell ? '& ' : '';
	const formattedArgs = args.map((a) => a.toCommandArgument());
	return `${commandPrefix}${command.fileToCommandArgument()} ${formattedArgs.join(' ')}`.trim();
	}

Seems like a bug that we somehow still add & if terminal type is other.

[edemaine]

Actually, I think what's going on is that my shell is detected as type "other", so VSCode falls back to using a default for Windows. The logs say:

Info 2021-09-15 15:59:46: Shell path 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'
Info 2021-09-15 15:59:46: Shell path identified as shell 'powershell'
Info 2021-09-15 15:59:46: Shell path from user env 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'

The fact that Shell path from user env gets logged means that userEnvironmentShellDetector.ts is getting activated, which means that the previous attempts failed; see this code:

vscode-python/src/client/common/terminal/shellDetector.ts

Lines 51 to 62 in 31ce50c

	// Sort in order of priority and then identify the shell.
	const shellDetectors = this.shellDetectors.slice().sort((a, b) => b.priority - a.priority);
	for (const detector of shellDetectors) {
	shell = detector.identify(telemetryProperties, terminal);
	traceVerbose(
	`${detector}. Shell identified as ${shell} ${terminal ? `(Terminal name is ${terminal.name})` : ''}`,
	);
	if (shell && shell !== TerminalShellType.other) {
	break;
	}
	}

What's weird is that it doesn't actually run the shell returned from UserEnvironmentShellDetector (PowerShell) and instead decides to run zsh instead.

But this actually seems to be how identifyTerminalShell is designed: it detects the shell type, but doesn't actually choose which shell to run. So even though userEnvironmentShellDetector.ts has nice code for locating the default Windows shell, it doesn't get chosen as the shell to use; instead, the extension uses my chosen shell.

[karrtikr]

The fact that Shell path from user env gets logged means that userEnvironmentShellDetector.ts is getting activated, which means that the previous attempts failed; see this code:

Ah, thanks for diagnosing the issue! The issue is we shouldn't fallback on using other detectors once VSCode has identified a terminal type as other.

Other shell detectors were in place before when we added the VSCode shell detector, which reliably tells us the shell path so we need not guess. We have an open issue #16023 to remove all other shell detectors which should solve this 🙂 You're very welcome to work on it if you would like.

[TylerLeonhardt]

@edemaine can you verify that this issue is fixed in the insider build? All you have to do is set the following setting: "python.insidersChannel": "daily" and reload vscode to get it.

[connor4312]

@TylerLeonhardt I verified just now :)

[edemaine]

I confirmed as well. 🥳