Skip to content

[machinelearningservices] Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace #10126

Open
#10127
Open
[machinelearningservices] Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace#10126
#10127
Assignees
a0x1abcopilot-swe-agent
Labels
Auto-AssignAuto assign by botMachine LearningService AttentionThis issue is responsible by Azure service team.act-experience-enablement-squadcustomer-reportedIssues that are reported by GitHub users external to the Azure organization.extension/mlquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Description

@x-engineering-agent
x-engineering-agent
bot
opened on Jul 21, 2026

Source: Azure/azure-cli#33756 (by vairakkumaarsvs)
Affected extension: machinelearningservices (src/machinelearningservices/)

<<UNTRUSTED:issue-33756>>

Issue #33756 (by vairakkumaarsvs)

Title

Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace

Body

Describe the bug

Azure CLI Version:
2.88.0

ML Extension:
2.44.1

Subscription:

Resource Group:
stmlwus2ws01-rg

Workspace:
stmlwus2ws01

Problem:

I deleted an Azure ML workspace.

When I attempt to recreate the workspace, deployment fails with:

"Soft-deleted workspace exists. Please purge or recover it."

However, attempting to permanently delete the workspace fails because the workspace cannot be found:

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Error:

(UserError) Unable to find workspace:
/subscriptions/.../resourcegroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/myws

This creates a deadlock:

  • Workspace creation fails because a soft-deleted workspace exists.
  • Permanent delete fails because the workspace cannot be found.
  • No CLI command appears available to enumerate or purge the soft-deleted workspace.

Expected behavior:

Either:

  1. az ml workspace delete --permanently-delete should purge the soft-deleted workspace, or
  2. The CLI should provide a supported command to list and purge soft-deleted workspaces.

Actual behavior:

Workspace name remains blocked but cannot be purged.

Related command

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Errors

(UserError) Unable to find workspace:
Code: UserError
Message: Unable to find workspace:

Issue script & Debug output

cli: Received HttpResponseError: Traceback (most recent call last):
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 677, in execute
raise ex
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 820, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 789, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 335, in call
return self.handler(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
return op(**command_args)
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 285, in ml_workspace_delete
log_and_raise_error(err, debug)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 193, in log_and_raise_error
raise cli_error
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/
... [truncated, original was 10958 chars]

Comments

Comment by Yong Zhang (@yonzhan)

Thank you for opening this issue, we will look into it.
<<END:issue-33756>>

Activity

  1. yonzhan commented on Jul 21, 2026

    @yonzhan
    Yong Zhang (yonzhan)
    on Jul 21, 2026
    Collaborator

    Thank you for opening this issue, we will look into it.

  2. Aditya Pujara (a0x1ab)
    assigned
    Copilot
    and
    Aditya Pujara (a0x1ab)
    on Jul 21, 2026

  3. x-engineering-agent commented on Jul 21, 2026

    @x-engineering-agent
    x-engineering-agentbot
    on Jul 21, 2026 – with X Engineering Agent
    Author

    Bug Analysis

    Root cause: ml_workspace_delete in
    src/machinelearningservices/azext_mlv2/manual/custom/workspace.py calls
    ml_client.workspaces.begin_delete(name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete) directly and only wraps the call with a
    generic except Exception that re-raises via log_and_raise_error. When a
    workspace is already soft-deleted, the underlying SDK's begin_delete first
    calls get(name) to fetch the workspace, which raises
    azure.core.exceptions.ResourceNotFoundError (404) because the active
    resource no longer exists (only the soft-deleted one does). This bubbles up as
    a confusing (UserError) Unable to find workspace even though the caller
    explicitly asked to --permanently-delete the workspace, creating a deadlock:
    create is blocked by the soft-deleted record, and delete can't find it to purge.

    Suggested fix: in ml_workspace_delete, when permanently_delete=True and the
    underlying call raises ResourceNotFoundError/HttpResponseError with a 404
    indicating the active workspace resource is missing, treat this as an
    expected condition for a workspace that only exists in soft-deleted state —
    either invoke the workspace's purge-only code path (if the SDK exposes one) or
    surface an actionable error message (instead of a generic traceback) telling
    the user the workspace is soft-deleted and how to purge it, e.g. via the ARM
    recover/purge control-plane operation for
    Microsoft.MachineLearningServices/locations/deletedWorkspaces. Add/adjust
    error handling so az ml workspace delete --permanently-delete succeeds (or
    gives clear guidance) when the target only exists as a soft-deleted resource.

    Use this EXACT PR title: [Machinelearningservices] Fix #10126: az ml workspace delete --permanently-delete: Handle ResourceNotFoundError when purging soft-deleted workspace

    PR title & description format (required)

    This repo enforces a PR format (guide). Please author the PR exactly as follows or CI's Check the Format of Pull Request Title and Content will fail.

    Use this EXACT PR title (copy verbatim, do not reword):

    [Machinelearningservices] Fix #10126: `az ml workspace delete --permanently-delete`: Handle ResourceNotFoundError when purging soft-deleted workspace

    Keep the backticks around the command and the Fix #10126: prefix. You may only adjust the wording after the command (the final summary) if the fix changes; the [Machinelearningservices] prefix, issue link, and backticked command must stay.

    Description — follow the PR template and fill in:

    • Link the issue — start the Description with a closing keyword so the PR auto-links and closes it: Fixes #10126.
    • Related command — the az ... command this affects.
    • Description (mandatory) — why the bug happens, what you changed, and the resulting behavior.
    • Testing Guide — example command(s) showing the fix works.
    • History Notes — leave the title to drive the history note, or add extra lines in the same format (component in brackets + the command in backticks), e.g. [Machinelearningservices] `az <command>`: <note>.
    • Keep the template checklist and tick the items you've satisfied.

    Posted by agent-assist (autonomous bug-fix pipeline).

  4. microsoft-github-policy-service
    added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    Auto-AssignAuto assign by bot
    Service AttentionThis issue is responsible by Azure service team.
    Machine Learning
    extension/ml
    act-experience-enablement-squad
    on Jul 21, 2026
  5. Copilot
    linked a pull request that will close this issuefix(ml): handle workspace stuck in soft-deleted state for permanently-delete and create commands #10127on Jul 21, 2026
  6. Copilot
    added a commit that references this issue on Jul 21, 2026

    fix(ml): handle soft-deleted workspace in permanently-delete and crea…

    597037a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Auto-AssignAuto assign by botMachine LearningService AttentionThis issue is responsible by Azure service team.act-experience-enablement-squadcustomer-reportedIssues that are reported by GitHub users external to the Azure organization.extension/mlquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions