Can't emit Python into a specific output directory

[mccoyp]

Context

I'm in the process of migrating a brownfield service (Key Vault) to TypeSpec (Azure/azure-rest-api-specs#28708) and have been struggling to emit Python code that matches our original SDK structure. In azure-keyvault-certificates we keep all generated code in a _generated module; we'd like to emit from TSP into the same directory.

I'm able to match our original SDK structure by placing tsp-location.yaml in _generated/, emitting with tsp-client update, and manually deleting everything that's emitted apart from source code. However, this isn't an ideal solution:

1. This manual work is tedious and can't be easily done with automation.
2. The modified SDK structure doesn't match the emitter's assumptions. For example, generated model references point to azure.keyvault.certificates.models, when the new location is azure.keyvault.certificates._generated.models. This would break references, e.g. in docs.
3. Packaging info we want to keep, like setup.py, isn't automatically updated. We'd like to emit from the root of the package so that packaging information gets correctly updated.

Attempted solutions

I've tried a number of configurations in tspconfig.yaml to achieve this, which haven't worked:

• Attempted values for "output-path":
  • "./azure/keyvault/certificates/_generated"
  • "azure-keyvault-certificates/azure/keyvault/certificates/_generated"
  • "azure.keyvault.certificates._generated"
• Attempted values for "emitter-output-dir":
  • "./azure/keyvault/certificates/_generated"
  • "azure-keyvault-certificates/azure/keyvault/certificates/_generated"
  • "azure.keyvault.certificates._generated"
• Attempted values for "namespace":
  • "azure.keyvault.certificates._generated"

In all of these attempts, running tsp-client update from the root of azure-keyvault-certificates emits the code into the package root, instead of into _generated/. Here are the files that are changed (moving tsp-location.yaml to the package root was done manually; everything else is emitted):

Additional information

In these attempts I've been using the --local-spec-repo option, which may or may not be relevant. The tsp-client output consistently indicates that the output directory is the package root:

mcpatino@DESKTOP-DPKUD1A:~/azure-sdk-for-python/sdk/keyvault/azure-keyvault-certificates$ tsp-client update --local-spec-repo ../../../../azure-rest-api-specs/specification/keyvault/Security.KeyVault.Certificates/
Using output directory '/home/mcpatino/azure-sdk-for-python/sdk/keyvault/azure-keyvault-certificates'
...

[msyyc]

Let me make an investigation.

[msyyc]

Update: since user only need function code instead of whole package, python emitter need new feature to support that.

[iscai-msft]

tldr: add generation-subdir. Here are the notes from our discussion

North Star

sdk/keyvault/azure-kevault-certificates
azure
-> keyvault
-> certificates
-> _version.py
-> _generated
-> _client.py
-> setup.py # needs to make sure that it packages everything in azure/keyvault/certificates, including _generated

• output root -> emitter-output-dir
• packaging root ->
• generation root -> generation-subfolder?

package-name: azure-keyvault-certificates
namespace: azure.keyvault.certificates
generation-subdir: /_generated

tsp-client uses service-dir and package-dir. package-dir is where the setup.py is, so in this case azure-keyvault-certificates.

• These are usually the same / inferred, here we're trying to customize generation root, while still keeping the packaging root the same, and including everything within it.

[maorleger]

👋 this looks very similar to what @dgetu was working on in the TypeScript emitter (Azure/autorest.typescript#3011) - maybe it makes sense to come up with a language / flag that is consistent across languages? @joheredi FYI

My workaround is similar to @mccoyp 's except I created a script to do the shuffling https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/keyvault/generate.js

[iscai-msft]

thanks for the heads up @maorleger. Let me make a note to talk about this in our group sync