[Design] SDK vs. MSBuild

[mattmasson]

Our default connector template comes with an msbuild file (connector.proj), but defaults to using the SDK build mechanism. This might be confusing for users.

We should remove the .proj from our template if we are defaulting to the SDK build.

That said, generating a new proj file from scratch can be difficult, and it would be great to replace the existing .mproj file (which targets an older msbuild) with a newer/simplified .proj file that uses modern msbuild features.

Design questions

1. Should we make the SDK build vs. MSBuild a choice when you create a connector?
2. Can we add an "enable for MSBuild" command that changes the default build mode to msbuild and generates a new proj file based on the current workspace contents?

I don't feel strongly about (1), but I would lean towards not making a new connector option, since the use of msbuild is likely a more advanced scenario. I do see value in (2) though, especially as we continue to improve the default build template (making it easier to sign extensions, insert localization steps, etc)... but you could argue that a developer could simply copy their msbuild definition from published templates or samples.

[bgribaudo]

Are you strongly married to keeping both build system options? For simplicity, would there be any chance to standardize on just one (i.e. MSBuild, if that is needed because if its support for advanced scenarios)?

Thinking that a beginner user may not need to touch the build file, so whether SDK or MSBuild is used doesn't affect them. For an advanced user, likely the additional SDK-related options they need to set wouldn't be too complex to set in either style build file, so maybe just supporting one style would be sufficient??

[mattmasson]

My preference would be to provide a single build option, but don't see a way to do that without requiring users to install Visual Studio (to get msbuild) or the .net core SDK (to get dotnet build) as a prerequisite. We could create our own .exe that calls into the MSBuild framework, but that's not something I'd want to maintain long term.

The main advantage of msbuild is that it is configurable - you can customize it (i.e. to add localization and/or signing) and easily plug it into your existing build processes. It's what we use for all internal connector development. But this approach:

1. Has external dependencies that would need to be installed separately
2. Doesn't integrate well with other build frameworks
3. Is fairly heavyweight when your build step is essentially just creating a .zip file

This led us towards adding an "SDK" build option, which was implemented with a call out to the makepqx.exe utility that comes with the SdkTools nuget.

We saw some advantages to letting the extension have full control over the build (knowing when a rebuild is needed, improved handling of build errors), but this ended up complicating evaluation scenarios - especially when you want to use a different build system. And I could imagine a number of issues arising from using a different build process during development vs. when you want to release the connector to "production".

[mattmasson]

When you click the "Run build task ..." command, the current behavior is to:

• Offer the SDK Build Task
• If the project contains a .proj or .mproj file, offer the MSBuild Build Task as well

If we're going to change this, at this point I'm leaning towards:

1. New connector projects default to SDK builds (no .proj included)
2. We have a command that lets you generate a default .proj file (which uses a modern msbuild format)
3. Docs that describe how to convert existing .mproj files into the newer .proj format
4. Docs that describe how to make msbuild your default build task

[bgribaudo]

Behind the scenes, does the MSBuild "package extension" task call makepqx.exe (or its backing DLL), implying that the build behavior is guaranteed to be the same between the two (assuming the same arguments are passed in)?

[bgribaudo]

In light of the large dependency cost that only supporting MSBuild would bring, your idea of having the default "new connector" template not include a .proj MSBuild file, and having some docs around setting up/converting to .proj files, sounds attractive to me.

I would hope that the MSBuild task used for packaging the connector could exactly mirror the SDK-style's MakePQX.exe compile behaviors (maybe by making the MSBuild package task actually call MakePQX.exe compile behind the scenes?).

[bgribaudo]

Related: How many of the legacy SDK's options do you anticipate the new SDK will support (perhaps with different syntax/command line args)?

  <AllowNativeQuery>True</AllowNativeQuery>
  <AsAction>True</AsAction>
  <FastCombine>True</FastCombine> -- irrelevant, as in the new SDK this behavior is enabled by default and cannot be turned off
  <ClearLog>True</ClearLog> -- irrelevant, as new SDK traces are output to files
  <ShowEngineTraces>True</ShowEngineTraces> -- handled by --logMashupEngineTraces
  <ShowUserTraces>True</ShowUserTraces> -- handled by --logMashupEngineTraces
  <LegacyRedirects>True</LegacyRedirects>
  <SuppressRowErrors>True</SuppressRowErrors>
  <SuppressCellErrors>True</SuppressCellErrors>
  <MaxRows>1000</MaxRows>
  <ThrowOnFoldingFailure>True</ThrowOnFoldingFailure> -- handled by --failOnFoldingFailure

[mattmasson]

AllowNativeQuery will become a new command-line option (and likely require some additional special handling).

The others will default to the Power Query behaviors.

[bgribaudo]

Thanks, @mattmasson.