Skip to content

Commit

Permalink
[vcpkg] Download vcpkg.exe rather than building it in bootstrap on Wi…
Browse files Browse the repository at this point in the history 
…ndows. (#15474)

This reduces bootstrap cost for Windows customers, resolving the issue initially submitted as #12502 .

The `toolsrc` tree was extracted to https://github.com/microsoft/vcpkg-tool. `bootstrap.sh` was changed to download the right source tarball, extract, and build it. This was chosen over the previous attempt, a submodule, over concerns of accidentally destroying people's local modifications.
  • Loading branch information
@BillyONeal
BillyONeal authored Feb 4, 2021
1 parent f226416 commit aa60b7e
Show file tree
Hide file tree
Showing 307 changed files with 46 additions and 69,859 deletions.
9 changes: 0 additions & 9 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,6 @@
*.userosscache
*.sln.docstates

toolsrc/out*
toolsrc/CMakeSettings.json
# fuzzing
sync_dir*

Expand All @@ -30,8 +28,6 @@ bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# VS Code build
toolsrc/build
# Ignore the executable
/vcpkg
/vcpkg.exe
Expand Down Expand Up @@ -290,11 +286,6 @@ __pycache__/
/installed*/
/packages/
/scripts/buildsystems/tmp/
/toolsrc/build.rel/
/toolsrc/windows-bootstrap/msbuild.x86.debug/
/toolsrc/windows-bootstrap/msbuild.x86.release/
/toolsrc/windows-bootstrap/msbuild.x64.debug/
/toolsrc/windows-bootstrap/msbuild.x64.release/
#ignore custom triplets
/triplets/*
#add vcpkg-designed triplets back in
Expand Down
2 changes: 1 addition & 1 deletion docs/about/privacy.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ We collect various telemetry events such as the command line used, the time of i

You can see the telemetry events any command by appending `--printmetrics` after the vcpkg command line.

In the source code (included in `toolsrc\`), you can search for calls to the functions `track_property()`, `track_feature()`, `track_metric()`, and `track_buildtime()`
In the source code (included at https://github.com/microsoft/vcpkg-tool/ ), you can search for calls to the functions `track_property()`, `track_feature()`, `track_metric()`, and `track_buildtime()`
to see every specific data point we collect.

## Avoid inadvertent disclosure information
Expand Down
195 changes: 0 additions & 195 deletions docs/tool-maintainers/benchmarking.md
View file

This file was deleted.

74 changes: 0 additions & 74 deletions docs/tool-maintainers/layout.md
View file

This file was deleted.

Loading

12 comments on commit aa60b7e

@hunt978
Copy link

@hunt978 hunt978 commented on aa60b7e Feb 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a good choice at all !!! bring toolsrc back pls...

@reynoldsbd
Copy link
Member

@reynoldsbd reynoldsbd commented on aa60b7e Jun 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replying to a related conversation from PR #12502:

Reason I ask is I have some local customizations to vcpkg source that would cause the product to be unusable without these customizations.

You might need to alter what you're doing to invoke cmake and ninja to build us rather than running bootstrap.bat/bootstrap.sh but the sources will still be present.

@BillyONeal / @StarGate-One my team is in the same boat, and as of aa60b7e the suggested workaround is no longer possible. I am going to try cloning/patching/building vcpkg-tool independently, essentially copying what bootstrap.sh is doing. Will update this comment with the results.

EDIT: Manually bootstrapping from vcpkg-tool works like any other CMake project: clone, patch, configure, build. Just make sure you're using the same permutation of compiler and generator as the official CI builds.

I can't help but feel that something has been lost here. Having everything self-contained in a single repo and built from source had several key benefits:

  • Only had a single commit hash representing the tool source and ports tree. Now I have to manage the two independently and figure out how to keep them synchronized.
  • Ability to easily patch the tool to work around issues important to our team without waiting for upstream to solve. Now we have to go above and beyond in order to patch things.
  • Perhaps this is subjective, but a locally-built EXE is always more trustworthy than downloading a prebuilt one. And honestly. bootstrap performance was never something that concerned me because vcpkg is not on the critical path of my "inner loop."

@BillyONeal
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reynoldsbd From a development team prospective, we also preferred everything being in one repo. Unfortunately, there are lots of people who want to run vcpkg on GitHub Actions or Azure Pipelines or similar CI systems, and for them our bootstrap was often the majority of their build time. We understand that this change isn't without costs, but we feel on balance that it is the right option for most customers. This also closes the door on one of our most common sources of customer breakage: customers trying to bootstrap with versions of Visual Studio different than the one we use for our day to day development.

Only had a single commit hash representing the tool source and ports tree. Now I have to manage the two independently and figure out how to keep them synchronized.

We are moving towards a model where the synchronization between these is less important; typically the tool version isn't changing more than once per month, and even then there's no significant loss in not being on the latest and greatest (other than not getting new features). This is something we would be wanting to do anyway to integrate into Visual Studio or CLion or similar.

Ability to easily patch the tool to work around issues important to our team without waiting for upstream to solve.

If you want you can still download from source; it just isn't our normal operating mode.

Perhaps this is subjective, but a locally-built EXE is always more trustworthy than downloading a prebuilt one.

They should be of similar trustworthyness because the same people with the power to merge changes here are also the people producing the binary in question and that binary has a digital signature that it is from us, but of course you can build your own copy. We stamp the commit SHA from vcpkg-tool into the resulting binary for you.

vcpkg is not on the critical path of my "inner loop."

It is in the inner loop of many customers though; feedback on this change has been mostly 'thank you' and 'when can you do macos and linux' thus far.

@reynoldsbd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the response Billy! I agree, this does sound like a net-positive for the community. And most of my concerns would vanish if we didn't have to patch.

Thanks for continuing to build this awesome tool! :)

@playgithub
Copy link
Contributor

@playgithub playgithub commented on aa60b7e Jul 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what is in the binary, I don't like the policy, I won't use vcpkg anymore due the kind of policies that even might happen in the future.

@BillyONeal
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what is in the binary, I don't like the policy, I won't use vcpkg anymore due the kind of policies that even might happen in the future.

As I indicated before, you will always be able to build it from source if you want.

@playgithub
Copy link
Contributor

@playgithub playgithub commented on aa60b7e Jul 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what is in the binary, I don't like the policy, I won't use vcpkg anymore due the kind of policies that even might happen in the future.

As I indicated before, you will always be able to build it from source if you want.

default matters to me
I'm not going to use it because I'm not sure what will happen in the future while considering this change.

@BillyONeal
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default matters to me
I'm not going to use it because I'm not sure what will happen in the future while considering this change.

I'm sorry to hear that, but hope you understand that we can't make the experience worse for 99.999% of our customers because you don't trust that we faithfully built the binary by default.

@playgithub
Copy link
Contributor

@playgithub playgithub commented on aa60b7e Jul 23, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hope you understand that we can't make the experience worse for 99.999% our customers because you don't trust that we faithfully built the binary by default.

It takes much more time to build libs than to build the vcpkg executable.
For the benefit of prebuilt binaries, conan is ok.

Reviewing Microsoft's Automatic Insertion of Telemetry into C++ Binaries

@BillyONeal
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It takes much more time to build libs than to build the vcpkg executable.

That is not true for many people. And it's true for almost noone when there is a cache hit.

For the benefit of prebuilt binaries, conan is ok.

So.... you trust conan to provide prebuilt libraries that you link in your program and ship to customers but you don't trust us to provide a courtesy copy of a thing we also provide all the source for?

Reviewing Microsoft's Automatic Insertion of Telemetry into C++ Binaries

That doesn't seem relevant.

@playgithub
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It takes much more time to build libs than to build the vcpkg executable.

That is not true for many people. And it's true for almost noone when there is a cache hit.

The vcpkg executable only take a little time to build, a miniute or so, and it needs rebuilding infrequently.

For the benefit of prebuilt binaries, conan is ok.

So.... you trust conan to provide prebuilt libraries that you link in your program and ship to customers but you don't trust us to provide a courtesy copy of a thing we also provide all the source for?

I don't use conan, though with conan I can build the binaries myself, it is just not high quality enough.
I've just learn cmake decently, I'll take try to use cmake only.
I'm using Linux now, the official repository is also convenient, more trustable.

Reviewing Microsoft's Automatic Insertion of Telemetry into C++ Binaries

That doesn't seem relevant.

It depends, the vcpkg binary use a flag to disable telemetry, but it still has the ability to telemetry, it's not as good as without source code of telemetry.

@playgithub
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why no option for building instead of downloading e.g.

$ bootstrap.sh
Options
1 Build from source
2 Download binary
3 Exit
Please select an option: _

Please sign in to comment.