Skip to content
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
4fe69cf
Create restore_spm_cache
spencertransier Feb 8, 2023
702c16e
Add separate save_spm_cache action
spencertransier Feb 9, 2023
5fb1859
Try different path to get relative path
spencertransier Feb 9, 2023
bf61231
Revert "Try different path to get relative path"
spencertransier Feb 9, 2023
0697fb3
Remove --quiet from aws commands
spencertransier Feb 10, 2023
ca48379
Test different tar option
spencertransier Feb 10, 2023
1f19980
Restore cache to correct location
spencertransier Feb 10, 2023
dc1ef56
Move the commands over to `install_swiftpm_dependencies`
spencertransier Feb 10, 2023
9c1565a
Fix linter issue
spencertransier Feb 10, 2023
4855a65
Automatically find Package.resolved
spencertransier Feb 10, 2023
ac1697b
Package.resolved variable isn't initialized yet
spencertransier Feb 11, 2023
776efe7
Exclude 'checkouts' and 'artifacts' from SPM cache
spencertransier Feb 11, 2023
e6813ff
Switch back to root project directory after restoring cache
spencertransier Feb 11, 2023
74e046b
Test with explicit workspace path
spencertransier Feb 11, 2023
7237304
Change location of SPM cache
spencertransier Feb 18, 2023
cdd1414
Change where to save cache from
spencertransier Feb 18, 2023
38c1a99
Install the SPM packages in the correct location
spencertransier Feb 18, 2023
543042e
Use `swift` or `xcodebuild` depending on the project type
spencertransier Feb 23, 2023
8bfe4f7
Use Apple Archive for archiving and extracting the cache
spencertransier Mar 1, 2023
68b361b
Update install_swiftpm_dependencies
spencertransier Mar 1, 2023
b3fe073
Add lines to verify archive/extract method
spencertransier Mar 1, 2023
d988855
Re-add --quiet to AWS upload
spencertransier Mar 1, 2023
1423e71
Add custom path
spencertransier Mar 1, 2023
9432ed6
Collapse aa archive into one line
spencertransier Mar 1, 2023
71f18ab
Use uname to determine whether to use Apple Archive
spencertransier Mar 1, 2023
f71842b
Continue iterating on Apple Archive support
spencertransier Mar 1, 2023
f3ba705
Remove Apple Archive option
spencertransier Mar 1, 2023
f08a599
Use $HOME instead of ~
spencertransier Mar 1, 2023
1b518d5
Use the correct TAR_OPTIONS
spencertransier Mar 1, 2023
bdaf289
Cleaned up debugging echos and added explanatory comments
spencertransier Mar 1, 2023
48bc2ac
Fix shellcheck warnings
spencertransier Mar 1, 2023
f8942a8
Fix shellcheck issue - SC2128
spencertransier Mar 1, 2023
8a6377f
Simplify saving cache by deleting unneeded cache files beforehand
spencertransier Mar 2, 2023
e8784f5
Fix indenting
spencertransier Mar 2, 2023
c05d5e8
Add curly braces around CACHE_KEY variables
spencertransier Mar 2, 2023
82af9b6
Add quotes around all variable names in `if` statement
spencertransier Mar 2, 2023
44785e4
Use default SPM location
spencertransier Mar 2, 2023
b919255
Update tar custom flag name
spencertransier Mar 2, 2023
9d49421
Exit if unable to find Package.resolved or if BUILD_TYPE is not set
spencertransier Mar 9, 2023
4cec047
Merge branch 'trunk' into add/spm-caching-actions
spencertransier Mar 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

.DS_Store
60 changes: 59 additions & 1 deletion bin/install_swiftpm_dependencies
Original file line number Diff line number Diff line change
@@ -1,9 +1,67 @@
#!/bin/bash -eu

CUSTOM_PACKAGE_RESOLVED_PATH=${1-}
BUILD_TYPE=${2-}
SPM_CACHE_LOCATION="$HOME/spm"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There exists a $HOME/.swiftpm/cache folder on my Mac that was probably created by SPM itself at some point.

I wonder if we shouldn't use that folder (for conventions) as opposed to ~/spm here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point! Jeremy and I had discussed using the cache that's within DerivedData, but I wasn't able to make that work because there's not a consistent path name within DerivedData. But it does work great with making the cache location in the script point to $HOME/Library/Caches/org.swift.swiftpm. And that has the added benefit of client repos not needing to specify a specific SPM cache folder in the Fastfile. Package resolution will pull from that org.swift.swiftpm folder automatically when resolving the packages.

Added in 44785e4


WORKSPACE_PACKAGE_RESOLVED_PATH=(*.xcworkspace/xcshareddata/swiftpm/Package.resolved)
ROOT_PACKAGE_RESOLVED_PATH=Package.resolved

# Find where Package.resolved is located
if [[ -n "$CUSTOM_PACKAGE_RESOLVED_PATH" ]]; then
PACKAGE_RESOLVED_LOCATION=$CUSTOM_PACKAGE_RESOLVED_PATH
elif [[ -f ${WORKSPACE_PACKAGE_RESOLVED_PATH[0]} ]]; then
BUILD_TYPE="XCODEBUILD"
PACKAGE_RESOLVED_LOCATION=${WORKSPACE_PACKAGE_RESOLVED_PATH[0]}
elif [[ -f $ROOT_PACKAGE_RESOLVED_PATH ]]; then
BUILD_TYPE="SWIFT"
PACKAGE_RESOLVED_LOCATION=$ROOT_PACKAGE_RESOLVED_PATH
else
echo "Unable to find Package.resolved"
fi

PACKAGE_RESOLVED_HASH=$(hash_file "$PACKAGE_RESOLVED_LOCATION")
CACHE_KEY="$BUILDKITE_PIPELINE_SLUG-spm-cache-$PACKAGE_RESOLVED_HASH"

# Restore SPM cache if it's available
mkdir -p "$SPM_CACHE_LOCATION"
cd "$SPM_CACHE_LOCATION"
restore_cache "$CACHE_KEY"
cd -

# This will let Xcode use the system SSH config for downloading packages
sudo defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not part of this PR. Is this preference same as xcodebuild -scmProvider system?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@crazytonyli I think so but I'm not sure. I will look into this and see about adding in the next update of the SPM caching action.


# Trust all GitHub.com and BitBucket.org keys – this allows checking out dependencies via SSH
add_host_to_ssh_known_hosts bitbucket.org
add_host_to_ssh_known_hosts github.com

xcodebuild -resolvePackageDependencies
# Based on the project type, resolve the packages using the correct build type
if [[ "$BUILD_TYPE" == "XCODEBUILD" ]]; then
echo " Resolving packages with \`xcodebuild\`"
xcodebuild \
-resolvePackageDependencies \
-clonedSourcePackagesDirPath "$SPM_CACHE_LOCATION"
elif [[ "$BUILD_TYPE" == "SWIFT" ]]; then
echo " Resolving packages with \`swift\`"
swift package resolve \
--cache-path "$SPM_CACHE_LOCATION" \
--verbose
else
echo "BUILD_TYPE is not set"
fi

# `checkouts` can be removed because the system can quickly generate them
# instead of needing to download them in the cache each time.
#
# `artifacts` should be removed because it causes issues when downloading
# certain packages to have the artifacts already present after extracting
# cache
echo " Cleaning up cache files before saving cache"
rm -rf "$SPM_CACHE_LOCATION/checkouts" "$SPM_CACHE_LOCATION/artifacts"

# If this is the first time we've seen this particular cache key, save it for the future
echo " Saving SPM Cache"
save_cache "$SPM_CACHE_LOCATION" "$CACHE_KEY" false --use_alternate_tar_config


12 changes: 11 additions & 1 deletion bin/save_cache
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,17 @@ if ! aws s3api head-object --bucket "$S3_BUCKET_NAME" --key "$CACHE_KEY" > /dev/
echo "No existing cache entry for $CACHE_KEY – storing in cache"

echo " Compressing"
tar -czf "$CACHE_KEY" "$CACHE_FILE"
TAR_CONFIG=${4-}
if [[ "$TAR_CONFIG" == '--use_alternate_tar_config' ]]; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit but I'm not a fan of the name of this custom flag… what about --use_relative_path_in_tar or something like that instead?


Also, I'm wondering if it wouldn't make sense to adopt that -C behavior… in all circumstances after all (seems a nicer solution than including /User/builder in the cache path and risking the un-tar-ing to fail if used on a different builder which might not have a /User/builder folder (all our Mac VMs currently do, but we shouldn't need to make that assumption, and also nothing says the cache would be restored on a Mac VM vs an Android AMI or Docker container or whatnot, so…).

That one might be a broader investigation to verify that changing that behavior that save_cache was using so far (and you kept in the else) to instead use -C "$CACHE_FILE" . in all cases would not break existing setups depending on how they use restore_cache on their end; so could be saved for a subsequent PR if you're not sure about that. But would be a nice simplification if we can confirm it and go in that direction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I like --use_relative_path_in_tar better, too. Changed in b919255

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if it wouldn't make sense to adopt that -C behavior in all circumstances.

I agree that would be nice to use the same behavior for all cases if possible. Because I saw different behavior between the existing tar configuration versus the one I added in this PR, I had assumed the projects that use save_cache would need to be updated and cause a breaking change. I haven't tested that out yet, but I think it'd be outside the scope for this project. I'll create a separate issue for it.

# This is used by actions such as `install_swiftpm_dependencies`
# This configuration allows the tar to not include the full system path of the
# directory that's being archived. For example, this will save only the
# "DIRECTORY_BEING_ARCHIVED" in `/User/builder/DIRECTORY_BEING_ARCHIVED`
# instead of also creating `/User/builder` when extracting the archive
tar -czf "$CACHE_KEY" -C "$CACHE_FILE" .
else
tar -czf "$CACHE_KEY" "$CACHE_FILE"
fi

echo " Uploading"
# If the bucket has transfer acceleration enabled, use it!
Expand Down