Skip to content
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
4fe69cf
Create restore_spm_cache
spencertransier Feb 8, 2023
702c16e
Add separate save_spm_cache action
spencertransier Feb 9, 2023
5fb1859
Try different path to get relative path
spencertransier Feb 9, 2023
bf61231
Revert "Try different path to get relative path"
spencertransier Feb 9, 2023
0697fb3
Remove --quiet from aws commands
spencertransier Feb 10, 2023
ca48379
Test different tar option
spencertransier Feb 10, 2023
1f19980
Restore cache to correct location
spencertransier Feb 10, 2023
dc1ef56
Move the commands over to `install_swiftpm_dependencies`
spencertransier Feb 10, 2023
9c1565a
Fix linter issue
spencertransier Feb 10, 2023
4855a65
Automatically find Package.resolved
spencertransier Feb 10, 2023
ac1697b
Package.resolved variable isn't initialized yet
spencertransier Feb 11, 2023
776efe7
Exclude 'checkouts' and 'artifacts' from SPM cache
spencertransier Feb 11, 2023
e6813ff
Switch back to root project directory after restoring cache
spencertransier Feb 11, 2023
74e046b
Test with explicit workspace path
spencertransier Feb 11, 2023
7237304
Change location of SPM cache
spencertransier Feb 18, 2023
cdd1414
Change where to save cache from
spencertransier Feb 18, 2023
38c1a99
Install the SPM packages in the correct location
spencertransier Feb 18, 2023
543042e
Use `swift` or `xcodebuild` depending on the project type
spencertransier Feb 23, 2023
8bfe4f7
Use Apple Archive for archiving and extracting the cache
spencertransier Mar 1, 2023
68b361b
Update install_swiftpm_dependencies
spencertransier Mar 1, 2023
b3fe073
Add lines to verify archive/extract method
spencertransier Mar 1, 2023
d988855
Re-add --quiet to AWS upload
spencertransier Mar 1, 2023
1423e71
Add custom path
spencertransier Mar 1, 2023
9432ed6
Collapse aa archive into one line
spencertransier Mar 1, 2023
71f18ab
Use uname to determine whether to use Apple Archive
spencertransier Mar 1, 2023
f71842b
Continue iterating on Apple Archive support
spencertransier Mar 1, 2023
f3ba705
Remove Apple Archive option
spencertransier Mar 1, 2023
f08a599
Use $HOME instead of ~
spencertransier Mar 1, 2023
1b518d5
Use the correct TAR_OPTIONS
spencertransier Mar 1, 2023
bdaf289
Cleaned up debugging echos and added explanatory comments
spencertransier Mar 1, 2023
48bc2ac
Fix shellcheck warnings
spencertransier Mar 1, 2023
f8942a8
Fix shellcheck issue - SC2128
spencertransier Mar 1, 2023
8a6377f
Simplify saving cache by deleting unneeded cache files beforehand
spencertransier Mar 2, 2023
e8784f5
Fix indenting
spencertransier Mar 2, 2023
c05d5e8
Add curly braces around CACHE_KEY variables
spencertransier Mar 2, 2023
82af9b6
Add quotes around all variable names in `if` statement
spencertransier Mar 2, 2023
44785e4
Use default SPM location
spencertransier Mar 2, 2023
b919255
Update tar custom flag name
spencertransier Mar 2, 2023
9d49421
Exit if unable to find Package.resolved or if BUILD_TYPE is not set
spencertransier Mar 9, 2023
4cec047
Merge branch 'trunk' into add/spm-caching-actions
spencertransier Mar 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@

.DS_Store
58 changes: 57 additions & 1 deletion bin/install_swiftpm_dependencies
Original file line number Diff line number Diff line change
@@ -1,9 +1,65 @@
#!/bin/bash -eu

CUSTOM_PACKAGE_RESOLVED_PATH=${1-}
BUILD_TYPE=${2-}
SPM_CACHE_LOCATION="$HOME/spm"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There exists a $HOME/.swiftpm/cache folder on my Mac that was probably created by SPM itself at some point.

I wonder if we shouldn't use that folder (for conventions) as opposed to ~/spm here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point! Jeremy and I had discussed using the cache that's within DerivedData, but I wasn't able to make that work because there's not a consistent path name within DerivedData. But it does work great with making the cache location in the script point to $HOME/Library/Caches/org.swift.swiftpm. And that has the added benefit of client repos not needing to specify a specific SPM cache folder in the Fastfile. Package resolution will pull from that org.swift.swiftpm folder automatically when resolving the packages.

Added in 44785e4


WORKSPACE_PACKAGE_RESOLVED_PATH=(*.xcworkspace/xcshareddata/swiftpm/Package.resolved)
ROOT_PACKAGE_RESOLVED_PATH=Package.resolved

# Find where Package.resolved is located
if [[ -n "$CUSTOM_PACKAGE_RESOLVED_PATH" ]]; then
PACKAGE_RESOLVED_LOCATION=$CUSTOM_PACKAGE_RESOLVED_PATH
elif [[ -f ${WORKSPACE_PACKAGE_RESOLVED_PATH[0]} ]]; then
BUILD_TYPE="XCODEBUILD"
PACKAGE_RESOLVED_LOCATION=${WORKSPACE_PACKAGE_RESOLVED_PATH[0]}
elif [[ -f $ROOT_PACKAGE_RESOLVED_PATH ]]; then
BUILD_TYPE="SWIFT"
PACKAGE_RESOLVED_LOCATION=$ROOT_PACKAGE_RESOLVED_PATH
else
echo "Unable to find Package.resolved"
fi

PACKAGE_RESOLVED_HASH=$(hash_file "$PACKAGE_RESOLVED_LOCATION")
CACHE_KEY="$BUILDKITE_PIPELINE_SLUG-spm-cache-$PACKAGE_RESOLVED_HASH"

# Restore SPM cache if it's available
mkdir -p "$SPM_CACHE_LOCATION"
cd "$SPM_CACHE_LOCATION"
restore_cache "$CACHE_KEY"
cd -

# This will let Xcode use the system SSH config for downloading packages
sudo defaults write com.apple.dt.Xcode IDEPackageSupportUseBuiltinSCM YES
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not part of this PR. Is this preference same as xcodebuild -scmProvider system?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@crazytonyli I think so but I'm not sure. I will look into this and see about adding in the next update of the SPM caching action.


# Trust all GitHub.com and BitBucket.org keys – this allows checking out dependencies via SSH
add_host_to_ssh_known_hosts bitbucket.org
add_host_to_ssh_known_hosts github.com

xcodebuild -resolvePackageDependencies
# Based on the project type, resolve the packages using the correct build type
if [[ "$BUILD_TYPE" == "XCODEBUILD" ]]; then
echo "Resolving packages with \`xcodebuild\`"
xcodebuild \
-resolvePackageDependencies \
-clonedSourcePackagesDirPath "$SPM_CACHE_LOCATION"
elif [[ "$BUILD_TYPE" == "SWIFT" ]]; then
echo "Resolving packages with \`swift\`"
swift package resolve \
--cache-path "$SPM_CACHE_LOCATION" \
--verbose
else
echo "BUILD_TYPE is not set"
fi

# `checkouts` can be excluded because the system can quickly generate them
# instead of needing to download them in the cache each time.
#
# `artifacts` should be excluded because it causes issues when downloading
# certain packages to have the artifacts already present after extracting
# cache
TAR_OPTIONS="--exclude=./checkouts --exclude=./artifacts"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than fighting with bash syntax regarding using quote or not, I'm wondering if it'd be simpler to just delete these unwanted files from the SPM_CACHE_LOCATION?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, copy the "SPM_CACHE_LOCATION", delete the unwanted files from the new copy, call save_cache on the new copy, and delete the new copy afterwards.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AliSoftware Thank you for the explanation! The array solution makes a lot of sense after the behavior I saw while debugging this.

@crazytonyli Good idea! I think I'm going to go with deleting those directories before saving the cache. It removes the complexity of passing the tar options along with simplifying the changes to save_cache.


# If this is the first time we've seen this particular cache key, save it for the future
save_cache "$SPM_CACHE_LOCATION" "$CACHE_KEY" false "$TAR_OPTIONS"


12 changes: 11 additions & 1 deletion bin/save_cache
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,17 @@ if ! aws s3api head-object --bucket "$S3_BUCKET_NAME" --key "$CACHE_KEY" > /dev/
echo "No existing cache entry for $CACHE_KEY – storing in cache"

echo " Compressing"
tar -czf "$CACHE_KEY" "$CACHE_FILE"
TAR_OPTIONS=${4-foo}
if [[ -n "$TAR_OPTIONS" ]]; then
# This is used by actions such as `install_swiftpm_dependencies`
# These options allow the tar to not include the full system path of the
# directory that's being archived. For example, this will save only the
# "DIRECTORY_BEING_ARCHIVED" in `/User/builder/DIRECTORY_BEING_ARCHIVED`
# instead of also creating `/User/builder` when extracting the archive
tar $TAR_OPTIONS -czf "$CACHE_KEY" -C "$CACHE_FILE" .
else
tar -czf "$CACHE_KEY" "$CACHE_FILE"
fi

echo " Uploading"
# If the bucket has transfer acceleration enabled, use it!
Expand Down