Add support for GIT LFS checkout#3909
Conversation
lizrabuya
changed the title
lizrabuya
changed the title
lizrabuya
force-pushed
the
feat/sup-6529/lfs-checkout
branch
from
f9e4186 to
255aa11
Compare
…y are not runnable on windows
| if args.Retry { | ||
| retrier = roko.NewRetrier( | ||
| roko.WithStrategy(roko.ExponentialSubsecond(1*time.Second)), | ||
| roko.WithMaxAttempts(10), // 10 attempts will take ~2 minutes 17s | ||
| roko.WithJitter(), | ||
| ) | ||
| } |
There was a problem hiding this comment.
The defaultCheckoutPhase is already wrapped in an outer roko retrier (default 6 attempts). The new inner LFS retrier adds 10 more. On a persistent LFS failure: 10 inner attempts (~2m17s) × up to 6 outer = potentially ~13+ minutes of retries, and each outer retry re-runs the entire clone/fetch/checkout/submodule sequence. We should consider whether the inner retry should be lighter, or whether LFS failures should Break() the outer retrier.
| { | ||
| name: "LFS enabled git lfs fetch fails", | ||
| lfsEnabled: true, | ||
| setupPath: func(t *testing.T) { | ||
| if runtime.GOOS == "windows" { | ||
| t.Skip("Not runnable on Windows: git for Windows uses bundled git-lfs.exe regardless of PATH") | ||
| } | ||
| t.Setenv("PATH", fakeLFSBinDir(t, | ||
| "#!/bin/sh\ncase \"$1\" in\n install) exit 0 ;;\n *) exit 1 ;;\nesac\n", | ||
| "@echo off\r\nif \"%1\"==\"install\" exit /b 0\r\nexit /b 1\r\n", | ||
| )) | ||
| }, | ||
| wantErr: "git lfs fetch", | ||
| }, |
There was a problem hiding this comment.
This test case runs for ~95s because the failure path results burns all 10 real-backoff retries. There is no way to inject a fast strategy. We should consider making the retry strategy injectable so the failure path tests in milliseconds.
There was a problem hiding this comment.
I think there is no need to implement an injectable retry strategy now. Dropping the max attempts to half will now reduce the tests significantly.
|
@lizrabuya could you ask for a review in #ask-agents or in our project channel tagging Josh/Ming once you address Dahtey's comments? |
3 participants
Description
Add configuration and environment variables when the backend sets
BUILDKITE_GIT_LFS_ENABLED=truein the job environment. WhenBUILDKITE_GIT_LFS_ENABLED=trueis set, the agent will check if thegit-lfsbinary is installed and proceed to perform agit lfs fetch+git lfs checkout.Changes
Configuration & CLI (clicommand/bootstrap.go, internal/job/config.go)
Checkout behaviour (internal/job/checkout.go)
LFS fetch helper (internal/job/git.go)
Smudge bypass (internal/job/executor.go)
Tests (internal/job/checkout_test.go)
Windows test caveats
Two of the LFS error-path subtests (
LFS enabled git lfs command failsandLFS enabled git lfs fetch fails) are skipped on Windows. They cannot run there for a fundamental reason, not flakiness:PATHto point at a fakegit-lfsscript that exits non-zero, so we can assert our error-wrapping behaviour.git-lfs.exeinsideGIT_EXEC_PATH(the Git installation'slibexec/git-coredirectory). Whengitresolves a subcommand likegit lfs ..., it looks inGIT_EXEC_PATHbefore falling back toPATH.git-lfs.exeruns, succeeds, and the test fails because we expected our fake to return an error.gitLFSFetchCheckoutandgit lfs install --localfailure handling) is platform-agnostic, so the Linux runs give us the real coverage. Skipping on Windows is a test-harness limitation, not a behavioural gap.Additionally, the two happy-path subtests (
LFS disabled,LFS enabled binary present) useos.MkdirTemp+ a best-effortRemoveAllint.Cleanuprather thant.TempDir(). On Windows, git's child processes (credential helpers, thegit-lfsfilter-process) can keep file handles open past their parent's exit.t.TempDir()'s cleanup is strict and fails the test onunlinkat ... being used by another processerrors; the best-effort variant lets the test pass while still cleaning up what it can.Testing
go test ./...). Buildkite employees may check this if the pipeline has run automatically.go tool gofumpt -extra -w .)Disclosures / Credits
The main function changes were implemented by me based on the recommendation from the research done for this function. I used Claude to review the code and to assist with writing the tests.