Handle kv.sqlite database contention/incomplete transactions - #2790
Conversation
|
Oh interesting. Are the errors mostly Windows? I could see multiple threads attempting concurrent sqlite opens; that may need a separate fix. I've seen it on MacOS before too, come to think of it, coming up as EBUSY from the OS. |
brhoades
left a comment
brhoades
left a comment
There was a problem hiding this comment.
LGTM. This is a good add. I didn't realize this was a modern feature?
| require.NoError(t, s.Close()) | ||
| } | ||
|
|
||
| func TestGetSet_Simultaneous(t *testing.T) { |
There was a problem hiding this comment.
I think testing this with N parallel tests against a single database across platforms might be interesting. Unless we're doing that somewhere.
There was a problem hiding this comment.
I don't think we have anything with a shared test database at the moment!
Windows is overrepresented in some of these specific error logs because it's the only OS where we use the launcher watchdog, so the only place where we're writing to the watchdog logs store in kv.sqlite. For error |
|
Got it. I don't know the proportional fleet size, but if it is a file handle contention problem it'd have to be disproportionate to track with my experience. Just a thought, no blockers from me here. |
3 participants
A recent update to modernc.org/sqlite makes
_busy_timeoutavailable. When_busy_timeoutis set, sqlite will wait for the timeout instead of immediately returning aSQLITE_BUSYerror. I am hopeful that this will resolve some of the contention-related error logs that we see.I chose a timeout of 10 seconds based solely on
rungroup.InterruptTimeout.