Sync with upstream 2.33.0 #293
Open
+22,806
−9,104
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File-system-object-layer functionality doesn't depend on store-layer concets, and therefore doesn't need to live inside there.
A shared pointer is not needed.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it uses `MemorySourceAccessor::File`, or rather variations of it, to go to a C++ data type first, and only JSON second. To accomplish this we add some type parameters to the `File` data type. Actually, we need to do two rounds of this, because shallow NAR listings. There is `FileT` and `DirectoryT` accordingly.
Also do a better JSON and testing for deep and shallow NAR listings. As documented, this for file system objects themselves, since `MemorySourceAccessor` is an implementation detail.
Deduplicate `listNar` and `MemorySourceAccessor::File`
The use of sourceToSink is an unnecessary serialization bottleneck. While we are at it, fix the copyRecursive implementation to actually copy the whole directory. It wasn't used for anything prior, but now it has a use and accompanying tests for flake clone.
Turn one unsafe C cast into a safe `static_cast`
`nlohmann::json` instance and JSON Schema for `MemorySourceAccessor`
libutil: Fix copyRecursive and use for nix flake clone
As a precaution. This function might get used for some long persisted file descriptor and we need good defaults.
libutil/unix: Add O_CLOEXEC to openDirectory
this is a painful change. we should really add EvalState or EvalMemory as an argument to various functions as we need it, but because we want to preserve the stablity API, we hack it in as a field of nix_value.
Introduce GitAccessorOptions
builtins.deepSeq on deeply nested structures (e.g., a linked list with 100,000 elements) caused an uncontrolled OS-level stack overflow with no Nix stack trace. Fix by adding call depth tracking to forceValueDeep, integrating with Nix's existing max-call-depth mechanism. Now produces a controlled "stack overflow; max-call-depth exceeded" error with a proper stack trace. Closes: NixOS#7816
When deepSeq encounters an error while evaluating a list element, the error trace now includes the list index, making it easier to locate the problematic element.
Similar to the deepSeq fix, toJSON on deeply nested structures caused an uncontrolled OS-level stack overflow. Fix by adding call depth tracking to printValueAsJSON.
Those can never be nullptr, so we should use the type system to ensure this invariant.
This is necessary to make pausing/unpausing possible in a follow-up commit.
…ad thread Instead of naively stalling the download thread we can instead stop the transfer. This allows the other multiplexed connections to continue downloading (and unpacking), if the result of the download gets piped into a GitFileSystemObjectSink. Prior art in lix project: - https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc - https://git.lix.systems/lix-project/lix/commit/12156d3beb8a16c0e2e8cf7180e1fbf27280a669 This patch is very different from the lix one, since we are using a decompression sink in the middle of the pipeline but the co-authored-by is there since I was motivated to implement this by looking at the lix side of things. Co-authored-by: eldritch horrors <[email protected]>
Since the root cause (the lack of backpressure control) has been fixed in the previous commit we can revert the change from 8ffea0a and make the default size much smaller.
libstore/filetransfer: Pause transfers instead of stalling the download thread
Updated documentation to clarify that building without optimization can lead to faster builds.
Clarify build options in debugging documentation
On FreeBSD, sysctl(KERN_PROC_PATHNAME) returns a null-terminated
string with pathLen including the terminator. This causes Nix to
fail during manual generation with:
error:
… while calling the 'concatStringsSep' builtin
at /nix/var/nix/builds/nix-63232-402489527/source/doc/manual/generate-settings.nix:99:1:
98| in
99| concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo))
| ^
100|
error: input string '/nix/store/gq89cj02b5zs67cbd85vzg5cgsgnd8mj-nix-2.31.2/bin/nix␀'
cannot be represented as Nix string because it contains null bytes
The issue occurs because generate-settings.nix reads the nix binary
path from JSON and evaluates it as a Nix string, which cannot contain
null bytes. Normal C++ string operations don't trigger this since they
handle null-terminated strings correctly.
Strip the null terminator on FreeBSD to match other platforms (Linux
uses /proc/self/exe, macOS uses _NSGetExecutablePath).
Credit: @wahjava (FreeBSD ports and Nixpkgs contributor)
…rminator fix(FreeBSD): remove null terminator from executable path
`deepSeq`, json: handle stack overflow, report list index
turn 'derivation has incorrect deferred output' into warning
The fact that we were introducing a conversion from the output of `nix path-info` into the input of `builtins.fetchTree` was the deciding factor. We want scripting outputs into inputs like that to be easy. Since JSON strings and objects are trivially distinguishable, we still have the option of introducing the JSON format as an alternative input scheme in the future, should we want to. (The output format would still be SRI in that case, presumably.)
builtins.path: Propagate references from derivation outputs
Use SRI hash (strings) as the official JSON format for Hash after all
Previously the daemon didn't validate that it got a valid GCAction and did a naive C-style cast to the enum. This is certainly unintentional, albeit mostly harmless in practice.
Fix `MAKE_WRAPPER_CONSTRUCTOR` to not override special constructors
This fixes out-of-date information that is no longer true, and makes the up-to-date information more accessible.
daemon: Add WorkerProto serialiser for GCAction
Correct `build-dir` error in manual, link relevant settings
std::filesystem::path does quoting by default so it resulted in: > netrc-file = "/etc/nix"/netrc
globals: Fix netrc-file default value
2.33 release notes
This matches what we just did for `nix path-info`, and I hope will allow us to avoiding any more breaking changes to this command for the foreseeable future. (cherry picked from commit 0f18076)
edolstra
added
the
flake-regression-test
Tagging release 2.33.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Context