Repository navigation
Conversation
Add a --data-dir command-line flag (and corresponding VECTOR_DATA_DIR environment variable) to override the data_dir global configuration option. When set, it takes precedence over any data_dir value in the configuration file. This keeps deployment-specific paths out of the configuration file and makes it possible to validate a configuration in a CI environment where the configured data_dir may not exist. Closes vectordotdev#9766
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 946a6e4464
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| #[arg(long, env = "VECTOR_DATA_DIR")] | ||
| pub data_dir: Option<PathBuf>, |
There was a problem hiding this comment.
This option is added only to RootOpts, but the validate subcommand is executed with its own validate::Opts and SubCommand::execute passes only those subcommand opts into validate::validate. As a result, VECTOR_DATA_DIR=/tmp vector validate ... (and vector --data-dir /tmp validate ...) parses the value but never applies it before validate_config/create_tmp_directory, so the documented CI validation case still uses the config file's data_dir and can fail on the nonexistent path.
Useful? React with 👍 / 👎.
| if let Some(data_dir) = data_dir { | ||
| debug!( | ||
| message = "Overriding data_dir from command line.", | ||
| ?data_dir | ||
| ); | ||
| config.global.data_dir = Some(data_dir); |
There was a problem hiding this comment.
This override is applied only to the initial config returned by load_configs; reloads in handle_signal call config::load_from_paths_with_provider_and_secrets directly and TopologyController::reload rejects changes to global options. With --watch-config/SIGHUP and a CLI/env data_dir that differs from the file (or default), the reloaded config loses this assignment, differs from the running global options, and reload fails with GlobalOptionsChanged instead of continuing to use the requested deployment-level data directory.
Useful? React with 👍 / 👎.
| #[arg(long, env = "VECTOR_DATA_DIR")] | ||
| pub data_dir: Option<PathBuf>, |
There was a problem hiding this comment.
Because this is a top-level RootOpts argument without global = true, clap only accepts it before a subcommand, so the advertised vector validate --data-dir /tmp vector.yaml form is rejected as an unexpected argument before any override can happen. If this flag is meant to work with subcommands in the position shown in the PR description, it needs to be global or defined on those subcommands.
Useful? React with 👍 / 👎.
| if let Some(data_dir) = data_dir { | ||
| debug!( | ||
| message = "Overriding data_dir from command line.", | ||
| ?data_dir | ||
| ); | ||
| config.global.data_dir = Some(data_dir); |
There was a problem hiding this comment.
The override happens only after load_from_paths_with_provider_and_secrets has fully built the config, but GlobalOptions::merge rejects two config files that both set different non-default data_dir values during that build. In a split configuration where --data-dir/VECTOR_DATA_DIR is supplied specifically to take precedence, startup still fails with conflicting values for 'data_dir' found before this assignment runs, so the deployment-level override cannot actually supersede all config-file values.
Useful? React with 👍 / 👎.
pront
added
the
type: feature
pront
added
the
meta: awaiting author
github-actions
Bot
added
docs review on hold
Preserve data-directory overrides and upstream config-loading changes. Keep both load_configs lint allowances, mark the consuming data_dir builder as must_use, and extract the reload test's subprocess launcher for Clippy.
github-actions
Bot
removed
the
meta: awaiting author
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17fe374dfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Keep option and path as separate arguments. The service library owns | ||
| // Windows command-line quoting, including spaces and trailing backslashes. | ||
| args.push(OsString::from("--data-dir")); | ||
| args.push(data_dir.as_os_str().to_owned()); |
There was a problem hiding this comment.
On Windows, when vector service install is given a relative override, this stores it verbatim in the SCM launch arguments. The inspected service path in src/vector_windows.rs passes those arguments to SCM and run_service never restores the installer's working directory, so whenever the service starts in a different working directory, a value such as ./state resolves elsewhere and Vector can write checkpoints/buffers to the wrong location or fail to start. Resolve the override against the installer working directory before registering the service.
Useful? React with 👍 / 👎.
Resolve relative data directory overrides against the installer's working directory before loading configuration and persisting service arguments. Preserve native and absolute paths without requiring the directory to exist. Add isolated working-directory regression tests and document the service installation behavior.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c83ad03a13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| _core_config_options: { | ||
| "data-dir": { |
There was a problem hiding this comment.
The CLI page template renders each subcommand from that command's own options map (website/layouts/shortcodes/cli/commands.html:66-69), and the validate map at website/cue/reference/cli.cue:483-505 does not reference this new shared option. As a result, the generated vector validate section—the primary usage example in this change—omits --data-dir, while graph and test expose it; add "data-dir": _core_config_options["data-dir"] to validate's options as well.
Useful? React with 👍 / 👎.
This branch has not been deployed
2 participants
Summary
Closes #9766. Adds the global
--data-diroption andVECTOR_DATA_DIRenvironment variable to override the deployment's data directory without editing pipeline configuration.Precedence is CLI, then environment, then configuration/default. The option is accepted before or after a subcommand. Each configuration file receives the override before global options are merged, so split files with differing
data_dirvalues can use one deployment override.The override reaches run, validate, graph and config unit-test loading, persists through disk/component reload and provider-builder replacement, and is included in Windows service-install launch arguments. Windows arguments remain separate
OsStringvalues; service-control behavior is unchanged. With neither option nor environment variable set, existing configuration/default behavior is unchanged. Website CLI reference entries document precedence, usage and unset behavior.Usage
This change covers
data_dironly; other global settings are outside its scope.How did you test this PR?
On Rust 1.96.1, against master
423c00203ee834496d0b6bc1948589b120b45e9cplus this change:cli-tests,--lib --tests, warnings denied) andcargo fmt --all -- --checkpassedWindows compilation and three Windows-only argument-builder tests remain unverified. Windows service execution/SCM argument handling also needs Windows CI. OS signal delivery and remote provider transport are outside the reload-handler test's coverage.
Official documentation/project-format/changelog checks remain outstanding; pinned CUE 0.17.1 and Prettier 3.8.1 were not run. The contributor reviewed the changes; the reported Rust checks were executed with agent assistance.
Does this PR include user facing changes?
Yes. A feature changelog fragment and CLI reference documentation are included.
These updates were prepared with OpenAI assistance. The exact model identifier was not available in this session. The PR author reviewed and approved the changes; remaining validation is listed above.