Research into memory consumption of hashicorp/terraform-provider-archive, specifically the archive_file data source.
archive_file reads each source file entirely into memory via os.ReadFile() before writing it to the zip archive. When Terraform evaluates multiple archive_file data sources concurrently (default parallelism=10), all allocations are live simultaneously in the single provider process. For large payloads (e.g. AWS Lambda deployment packages), this causes OOM kills in CI environments with constrained memory.
The TarArchiver in the same codebase already uses streaming I/O (os.Open + io.Copy). The zip archiver does not.
Go benchmark (direct ZipArchiver calls, 10 concurrent × 50 MB archives):
| Implementation | HeapΔ | MaxRSS |
|---|---|---|
| Buffered (upstream) | 508 MB | 518 MB |
| Streaming (proposed) | 8 MB | 18 MB |
Memory scales linearly with concurrency. GC cannot help — all []byte slices are simultaneously live.
Real Terraform provider (terraform plan -parallelism=10, 10 × 50 MB archives):
| Implementation | Peak RSS |
|---|---|
| Buffered (upstream) | 1034 MB |
| Streaming (input-side fix) | 533 MB |
The streaming fix eliminates input-side buffering. Remaining RSS is from genFileChecksums, which reads the output zip back via os.ReadFile — a separate optimization.
├── archive-bench/ Go benchmark: measures ZipArchiver memory in isolation
│ ├── zip_archiver.go Copied upstream code + streaming alternative
│ ├── main.go Benchmark harness (single-file, directory, concurrent)
│ ├── run_benchmarks.sh Full matrix runner (both implementations)
│ └── analyze.py Aggregates JSONL results into comparison tables
│
└── tf-validation/ Terraform integration test: validates benchmark vs real provider
├── build_providers.sh Builds upstream + patched provider binaries
├── setup.sh Generates test data + terraform config
├── monitor.sh Runs terraform plan + samples provider RSS
└── run_validation.sh Full matrix (both impls, parallelism 1-10)
cd archive-bench
go build -o archive-bench .
# Quick comparison
./archive-bench -mode=concurrent -size=50 -concurrency=10 -runs=5
./archive-bench -mode=concurrent -size=50 -concurrency=10 -runs=5 -streaming
# Full matrix
./run_benchmarks.sh
python3 analyze.py results/*/*.jsonlcd tf-validation
./build_providers.sh # builds ~/bin/terraform-provider-archive-{buffered,streaming}
./setup.sh 10 50 # 10 archives × 50 MB
./run_validation.sh # full matrixBenchmark code and scripts: MIT (see LICENSE).
zip_archiver.go contains code copied from hashicorp/terraform-provider-archive, licensed under MPL-2.0 (see NOTICE).