Skip to content

Add date-based partitioning to parquet file paths#6

Merged
xDarksome merged 6 commits into
masterfrom
copilot/update-key-suffix-format
Feb 3, 2026
Merged

Add date-based partitioning to parquet file paths#6
xDarksome merged 6 commits into
masterfrom
copilot/update-key-suffix-format

Conversation

Copilot AI commented Feb 3, 2026

Copy link
Copy Markdown

Adds YYYY-MM-DD date prefix to parquet file key structure for date-based partitioning.

Changes

  • pkg/parquet/writer.go: Modified createBackendWriter() to insert date between prefix and metric name using strings.LastIndex() to split the path
  • pkg/parquet/writer_test.go: Updated test expectations for new path structure
  • README.md: Updated directory structure example

Path Structure

// Before
self.currentFile = fmt.Sprintf("%s/%s.parquet", self.prefix, basename)
// Example: prefix/kube_node_stuff/20240307101250.parquet

// After  
// Splits prefix at last "/" and inserts date between prefix and metric name
lastSlash := strings.LastIndex(self.prefix, "/")
// Example: prefix/2024-03-07/kube_node_stuff/20240307101250.parquet

The implementation correctly places the prefix before the date, inserting the date between the prefix portion and the metric name. This enables date-based S3 lifecycle policies and simplifies retention management while maintaining the prefix structure.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits February 3, 2026 15:42
Co-authored-by: xDarksome <22872975+xDarksome@users.noreply.github.com>
…stamp format

Co-authored-by: xDarksome <22872975+xDarksome@users.noreply.github.com>
Copilot AI changed the title [WIP] Update key suffix format for parquet files Add date-based partitioning to parquet file paths Feb 3, 2026
Copilot AI requested a review from xDarksome February 3, 2026 15:46
@xDarksome xDarksome self-requested a review February 3, 2026 15:55
Comment thread README.md Outdated

```
/data/<prefix>/<metric name>/2024022021.parquet
/data/<YYYY-MM-DD>/<prefix>/<metric name>/20240220211430.parquet

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefix should be before the date

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7ac4619. The path structure now correctly places the prefix before the date: /data/<prefix>/<YYYY-MM-DD>/<metric name>/timestamp.parquet

Copilot AI and others added 2 commits February 3, 2026 16:00
Co-authored-by: xDarksome <22872975+xDarksome@users.noreply.github.com>
Co-authored-by: xDarksome <22872975+xDarksome@users.noreply.github.com>
@xDarksome xDarksome marked this pull request as ready for review February 3, 2026 16:17
@xDarksome xDarksome merged commit 0cd5c42 into master Feb 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants