Conversation
Update Postman Documentation
Add CITATION file
Handle time zones and expose format_entsoe_datetime to user as helper function
Update Postman Documentation
…elements within a time_series entry Until now I did not discover nested elements within a time_series entry but the ProductionAndGenerationUnits endpoint returns those. This change may increase the number of rows and columns for existing queries but its absolutely necessary for not silently dropping data. Endpoints like Load and EnergyPrices are not affected as they do not have nested structures in the `time_series` objects.
Agent-Logs-Url: https://github.com/BerriJ/entsoe-apy/sessions/ba7dfbda-8cd3-474b-be4c-a2c1261f52e2 Co-authored-by: BerriJ <37799358+BerriJ@users.noreply.github.com>
Fix normalize_to_records function
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prepares the 1.1.0 release by improving record normalization (correct nested-list expansion), adding a timezone-aware ENTSO-E datetime formatter utility, and expanding Balancing endpoint coverage (including archived “Changes to Bid Availability” support). It also updates endpoint metadata/docs and adds citation metadata automation.
Changes:
- Refactors
normalize_to_recordsto cross-join/expand nested lists correctly and adds comprehensive unit tests. - Adds
format_entsoe_datetimewith UTC normalization for tz-aware datetimes and exports/documents it. - Introduces a new Balancing parameter class + endpoint JSON for “Changes to Bid Availability Archives”, plus metadata/docs updates.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_normalize_to_records.py | Adds regression and feature tests for nested list expansion and cross-joins. |
| src/entsoe/utils/utils.py | Updates format_entsoe_datetime to handle tz-aware datetimes (convert to UTC). |
| src/entsoe/utils/records.py | Refactors normalization logic to collect expansions and perform cross-joins. |
| src/entsoe/utils/init.py | Exposes format_entsoe_datetime via entsoe.utils. |
| src/entsoe/Balancing/specific_params.py | Adds ChangesToBidAvailabilityArchives parameter class. |
| src/entsoe/Balancing/init.py | Exports ChangesToBidAvailabilityArchives. |
| misc/endpoints/README.md | Updates endpoint listing (adds archives endpoint, reorders one Load entry). |
| misc/endpoints/Market/11.1.B Flow Based Allocations.json | Adjusts example query parameters/values. |
| misc/endpoints/Market/11.1 Continuous Allocations - Offered Transfer Capacity.json | Removes an unused/disabled offset entry from the example. |
| misc/endpoints/Balancing/IFs mFRR 9.9, aFRR 9.6&9.8 Changes to Bid Availability Archives.json | Adds new endpoint definition JSON for the archives variant. |
| misc/endpoints/all_endpoints.json | Adds the new Balancing archives endpoint name and reorders one Load entry. |
| docs/utilities.md | Documents the new format_entsoe_datetime utility. |
| docs/examples.md | Adds a datetime/timezone usage example for API calls. |
| CITATION.cff | Adds citation metadata file. |
| .github/workflows/citation-sync.yml | Adds workflow intended to sync/update CITATION on tag pushes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and new features across the codebase, focusing on enhancing data normalization, time zone handling, API endpoint coverage, and project metadata. The most notable changes are the robust overhaul of the
normalize_to_recordsutility to correctly expand nested lists into multiple records, the addition of a utility for formatting ENTSO-E datetimes with proper timezone handling, and the implementation of a new API endpoint for balancing bid availability archives. Documentation and metadata have also been updated to reflect these enhancements.Data normalization and testing improvements:
normalize_to_recordsinentsoe.utils.recordsto correctly handle nested lists, ensuring that all nested data is expanded into multiple records rather than dropping data. Added comprehensive unit tests intests/test_normalize_to_records.pyto cover flat dicts, nested dicts, lists, cross-joins, and field ignoring. [1] [2] [3]Datetime and timezone utilities:
format_entsoe_datetimetoentsoe.utils, which converts both timezone-aware and naive datetimes (including pandas Timestamps) to the ENTSO-E API integer format, assuming UTC for naive datetimes. Updated documentation and utility exports accordingly. [1] [2] [3] [4] [5]API endpoint and parameter coverage:
ChangesToBidAvailabilityArchivesinentsoe.Balancing, updates to__init__.pyexports, and a corresponding endpoint JSON definition. Updated endpoint listings inall_endpoints.jsonandREADME.md. (misc/endpoints/Balancing/IFs mFRR 9.9, aFRR 9.6&9.8 Changes to Bid Availability Archives.jsonR1-R48, [1] [2] [3] [4] [5]Documentation and metadata updates:
docs/utilities.mdanddocs/examples.md. Added aCITATION.cfffile for software citation metadata and a GitHub workflow to automatically update it on new tags. [1] [2] [3]Endpoint corrections and maintenance: