Skip to content

feat: new Fastly plugin #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: new Fastly plugin #67

wants to merge 8 commits into from

Conversation

spa-raj
Copy link

@spa-raj spa-raj commented Aug 7, 2025

What does this PR change?

  • This pull request introduces the initial implementation of the Fastly plugin for OpenCost, including configuration, invoice parsing, test harness integration, and basic validation. The most important changes are grouped below:

Core Plugin Implementation:

  • Added the initial FastlyConfig struct in fastlyconfig.go to hold plugin configuration such as API key and log level.
  • Implemented invoice data structures and custom JSON unmarshaling logic in invoices.go, including robust handling for the monthly_transaction_amount field which may arrive as a number or string from the API.

Validation and Testing:

  • Added a standalone validator CLI (main.go) that reads daily and hourly protobuf files, validates the responses for expected structure and values, and checks for errors or anomalies in cost data.
  • Introduced comprehensive Go-based integration and unit tests in fastly_test.go to verify correct plugin behavior across daily, hourly, future, and month-to-date cost retrieval scenarios.

Supporting Files and Dependencies:

  • Added sample daily and hourly test JSON files (fastly_daily_test.json, fastly_hourly_test.json) for use in validation and testing. [1] [2]
  • Created a new Go module and added all required dependencies for the Fastly plugin in go.mod.

Does this PR relate to any other PRs?

  • No

How will this PR impact users?

  • This PR adds a new Fastly cost integration plugin that enables OpenCost users to:
  • Monitor Fastly CDN costs alongside other cloud expenses in their OpenCost dashboard
  • View detailed cost breakdowns by service type (bandwidth, requests, compute, support, etc.)
  • Track both historical and real-time costs with support for hourly and daily granularity
  • Benefit from improved performance with invoice caching and optimized API rate limiting (5400 req/min)
  • Get accurate cost attribution with unique UUIDs for each cost item and proper proration when invoice periods don't align with requested time ranges
  • Experience better reliability with comprehensive error handling, request validation, and future date handling

Does this PR address any GitHub or Zendesk issues?

How was this PR tested?

  • Comprehensive multi-layered testing approach:
  1. Unit Tests (25 tests) - All individual functions tested with mock HTTP responses:
    • Request validation tests (6 sub-tests covering edge cases)
    • Configuration loading and error handling tests
    • All three Fastly invoice API endpoints tested:
      • GET /billing/v3/invoices (list invoices + pagination)
      • GET /billing/v3/invoices/month-to-date (current month data)
      • GET /billing/v3/invoices/{invoice_id} (specific invoice) ✅ New coverage added
    • UUID generation validation tests
    • Caching behavior tests
    • Future date handling tests
    • Complete error scenario coverage
  2. Integration/Harness Tests (4 tests) - End-to-end plugin testing with real Fastly API:
    • Real API calls using live Fastly API keys
    • Tests monthly, daily, and hourly data retrieval
    • Validates complete workflow: config → API → response parsing → cost attribution
    • Future window handling validation
  3. Validator Tests - Plugin response validation:
    • Tested with both daily and hourly mock data files
    • Validates cost response structure and business logic

All tests pass: ✅ 25/25 unit tests + 4/4 harness tests + validator tests

Does this PR require changes to documentation?

  • Yes, this PR requires documentation updates:
    • Installation guide - How to install and configure the Fastly plugin with OpenCost
    • Configuration documentation - Required API key setup and environment variables
    • API permissions - Document that users need a Fastly API token with billing read permissions
    • Usage examples - Sample configurations and expected output formats
    • Limitations - Document that minimum resolution is 1 hour and data is available for the last 12 months
    • Plugin registry - Add Fastly to the list of supported OpenCost plugins

spa-raj added 7 commits August 7, 2025 16:14
@spa-raj
Initial commit for the fastly plugin
f0dfec8 
Signed-off-by: Sparsh <[email protected]>
@spa-raj
Add Fastly cost integration plugin for OpenCost
9f9d237 
This commit implements a Fastly plugin for OpenCost that integrates with the Fastly billing API to retrieve and display cloud costs. The plugin:

1. Fetches invoice data from Fastly's API (both historical and month-to-date)
2. Converts Fastly billing items to OpenCost's CustomCost model
3. Implements proper rate limiting to respect Fastly API constraints
4. Caches invoice data to minimize redundant API calls
5. Prorates costs across time windows when invoice periods don't align with requested time ranges

The plugin allows OpenCost users to monitor and analyze their Fastly CDN costs alongside other cloud expenses.

Signed-off-by: Sparsh <[email protected]>
@spa-raj
fixed the api request header and updated test cases
bd2dd7a 
Signed-off-by: Sparsh <[email protected]>
@spa-raj
made harness tests work
7f3c43c 
Signed-off-by: Sparsh <[email protected]>
@spa-raj
Enhance Fastly plugin with UUID generation and improved rate limiting
8d9b2f2 
  This commit implements several key improvements to the Fastly OpenCost plugin:

  Core Enhancements:
  - Replace shared invoice IDs with unique UUIDs for each cost item using github.com/google/uuid
  - Update rate limiting from 600 to 5400 requests per minute to better utilize Fastly's 6000 req/min limit
  - Add comprehensive request validation with proper error handling for edge cases

  Performance & Reliability:
  - Implement invoice caching to reduce redundant API calls
  - Batch invoice fetching for entire periods instead of per-window requests
  - Add proper future date handling (return empty response instead of errors)
  - Enhanced error logging with more descriptive messages

  Testing Improvements:
  - Add comprehensive test coverage with TestUniqueUUIDGeneration
  - Replace transport mocking with HTTPClient interface for better testability
  - Add request validation tests covering all edge cases
  - Include test utilities and mock data generation tools

  API Integration:
  - Remove redundant Host headers from API requests
  - Improve charge categorization (commitment, credit, support, tax, usage)
  - Enhanced usage unit mapping for better cost attribution
  - Add support for credits/discounts with proper labeling

Signed-off-by: Sparsh <[email protected]>
@spa-raj
Added missing tests for get invoice by ID
bcc8eb1 
Signed-off-by: Sparsh <[email protected]>
@spa-raj
Added fastly invoices api data model mapping with FOCUS and added ext…
6ecb185 
…ended attributes for the same

Signed-off-by: Sparsh <[email protected]>
@spa-raj
Copy link
Author

spa-raj commented Sep 18, 2025

@ameijer Can you approve this backend workflow? It's been a while.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
needs-follow-up
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fastly Plugin
1 participant
@spa-raj