Skip to content

Conversation

overcat
Copy link
Contributor

@overcat overcat commented Sep 3, 2025

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

Support HTTP as a read-only data source

Why

See #5808

Known limitations

N/A

@overcat overcat marked this pull request as ready for review September 3, 2025 03:14
@Copilot Copilot AI review requested due to automatic review settings September 3, 2025 03:14
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds HTTP as a read-only data source to the datastore package, allowing access to files hosted on HTTP(S) endpoints. The implementation supports authentication via custom headers and provides all standard read operations while explicitly rejecting write operations.

  • Adds HTTPDataStore implementation with configurable base URL, timeout, and custom headers
  • Implements all read operations (GetFile, Exists, Size, GetFileMetadata, etc.) using HTTP HEAD/GET requests
  • Provides comprehensive test coverage with mock HTTP server for various scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
support/datastore/http.go Core HTTP datastore implementation with read-only operations
support/datastore/http_test.go Comprehensive test suite with mock HTTP server
support/datastore/datastore.go Integration point adding HTTP type to datastore factory

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment on lines +61 to +62
client := &http.Client{
Timeout: timeout,
Copy link
Preview

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HTTP client lacks important security configurations. Consider adding MaxIdleConns, MaxIdleConnsPerHost, and IdleConnTimeout to prevent connection pool exhaustion, and disable automatic redirect following for better security control.

Suggested change
client := &http.Client{
Timeout: timeout,
transport := &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
}
client := &http.Client{
Timeout: timeout,
Transport: transport,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},

Copilot uses AI. Check for mistakes.


resp, err := h.client.Do(req)
if err != nil {
log.Debugf("Error retrieving file '%s': %v", filePath, err)
Copy link
Preview

Copilot AI Sep 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging the full error message may expose sensitive information like internal URLs or authentication details. Consider sanitizing the error message before logging or reducing the log level.

Suggested change
log.Debugf("Error retrieving file '%s': %v", filePath, err)
log.Debugf("Error retrieving file '%s'", filePath)

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant