Renamed the audit_log_api source to logging_log_entry and implement the Retry mechanism

[ParthaI]

Example query results

Results

Add example SQL query results here (please include the input queries as well)

[ParthaI]

Summary of Changes

This PR migrates the GCP logging log entry source from the legacy cloud.google.com/go/logging/logadmin client to the modern cloud.google.com/go/logging/apiv2 (GAPIC) client and implements a configurable retry mechanism with exponential backoff.

Key Changes:

1. Client Migration

• Replaced cloud.google.com/go/logging/logadmin with cloud.google.com/go/logging/apiv2 (GAPIC client)
• Updated API calls from client.Entries() to client.ListLogEntries()
• Modified data mapping to handle protobuf structures (loggingpb.LogEntry, audit.AuditLog)
• Updated field access patterns to use protobuf getter methods

2. Retry Mechanism Implementation

• Added configurable retry parameters at the connection level (config/gcp_connection.go):
  • min_retry_delay (Initial delay, default: 500ms)
  • max_retry_delay (Maximum delay, default: 60s)
  • backoff_multiplier (Exponential growth factor, default: 1.30)
• Implemented exponential backoff with jitter using gax.WithRetry and gax.Backoff
• Added retry for specific error codes: Unavailable, DeadlineExceeded, ResourceExhausted, Aborted

3. Data Mapping Updates

• Updated tables/audit_log/audit_log_mapper.go to handle both GAPIC and logadmin client data formats
• Fixed protobuf payload extraction using proto.Unmarshal for anypb.Any fields
• Corrected HTTP request field mapping for protobuf structures
• Maintained backward compatibility for existing log data

4. Configuration Improvements

• Aligned parameter naming with GCP SDK defaults
• Added validation for retry parameters

5. Documentation Updates

• Updated source documentation to reflect new retry configuration
• Updated example configurations to show connection-level retry settings

Benefits:

• Improved Reliability: Robust retry mechanism handles transient network and service errors
• Better Performance: GAPIC client provides improved performance and features
• Enhanced Maintainability: Modern client with better long-term support
• Configurable Resilience: Users can customize retry behavior based on their needs

Breaking Changes:

• The source name is only the breaking changes.

[Copilot]

Pull Request Overview

This PR renames the audit_log_api source to logging_log_entry and implements comprehensive retry mechanisms for improved reliability. The change modernizes the GCP logging integration by migrating from the logadmin client to the GAPIC (Google API Go Client) logging client with configurable retry capabilities.

Key changes include:

• Complete replacement of the audit_log_api source with a new logging_log_entry source featuring configurable retry mechanisms
• Migration from cloud.google.com/go/logging to cloud.google.com/go/logging/apiv2/loggingpb for better API support
• Introduction of a flexible log type filter mapping system for table-specific log filtering

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tables/audit_log/audit_log_table.go	Updates source reference and adds table-specific options for log type filtering
tables/audit_log/audit_log_mapper.go	Migrates from logadmin to GAPIC client types with protobuf message handling
sources/logging_log_entry/logging_log_type_filter_map.go	New file defining table-to-log-type mappings for flexible filtering
sources/logging_log_entry/logging_log_entry_source_config.go	New configuration structure with table-specific validation
sources/logging_log_entry/logging_log_entry_source.go	New source implementation with GAPIC client and retry mechanisms
config/gcp_connection.go	Adds configurable retry parameters for GAPIC clients
Various documentation files	Updates documentation to reflect the source rename and new capabilities

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[recharge-jmog]

@ParthaI I rebased my local PR branch with this PR and did a little testing. Here's what I've seen:

• I was able to pull over 2M logs from the Cloud Logging API with no rate errors, so that seems to be working well
• I received this error on 2 rows: 2 rows with error: error decoding service data: unsupported type: type.googleapis.com/google.cloud.bigquery.logging.v1.AuditData indicating there may be some addition needed to get Big Query audit logs supported

So overall, looking pretty good. I presume the use pattern will be just setting up different partitions for the other log types (requests, etc) and collecting them separately?

[ParthaI]

Hello @recharge-jmog,

Thank you for testing the changes and sharing your feedback — it's much appreciated!

• I’ve addressed the error decoding service data issue, so this error should no longer occur going forward.

"So overall, looking pretty good. I presume the use pattern will be just setting up different partitions for the other log types (requests, etc) and collecting them separately?"

Yes, exactly. Each log type (e.g., request logs, etc.) will be handled via its own table and partition. In the original PR, I had reverted the changes related to the request_logs table, but you can refer to this commit as an example of how it was structured earlier.

Thanks again!

[recharge-jmog]

Yes, exactly. Each log type (e.g., request logs, etc.) will be handled via its own table and partition. In the original PR, I had reverted the changes related to the request_logs table, but you can refer to this commit as an example of how it was structured earlier.

If I understand correctly then, once this PR is merged I should open a new PR (or rebase and revise my previous PR) to implement the requests table separately?

[ParthaI]

Yes, correct @recharge-jmog!