feature(connectors): add GBIF occurrences connector - #56
Merged
kellykohlleffel merged 4 commits intoAug 17, 2026
Conversation
Sync species occurrence records from the public GBIF occurrence search API. Offset cursor over a bounded query with the deep-paging cap handled explicitly; the reserved taxonomic fields class and order are renamed to taxon_class and taxon_order. No credentials required.
Contributor
There was a problem hiding this comment.
🟡 Not ready to approve
The gbif README is not fully compliant with the repo’s README requirements, and connector.py needs small consistency fixes for checkpoint/state handling and standard end-of-file comments.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds a new gbif Connector SDK example that syncs GBIF occurrence records into a single occurrence table using offset-based pagination with a deep-paging cap safeguard and resumable state.
Changes:
- Added
gbif/connector.pyimplementing schema, configuration validation, retry/backoff, offset cursoring, and checkpointing. - Added
gbif/configuration.jsonandgbif/README.mddocumenting setup, configuration, pagination, and delivered schema. - Listed the new
gbifconnector in the repository rootREADME.md.
File summaries
| File | Description |
|---|---|
| README.md | Adds the gbif connector to the top-level connector index. |
| gbif/README.md | Documents the connector’s purpose, configuration, pagination approach, and table schema. |
| gbif/connector.py | Implements the GBIF occurrence sync logic, schema, and operational behaviors (retry, pagination cap, state). |
| gbif/configuration.json | Provides placeholder configuration keys for paging and optional query filters. |
Review details
Comments suppressed due to low confidence (1)
gbif/connector.py:418
- Same as above: update the existing
statedict before checkpointing to avoid discarding any other state keys (repo examples generally usestate[...] = ...thenop.checkpoint(state)).
op.checkpoint(state={"offset": offset})
- Files reviewed: 4/4 changed files
- Comments generated: 4
- Review effort level: Low
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (3)
gbif/connector.py:312
- Same issue as in
validate_configuration():taxon_key = configuration.get(...).strip()andcountry = ...strip()will raiseAttributeErrorif a user supplies a numeric JSON value ornull. Coerce to string (or empty string) before stripping.
page_size = int(configuration.get("page_size", __DEFAULT_PAGE_SIZE))
max_records = int(configuration.get("max_records_per_sync", "0"))
taxon_key = configuration.get("taxon_key", "").strip()
country = configuration.get("country", "").strip()
gbif/connector.py:100
configuration.get("taxon_key", "").strip()and the similar country line assume these config values are always strings. If a user providestaxon_keyas a JSON number (e.g.212) ornull, this will raiseAttributeErrorand fail the sync. Coerce to string (or empty string) before calling.strip()so both string and numeric JSON values are accepted.
This issue also appears on line 308 of the same file.
taxon_key = configuration.get("taxon_key", "").strip()
if taxon_key and (not taxon_key.isdigit() or int(taxon_key) <= 0):
raise ValueError(
f"Invalid configuration value for taxon_key: {taxon_key}. "
"Must be a positive integer GBIF taxon key, for example 212 for birds."
)
country = configuration.get("country", "").strip()
if country and not re_two_letter(country):
gbif/README.md:95
- In the Tables created section, the table name is shown as
OCCURRENCE, but the connector schema defines the table name asoccurrence. Using the exact schema name here avoids confusion for users searching for the table in their destination.
`OCCURRENCE`
fivetran-sahilkhirwal
approved these changes
Aug 11, 2026
Contributor
|
@kellykohlleffel please resolve all you comments and request for a re-review as it is not allowing us to merge the PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this connector does
Syncs species occurrence records from the public GBIF occurrence search API into a single
occurrencetable keyed ongbif_id. GBIF (Global Biodiversity Information Facility) aggregates biodiversity data from thousands of institutions; each record is one observation or specimen of a species with its taxonomy, coordinates, and event date. The API is public and requires no credentials.Hazards found while profiling the live API (2026-07-29)
Profiling was done against the live endpoint before any code was written. What it found:
class(e.g. "Magnoliopsida") andorder(e.g. "Rosales") as top-level taxonomic fields, and a numerickeythat duplicatesgbifID. All three are reserved SQL keywords that fail an unquotedCREATE TABLEon most warehouses. Handled by renamingclassandordertotaxon_classandtaxon_orderat the source, and droppingkeyin favour of the stringgbif_idprimary key. Verified by the shared contract's static reserved-word check and a live DuckDBCREATE TABLEof the delivered schema.offset + limit <= 100000; live,offset=100001returns HTTP 400, and the unfiltered corpus reports 3,909,145,444 matches. A connector that paged blindly would either 400 mid-sync or silently deliver a truncated table. Handled by reading the totalcounton the first page, warning loudly when it exceeds the cap, stopping at the cap, and shrinking the final page's limit so no request is ever issued pastoffset + limit = 100000. The remedy for a larger set is to narrow the query withtaxon_keyorcountry. Both debug runs below show the cap warning firing live.max_records_per_syncis a true ceiling. Covered by a drain regression test asserting repeated bounded syncs cover every record exactly once.lastInterpretedis rewritten on reprocessing; a Jan-2024 window returned count 0), so a time cursor would skip or duplicate. The offset is the resume cursor for a bounded query. Hazard feature(connector_sdk): Initial community connectors commit #1 (inclusive-range-to-compound-cursor) does not apply: there is no inclusive time-range filter driving the cursor.limitat 300 regardless of what is requested;page_sizeis validated to 1-300.page_size,max_records_per_sync,taxon_key,country) is validated before any request; numeric checks reject zero where a positive integer is required; thecountryandtaxon_keyfilters are URL-encoded viaurllib.parse. There is no hostname config and no boolean flag, so those hazard classes do not apply.Debug evidence
Two live
fivetran debugruns. The second resumes from the first's checkpoint (offset 150 -> offset 300) rather than restarting:Checklist
op.upsert/op.checkpointcalled directly, noyield op.)validate_configuration()called first inupdate()configuration.jsonholds placeholder values only; no credentials in the diffconnector.py,configuration.json,README.md