diff --git a/docs/superpowers/specs/2026-10-07-oilgas-job-portal-adapters-design.md b/docs/superpowers/specs/2026-10-07-oilgas-job-portal-adapters-design.md new file mode 100644 index 000000000..67f18a868 --- /dev/null +++ b/docs/superpowers/specs/2026-10-07-oilgas-job-portal-adapters-design.md @@ -0,0 +1,137 @@ +# Oil & gas job portal adapter: EnergyJobline + +**Date:** 2026-10-07 +**Status:** design approved, scope narrowed after a Rigzone feasibility spike + +## The problem + +The user supplied ~40 oil & gas domains (agencies, job portals, majors, EPC/service +companies) to evaluate as new ingest sources. Recon across all of them, plus a pass over +the live `boards` table, narrowed things down fast: + +- 18 majors/EPC/service companies sit on a known ATS. 13 were already boarded + (Shell, BP, Baker Hughes, Eni, ExxonMobil, Fluor, Halliburton, KBR, McDermott, SLB, + TechnipFMC, Wood, Worley). 5 were added via `cmd/add-board` on 2026-10-06 (Subsea7, + Bechtel, ADNOC, QatarEnergy, TotalEnergies). Aramco is excluded — its site TCP-blocks + automated requests and the platform could not be confirmed. +- Of the 7 general/niche job portals (Rigzone, Oilcareers, EnergyJobline, + OilAndGasJobSearch, Bayt, GulfTalent, NaukriGulf, Laimoon — 8 names, 7 distinct + backends since Oilcareers shares Rigzone's infrastructure): + - **Bayt and GulfTalent already have adapters**, already registered, and already have + live `boards` rows. Nothing to build. + - **Laimoon's jobs backend is dead** — confirmed stub/kill-switch across the whole + `jobs.laimoon.com` subdomain, no link from the live homepage. Out of scope. + - **NaukriGulf and OilAndGasJobSearch are unreachable at the network level** from both + this environment and the prod host (tar-pit / connection refused). Out of scope. + - **Rigzone** looked buildable on first recon (AWS WAF challenge, same shape as + Bayt/GulfTalent's Cloudflare challenge) but a feasibility spike invalidated that — see + below. Moved to the deferred group. + - **EnergyJobline** is the one portal this design covers. + +## Rigzone feasibility spike (2026-10-07) — INVALIDATED for now + +Before committing to build Rigzone alongside EnergyJobline, its one open risk (would the +project's existing Chrome-fingerprint transport, `fingerprinthttp.go`, clear Rigzone's bot +defense the way it clears Bayt's and GulfTalent's?) was spiked directly against the live +site, using the actual transport code rather than a guess: + +- **Plain fingerprint transport (`newFingerprintHTTP`, the same one `bayt.go`/ + `gulftalent.go` use):** both `https://www.rigzone.com/` and `/sitemap.xml` returned a + flat `HTTP 403 Forbidden` (nginx-style, "a padding to disable MSIE and Chrome friendly + error page" body) with no `x-amzn-waf-action` header at all — i.e. not the solvable + "challenge" response seen from a plain `curl`, but a harder block that offers nothing to + pass. This matches the pattern this repo's own `firecrawltier.go` documents for + Bayt/GulfTalent/Wellfound: "the refusal comes before any challenge is offered" — a + datacenter-IP-level block, not a fingerprint check. +- **Proxied fingerprint transport (`newProxiedFingerprintHTTP`, over this repo's existing + `SOURCES_PROXY_URL`):** untestable — the proxy account returned `402 Payment Required`. + Determining whether a non-datacenter egress clears Rigzone's block would require + topping up that account first, a real spend decision. +- The further escalation this repo already has for exactly this shape of wall — the hosted + Firecrawl tier (`firecrawltier.go`) — also costs money per page and was not attempted. + +**Verdict: INVALIDATED within the current budget.** The free tier (fingerprint spoofing +alone) flatly fails, and the next tier that might work costs money to even test. Asked +directly, the user chose not to spend to find out — so Rigzone joins NaukriGulf and +OilAndGasJobSearch in the deferred group rather than being forced through. It can be +revisited later specifically because it is a 100%-on-topic oil & gas board (unlike Bayt/ +GulfTalent, whose broad general listings make the hosted tier's cost/relevance trade-off +much worse) — if the user later wants to fund a proxy top-up or a Firecrawl trial, this +spike's numbers are the starting point, not a dead end. + +## EnergyJobline + +Confirmed with a plain `curl` (Googlebot UA, no cookies, no JS) — no bot defense observed: + +- `https://www.energyjobline.com/sitemap.xml` is an open sitemap **index** of 4 + sub-sitemaps. +- A live posting, `https://www.energyjobline.com/job/controls-engineer-atlanta-31835232`, + server-renders three `` +} + +func TestEnergyjoblineFetchWalksIndexThenDetailAndMaps(t *testing.T) { + jobURL := "https://www.energyjobline.com/job/controls-engineer-atlanta-31835232" + detail := energyjoblineDetailHTML( + "Controls Engineer in Atlanta", "The AES Corporation", + "Field service controls engineering.", "2026-10-01", "Atlanta", "US") + + fake := (&routedHTTP{}). + route("sitemap.xml?page=1", energyjoblineURLSetXML( + jobURL, + "https://www.energyjobline.com/", // site root, not a job + "https://www.energyjobline.com/jobs", // listing root, not a job + "https://www.energyjobline.com/news-article/oil-and-gas/some-story", // unrelated + )). + route("sitemap.xml?page=2", energyjoblineURLSetXML()). + // Registered after the page-specific routes: routedHTTP matches by substring in + // append order, and "sitemap.xml" is itself a substring of "sitemap.xml?page=1" — + // this generic route must lose that race for the index URL to resolve correctly. + route("sitemap.xml", energyjoblineSitemapIndexXML( + "https://www.energyjobline.com/sitemap.xml?page=1", + "https://www.energyjobline.com/sitemap.xml?page=2", + )). + route("/job/controls-engineer-atlanta-31835232", detail) + + jobs, err := NewEnergyJobline(fake).Fetch(context.Background(), CompanyEntry{ + Company: "EnergyJobline", Provider: "energyjobline", + }) + if err != nil { + t.Fatalf("Fetch: %v", err) + } + if len(jobs) != 1 { + t.Fatalf("got %d jobs, want 1 (non-job sitemap entries must be excluded)", len(jobs)) + } + j := jobs[0] + if j.ExternalID != "31835232" { + t.Errorf("ExternalID = %q, want 31835232", j.ExternalID) + } + if j.Title != "Controls Engineer in Atlanta" { + t.Errorf("Title = %q", j.Title) + } + if j.Company != "The AES Corporation" { + t.Errorf("Company = %q, want the posting's own hiringOrganization", j.Company) + } + if j.URL != jobURL { + t.Errorf("URL = %q, want %q", j.URL, jobURL) + } + if j.Location != "Atlanta, US" { + t.Errorf("Location = %q", j.Location) + } + if j.PostedAt == nil || j.PostedAt.Format("2006-01-02") != "2026-10-01" { + t.Errorf("PostedAt = %v, want 2026-10-01", j.PostedAt) + } +} + +func TestEnergyjoblineBrokenSitemapIndexErrorsTheCrawl(t *testing.T) { + fake := (&routedHTTP{}).routeErr("sitemap.xml", errors.New("origin down")) + _, err := NewEnergyJobline(fake).Fetch(context.Background(), CompanyEntry{ + Company: "EnergyJobline", Provider: "energyjobline", + }) + if err == nil { + t.Fatal("Fetch: want error on a broken sitemap index, got nil") + } +} + +// The index itself can resolve fine while one of its sub-sitemaps fails; that failure +// propagates as a hard Fetch error too (no fullBoardListing is claimed here, so any shard +// failure means the enumeration is incomplete and the caller should not trust a partial +// result as "today's whole catalogue"). +func TestEnergyjoblineSubSitemapFailureFailsTheCrawl(t *testing.T) { + fake := (&routedHTTP{}). + route("sitemap.xml?page=1", energyjoblineURLSetXML("https://www.energyjobline.com/job/ok-1")). + route("sitemap.xml", energyjoblineSitemapIndexXML( + "https://www.energyjobline.com/sitemap.xml?page=1", + "https://www.energyjobline.com/sitemap.xml?page=2", + )). + routeErr("sitemap.xml?page=2", errors.New("origin down")) + + _, err := NewEnergyJobline(fake).Fetch(context.Background(), CompanyEntry{ + Company: "EnergyJobline", Provider: "energyjobline", + }) + if err == nil { + t.Fatal("Fetch: want error when one sub-sitemap fails, got nil") + } +} + +func TestEnergyjoblineCompanyComesFromThePosting(t *testing.T) { + jobURL := "https://www.energyjobline.com/job/welder-kent-27543774" + detail := energyjoblineDetailHTML("WELDER in Kent", "Energy Jobline ZR", + "desc", "2026-10-02", "Kent", "GB") + fake := (&routedHTTP{}).route("/job/welder-kent-27543774", detail) + + j, ok := energyjobline{http: fake}.detail(context.Background(), + CompanyEntry{Company: "EnergyJobline", Provider: "energyjobline"}, jobURL) + if !ok { + t.Fatal("detail returned ok=false") + } + // The site attributes agency-submitted postings to the agency's own brand, not a + // confidential end client — that is the real, honest value of hiringOrganization for + // this posting (verified against the live site's own human-visible company link, not + // just its ld+json), so it is stored as-is rather than filtered or guessed around. + if j.Company != "Energy Jobline ZR" { + t.Errorf("Company = %q, want the posting's own (agency) hiringOrganization", j.Company) + } +} + +func TestEnergyjoblineUnreadableDetailWhenHiringOrgEmpty(t *testing.T) { + jobURL := "https://www.energyjobline.com/job/no-employer-99999999" + detail := energyjoblineDetailHTML("Mystery Role", "", "desc", "2026-10-02", "Kent", "GB") + fake := (&routedHTTP{}).route("/job/no-employer-99999999", detail) + + j, ok := energyjobline{http: fake}.detail(context.Background(), + CompanyEntry{Company: "EnergyJobline", Provider: "energyjobline"}, jobURL) + if !ok { + t.Fatal("detail returned ok=false, want an unreadableDetail stub") + } + if !j.Unreadable { + t.Error("Unreadable = false, want true (empty hiringOrganization)") + } + if j.Company != "EnergyJobline" { + t.Errorf("Company = %q, want the configured fallback EnergyJobline", j.Company) + } +} + +func TestEnergyjoblineUnreadableDetailWhenNoJobPosting(t *testing.T) { + jobURL := "https://www.energyjobline.com/job/broken-page-11111111" + fake := (&routedHTTP{}).route("/job/broken-page-11111111", "no ld+json here") + + j, ok := energyjobline{http: fake}.detail(context.Background(), + CompanyEntry{Company: "EnergyJobline", Provider: "energyjobline"}, jobURL) + if !ok { + t.Fatal("detail returned ok=false, want an unreadableDetail stub") + } + if !j.Unreadable { + t.Error("Unreadable = false, want true (no JobPosting block)") + } +} + +// A posting can carry a JobPosting block with no jobLocation entries at all (the field is +// an array in the markup, but nothing guarantees the site always populates it) — this must +// not panic on an empty-slice index and should simply leave Location blank. +func TestEnergyjoblineDetailWithNoJobLocation(t *testing.T) { + jobURL := "https://www.energyjobline.com/job/no-location-66666666" + html := `` + fake := (&routedHTTP{}).route("/job/no-location-66666666", html) + + j, ok := energyjobline{http: fake}.detail(context.Background(), + CompanyEntry{Company: "EnergyJobline", Provider: "energyjobline"}, jobURL) + if !ok { + t.Fatal("detail returned ok=false") + } + if j.Location != "" { + t.Errorf("Location = %q, want empty (no jobLocation entries)", j.Location) + } + if j.Company != "Real Co" { + t.Errorf("Company = %q, want Real Co", j.Company) + } +} + +func TestEnergyjoblineJobID(t *testing.T) { + cases := map[string]string{ + "https://www.energyjobline.com/job/controls-engineer-atlanta-31835232": "31835232", + "https://www.energyjobline.com/job/welder-kent-27543774/": "27543774", + "https://www.energyjobline.com/": "", + "https://www.energyjobline.com/jobs": "", + "https://www.energyjobline.com/news-article/oil-and-gas/story": "", + "https://www.energyjobline.com/company/energy-jobline-o-p": "", + } + for u, want := range cases { + if got := energyjoblineJobID(u); got != want { + t.Errorf("energyjoblineJobID(%q) = %q, want %q", u, got, want) + } + } +} + +func TestEnergyjoblineOneBadDetailDoesNotFailTheCrawl(t *testing.T) { + goodURL := "https://www.energyjobline.com/job/good-job-22222222" + badURL := "https://www.energyjobline.com/job/bad-job-33333333" + good := energyjoblineDetailHTML("Good Job", "Real Co", "d", "2026-10-02", "Kent", "GB") + + fake := (&routedHTTP{}). + route("sitemap.xml?page=1", energyjoblineURLSetXML(goodURL, badURL)). + route("sitemap.xml", energyjoblineSitemapIndexXML("https://www.energyjobline.com/sitemap.xml?page=1")). + route("/job/good-job-22222222", good). + routeErr("/job/bad-job-33333333", errors.New("fetch failed")) + + jobs, err := NewEnergyJobline(fake).Fetch(context.Background(), CompanyEntry{ + Company: "EnergyJobline", Provider: "energyjobline", + }) + if err != nil { + t.Fatalf("Fetch: %v", err) + } + // A plain transport error states nothing about the posting (it might be a timeout, a + // transient 5xx — anything short of a confirmed-gone answer), so it must not be + // mistaken for the posting having been removed: it survives as an Unreadable stub, the + // same contract fetchDetails documents and bayt.go/gulftalent.go already honour. + if len(jobs) != 2 { + t.Fatalf("got %d jobs, want 2 (the good posting plus an Unreadable stub for the one that failed to fetch)", len(jobs)) + } + markers := unreadableMarkers(jobs) + if len(markers) != 1 || markers[0].ExternalID != "33333333" { + t.Fatalf("unreadable markers = %v, want one for the posting whose detail fetch errored", markers) + } + if markers[0].Company != "EnergyJobline" { + t.Errorf("marker Company = %q, want the entry's configured company", markers[0].Company) + } + for _, j := range jobs { + if j.ExternalID == "22222222" && j.Title != "Good Job" { + t.Errorf("good posting Title = %q, want Good Job", j.Title) + } + } +} + +// The other half of the distinction (see bayt_test.go's TestBaytGoneDetailDropsThePosting): +// a 404/410 is the platform's own answer that the posting is gone, so the crawl drops it +// rather than marking it unreadable. +func TestEnergyjoblineGoneDetailDropsThePosting(t *testing.T) { + goodURL := "https://www.energyjobline.com/job/good-job-44444444" + goneURL := "https://www.energyjobline.com/job/gone-job-55555555" + good := energyjoblineDetailHTML("Good Job", "Real Co", "d", "2026-10-02", "Kent", "GB") + + fake := (&routedHTTP{}). + route("sitemap.xml?page=1", energyjoblineURLSetXML(goodURL, goneURL)). + route("sitemap.xml", energyjoblineSitemapIndexXML("https://www.energyjobline.com/sitemap.xml?page=1")). + route("/job/good-job-44444444", good). + routeErr("/job/gone-job-55555555", &StatusError{Method: "GET", Code: 404, URL: goneURL}) + + jobs, err := NewEnergyJobline(fake).Fetch(context.Background(), CompanyEntry{ + Company: "EnergyJobline", Provider: "energyjobline", + }) + if err != nil { + t.Fatalf("Fetch: %v", err) + } + if len(jobs) != 1 || jobs[0].ExternalID != "44444444" { + t.Fatalf("got %v, want only the posting whose detail answered — the 404'd one dropped, not marked", jobs) + } +} + +func TestEnergyjoblineProviderAndBoardless(t *testing.T) { + s := NewEnergyJobline(nil) + if s.Provider() != "energyjobline" { + t.Errorf("Provider() = %q, want energyjobline", s.Provider()) + } + if _, ok := s.(boardless); !ok { + t.Error("energyjobline must implement boardless") + } + if _, ok := s.(aggregator); !ok { + t.Error("energyjobline must implement aggregator") + } +} diff --git a/internal/ingest/sources/registry.go b/internal/ingest/sources/registry.go index 019dff1ba..4e24bdb15 100644 --- a/internal/ingest/sources/registry.go +++ b/internal/ingest/sources/registry.go @@ -354,6 +354,7 @@ func All(c HTTPClient) map[string]Source { // listing and detail pages are fully static server-rendered HTML. NewStaffy(c), NewDataArt(c), + NewEnergyJobline(c), // JettyCloud: its postings exist only on its own careers site — the employer runs no // ATS a crawl can reach, so this adapter is the only way the catalogue can carry it. NewJettyCloud(c), diff --git a/openspec/changes/energyjobline-ingest-adapter/.openspec.yaml b/openspec/changes/energyjobline-ingest-adapter/.openspec.yaml new file mode 100644 index 000000000..a26662f7a --- /dev/null +++ b/openspec/changes/energyjobline-ingest-adapter/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-07 diff --git a/openspec/changes/energyjobline-ingest-adapter/design.md b/openspec/changes/energyjobline-ingest-adapter/design.md new file mode 100644 index 000000000..349e33328 --- /dev/null +++ b/openspec/changes/energyjobline-ingest-adapter/design.md @@ -0,0 +1,100 @@ +## Context + +Full recon and the Rigzone feasibility spike live in +`docs/superpowers/specs/2026-10-07-oilgas-job-portal-adapters-design.md`; this design +covers only the part that ships: the `energyjobline` adapter. + +`https://www.energyjobline.com/sitemap.xml` is an open sitemap **index** of 4 +sub-sitemaps (confirmed via plain `curl` with a Googlebot UA — no bot defense observed). A +live posting, `https://www.energyjobline.com/job/controls-engineer-atlanta-31835232`, +server-renders three `