Repository navigation
feat(sources): add Cochrane Library browser source (authenticated session) #27
Description
Summary
Cochrane Library is the gold-standard database for systematic reviews and meta-analyses (Cochrane Database of Systematic Reviews, CDSR). Several colleagues and research groups rely on it as a primary source and would benefit from being able to search it directly via MOSAIC.
Why a dedicated source is needed
Cochrane reviews are indexed by PubMed and Europe PMC (both already MOSAIC sources), but:
- Coverage lag: newly published reviews can take days/weeks to appear in PubMed.
- Cochrane-specific filters (review group, protocol stage, CD number) are not available via PubMed/EPMC.
- Direct access provides richer metadata: review stage, protocol status, Cochrane group.
- Users who work primarily with CDSR prefer a dedicated, clearly labelled source in MOSAIC output.
Why a public API is not an option
Cochrane exposes two private REST APIs (archie.cochrane.org and api.cochrane.org), both protected by OAuth 2.0 credentials obtainable only by contacting Cochrane directly — no self-service signup exists. Scraping cochranelibrary.com is blocked at the edge (HTTP 403 for all automated clients) and prohibited by Wiley's Terms of Use.
Proposed approach: authenticated browser session
Follow the existing pattern used by ScienceDirectBrowserSource (mosaic/sources/sciencedirect_browser.py): a Playwright-based source that reuses a saved login session, activated via mosaic auth login.
Login flow
The Cochrane Library uses a Wiley / institutional SSO login at:
https://www.cochranelibrary.com/
Users log in once through the browser (institutional login, personal Wiley account, or NHS/Athens):
mosaic auth login cochrane --url https://www.cochranelibrary.comThis opens a headed browser window, lets the user log in interactively, then saves the Playwright storage state to ~/.config/mosaic/sessions/cochrane.json — identical to the ScienceDirect flow.
Search flow
- Load saved session into a headless Playwright context.
- Navigate to
https://www.cochranelibrary.com/search. - Fill the search input and submit the form.
- Wait for result cards (
li.search-results-itemor equivalent selector — needs verification against live DOM). - Extract per-result: title, authors, DOI (derivable from CD number via
10.1002/14651858.CDxxxxxx), year, review type (systematic review / protocol), open-access status, URL. - Detect expired session by checking for redirect to
login.cochrane.orgoridp.wiley.com.
Suggested DOM selectors (to be verified)
| Field | Selector |
|---|---|
| Result container | li.search-results-item |
| Title | a.result-title |
| Authors | .search-result-authors |
| Year | .search-result-date |
| CD number → DOI | [data-cdnumber] or URL path |
| OA badge | .open-access-label |
| PDF link | a[href*='.pdf'] |
Note: selectors must be validated against the live site DOM before implementation. The Cochrane Library redesigns its frontend periodically.
Session expiry detection
COCHRANE_BASE = "https://www.cochranelibrary.com"
SSO_DOMAINS = ("login.cochrane.org", "idp.wiley.com", "idp.cochrane.org")
if any(d in page.url for d in SSO_DOMAINS):
rprint(
"[yellow]Cochrane session expired.[/yellow] "
"Run: [bold]mosaic auth login cochrane "
"--url https://www.cochranelibrary.com[/bold]"
)
return []Skeleton implementation
# mosaic/sources/cochrane_browser.py
from __future__ import annotations
import asyncio, re
from mosaic.models import Paper, SearchFilters
from mosaic.sources.base import BaseSource
_COCHRANE_BASE = "https://www.cochranelibrary.com"
_SSO_DOMAINS = ("login.cochrane.org", "idp.wiley.com", "idp.cochrane.org")
def _cd_to_doi(cd: str) -> str | None:
"""Convert a Cochrane CD number to its DOI."""
m = re.match(r"(CD\d+)", cd, re.IGNORECASE)
return f"10.1002/14651858.{m.group(1).upper()}" if m else None
class CochraneBrowserSource(BaseSource):
"""Search Cochrane Library via a saved Playwright session."""
name = "cochrane"
def available(self) -> bool:
try:
from mosaic.auth import find_session_for_url, session_is_valid
session_name = find_session_for_url(_COCHRANE_BASE)
return session_name is not None and session_is_valid(session_name)
except Exception:
return False
def search(self, query: str, max_results: int = 25,
filters: SearchFilters | None = None) -> list[Paper]:
try:
from mosaic.auth import find_session_for_url, _require_playwright
_require_playwright()
session_name = find_session_for_url(_COCHRANE_BASE)
if not session_name:
return []
return asyncio.run(
self._browser_search(query, max_results, session_name, filters)
)
except Exception:
return []
async def _browser_search(self, query, max_results, session_name, filters):
from mosaic.auth import session_path, _launch_browser
from playwright.async_api import async_playwright
from rich import print as rprint
state_file = session_path(session_name)
papers: list[Paper] = []
async with async_playwright() as p:
browser = await _launch_browser(p, headless=True)
context = await browser.new_context(storage_state=str(state_file))
page = await context.new_page()
await page.add_init_script(
"Object.defineProperty(navigator,'webdriver',{get:()=>undefined})"
)
try:
await page.goto(
f"{_COCHRANE_BASE}/search",
wait_until="networkidle",
timeout=30_000,
)
if any(d in page.url for d in _SSO_DOMAINS):
rprint(
"[yellow]Cochrane session expired.[/yellow] "
"Run: [bold]mosaic auth login cochrane "
"--url https://www.cochranelibrary.com[/bold]"
)
return []
# TODO: fill search form — selectors to be verified
await page.fill("input[name='searchText'], #searchText", query)
await page.press( "input[name='searchText'], #searchText", "Enter")
await page.wait_for_load_state("networkidle", timeout=30_000)
if any(d in page.url for d in _SSO_DOMAINS):
rprint(
"[yellow]Cochrane session expired.[/yellow] "
"Run: [bold]mosaic auth login cochrane "
"--url https://www.cochranelibrary.com[/bold]"
)
return []
try:
await page.wait_for_selector(
"li.search-results-item", timeout=12_000
)
except Exception:
pass
papers = await self._extract_results(page, max_results)
except Exception:
pass
finally:
await browser.close()
return papers
async def _extract_results(self, page, max_results: int) -> list[Paper]:
items = await page.query_selector_all("li.search-results-item")
papers = []
for item in items[:max_results]:
try:
paper = await self._parse_item(item)
if paper:
papers.append(paper)
except Exception:
continue
return papers
async def _parse_item(self, item) -> Paper | None:
# TODO: verify all selectors against live DOM
title_el = await item.query_selector("a.result-title")
if not title_el:
return None
title = (await title_el.inner_text()).strip()
if not title:
return None
href = await title_el.get_attribute("href") or ""
url = f"{_COCHRANE_BASE}{href}" if href.startswith("/") else href or None
# CD number → DOI
doi: str | None = None
m = re.search(r"(CD\d+)", href, re.IGNORECASE)
if m:
doi = _cd_to_doi(m.group(1))
# Authors
authors: list[str] = []
author_el = await item.query_selector(".search-result-authors")
if author_el:
raw = (await author_el.inner_text()).strip()
authors = [a.strip() for a in raw.split(",") if a.strip()]
# Year
year: int | None = None
date_el = await item.query_selector(".search-result-date")
if date_el:
y_match = re.search(r"\b(19|20)\d{2}\b", await date_el.inner_text())
if y_match:
year = int(y_match.group(0))
# OA
is_oa = bool(await item.query_selector(".open-access-label"))
# PDF
pdf_url: str | None = None
pdf_el = await item.query_selector("a[href*='.pdf'], a[href*='/pdf/']")
if pdf_el:
pdf_href = await pdf_el.get_attribute("href") or ""
pdf_url = (
f"{_COCHRANE_BASE}{pdf_href}"
if pdf_href.startswith("/") else pdf_href
)
return Paper(
title=title, authors=authors, year=year,
doi=doi, url=url, pdf_url=pdf_url,
journal="Cochrane Database of Systematic Reviews",
source=self.name, is_open_access=is_oa,
)Wiring checklist
- Add
CochraneBrowserSourcetomosaic/sources/__init__.py - Wire into
cli.py:_build_sources()under thebrowserextra guard (same asScienceDirectBrowserSource) - Add shorthand
cochrane(orcdsr) to CLI source list - Add
cochraneto the Sources table inREADME.mdanddocs/guide/sources.md - Add
cochraneentry todocs/guide/web-ui.mdanddocs/guide/authenticated-access.md - Write tests with
unittest.mockpatching Playwright calls (followtest_sciencedirect_browser.pypattern) - Verify all DOM selectors against live
cochranelibrary.combefore finalising
Prerequisites
- Requires
[browser]extra (playwright>=1.40) - User must have a valid Cochrane Library / Wiley account (institutional, NHS/Athens, or personal)
- Login:
mosaic auth login cochrane --url https://www.cochranelibrary.com
Activity
szaghi commentedon Oct 14, 2026
Re-scoped. A Playwright scraper of cochranelibrary.com would still be automated access that Wiley's terms prohibit (the issue itself notes this), its DOM selectors were never verified, and it would need continuous maintenance.
Implemented instead: a
cochranesource that queries Europe PMC restricted to the CDSR (ISSN:"1469-493X"), labels results as Cochrane, and links each one to itscochranelibrary.com/cdsr/doi/<DOI>/fullpage. No auth, no Playwright. All Europe PMC filters work (year, author, field, raw query). A live check returned reviews published in 2026, so the indexing lag cited as the motivation is small in practice.Not covered: Cochrane-only metadata (review group, protocol stage). Those would need Cochrane's OAuth API, which is credential-gated. This issue will close with the commit that adds the source.