Skip to content

feat(sources): add Cochrane Library browser source (authenticated session) #27

Closed

Description

@szaghi

Summary

Cochrane Library is the gold-standard database for systematic reviews and meta-analyses (Cochrane Database of Systematic Reviews, CDSR). Several colleagues and research groups rely on it as a primary source and would benefit from being able to search it directly via MOSAIC.

Why a dedicated source is needed

Cochrane reviews are indexed by PubMed and Europe PMC (both already MOSAIC sources), but:

  • Coverage lag: newly published reviews can take days/weeks to appear in PubMed.
  • Cochrane-specific filters (review group, protocol stage, CD number) are not available via PubMed/EPMC.
  • Direct access provides richer metadata: review stage, protocol status, Cochrane group.
  • Users who work primarily with CDSR prefer a dedicated, clearly labelled source in MOSAIC output.

Why a public API is not an option

Cochrane exposes two private REST APIs (archie.cochrane.org and api.cochrane.org), both protected by OAuth 2.0 credentials obtainable only by contacting Cochrane directly — no self-service signup exists. Scraping cochranelibrary.com is blocked at the edge (HTTP 403 for all automated clients) and prohibited by Wiley's Terms of Use.

Proposed approach: authenticated browser session

Follow the existing pattern used by ScienceDirectBrowserSource (mosaic/sources/sciencedirect_browser.py): a Playwright-based source that reuses a saved login session, activated via mosaic auth login.

Login flow

The Cochrane Library uses a Wiley / institutional SSO login at:

https://www.cochranelibrary.com/

Users log in once through the browser (institutional login, personal Wiley account, or NHS/Athens):

mosaic auth login cochrane --url https://www.cochranelibrary.com

This opens a headed browser window, lets the user log in interactively, then saves the Playwright storage state to ~/.config/mosaic/sessions/cochrane.json — identical to the ScienceDirect flow.

Search flow

  1. Load saved session into a headless Playwright context.
  2. Navigate to https://www.cochranelibrary.com/search.
  3. Fill the search input and submit the form.
  4. Wait for result cards (li.search-results-item or equivalent selector — needs verification against live DOM).
  5. Extract per-result: title, authors, DOI (derivable from CD number via 10.1002/14651858.CDxxxxxx), year, review type (systematic review / protocol), open-access status, URL.
  6. Detect expired session by checking for redirect to login.cochrane.org or idp.wiley.com.

Suggested DOM selectors (to be verified)

Field Selector
Result container li.search-results-item
Title a.result-title
Authors .search-result-authors
Year .search-result-date
CD number → DOI [data-cdnumber] or URL path
OA badge .open-access-label
PDF link a[href*='.pdf']

Note: selectors must be validated against the live site DOM before implementation. The Cochrane Library redesigns its frontend periodically.

Session expiry detection

COCHRANE_BASE = "https://www.cochranelibrary.com"
SSO_DOMAINS = ("login.cochrane.org", "idp.wiley.com", "idp.cochrane.org")

if any(d in page.url for d in SSO_DOMAINS):
    rprint(
        "[yellow]Cochrane session expired.[/yellow] "
        "Run: [bold]mosaic auth login cochrane "
        "--url https://www.cochranelibrary.com[/bold]"
    )
    return []

Skeleton implementation

# mosaic/sources/cochrane_browser.py

from __future__ import annotations
import asyncio, re
from mosaic.models import Paper, SearchFilters
from mosaic.sources.base import BaseSource

_COCHRANE_BASE = "https://www.cochranelibrary.com"
_SSO_DOMAINS   = ("login.cochrane.org", "idp.wiley.com", "idp.cochrane.org")

def _cd_to_doi(cd: str) -> str | None:
    """Convert a Cochrane CD number to its DOI."""
    m = re.match(r"(CD\d+)", cd, re.IGNORECASE)
    return f"10.1002/14651858.{m.group(1).upper()}" if m else None


class CochraneBrowserSource(BaseSource):
    """Search Cochrane Library via a saved Playwright session."""

    name = "cochrane"

    def available(self) -> bool:
        try:
            from mosaic.auth import find_session_for_url, session_is_valid
            session_name = find_session_for_url(_COCHRANE_BASE)
            return session_name is not None and session_is_valid(session_name)
        except Exception:
            return False

    def search(self, query: str, max_results: int = 25,
               filters: SearchFilters | None = None) -> list[Paper]:
        try:
            from mosaic.auth import find_session_for_url, _require_playwright
            _require_playwright()
            session_name = find_session_for_url(_COCHRANE_BASE)
            if not session_name:
                return []
            return asyncio.run(
                self._browser_search(query, max_results, session_name, filters)
            )
        except Exception:
            return []

    async def _browser_search(self, query, max_results, session_name, filters):
        from mosaic.auth import session_path, _launch_browser
        from playwright.async_api import async_playwright
        from rich import print as rprint

        state_file = session_path(session_name)
        papers: list[Paper] = []

        async with async_playwright() as p:
            browser = await _launch_browser(p, headless=True)
            context = await browser.new_context(storage_state=str(state_file))
            page    = await context.new_page()
            await page.add_init_script(
                "Object.defineProperty(navigator,'webdriver',{get:()=>undefined})"
            )
            try:
                await page.goto(
                    f"{_COCHRANE_BASE}/search",
                    wait_until="networkidle",
                    timeout=30_000,
                )
                if any(d in page.url for d in _SSO_DOMAINS):
                    rprint(
                        "[yellow]Cochrane session expired.[/yellow] "
                        "Run: [bold]mosaic auth login cochrane "
                        "--url https://www.cochranelibrary.com[/bold]"
                    )
                    return []

                # TODO: fill search form — selectors to be verified
                await page.fill("input[name='searchText'], #searchText", query)
                await page.press( "input[name='searchText'], #searchText", "Enter")
                await page.wait_for_load_state("networkidle", timeout=30_000)

                if any(d in page.url for d in _SSO_DOMAINS):
                    rprint(
                        "[yellow]Cochrane session expired.[/yellow] "
                        "Run: [bold]mosaic auth login cochrane "
                        "--url https://www.cochranelibrary.com[/bold]"
                    )
                    return []

                try:
                    await page.wait_for_selector(
                        "li.search-results-item", timeout=12_000
                    )
                except Exception:
                    pass

                papers = await self._extract_results(page, max_results)
            except Exception:
                pass
            finally:
                await browser.close()
        return papers

    async def _extract_results(self, page, max_results: int) -> list[Paper]:
        items = await page.query_selector_all("li.search-results-item")
        papers = []
        for item in items[:max_results]:
            try:
                paper = await self._parse_item(item)
                if paper:
                    papers.append(paper)
            except Exception:
                continue
        return papers

    async def _parse_item(self, item) -> Paper | None:
        # TODO: verify all selectors against live DOM
        title_el = await item.query_selector("a.result-title")
        if not title_el:
            return None
        title = (await title_el.inner_text()).strip()
        if not title:
            return None

        href = await title_el.get_attribute("href") or ""
        url  = f"{_COCHRANE_BASE}{href}" if href.startswith("/") else href or None

        # CD number → DOI
        doi: str | None = None
        m = re.search(r"(CD\d+)", href, re.IGNORECASE)
        if m:
            doi = _cd_to_doi(m.group(1))

        # Authors
        authors: list[str] = []
        author_el = await item.query_selector(".search-result-authors")
        if author_el:
            raw = (await author_el.inner_text()).strip()
            authors = [a.strip() for a in raw.split(",") if a.strip()]

        # Year
        year: int | None = None
        date_el = await item.query_selector(".search-result-date")
        if date_el:
            y_match = re.search(r"\b(19|20)\d{2}\b", await date_el.inner_text())
            if y_match:
                year = int(y_match.group(0))

        # OA
        is_oa = bool(await item.query_selector(".open-access-label"))

        # PDF
        pdf_url: str | None = None
        pdf_el = await item.query_selector("a[href*='.pdf'], a[href*='/pdf/']")
        if pdf_el:
            pdf_href = await pdf_el.get_attribute("href") or ""
            pdf_url  = (
                f"{_COCHRANE_BASE}{pdf_href}"
                if pdf_href.startswith("/") else pdf_href
            )

        return Paper(
            title=title, authors=authors, year=year,
            doi=doi, url=url, pdf_url=pdf_url,
            journal="Cochrane Database of Systematic Reviews",
            source=self.name, is_open_access=is_oa,
        )

Wiring checklist

  • Add CochraneBrowserSource to mosaic/sources/__init__.py
  • Wire into cli.py:_build_sources() under the browser extra guard (same as ScienceDirectBrowserSource)
  • Add shorthand cochrane (or cdsr) to CLI source list
  • Add cochrane to the Sources table in README.md and docs/guide/sources.md
  • Add cochrane entry to docs/guide/web-ui.md and docs/guide/authenticated-access.md
  • Write tests with unittest.mock patching Playwright calls (follow test_sciencedirect_browser.py pattern)
  • Verify all DOM selectors against live cochranelibrary.com before finalising

Prerequisites

  • Requires [browser] extra (playwright>=1.40)
  • User must have a valid Cochrane Library / Wiley account (institutional, NHS/Athens, or personal)
  • Login: mosaic auth login cochrane --url https://www.cochranelibrary.com

References

Activity

  1. szaghi commented on Oct 14, 2026

    @szaghi
    OwnerAuthor

    Re-scoped. A Playwright scraper of cochranelibrary.com would still be automated access that Wiley's terms prohibit (the issue itself notes this), its DOM selectors were never verified, and it would need continuous maintenance.

    Implemented instead: a cochrane source that queries Europe PMC restricted to the CDSR (ISSN:"1469-493X"), labels results as Cochrane, and links each one to its cochranelibrary.com/cdsr/doi/<DOI>/full page. No auth, no Playwright. All Europe PMC filters work (year, author, field, raw query). A live check returned reviews published in 2026, so the indexing lag cited as the motivation is small in practice.

    mosaic search "exercise for chronic low back pain" --source cochrane
    

    Not covered: Cochrane-only metadata (review group, protocol stage). Those would need Cochrane's OAuth API, which is credential-gated. This issue will close with the commit that adds the source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions