Our API is under a lot of stress lately and a large contributing factor to this seems to be the proliferation of AI generated "threat intel" dashboards which make use of it.
For example if you prompt claude to generate a dashboard that makes use of open datasets to create a comprehensive threat intel dashboard, the OONI API ends up being part of it, yet the code it will generate is not at all respectful of our API and generates a lot of unnecessary load, see for example this snippet that was produced by claude:
def get_ooni_censorship(iso2: str) -> dict:
"""OONI API — recent censorship measurements for this country."""
result = {"total": 0, "anomalies": 0, "confirmed": 0, "since": ""}
since = (datetime.now(timezone.utc) - timedelta(days=7)).strftime("%Y-%m-%d")
url = (f"https://api.ooni.io/api/v1/measurements?probe_cc={iso2}"
f"&since={since}&limit=100&confirmed=true")
data = fetch(url)
if data and "results" in data:
confirmed = data["results"]
result["confirmed"] = len(confirmed)
url2 = (f"https://api.ooni.io/api/v1/measurements?probe_cc={iso2}"
f"&since={since}&limit=200&anomaly=true")
data2 = fetch(url2)
if data2 and "results" in data2:
result["anomalies"] = len(data2["results"])
# Also fetch aggregate stats
url3 = (f"https://api.ooni.io/api/v1/aggregation?probe_cc={iso2}"
f"&since={since}&axis_x=measurement_start_day")
data3 = fetch(url3)
if data3 and "result" in data3:
rows = data3["result"]
if isinstance(rows, list):
result["total"] = sum(r.get("measurement_count", 0) for r in rows)
elif isinstance(rows, dict):
result["total"] = rows.get("measurement_count", 0)
result["since"] = since
return resultWe can see, as an example, that in this snippet the measurements request is being duplicated twice to retrieve the anomaly and confirmed flag values, yet all this data could be much more efficiently retrieved from the single aggregation query that's performed at the end of the block.
The AI agents are also not smart enough to figure out the semantics of the OONI dataset, so they end up doing queries that will never produce any meaningful results wasting our resources for looking up things that are not part of our data schematic (eg. searching for confirmed keys in measurements that can never produce a confirmed flag: https://github.com/lina767/digital-war-room/blob/425ec257613f849e20e1c9a28065a036b31f158c/backend/agents/techint_agent.py#L281)
To more concretely assess the impact of this, I processed our access logs looking for user-agent strings which look AI generated and found at least 41 suspicious user agent strings, which I copy below for reference:
warzone-worker/1.0
SamizdatProbe/1.0
JavidWatch/1.0 (Iran Internet Freedom Monitor)
Conflictly/1.0 (https://conflictly.app)
Voidly/1.0
IranInternetMonitor/2.2
IranFreedomHub/1.0
VM-Creative-Internet-Radar/1.0
ONIST/1.0 Open-Intelligence-Platform
WarMCPMonitor/0.1 (contact@kezify.com)
Forager Research data-courier@localhost
Voidly/1.0 (censorship-index)
IncidentMonitor/1.0
ninja-fusion-ooni/1.0
SupplyChainRiskIntelligence/1.0
MarengoTech-CIVCOM-Intel/1.0 (contact: marengotech.com)
ooni-monitor/1.0
dns-block-checker-saas/0.1
ATLAS/1.0 (Geopolitical Dashboard)
WWO-Monitor/4.0 (WorldWatchOne)
ferran-data-pipeline/1.0
azure-data-factory/2.0
InternetChowkidar/1.0.0 ooniprobe-engine/3.29.0
IsItBlocked/1.0 (+https://isitblocked.com)
BunkerPI Research (admin@bunkerpi.com)
CsibuIntel/0.1 (https://github.com/bravenewxyz/Csibu)
IsItBlockedAudit/1.0 (+https://isitblocked.com)
SAHEL-EYE/1.0 (Cyber-Geopolitical OSINT; +https://saheleye.joranbatty.fr)
BlackoutObservatory/1.0 (+https://blackoutobservatory.org)
Intel-Dashboard/2.0 (research)
NarrativeOS-DMNC/1.0 (Pakistan Media Monitor)
ScoutIntelFeed/1.0 (personal intelligence aggregator)
WorldviewDashboard/1.0 (+http://localhost)
WorldView/1.0 (situational-awareness)
WARWATCH-GEP/7.0 (+https://warwatch.lovable.app)
claude
OMK-Sentinel/1.0 (+ooni_weak)
osint/0.1 (https://github.com/mfx/osint)
OSINT-Dashboard/1.0
IranSimulator/1.0
Lyqent Security Research (https://lyqent.com)
The total API response time for all these user agents adds up to 123155.48 seconds or ~34h in the sampled window of 7 days. To put this into perspective the total response time for legitimate users of OONI Explorer in the same window of time was 40h.
We should think of ways to limit the impact of these scrapers hitting our infrastructure, since it has lead to specific incidents affecting the availability of measurement collection:
Some ideas worth exploring are:
- Make documentation more clear about best practices for retrieving the kind of information these dashboards need so that AI generated code ends up being correct
- Setting up an MCP server for OONI data
- Apply more aggressive rate limiting to unauthenticated requests that look like they are coming from an AI scraper
- Search for
https://api.ooni.org in github looking for open source codebases that use our API and file an issue there pointing them to the relevant documentation
Our API is under a lot of stress lately and a large contributing factor to this seems to be the proliferation of AI generated "threat intel" dashboards which make use of it.
For example if you prompt claude to generate a dashboard that makes use of open datasets to create a comprehensive threat intel dashboard, the OONI API ends up being part of it, yet the code it will generate is not at all respectful of our API and generates a lot of unnecessary load, see for example this snippet that was produced by claude:
We can see, as an example, that in this snippet the
measurementsrequest is being duplicated twice to retrieve the anomaly and confirmed flag values, yet all this data could be much more efficiently retrieved from the singleaggregationquery that's performed at the end of the block.The AI agents are also not smart enough to figure out the semantics of the OONI dataset, so they end up doing queries that will never produce any meaningful results wasting our resources for looking up things that are not part of our data schematic (eg. searching for
confirmedkeys in measurements that can never produce a confirmed flag: https://github.com/lina767/digital-war-room/blob/425ec257613f849e20e1c9a28065a036b31f158c/backend/agents/techint_agent.py#L281)To more concretely assess the impact of this, I processed our access logs looking for user-agent strings which look AI generated and found at least 41 suspicious user agent strings, which I copy below for reference:
The total API response time for all these user agents adds up to 123155.48 seconds or ~34h in the sampled window of 7 days. To put this into perspective the total response time for legitimate users of OONI Explorer in the same window of time was 40h.
We should think of ways to limit the impact of these scrapers hitting our infrastructure, since it has lead to specific incidents affecting the availability of measurement collection:
Some ideas worth exploring are:
https://api.ooni.orgin github looking for open source codebases that use our API and file an issue there pointing them to the relevant documentation