Just a few days ago, the pipeline started failing. It happened to coincide with us making the repo private, but looking at the failed workflows, the first step of the ingest, scraping the funding opportunity page, was failing.
Here's what visiting the site in the pipeline returns.
<!DOCTYPE html>
<html class="no-js" lang="en-US">
<!--<!
[endif]--><head>
<title>Attention Required! | Cloudflare</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link
rel="stylesheet"
id="cf_styles-css"
href="/cdn-cgi/styles/cf.errors.css"
/>
<!--[if lt IE 9
]><link
rel="stylesheet"
id="cf_styles-ie-css"
href="/cdn-cgi/styles/cf.errors.ie.css"
/><![endif]-->
<style>
body {
margin: 0;
padding: 0;
}
</style>
<!--[if gte IE 10]><!-->
<script>
if (!navigator.cookieEnabled) {
window.addEventListener("DOMContentLoaded", function () {
var cookieEl = document.getElementById("cookie-alert");
cookieEl.style.display = "block";
});
}
</script>
<!--<![endif]-->
</head>
<body>
<div id="cf-wrapper">
<div
class="cf-alert cf-alert-error cf-cookie-error"
id="cookie-alert"
data-translate="enable_cookies"
>
Please enable cookies.
</div>
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-header cf-error-overview">
<h1 data-translate="block_headline">Sorry, you have been blocked</h1>
<h2 class="cf-subheadline">
<span data-translate="unable_to_access"
>You are unable to access</span
>
nih.gov
</h2>
</div>
<!-- /.header -->
<div class="cf-section cf-highlight">
<div class="cf-wrapper">
<div class="cf-screenshot-container cf-screenshot-full">
<span class="cf-no-screenshot error"></span>
</div>
</div>
</div>
<!-- /.captcha-container -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="blocked_why_headline">
Why have I been blocked?
</h2>
<p data-translate="blocked_why_detail">
This website is using a security service to protect itself from
online attacks. The action you just performed triggered the
security solution. There are several actions that could trigger
this block including submitting a certain word or phrase, a SQL
command or malformed data.
</p>
</div>
<div class="cf-column">
<h2 data-translate="blocked_resolve_headline">
What can I do to resolve this?
</h2>
<p data-translate="blocked_resolve_detail">
You can email the site owner to let them know you were blocked.
Please include what you were doing when this page came up and
the Cloudflare Ray ID found at the bottom of this page.
</p>
</div>
</div>
</div>
<!-- /.section -->
<div
class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"
>
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1"
>Cloudflare Ray ID:
<strong class="font-semibold">9038d350bea8f791</strong></span
>
<span class="cf-footer-separator sm:hidden">•</span>
<span
id="cf-footer-item-ip"
class="cf-footer-item sm:block sm:mb-1"
>
Your IP:
<button
type="button"
id="cf-footer-ip-reveal"
class="cf-footer-ip-reveal-btn"
>
Click to reveal
</button>
<span class="hidden" id="cf-footer-ip">69.253.52.214</span>
<span class="cf-footer-separator sm:hidden">•</span>
</span>
<span class="cf-footer-item sm:block sm:mb-1"
><span>Performance & security by</span>
<a
rel="noopener noreferrer"
href="https://www.cloudflare.com/5xx-error-landing"
id="brand_link"
target="_blank"
>Cloudflare</a
></span
>
</p>
<script>
(function () {
function d() {
var b = a.getElementById("cf-footer-item-ip"),
c = a.getElementById("cf-footer-ip-reveal");
b &&
"classList" in b &&
(b.classList.remove("hidden"),
c.addEventListener("click", function () {
c.classList.add("hidden");
a.getElementById("cf-footer-ip").classList.remove("hidden");
}));
}
var a = document;
document.addEventListener &&
a.addEventListener("DOMContentLoaded", d);
})();
</script>
</div>
<!-- /.error-footer -->
</div>
<!-- /#cf-error-details -->
</div>
<!-- /#cf-wrapper -->
<script>
window._cf_translation = {};
</script>
<script
defer=""
src="https://static.cloudflareinsights.com/beacon.min.js/vcd15cbe7772f49c399c6a5babf22c1241717689176015"
integrity="sha512-ZpsOmlRQV6y907TI0dKBHq9Md29nnaEIPlkf84rnaERnq6zvWvPUqr2ft8M1aS28oN72PdrCzSjY4U6VaAw1EQ=="
data-cf-beacon='{"rayId":"9038d350bea8f791","version":"2025.1.0","serverTiming":{"name":{"cfExtPri":true,"cfL4":true,"cfSpeedBrain":true,"cfCacheStatus":true}},"token":"4da10bb88a2f4316a9de363d388a5456","b":1}'
crossorigin="anonymous"
></script>
</body>
</html>Idk how making the repo private could've caused this. Running the pipeline locally produces the same result, and the pipeline itself isn't aware of whether the repo is public/private (aside from the repo ingest step, where this repo is tagged for ingest).
I also don't know what changed in the last few days that would cause this. It might just be that we scraped the page one too many times and got blocked.
curl returns the same thing, on my machine.
Will look into if there's anything I can do to appease the firewall, and will try to contact the appropriate admins in the meantime.
Just a few days ago, the pipeline started failing. It happened to coincide with us making the repo private, but looking at the failed workflows, the first step of the ingest, scraping the funding opportunity page, was failing.
Here's what visiting the site in the pipeline returns.
Idk how making the repo private could've caused this. Running the pipeline locally produces the same result, and the pipeline itself isn't aware of whether the repo is public/private (aside from the repo ingest step, where this repo is tagged for ingest).
I also don't know what changed in the last few days that would cause this. It might just be that we scraped the page one too many times and got blocked.
curlreturns the same thing, on my machine.Will look into if there's anything I can do to appease the firewall, and will try to contact the appropriate admins in the meantime.