Skip to content

Build: Skip checking oembed wp-json links in the hydra crawler #1275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 12, 2025

Conversation

mgol
Copy link
Member

@mgol mgol commented Apr 12, 2025

The oembed endpoint responds HTTP 429 Too Many Requests too often; perhaps because almost every page links to one.

@mgol mgol self-assigned this Apr 12, 2025
@mgol mgol force-pushed the hydra-skip-oembed branch from 47cf94a to 6538124 Compare April 12, 2025 11:19
@mgol
Copy link
Member Author

mgol commented Apr 12, 2025

The remaining failure is legitimate and I'm fixing it in jquery/jquery-wp-content#482. Please approve that other PR and I can re-run the check here before merging.

@mgol mgol requested a review from Krinkle April 12, 2025 11:25
@Krinkle Krinkle closed this Apr 12, 2025
@Krinkle Krinkle reopened this Apr 12, 2025
@Krinkle
Copy link
Member

Krinkle commented Apr 12, 2025

Looks like Google is having issues.

broken: 1
---

- code:    500
  url:     https://code.google.com/p/html5shiv/
  parent:  https://api.jquery.com/html/
  error:   Internal Server Error

About 1 in 10 times reproducible locally with curl. Ha!

$ curl -I https://code.google.com/p/html5shiv/
HTTP/2 500 
…
x-cloud-trace-context: a4027b46da15f8d429f62efd19f6f836
…
server: Google Frontend

The oembed endpoint responds HTTP 429 Too Many Requests too often; perhaps
because almost every page links to one.

Also, run the spider check on PRs that modify the hydra config file.
@mgol mgol force-pushed the hydra-skip-oembed branch from 6538124 to eac2211 Compare April 12, 2025 17:08
@mgol
Copy link
Member Author

mgol commented Apr 12, 2025

@Krinkle all green now.

@Krinkle Krinkle merged commit ff7994e into jquery:main Apr 12, 2025
4 checks passed
@mgol mgol deleted the hydra-skip-oembed branch April 12, 2025 17:12
@mgol mgol removed the Needs review label Apr 12, 2025
mgol added a commit to mgol/api.jquery.com that referenced this pull request Apr 14, 2025
Change the html5shiv link from https://code.google.com/p/html5shiv/ to
https://github.com/afarkas/html5shiv to which the former already 301-redirects.

We've had some code.google.com-related spider-check crawler errors which this
may also eliminate.

Ref jquerygh-1275
@mgol
Copy link
Member Author

mgol commented Apr 14, 2025

Looks like Google is having issues.

broken: 1
---

- code:    500
  url:     https://code.google.com/p/html5shiv/
  parent:  https://api.jquery.com/html/
  error:   Internal Server Error

About 1 in 10 times reproducible locally with curl. Ha!

$ curl -I https://code.google.com/p/html5shiv/
HTTP/2 500 
…
x-cloud-trace-context: a4027b46da15f8d429f62efd19f6f836
…
server: Google Frontend

@Krinkle This link already 301-redirects to GitHub so I updated it in PR #1276, PTAL.

Krinkle pushed a commit that referenced this pull request Apr 14, 2025
Change the html5shiv link from https://code.google.com/p/html5shiv/ to
https://github.com/afarkas/html5shiv to which the former already 301-redirects.

We've had some code.google.com-related spider-check crawler errors which this
may also eliminate.

Ref gh-1275
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants