Skip to content

🐌 Handle MyPurdue request throttling #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
haydenmc merged 1 commit into from
Jan 8, 2025
Merged

🐌 Handle MyPurdue request throttling #78

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions src/Scraper/Connections/MyPurdueConnection.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,11 @@ private enum HttpMethod { GET, POST };
private readonly HttpClient httpClient;

// How many request attempts should be made before failure
private const int MAX_RETRIES = 5;
private const int MAX_RETRIES = 6;

// MyPurdue now throttles our requests, and only seems to let us through
// after waiting a minimum of one minute.
private const int THROTTLE_DELAY_BASE_MS = 60000;

public MyPurdueConnection(ILogger<MyPurdueConnection> logger)
{
Expand Down Expand Up @@ -146,11 +150,19 @@ public async Task<string> GetSectionListPageAsync(string termCode, string subjec
{
this.logger.LogError("Received non-success status code '{}' on " +
"GetSectionListPageAsync.", request.StatusCode);
continue;
}
else

var content = await request.Content.ReadAsStringAsync();
if (content.Contains("We are sorry, but the site has received too many requests."))
{
return await request.Content.ReadAsStringAsync();
var retryDelayMs = THROTTLE_DELAY_BASE_MS + ExponentialRetryDelayMs(attempts);
this.logger.LogError("MyPurdue is throttling requests. Retrying in {}ms...",
retryDelayMs);
await Task.Delay(retryDelayMs);
continue;
}
return content;
}
throw new ApplicationException(
"Exceeded retries attempting to query MyPurdue section list");
Expand Down Expand Up @@ -315,5 +327,10 @@ private async Task<HttpResponseMessage> Request(HttpMethod method, string url,
}
return result;
}

private static int ExponentialRetryDelayMs(int retryNumber)
{
return (int)(Math.Pow(2, retryNumber) * 1000);
}
}
}
Loading