Custom Scratch website scraper (Voyager)

[NotFenixio]

Recently, ScratchDB has been acting up, causing problems for Snazzle, which relies on it. But we can't just toss ScratchDB aside because it's our main source of info. So, here's an idea: Let's create our own ScratchDB.

To do this, we'd learn from ScratchDB's way of doing things. We can use PlayWright, Selenium, and/or Atoma to grab info from Scratch, and then BeautifulSoup to clean it up and get the data we need.

Now, this idea is kind of like a trial run, like taking a poll. I want to know what y'all think about it. Would this be a good move?

[EngineerRunner]

i mean, i'm mostly working on Pyratch now, but this'd be really helpful for all alternative frontends, so i support this idea.

[NotFenixio]

Awesome! We need a name... Any suggestions?

[NotFenixio]

I'll just call it ScratchedDB for now,

[redstone-dev]

I really like this idea, however, we'd need reliable hosting with the closest to 100% uptime we can get. I have an AWS account so we could try that, but it's really expensive so we'd need to get our money's worth out of it.

As for the name, we could call it Voyager. (Thanks ChatGPT)

We could also try writing it in Rust for funzies.

[NotFenixio]

Voyager then! I'll rename the repo in a moment.

The idea is to create a locally-deployable ScratchDB, so whoever downloads Snazzle or any other alternative frontend, will be hosting its own ScratchDB.

Also, I discovered that AWS has a 12-month free tier for Amazon EC2 which we can use to deploy this new thing for 1 year. (A simple Glitch project with UptimeRobot could do the thing too)

And for the Rust thing, I don't know... Let's try doing it in Python and leaving that for the future Snazzle Svelte/Rust port.

[redstone-dev]

And for the Rust thing, I don't know... Let's try doing it in Python and leaving that for the future Snazzle Svelte/Rust port.

Rust rewrite of Svelte??? /j </offtopic>

[davidtheplatform]

Random suggestions:
Use requests and beautiful soup since it uses way less ram (also there are rss feeds but they don’t have every post)

Have a centralized server to reduce load on scratch but clients have a local cache/scraper in case the server goes down

Clients can choose whether they want stale data immediately or updates data that takes longer to get

[redstone-dev]

Clients can choose whether they want stale data immediately or updates data that takes longer to get

I think we could combine Voyager with a system on the client that checks if the RSS data has new posts that Voyager doesn't have yet, in which case it sends this data to the central Voyager server and then displays the new data to the user.

[davidtheplatform]

Clients can choose whether they want stale data immediately or updates data that takes longer to get

I think we could combine Voyager with a system on the client that checks if the RSS data has new posts that Voyager doesn't have yet, in which case it sends this data to the central Voyager server and then displays the new data to the user.

Its probably better to avoid sending requests to Scratch if we don't have to.
What I meant was that if the client doesn't care about having the most up-to-date data it can tell the server that so the server doesn't have to make a request to the Scratch servers.

[redstone-dev]

Clients can choose whether they want stale data immediately or updates data that takes longer to get

I think we could combine Voyager with a system on the client that checks if the RSS data has new posts that Voyager doesn't have yet, in which case it sends this data to the central Voyager server and then displays the new data to the user.

Its probably better to avoid sending requests to Scratch if we don't have to. What I meant was that if the client doesn't care about having the most up-to-date data it can tell the server that so the server doesn't have to make a request to the Scratch servers.

To that end, we should also add rate-limiting (maybe only 3 requests a second?) to avoid stressing the Scratch servers. We may need to increase this number based on website traffic, though. Ideally the server should do this automatically somehow.

[ajskateboarder]

We don't need any browser automation tools, Scratch forums are easy to fetch over HTTP requests

If we are using Rust, we can use the reqwest and scraper crates for data and serve it over actix. I can work on it whenever I have free time

To that end, we should also add rate-limiting (maybe only 3 requests a second?) to avoid stressing the Scratch servers. We may need to increase this number based on website traffic, though. Ideally the server should do this automatically somehow.

I think 3 requests/second is fine

[NotFenixio]

We're building it on Python, but we need some help with specific functions that require indexing Scratch. Voyager Development (view)