Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create endpoint for downloading all data #6

Open
zbryikt opened this issue Feb 26, 2014 · 3 comments
Open

create endpoint for downloading all data #6

zbryikt opened this issue Feb 26, 2014 · 3 comments

Comments

@zbryikt
Copy link
Member

zbryikt commented Feb 26, 2014

add url like /channel/xxx/all.json to get all formatted data in xxx channel

@godfat
Copy link
Member

godfat commented Mar 30, 2014

Since we're storing data in redis lists, named with: "irclog:channel:##{channel}:#{date}", that means we're grouping with channel and date, there might not have an easy way to extract all data for a given channel.

We either:

  • Scan over a time period
  • Migrate the data to hash based
  • Add additional data for this query

Ideally, using hash might be simpler, but then we need to migrate the data.

@dannvix
Copy link
Member

dannvix commented Sep 1, 2014

Another possible approach – provide a public link to the Redis RDB persistence (i.e. dump.rdb) and provide a script to extract logs of specified channel from the file. However, this might exhaust the network transfer quota :P

@caasi
Copy link
Member

caasi commented Mar 22, 2020

FYI: @kiang creates a repo to fetch all logs in JSON: https://github.com/g0v/logbot-logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants