Wikidata indexing with qEndpoint

[ate47]

Hi all,

Recently we used qEndpoint to index a recent version of the Wikidata-all dataset, it was done on a Windows computer with a Ryzen 5 5600 CPU with 600GB SSD using 10 cores and 10GB of RAM.

The dataset was found here

dataset	size (gz)	triples	time to index
wikidata-all	148GB	17.5B	50h
wikidata-truthy	58GB	7.3B	14h

Wikidata-all

Here the times to compress the Wikidata all dataset with qEndpoint:

# dictionary compression + SPO index

## Sub chunks creation
section compression -- : 23.86 h
bit triple creation -- :  0.41 h
triple compression/map :  2.01 h
dictionary write ----- :  4.41 h
## Sub chunks merge
sub dict merge ------- :  4.57 h
sub triple merge ----- :  7.85 h

# OPS/PSO/POS indexes
OPS index ------------ :  3.33 h
P.. index ------------ :  1.50 h

Final size:

total 296G
183G index_dev.hdt
113G index_dev.hdt.index.v1-1

Try by yourself

You can try by yourself using the wikidata docker image or with the latest release of qEndpoint.

[D063520]

This is really cool!

[namespace-Pt]

@ate47 @D063520 Cool! I have several problems:

1. which file should I download for the latest wiki-all? I downloaded latest-all.ttl.gz, is that okay?
2. how to convert the file to index_dev.hdt and index_dev.hdt.index.v1-1? I'm using rdf2hdt.sh from the cli but I don't know how to config the index to build. I want to build all indexes (spo, ops, etc.) so that the 'reverse' queries can be executed. My current solution is running export JAVA_OPTIONS="-Dbitmaptriples.index.others=ops,osp,pos,pso,sop -Xmx1g" && bash /opt/tiger/qendpoint/qendpoint-cli-2.5.2/bin/rdf2hdt.sh /opt/tiger/qendpoint/example.ttl /opt/tiger/qendpoint/example.hdt -rdftype turtle -index -multithread. Is that correct?
3. in the current wikidata docker I found the wikidata is a 2-year-ago dump, I cannot execute several queries as stated here. I don't know why they occur.

If you could upload the files you built that would be super nice!

Really appreaciate it if you would help me.

[ate47]

Hi, it seems that you have a small error in your command, to add hdt options for the generation, you have to use the rdf2hdt options

• -config [file] load a file of options, one per line
• -options [options] load options from string, semicolon separated

[namespace-Pt]

Thanks for your reply! Do you mean that I should run the following command to create all indexes?

bash bin/rdf2hdt.sh example.ttl example.hdt \
  -rdftype turtle \
  -index \
  -multithread \
  -options "bitmaptriples.index.others=ops,osp,pos,pso,sop"

BTW, I'm I downloading latest-all.ttl.gz, is that correct one for the latest wiki dump including all properties?

Another question is that when running the command in my previous comment, my machine gets OOM error after reading 1B lines (I specified -Xmx100g and my machine has 200G memory). How do you manage to run indexing using 10G of RAM?

Thank you very much.

[ate47]

May I suggest you to read this part of the Wiki? https://github.com/the-qa-company/qEndpoint/wiki/qEndpoint-CLI-Indexing-datasets.

With the default memory loader, you can't go above 1 or 2B triples, this wiki explains how to switch to the disk loader.

If you are using all the indexes, you can also disable the FoQ index (.index.hdt.v1-1 file) generation with this option:

bitmaptriples.index.noFoQ=true

It will be faster and the FoQ index default loader is also not scaling.

[namespace-Pt]

Cool! Didn't notice this wiki before. Thank you very much!