fix(cache): Estimate size of posting lists #9515
Conversation
xqqp
force-pushed
the
fix_cache_costs
branch
6 times, most recently
from
a933320 to
7bbc497
Compare
|
Test failures seem unrelated to this change. @matthewmcneely when you have time please take a look, without this patch v25 is unusable for me, because memory quickly fills up. |
@xqqp Yeah, the shared runners we are now using don't seem to be up to the task. We're working on it. |
|
@xqqp Thank again for this PR and your patience. We're still trying to get our runners configured thru a third party following the move back to dgraph-io. In the meantime, a few questions regarding your PR.
Thanks again, and I'll let you know when we get our runners configured and get this PR successfully thru CI |
|
@matthewmcneely Thanks for looking into this.
See predictable-labs#6 (comment) regarding this. Try to query a few million unique nodes and the issue should pop up very quickly. I implemented short program which shows the issue on the 21 million dataset below. If you load more data the issue will become more prevalent. Golang Query examplepackage main
import (
"context"
"encoding/json"
"fmt"
"log"
"strconv"
"github.com/dgraph-io/dgo/v250"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
func main() {
dgraphClient, err := dgo.NewClient("localhost:9080",
dgo.WithGrpcOption(grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(1024*1024*1024))),
dgo.WithGrpcOption(grpc.WithTransportCredentials(insecure.NewCredentials())))
if err != nil {
log.Fatalf("Failed to connect to Dgraph: %v", err)
}
defer dgraphClient.Close()
ctx := context.Background()
step := 50000
lastNodeUID := "0x0"
for i := 0; ; i += step {
query := `query Q($first:int,$after:string){
q(func: has(dgraph.type),first:$first,after:$after){
uid
performance.film {
uid
}
performance.actor {
uid
}
performance.character{
uid
}
}
}`
resp, err := dgraphClient.NewTxn().QueryWithVars(ctx, query, map[string]string{
"$first": strconv.Itoa(step), "$after": lastNodeUID})
if err != nil {
log.Fatalf("Failed to execute query: %v", err)
}
var r struct {
Q []struct {
UID string `json:"uid"`
} `json:"q"`
}
if err = json.Unmarshal(resp.GetJson(), &r); err != nil {
log.Fatalf("Failed to unmarshal json: %v", err)
}
if len(r.Q) == 0 {
break
}
lastNodeUID = r.Q[len(r.Q)-1].UID
fmt.Println("step", step)
}
}
With this PR an approximation of the actual cache item size will be set for the cache cost. Without this PR the cost of each cache item is 1, which means that the cache will keep up to 1099511627776 items (default 1024 MiB max cost). So items will be rarely evicted from the cache. If this PR is applied and the default 1024 MiB size of the cache is kept, then the cache will keep much less items than before (because the approximated cost is now set), which could decrease performance. For that reason I increased the default cache size. If you want to keep the default cache size that is also fine with me. |
2 participants
Description
The maximum cost of the Ristretto cache is set to a number of megabytes, but when inserting a posting list item into the cache, the cost is set to 1. Because of this the cache only grows and no items get evicted. This results in high memory consumption, especially with the new UID cache (see #9513).
This PR introduces a cost function which estimates the memory size of each item. For more details see predictable-labs#6. Credits to @darkcoderrises for the implementation.
The default cache size is changed from 1024 to 4096, to reflect the more accurate cost estimation. See below for how the size of the cache relates to the occupied memory by the Dgrpah process, when this PR is applied.
size-mb=1024
Dgraph process occupies up to 3.7 GiB of memory
heap:
size-mb=2048
Dgraph process occupies up to 5.4 GiB of memory
heap:
size-mb=4096
Dgraph process occupies up to 8.4 GiB of memory
heap:
size-mb=8192
Dgraph process occupies up to 13.3 GiB of memory
heap:
Closes #9513