Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak when creating a lot of trees #6

Open
rickbeeloo opened this issue Nov 25, 2020 · 1 comment
Open

Memory leak when creating a lot of trees #6

rickbeeloo opened this issue Nov 25, 2020 · 1 comment

Comments

@rickbeeloo
Copy link

rickbeeloo commented Nov 25, 2020

I also posted this issue here as I'm not sure which of the two is maintained.

Simple example and look at top/htop
Take this simple example where we create a tree in each iteration:

from ailist import AIList
import time

for i in range(7000000):
    a = AIList()
    a.add(i,i+1)
    time.sleep(0.0000001)

Then when look at top/htop we clearly see that memory is not freed and gradually builds up after each iteration.

We can see memory usage with this simple example
We can clearly see this when we look at the memory usage increase after each iteration (very small example):

from ailist import AIList
import time
import numpy as np
from tqdm import tqdm
import matplotlib.pyplot as plt

def getCurrentMemoryUsage():
    # from: https://stackoverflow.com/questions/938733/total-memory-used-by-python-process
    with open('/proc/self/status') as f:
        memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]
    return int(memusage.strip()) # in kb

r = 50000
memory_usage = np.empty(r, dtype = 'int32')
for i in tqdm(range(r)):
    a = AIList()
    a.add(i,i+1)
    time.sleep(0.0000001)
    memory_usage[i] = getCurrentMemoryUsage()

plt.plot(range(r), memory_usage)

Which shows us:
image

Not a garbage collection thing
When we call garbage collection within the loop (which we of course shouldn't have to) it also does not resolve the issue:
image

This works fine with other interval libraries, such as IntervalTree

from intervaltree import Interval, IntervalTree

r = 5000
memory_usage = np.empty(r, dtype = 'int32')
for i in tqdm(range(r)):
    t = IntervalTree()
    t.add(Interval(i,i+1))
    time.sleep(0.0000001)
    memory_usage[i] = getCurrentMemoryUsage()

plt.plot(range(r), memory_usage)

image

Compared to intervaltree AIlist is much faster so I would love to use this instead, however, currently we can't with the memory issue :(

@kylessmith
Copy link
Owner

I have recently push some major edits to this github repo. Let me know if this problem continues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants