Skip to content

Conversation

@sunqm
Copy link

@sunqm sunqm commented Jul 14, 2019

Automatic gc may not be called before the child process exits. Resources
may not be recycled properly without calling gc in the child process.

Automatic gc may not be called before the child process exits. Resources
may not be recycled properly without calling gc in the child process.
@pgiri
Copy link
Owner

pgiri commented Jul 17, 2019

Thanks for the patch.

I was not aware of this issue. Can you give some more info / details on what resources may be affected in parent process? An example demonstrating it would be helpful, as similar issue may be with pycos project. Thinking about it now, may be destrcutors in job may not be executed?

@sunqm
Copy link
Author

sunqm commented Jul 21, 2019

When the subprocess gets reference counts problem, the exec part does not release memory. It's common that code has reference counting problem, like the code below does not remove the temporary file after running dispy job but in a regular python program, the temporary file can be removed automatically. When dispy executes thousands of jobs without calling gc to recycle bad referenced objects, the small leaks in each execution cannot be ignored.

import dispy
def f():
    import tempfile
    tf = tempfile.NamedTemporaryFile()
    tf.tf = tf
    print(tf.name)

c = dispy.JobCluster(f, nodes='127.0.0.1')
job = c.submit()
c.wait()
print(job.stdout)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants