Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users

Dashboard
Notifications
Mark all as read
Q&A

Freeing sub-process resources?

+5
−0

I'm using Python to invoke another program in a sub-process. I've noticed my memory sometimes gets so large as to crash the system, and I'm wondering if I'm not correctly cleaning up the memory somehow.

Here's what I have:

children = []

fruits = ["apple", "banana", "cherry"]

for fruit in fruits:
	children.append(subprocess.Popen(["fruitProgram", fruit]))

while len(children) > 0:
	child = children.pop()
	child.wait()

subprocess.run("doStuffWithFruitProgramOutput")

I've had my doStuffWithFruitProgramOutput crash due to lack of memory when run like this, but not when run by itself. Is child.wait() not enough to free the child process resources?

I tried adding gc.collect() after clearing children, and that doesn't seem to have helped.

Is there something else I need to be doing? Or am I barking up the wrong tree here, and my problem is somewhere else?

Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

2 comments

Can you confirm the same behaviour occurs if you use communicate instead of wait? If yes, what about a potential memory leak from fruitProgram? If you run fruitProgram directly from the shell, do you see a significant memory increase between the start and end of the program? Patol75‭ 10 days ago

It can't be fruitProgram itself leaking. When a process is terminated and closed, the OS itself frees the memory it used. Leaks only persist until a process terminates, unless the OS itself actually has a leak (extraordinarily unlikely). I'll give communicate a go, but the wording at https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait is exactly the same for what it does with the process: waits for it to terminate. Hyperlynx‭ 2 days ago

0 answers

Sign up to answer this question »