Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Convert .npy files in a directory to images (.png)
I have around 20,000 .npy files in a directory. That main directory has no subfolders:-
Main_dir
|
|--1.npy
|--2.npy
|--3.npy
|--........
The absolute file paths are stored in a list named paths.
What is the fastest and most efficient method (time and memory-wise) to convert all these .npy files to image (.png preferable)?
I don't care if the solution would use python or bash.
2 answers
You are accessing this answer with a direct link, so it's being shown above all other answers regardless of its score. You can return to the normal view.
fastest and most efficient method (time and memory-wise)
Time and memory go against each other in a simple way: amount of jobs divides the time (up to amount of cores, sans the overhead, assuming no I/O bottlenecks e.g. an SSD), and multiplies the peak memory used.
stored in a list named paths
Is that a Python list, or do you mean a newline-delimited file?
For the latter, an easy bash way to parallelize is xargs --max-procs=16 --max-args=1 command_to_convert_one < pathlist_file
Alternatively, to list all files recursively (just in case), you can use find Main_dir -type f -name '*.npy'.
convert all these
.npyfiles to image
Here is the most unclear part of your question: npy files are datafiles (claimed as alternative to CSV).
There are many ways to convert them to images, such as plotting the data with matplotlib or any other library; or interpreting the data as image pixels; or anything else.
Without knowing what data is there and what you are trying to do, it is impossible to give a good answer.
0 comment threads
A possible parallel solution using imageio:
from imageio import imwrite
from multiprocessing import Pool
from numpy import load
from pathlib import Path
def npy2png(npyFile):
imwrite(npyFile.stem + '.png', load(npyFile))
rootPath = Path('/path/to/Main_dir')
if __name__ == '__main__':
with Pool(processes=8) as pool:
pool.map(npy2png, rootPath.glob('*npy'))
Additional arguments can be passed to imwrite; in the case of the PNG format, execute
import imageio
imageio.help(name='png')
to know which ones are available.
0 comment threads
0 comment threads