Convert .npy files in a directory to images (.png)

A possible parallel solution using imageio:

from imageio import imwrite
from multiprocessing import Pool
from numpy import load
from pathlib import Path


def npy2png(npyFile):
    imwrite(npyFile.stem + '.png', load(npyFile))


rootPath = Path('/path/to/Main_dir')
if __name__ == '__main__':
    with Pool(processes=8) as pool:
        pool.map(npy2png, rootPath.glob('*npy'))

Additional arguments can be passed to imwrite; in the case of the PNG format, execute

import imageio
imageio.help(name='png')

to know which ones are available.

posted over 3 years ago

CC BY-SA 4.0

Patol75‭

91 reputation 0 2 9 2

fastest and most efficient method (time and memory-wise)

Time and memory go against each other in a simple way: amount of jobs divides the time (up to amount of cores, sans the overhead, assuming no I/O bottlenecks e.g. an SSD), and multiplies the peak memory used.

stored in a list named paths

Is that a Python list, or do you mean a newline-delimited file?

For the latter, an easy bash way to parallelize is xargs --max-procs=16 --max-args=1 command_to_convert_one < pathlist_file

Alternatively, to list all files recursively (just in case), you can use find Main_dir -type f -name '*.npy'.

convert all these .npy files to image

Here is the most unclear part of your question: npy files are datafiles (claimed as alternative to CSV).

There are many ways to convert them to images, such as plotting the data with matplotlib or any other library; or interpreting the data as image pixels; or anything else.

Without knowing what data is there and what you are trying to do, it is impossible to give a good answer.

posted over 3 years ago

CC BY-SA 4.0

hoverhell‭

31 reputation 0 2 3 0