How can I get a value as a node:stream from the Nest.js cache-manager?

I'm using Nest.js and its @nestjs/cache-manager (using the Redis implementation), I need to read data I have stored in the cache as a stream, but I'm not entirely sure how to go about it. I can't find much of a hint in the type deceleration file for cache-manager. Has anyone done anything like this?

Context

I have some large data split into pages amongst multiple keys (there is lots of data), and I want to stream multiple of the pages into a pipeline that further filters all that data to a much smaller data set.

The base query that populates the data was rather complex and time intensive to run, so hence I am caching it to not have to run that query again (frequently accessed data), but now I have a problem where there are 1.6 million records in one cache-key, and it's a memory bottleneck. So I figured streaming was in order, and I'm planning to split that large chunk of data up into many smaller pages I will place in the cache, but if I need to filter all of them again I want to filter them in a stream/pipeline.

is a duplicate

This question has been asked before and has already been answered. It should be marked as a duplicate.

Please enter the URL of the proposed duplicate in the details field below.

not constructive

This question cannot be answered in a way that is helpful to anyone. It's not possible to learn something from possible answers, except for the solution for the specific problem of the asker.

Communities

How can I get a value as a node:stream from the Nest.js cache-manager?

Question

Context

1 comment thread

0 answers