Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

Review Suggested Edit

You can't approve or reject suggested edits because you haven't yet earned the Edit Posts ability.

How can I get a value as a node:stream from the Nest.js cache-manager?

I'm using Nest.js and it's @nestjs/cache-manager (using the Redis implementation), I need to read data I have stored in the cache as a stream, but I'm not entirely sure how to go about it. I can't find much of a hint in the type deceleration file for cache-manager. Has anyone done anything like this?
A little Context: I have some large data split into pages amongst multiple keys (there is lots of data), and I want to stream multiple of the pages into a pipeline that further filters all that data to a much smaller data set.
The base query that populates the data was rather complex and time intensive to run, so hence I am caching it to not have to run that query again (frequently accessed data), but now the problem is that I have a situation where there is 1.6 million records in one cache-key, and it's a memory bottleneck. So I figured streaming was in order; and I'm planning to split that large chunk of data up into many smaller pages I will place in the cache, but if I need to filter all of them again I want to filter them in a stream/pipeline.

# Question
**How can I read data from a NestJS Redis Cache as a stream?**
I'm using [Nest.js](https://nestjs.com) and its [@nestjs/cache-manager](https://www.npmjs.com/package/@nestjs/cache-manager) (using the Redis implementation), I need to read data I have stored in the cache as a stream, but I'm not entirely sure how to go about it. I can't find much of a hint in the type deceleration file for cache-manager. Has anyone done anything like this?
# Context
I have some large data split into pages amongst multiple keys (there is lots of data), and I want to stream multiple of the pages into a pipeline that further filters all that data to a much smaller data set.
The base query that populates the data was rather complex and time intensive to run, so hence I am caching it to not have to run that query again (frequently accessed data), but now I have a problem where there are 1.6 million records in one cache-key, and it's a memory bottleneck. So I figured streaming was in order, and I'm planning to split that large chunk of data up into many smaller pages I will place in the cache, but if I need to filter all of them again I want to filter them in a stream/pipeline.

node.js caching redis nest.js node:stream

Communities

Review Suggested Edit