Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Comments on How can I get a value as a node:stream from the Nest.js cache-manager?
Post
How can I get a value as a node:stream from the Nest.js cache-manager?
Question
How can I read data from a NestJS Redis Cache as a stream?
I'm using Nest.js and its @nestjs/cache-manager (using the Redis implementation), I need to read data I have stored in the cache as a stream, but I'm not entirely sure how to go about it. I can't find much of a hint in the type deceleration file for cache-manager. Has anyone done anything like this?
Context
I have some large data split into pages amongst multiple keys (there is lots of data), and I want to stream multiple of the pages into a pipeline that further filters all that data to a much smaller data set.
The base query that populates the data was rather complex and time intensive to run, so hence I am caching it to not have to run that query again (frequently accessed data), but now I have a problem where there are 1.6 million records in one cache-key, and it's a memory bottleneck. So I figured streaming was in order, and I'm planning to split that large chunk of data up into many smaller pages I will place in the cache, but if I need to filter all of them again I want to filter them in a stream/pipeline.
1 comment thread