It would be more useful to split this into two questions:

1. How to filter rows based on a condition
2. How to do a cumsum

Combining them makes things more confusing, especially to people who want to know only one and not the other.

Also, the following code is simpler:

# Part 1: Filter
filtered = [i for i in all_data if i["value"] > i["threshold"]]

# Part 2: Cumsum
s = 0
for i in filtered:
      s += i["value"]
      i["cumsum"] = s

you have misinterpreted the question; the condition for the cumsum involves different rows for "value" and "threshold", therefore the filter and cumsum cannot be split into two parts, as it would lead to the incorrect result of "3, 4, 6" instead of "3, 1, 3" as expected. Further, and this may not have been your intention, but the question specifies apache-spark/pyspark so plain python code cannot be the answer as it would be inefficient, although it can be used to clarify the question.

Oh, I see now. You're not taking a cumulative sum, you're taking a sum of all values below each threshold. So there's a self-join.

You're right, my Spark-less code won't work. The one you have in the question will.

You should remove mentions of "cumulative sum" because that's not what you are trying to do.

Still 2 questions btw:

1. Self-join all values that are more than threshold.
2. Sum them (with a group by).

I don't think it's necessarily those two questions, since there are alternate solutions that don't involve a self-join and sum (for example, 2 nested for loops, although possibly less efficient, would be a solution too).