Q&A
Post

# Python looping 300 000 rows

+3
−0

Based on my last question comes new one.
How to loop over 300 000 rows and edit each row string one by one? I have a list of 11-digit numbers stored in one single column in Excel, and I need to separate the digits according to this pattern: `2-2-1-3-3`.

I use the code below to loop to test the solution for only 20 rows and it's working.

Example: `00002451018` becomes `00 00 2 451 018`.

`priceListTest` contains the column `Column1` which has these 11 digit numbers. Somehow I need to loop all over these 300 000 rows and use the `get_slices` to change the pattern for each row like from the example above and store it into the new column `New Value`.

The `for index, row` it's working very slowly when I have to use it for 300 000 rows. Maybe there is a better method, but I'm new to python.

``````for index, row in priceListTest.iterrows():
#print(index,row)
def get_slices(n, sizes, n_digits=11):
for size in sizes:
n_digits -= size

val, n = divmod(n, 10 ** n_digits)
yield f'{val:0{size}}'

n = row['Column1']
newVar = (' '.join(get_slices(n, [2, 2, 1, 3, 3])))
priceListTest.at[index,['New Value']] = newVar``````
Why does this post require moderator attention?
Why should this post be closed?

Parallel execution (1 comment)
The actual performance issue (5 comments)
Types (1 comment)
Create the function just once (3 comments)
A small note regarding MCVE (1 comment)

# Comments on Python looping 300 000 rows

A small note regarding MCVE
FoggyFinder‭ wrote 2 months ago:

Disclaimer: I don't know enough about Python to suggest alternative approaches.

Anyway, when it comes to performance Qs I'd recommend providing full working MCVE since it's easily to overlook something. It might turn out that a culprit lays in a totally different area.

This community is part of the Codidact network. We have other communities too — take a look!