Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

Post History

62%

+3 −1

Q&A Python looping 300 000 rows

Based on my last question comes new one. How to loop over 300 000 rows and edit each row string one by one? I have a list of 11-digit numbers stored in one single column in Excel, and I need to s...

1 answer · posted 2y ago by sfrow‭ · last activity 2y ago by NoahTheDuke‭

Question python performance loop

#5: Post edited by

Alexei‭ · 2021-11-10T07:52:17Z (over 2 years ago)
removed non-applicable tag

Copy Link

Raw

Markdown

python performance loops row

python performance row loop

#4: Post edited by

Alexei‭ · 2021-11-09T06:23:34Z (over 2 years ago)
added relevant tag

Copy Link

Raw

Markdown

python loops row

python performance loops row

#3: Post edited by

sfrow‭ · 2021-11-08T20:39:29Z (over 2 years ago)

Copy Link

Raw

Markdown

Based on [my last question](https://software.codidact.com/posts/284803) comes new one.
~~How to loop over 300 000 rows and edit the string one by one?~~
I have a list of 11-digit numbers stored in one single column in Excel, and I need to separate the digits according to this pattern: `2-2-1-3-3`.
I use the code below to loop to test the solution for only 20 rows and it's working.
Example: `00002451018` becomes `00 00 2 451 018`.
`priceListTest` contains the column `Column1` which has these 11 digit numbers. Somehow I need to loop all over these 300 000 rows and use the `get_slices` to change the pattern for each row like from the example above and store it into the new column `New Value`.
The `for index, row` it's working very slowly when I have to use it for 300 000 rows. Maybe there is a better method, but I'm new to python.
Thanks in advance!
```Python
for index, row in priceListTest.iterrows():
#print(index,row)
def get_slices(n, sizes, n_digits=11):
for size in sizes:
n_digits -= size
val, n = divmod(n, 10 ** n_digits)
yield f'{val:0{size}}'
n = row['Column1']
newVar = (' '.join(get_slices(n, [2, 2, 1, 3, 3])))
priceListTest.at[index,['New Value']] = newVar

Based on [my last question](https://software.codidact.com/posts/284803) comes new one.
How to loop over 300 000 rows and edit each row string one by one?
I have a list of 11-digit numbers stored in one single column in Excel, and I need to separate the digits according to this pattern: `2-2-1-3-3`.
I use the code below to loop to test the solution for only 20 rows and it's working.
Example: `00002451018` becomes `00 00 2 451 018`.
`priceListTest` contains the column `Column1` which has these 11 digit numbers. Somehow I need to loop all over these 300 000 rows and use the `get_slices` to change the pattern for each row like from the example above and store it into the new column `New Value`.
The `for index, row` it's working very slowly when I have to use it for 300 000 rows. Maybe there is a better method, but I'm new to python.
Thanks in advance!
```Python
for index, row in priceListTest.iterrows():
#print(index,row)
def get_slices(n, sizes, n_digits=11):
for size in sizes:
n_digits -= size
val, n = divmod(n, 10 ** n_digits)
yield f'{val:0{size}}'
n = row['Column1']
newVar = (' '.join(get_slices(n, [2, 2, 1, 3, 3])))
priceListTest.at[index,['New Value']] = newVar

#2: Post edited by

sfrow‭ · 2021-11-08T20:38:48Z (over 2 years ago)

Copy Link

Raw

Markdown

~~python looping 300 000 rows~~

Python looping 300 000 rows

#1: Initial revision by

sfrow‭ · 2021-11-08T20:38:30Z (over 2 years ago)

Copy Link

Raw

Markdown

python looping 300 000 rows

Based on [my last question](https://software.codidact.com/posts/284803) comes new one.<br>
How to loop over 300 000 rows and edit the string one by one?
I have a list of 11-digit numbers stored in one single column in Excel, and I need to separate the digits according to this pattern: `2-2-1-3-3`.<br>
<br>
I use the code below to loop to test the solution for only 20 rows and it's working.<br>
<br>
Example: `00002451018` becomes `00 00 2 451 018`.<br>
<br>
`priceListTest` contains the column `Column1` which has these 11 digit numbers. Somehow I need to loop all over these 300 000 rows and use the `get_slices` to change the pattern for each row like from the example above and store it into the new column `New Value`.<br>

The `for index, row` it's working very slowly when I have to use it for 300 000 rows. Maybe there is a better method, but I'm new to python.<br>
<br>
Thanks in advance!


```Python
for index, row in priceListTest.iterrows(): 
    #print(index,row)
    def get_slices(n, sizes, n_digits=11):
        for size in sizes:
            n_digits -= size
            
            val, n = divmod(n, 10 ** n_digits)
            yield f'{val:0{size}}'

    n = row['Column1']
    newVar = (' '.join(get_slices(n, [2, 2, 1, 3, 3])))
    priceListTest.at[index,['New Value']] = newVar

python loops row

Communities

Post History