The Author Online Book Forums are Moving

The Author Online Book Forums will soon redirect to Manning's liveBook and liveVideo. All book forum content will migrate to liveBook's discussion forum and all video forum content will migrate to liveVideo. Log in to liveBook or liveVideo with your Manning credentials to join the discussion!

Thank you for your engagement in the AoF over the years! We look forward to offering you a more enhanced forum experience.

import-bot (20211) [Avatar] Offline
#1
[Originally posted by mpwright]

Hello. I need to find and replace 2,000,000 numbers in 24 files each 150mb in
size. A normal s/// method works far to slowly. Do you know any modules or
tree search algorithms that could work on files of this size and not take a
week to run! Thanks, Matt.
import-bot (20211) [Avatar] Offline
#2
Re: Search techniques
[Originally posted by dave]

> Hello. I need to find and replace 2,000,000 numbers in 24 files each 150mb in
> size. A normal s/// method works far to slowly. Do you know any modules or
> tree search algorithms that could work on files of this size and not take a
> week to run! Thanks, Matt.

Matt,

The problem probably isn't with the s/// part of your code, but with the file
accesses. Are you reading the files a line at a time? That's going to be very
slow. Is there a larger record that you count read in at one time? Or could
you just read the file 500k at a time?

Take a look at the $/ variable (either in perldoc perlvar or on pp111-114 of
my book.

Let me know if you need any more help.

Dave...