Gossamer Forum: General: Perl Programming: Whys this take so long to run?

Nov 7, 2003, 8:41 AM

Andy

Veteran / Moderator (18436 posts)

Nov 7, 2003, 8:41 AM

Post #1 of 3

Shortcut

Whys this take so long to run?

Hi. Can anyone give me any pointers as to why this is so slow to run the updates?

Code:
open(GRAB, "Hotel_Description_Cut_1.txt") or die "Error reading Hotel_Description_Cut_1.txt. Reason: $!"; 
    my ($old_id,$image_name,$field); 
    my $i = 0; 
    my $count = 0; 
    while (<GRAB>) { 

         chomp; 

         my $_update; 
         my @cut = split /\|/, $_; 

        # skip the beginning line....  
         if (/HotelID/) { next; }          

         #print "$count: HotelID: " . $cut[0] . "\n"; 

         unless ($count % 10) { print "> $count\n"; } 

         my $hotel_id = $cut[0]; 
         my $desc     = $cut[1]; 

         undef @cut; 

         $count++; 

       if ($DB->table('Links')->count( { HotelID => $hotel_id } )) {   
         $DB->table('Links')->update( { Description => $desc } , { HotelID => $hotel_id } ) or die $GT::SQL::error; 
       } 

     } 
    close(GRAB);

It is taking about 1 minute to complete 160 entries, which is far too slow. There are almost 50,000 to run... which would take about 5 1/2 hours!

I just can't see why its going so slow.

Anyone got any ideas? Unsure

Cheers

Andy (mod)
andy@ultranerds.co.uk

Nov 7, 2003, 9:07 AM

fuzzy logic

Enthusiast (854 posts)

Nov 7, 2003, 9:07 AM

Post #2 of 3

Shortcut

Re: [Andy] Whys this take so long to run? In reply to

I'd love to help but my upstairs neighbor has been playing this horrible Indian music on her stereo for the past 2 hours and it's making my head hurt.

Philip
------------------
Limecat is not pleased.

Nov 7, 2003, 9:08 AM

Andy

Veteran / Moderator (18436 posts)

Nov 7, 2003, 9:08 AM

Post #3 of 3

Shortcut

Re: [Andy] Whys this take so long to run? In reply to

I don't know how, or why this is possible... but using the below code seems to have solved the problem;

Code:
    open(GRAB, "Hotel_Description.txt") or die "Error reading Hotel_Description.txt. Reason: $!"; 
    my ($old_id,$image_name,$field); 
    my $i = 0; 
    my $count = 0; 
    my $pushed; 
    my @_ids; 
    while (<GRAB>) { 

         chomp; 

         my $_update; 
         my @cut = split /\|/, $_; 

        # skip the beginning line....  
         if (/HotelID/) { next; }          

         push(@_ids,$cut[0]); 
         $pushed->{$cut[0]} = $cut[1]; 

         unless($count % 1000) { print "Reading IDs group: $count \n"; } 

         $count++; 

     } 
    close(GRAB);  


    my $count = 0; 
    foreach (@_ids) { 

         if ($DB->table('Links')->count( { ID => $_ } )) {   
           $DB->table('Links')->update( { Description => $pushed->{$_} } , { ID => $_ } ) or die $GT::SQL::error; 
         } 
         unless($count % 100) { print "Writing: $count \n"; } 
         $count++; 

    }

It now completes in about 5 minutes! Cool

Cheers

Andy (mod)
andy@ultranerds.co.uk