Gossamer Forum: General: Perl Programming: What's the best way to open a large file?

Mar 12, 2001, 4:27 AM

inbuco

User (68 posts)

Mar 12, 2001, 4:27 AM

Post #1 of 9

Shortcut

What's the best way to open a large file?

I need to unzip a 2GB file on my unix server using gunzip.

I need to read each line and print only every other line.

The server is a Dual Xeon Pentium with 2GB of ram.

Is this possible?

Mar 12, 2001, 5:13 AM

Andy

Veteran / Moderator (18441 posts)

Mar 12, 2001, 5:13 AM

Post #2 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

Do you mind me asking, WHY????

Andy

webmaster@ace-installer.com
http://www.ace-installer.com

Mar 12, 2001, 7:42 AM

GClemmons

User / Moderator (314 posts)

Mar 12, 2001, 7:42 AM

Post #3 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

this is a prime example of why slurping is a bad thing.

Code:

my $t = 0;
open(LF, "your.file") or die "error opening file: $!\n";
print ++$t % 2 ? $_ : '' while (<LF>);
close(LF);

this will give you every other line starting with the first.
if you want it to start with the second, put the ++ on the other side of $t.

-g

s/(\d{2})/chr($1)/ge + print if $_ = '8284703280698276687967';

Mar 12, 2001, 8:46 AM

Paul

Veteran (19537 posts)

Mar 12, 2001, 8:46 AM

Post #4 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

Holy smokes - 2GB - thats pretty impressive - can I put my site on your server? Smile

(joking)

Paul Wilson.
http://www.wiredon.net/gt/
http://www.perlmad.com/

Mar 12, 2001, 3:12 PM

inbuco

User (68 posts)

Mar 12, 2001, 3:12 PM

Post #5 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

When I try gunzip -d it tells me that the file is too big, however, there is more than enough space on the drive. Any ideas?

Mar 13, 2001, 5:13 AM

Andy

Veteran / Moderator (18441 posts)

Mar 13, 2001, 5:13 AM

Post #6 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

Probably cos you only have 2GB of RAM, and some of that will be used on the OS (operating system), so you will probably only have about 1.95GB RAM, not really enough to open the fill fully is it? (not sure, just having an educated guess, am i right?) Cool

Andy

webmaster@ace-installer.com
http://www.ace-installer.com

Mar 13, 2001, 7:31 AM

Paul

Veteran (19537 posts)

Mar 13, 2001, 7:31 AM

Post #7 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

In Reply To:

Probably cos you only have 2GB of RAM

What on earth are you on about???????

2GIG of RAM is an AWESOME amount!!!

I think you are getting confused between RAM and disk space.

Paul Wilson.
http://www.wiredon.net/gt/
http://www.perlmad.com/

Mar 13, 2001, 8:16 AM

GClemmons

User / Moderator (314 posts)

Mar 13, 2001, 8:16 AM

Post #8 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

i'll take a guess that this is a linux box?
linux on 32 bit OSs doesn't support files larger than 2 gigs and because of that gunzip can't handle it.

try using zcat to split it into smaller sections

-g

s/(\d{2})/chr($1)/ge + print if $_ = '8284703280698276687967';

Mar 13, 2001, 10:23 AM

Andy

Veteran / Moderator (18441 posts)

Mar 13, 2001, 10:23 AM

Post #9 of 9

Shortcut

Re: What's the best way to open a large file? In reply to

Oh well...thought i would have it completley wrong...lol...thats why i said it was an educated guess....maybe not so educated as i thought Cool

Andy

webmaster@ace-installer.com
http://www.ace-installer.com