davem at iabyn
Jan 2, 2010, 4:17 PM
Post #2 of 4
(367 views)
Permalink
|
|
Re: memory problem of "eval" in perl 5.10?
[In reply to]
|
|
On Thu, Nov 26, 2009 at 08:54:48PM +0100, christian4perl [at] lists wrote:
> Hello,
>
> We have a problem with the eval of the "new" perl 5.10.
>
> Before, a brief background. We have run several applications which were
> developed with design patterns, especially the state machine. In
> conjunction with a data flow logic, analogous to the Unix filter. Perl’s
> internal state of the state machine will be dumped at the end of every
> "step" (Data:: Dumper), and stored in flat files or database CLOB
> columns. Later, these dumps are reread, evaluated and then the work on
> the data will continue.
>
> Now it seems, that the "eval" from perl 5.10 are partially being very
> rude with the available memory resources. It gives a mismatch of 1:1000;
> every byte to be evaluated, needs 1Kbyte main memory. That means, the
> data which could be processed with perl 5.8.8, with perl 5.10.0 creates
> an "out of memory" (just as all makable 5.9.*). For reproducing tests,
> the script eval.pl attached. The offender (eval) is on line 65.
>
> The function memory () and expandSI () are to determine the memory
> consumption, as it sees the operating system (it works under linux
> only). So that eval.pl has something to do, we also attached the support
> script get.pl. It loads eight outer space images from the Internet and
> creates files with the changed Data::Dumper HTTP::Response instances
> (lines 72-75 of get.pl).
>
> We presume that the evaluation has a caching mechanism. Because a
> functional evaluation repeats often; no new memory is required. Even two
> different evaluations are possible repeatedly, therefore there is enough
> space for everyone alone.
>
> However, most of our application steps evaluate smaller structures. It
> never takes a long time. The computer will interactively be tough and
> will relatively quickly get the application from linux as an "out of
> memory".
>
> Can you help us?
>
> (a) Is there an error in the procedure? Although the applications run
> under perl 5.6 and 5.8 for many of years.
>
> (b) Does it require different "parameters" of the modules or compiling
> it?
>
> (c) How can we find a solution to our problem.
Short answer: when evaling the string which was read in from the file that
was created using Data::Dumper, avoid using using a variable called '$obj';
i.e. in these lines:
my $obj ;
...
$obj = <$fh> ;
...
$obj = eval $obj ;
...
return $obj ;
rename the variable, eg
my $data ;
...
$data = <$fh> ;
...
{
my $obj;
$data = eval $data ;
}
...
return $data ;
Long answer:
The file created by Data::Dumper looks roughly like:
$obj = {
content => "multi-megabyte string containing 1000s of ' chars",
foo => $obj->{bar},
}
You read the file into $obj then eval it. The "$obj->{bar}" bit makes the
string be treated as a symbolic reference, so it is treated as a variable
name where "'" is the old perl-4 equivalent of the package separator "::",
so it tries to create a new variable that is nested thousands deep
package-wise, and uses all available memory to create all the packages.
Other answer:
Don't use Data::Dumper for dumping large binary data; you're probably
better off using Storable.
p5pers:
I'm still a bit confused by the fact that the OP's code worked in 5.8.x
but not 5.10.x; all my attempts to reduce the code soon got 5.8.x eating
memory too. Also, the memory usage seems proportional to the *square* of
the number of "'" in the variable name, e.g. in the following code:
my $quotes = "'" x 4000;
$quotes->{foo};
system "ps -flyp $$";
double it to 8000 and memory usage quadruples.
Also, I'm not clear why Data::Dumper produces output like
$obj = { foo => $obj->{bar} }
since evalling this doesn't regenerate the circular reference.
--
Overhead, without any fuss, the stars were going out.
-- Arthur C Clarke
|
attachment.tgz
