vitalyb at telenet
Apr 22, 2012, 9:58 AM
Views: 233
Permalink
|
|
PATCH reduce sa-awl memory usage
|
|
Hello,
current version of sa-awl loads full database key list to memory before showing any
stats or performing maintenance. I believe it's obvious that this behavior is
undesirable and makes large databases impossible to handle.
The patch below improves sa-awl scaling and responsiveness by scanning database
row-by-row basis instead of loading all keys to memory first.
Tested cleaning db with over 8 million rows.
For a cached db with 850K rows memory usage lowers from 1G to 6M, execution time
is around 12% slower, though.
I'm not a perl expert, please review.
Thanks.
--- sa-awl.orig 2012-04-22 18:38:55.000000000 +0300
+++ sa-awl 2012-04-22 18:59:10.527228442 +0300
@@ -82,11 +82,10 @@
or die "Cannot open file $db: $!\n";
}
-my @k = grep(!/totscore$/,keys(%h));
-for my $key (@k)
+while (my ($key, $count) = each %h)
{
+ next if $key =~ /totscore$/;
my $totscore = $h{"$key|totscore"};
- my $count = $h{$key};
next unless defined($totscore);
if ($opt_clean) {
|
PATCH reduce sa-awl memory usage
