Mailing List Archive: Analog: Help

Getting slow

Index | Next | Previous | View Threaded

os at sime

Dec 3, 1998, 3:53 AM

Post #1 of 4 (505 views)
Permalink

Getting slow

Hi all!

I am using Analog 3.11 on a Linux system.

Everything works fine until I try to process larger quantities of log lines.

After processing about 500.000 lines the whole system is getting *really*
slow almost halting my web server - even a nice/renice doesn't help. DNS
lookups are already turned off.

Any idea what to do?

- Otto -

---------------------------------------------------------
Otto J. Simon CEO of simon media GmbH
Andreas-Hofer-Platz 9 A-8010 Graz / Austria, Europe
vox: +43/316/813 8240 fax: +43/316/813 8246
os [at] sime http://www.sime.com
---------------------------------------------------------

--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------

sret1 at mathstat

Dec 3, 1998, 6:49 AM

Post #2 of 4 (487 views)
Permalink

Getting slow [In reply to]

On Thu, 3 Dec 1998, Otto J. Simon wrote:

> Hi all!
>
> I am using Analog 3.11 on a Linux system.
>
> Everything works fine until I try to process larger quantities of log lines.
>
> After processing about 500.000 lines the whole system is getting *really*
> slow almost halting my web server - even a nice/renice doesn't help. DNS
> lookups are already turned off.
>
> Any idea what to do?
>

Try using the LOWMEM options. See docs/lowmem.html.

--
Stephen Turner sret1 [at] cam http://www.statslab.cam.ac.uk/~sret1/
Normally: Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
Until 12/98: Dept of Math & Stats, 585 King Edward Ave, Ottawa K1N 6N5, Canada
Microsoft: Where am I allowed to go today?

--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------

mark.kaprow at gsa

Dec 3, 1998, 7:14 AM

Post #3 of 4 (486 views)
Permalink

Getting slow [In reply to]

Do you have enough RAM/SWAP? I use Analog on Solaris and have
never had a problem with large log files. My current logfile
is over 750MB. I just ran an analysis using Analog 3.0 on the
750+ MB file and it only took 11 minutes 39 seconds.

One thing I have found in general, is that if you cut the log,
and gzip it the zcat it into analog, it runs much faster.

You can write a script to copy the logfile to a safe place,
gzip it, then zcat it into analog and produce an output file.
You may or may not want to zero out the log file after you copy
it but before you gzip it.

Regards,

mark

From: "otto j. simon" <os [at] sime> AT internet on 12/03/98
12:53 PM

To: analog-help [at] lists AT internet [at] ccMTA-GEMS-CO-0
cc: (bcc: Mark S. Kaprow/KGMP/CO/GSA/GOV)

Subject: [analog-help] Getting slow

Hi all!

I am using Analog 3.11 on a Linux system.

Everything works fine until I try to process larger quantities
of log lines.

After processing about 500.000 lines the whole system is
getting *really* slow almost halting my web server - even a
nice/renice doesn't help. DNS lookups are already turned off.

Any idea what to do?

- Otto -

---------------------------------------------------------
Otto J. Simon CEO of simon media GmbH
Andreas-Hofer-Platz 9 A-8010 Graz / Austria, Europe
vox: +43/316/813 8240 fax: +43/316/813 8246
os [at] sime http://www.sime.com
---------------------------------------------------------

---------------------------------------------------------------
-----
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
---------------------------------------------------------------
-----

os at sime

Dec 4, 1998, 3:54 AM

Post #4 of 4 (486 views)
Permalink

Getting slow [In reply to]

Hi!

Thank you very much for your hint.

In fact almost every request to our webserver comes in the form

/foo-bar/foo?session-id=bar

So the following config line finally solved the problem:

ARGSEXCLUDE *

- Otto -

At 15:49 Uhr +0100 on 03.12.1998, Stephen Turner wrote:

>On Thu, 3 Dec 1998, Otto J. Simon wrote:
>
>> Hi all!
>>
>> I am using Analog 3.11 on a Linux system.
>>
>> Everything works fine until I try to process larger quantities of log lines.
>>
>> After processing about 500.000 lines the whole system is getting *really*
>> slow almost halting my web server - even a nice/renice doesn't help. DNS
>> lookups are already turned off.
>>
>> Any idea what to do?
>>
>
>Try using the LOWMEM options. See docs/lowmem.html.
>
>--
>Stephen Turner sret1 [at] cam http://www.statslab.cam.ac.uk/~sret1/
>Normally: Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
>Until 12/98: Dept of Math & Stats, 585 King Edward Ave, Ottawa K1N 6N5, Canada
> Microsoft: Where am I allowed to go today?
>
>
>--------------------------------------------------------------------
>This is the analog-help mailing list. To unsubscribe from this
>mailing list, send mail to analog-help-request [at] lists
>with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
>--------------------------------------------------------------------

---------------------------------------------------------
Otto J. Simon CEO of simon media GmbH
Andreas-Hofer-Platz 9 A-8010 Graz / Austria, Europe
vox: +43/316/813 8240 fax: +43/316/813 8246
os [at] sime http://www.sime.com
---------------------------------------------------------

--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to analog-help-request [at] lists
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads