Mailing List Archive: SpamAssassin: devel

__FILL_THIS_FORM_SHORT* rule slowness

Index | Next | Previous | View Threaded

axb.lists at gmail

Feb 27, 2012, 6:47 AM

Post #1 of 11 (384 views)
Permalink

*__FILL_THIS_FORM_SHORT rule slowness**

These two rules seem to use as lot of processing time:

__FILL_THIS_FORM_SHORT2 0.8015
__FILL_THIS_FORM_LONG2 0.7652

nearly a second/rule which is only used in metas seems a bit heavy.

John, could you optimize them somewhat?

Thx

jhardin at impsec

Feb 27, 2012, 7:11 AM

Post #2 of 11 (363 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On Mon, 27 Feb 2012, Axb wrote:

> These two rules seem to use as lot of processing time:
>
> __FILL_THIS_FORM_SHORT2 0.8015
> __FILL_THIS_FORM_LONG2 0.7652
>
> nearly a second/rule which is only used in metas seems a bit heavy.
>
> John, could you optimize them somewhat?

Is this consistent, or do you have specific messages where it's unusually
slow? There are some edge cases that have triggered bad behavior in the
past that I have corrected for.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin [at] impsec FALaholic #11174 pgpk -a jhardin [at] impsec
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
15 days until Albert Einstein's 133rd Birthday

axb.lists at gmail

Feb 27, 2012, 7:19 AM

Post #3 of 11 (362 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On 02/27/2012 04:11 PM, John Hardin wrote:
> On Mon, 27 Feb 2012, Axb wrote:
>
>> These two rules seem to use as lot of processing time:
>>
>> __FILL_THIS_FORM_SHORT2 0.8015
>> __FILL_THIS_FORM_LONG2 0.7652
>>
>> nearly a second/rule which is only used in metas seems a bit heavy.
>>
>> John, could you optimize them somewhat?
>
> Is this consistent, or do you have specific messages where it's
> unusually slow? There are some edge cases that have triggered bad
> behavior in the past that I have corrected for.

I don't have any saved sample I could hand out but I mostly see it in
large msgs from corp users / lots of replies .
(ppl using Email as IM)
The common denominator is Outlook/Exchnage and many "parts"

darxus at chaosreigns

Feb 27, 2012, 4:59 PM

Post #4 of 11 (358 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On 02/27, Axb wrote:
> These two rules seem to use as lot of processing time:
>
> __FILL_THIS_FORM_SHORT2 0.8015
> __FILL_THIS_FORM_LONG2 0.7652

Please remind me, how can I check these times?

--
"Wash daily from nose-tip to tail-tip; drink deeply, but never too deep;
And remember the night is for hunting, and forget not the day is for sleep."
- The Law of the Jungle, Rudyard Kipling
http://www.ChaosReigns.com

jhardin at impsec

Feb 27, 2012, 8:38 PM

Post #5 of 11 (357 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On Mon, 27 Feb 2012, darxus [at] chaosreigns wrote:

> On 02/27, Axb wrote:
>> These two rules seem to use as lot of processing time:
>>
>> __FILL_THIS_FORM_SHORT2 0.8015
>> __FILL_THIS_FORM_LONG2 0.7652
>
> Please remind me, how can I check these times?

the HitFreqsRuleTiming plugin

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin [at] impsec FALaholic #11174 pgpk -a jhardin [at] impsec
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
A superior gunman is one who uses his superior judgment to keep
himself out of situations that would require the use of his
superior skills.
-----------------------------------------------------------------------
15 days until Albert Einstein's 133rd Birthday

darxus at chaosreigns

Feb 28, 2012, 6:03 AM

Post #6 of 11 (344 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On 02/27, John Hardin wrote:
> >These two rules seem to use as lot of processing time:
> >
> >__FILL_THIS_FORM_SHORT2 0.8015
> >__FILL_THIS_FORM_LONG2 0.7652

> Is this consistent, or do you have specific messages where it's
> unusually slow? There are some edge cases that have triggered bad
> behavior in the past that I have corrected for.

Somebody came into IRC a few days ago saying SA was hanging on some emails.
It turns out it was just taking "over 2 minutes". Not sure about the
hardware. But when I checked the timings on the two samples he provided, I
got this:

T __FILL_THIS_FORM_SHORT2 5.9169 5.9169 1
T __FILL_THIS_FORM_LONG2 5.9041 5.9041 1
T RAZOR2_CF_RANGE_E8_51_100 5.0010 5.0010 1
T __FILL_THIS_FORM_FRAUD_PHISH1 2.1362 2.1362 1

T __FILL_THIS_FORM_SHORT2 6.9687 6.9687 1
T __FILL_THIS_FORM_LONG2 6.8891 6.8891 1
T __FILL_THIS_FORM_FRAUD_PHISH1 2.5187 2.5187 1
T __FILL_THIS_FORM_LOAN1 1.6947 1.6947 1

The examples are here:
http://pastebin.com/RJBRQqjJ
http://pastebin.com/54pptaPb

(My timings were with an "AMD Phenom(tm) II X4 965 Processor" processor,
otherwise idle - home workstation.)

--
"A ship in a port is safe, but that's not what ships are built for."
-Grace Murray Hopper
http://www.ChaosReigns.com

axb.lists at gmail

Feb 28, 2012, 6:13 AM

Post #7 of 11 (347 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On 02/28/2012 03:03 PM, darxus [at] chaosreigns wrote:
> On 02/27, John Hardin wrote:
>>> These two rules seem to use as lot of processing time:
>>>
>>> __FILL_THIS_FORM_SHORT2 0.8015
>>> __FILL_THIS_FORM_LONG2 0.7652
>
>> Is this consistent, or do you have specific messages where it's
>> unusually slow? There are some edge cases that have triggered bad
>> behavior in the past that I have corrected for.
>
> Somebody came into IRC a few days ago saying SA was hanging on some emails.
> It turns out it was just taking "over 2 minutes". Not sure about the
> hardware. But when I checked the timings on the two samples he provided, I
> got this:
>
> T __FILL_THIS_FORM_SHORT2 5.9169 5.9169 1
> T __FILL_THIS_FORM_LONG2 5.9041 5.9041 1
> T RAZOR2_CF_RANGE_E8_51_100 5.0010 5.0010 1
> T __FILL_THIS_FORM_FRAUD_PHISH1 2.1362 2.1362 1
>
> T __FILL_THIS_FORM_SHORT2 6.9687 6.9687 1
> T __FILL_THIS_FORM_LONG2 6.8891 6.8891 1
> T __FILL_THIS_FORM_FRAUD_PHISH1 2.5187 2.5187 1
> T __FILL_THIS_FORM_LOAN1 1.6947 1.6947 1
>
> The examples are here:
> http://pastebin.com/RJBRQqjJ
> http://pastebin.com/54pptaPb
>
> (My timings were with an "AMD Phenom(tm) II X4 965 Processor" processor,
> otherwise idle - home workstation.)

THANKS! you made my day.

now: is this ReplaceTags slowness or .....

John,
what can you do to solve this? kill the rules?

jhardin at impsec

Feb 28, 2012, 6:22 AM

Post #8 of 11 (344 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On Tue, 28 Feb 2012, darxus [at] chaosreigns wrote:

> Somebody came into IRC a few days ago saying SA was hanging on some emails.
> It turns out it was just taking "over 2 minutes". Not sure about the
> hardware. But when I checked the timings on the two samples he provided, I
> got this:
>
> T __FILL_THIS_FORM_SHORT2 5.9169 5.9169 1
> T __FILL_THIS_FORM_LONG2 5.9041 5.9041 1
> T RAZOR2_CF_RANGE_E8_51_100 5.0010 5.0010 1
> T __FILL_THIS_FORM_FRAUD_PHISH1 2.1362 2.1362 1
>
> T __FILL_THIS_FORM_SHORT2 6.9687 6.9687 1
> T __FILL_THIS_FORM_LONG2 6.8891 6.8891 1
> T __FILL_THIS_FORM_FRAUD_PHISH1 2.5187 2.5187 1
> T __FILL_THIS_FORM_LOAN1 1.6947 1.6947 1
>
> The examples are here:
> http://pastebin.com/RJBRQqjJ
> http://pastebin.com/54pptaPb

I will take a look. Thanks!

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin [at] impsec FALaholic #11174 pgpk -a jhardin [at] impsec
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The difference is that Unix has had thirty years of technical
types demanding basic functionality of it. And the Macintosh has
had fifteen years of interface fascist users shaping its progress.
Windows has the hairpin turns of the Microsoft marketing machine
and that's all. -- Red Drag Diva
-----------------------------------------------------------------------
14 days until Albert Einstein's 133rd Birthday

jhardin at impsec

Feb 28, 2012, 6:27 AM

Post #9 of 11 (346 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On Tue, 28 Feb 2012, Axb wrote:

>> T __FILL_THIS_FORM_SHORT2 5.9169 5.9169 1
>> T __FILL_THIS_FORM_LONG2 5.9041 5.9041 1
>> T RAZOR2_CF_RANGE_E8_51_100 5.0010 5.0010 1
>> T __FILL_THIS_FORM_FRAUD_PHISH1 2.1362 2.1362 1
>>
>> T __FILL_THIS_FORM_SHORT2 6.9687 6.9687 1
>> T __FILL_THIS_FORM_LONG2 6.8891 6.8891 1
>> T __FILL_THIS_FORM_FRAUD_PHISH1 2.5187 2.5187 1
>> T __FILL_THIS_FORM_LOAN1 1.6947 1.6947 1
>
> THANKS! you made my day.
>
> now: is this ReplaceTags slowness or .....

Replacetags by itself is lightweight, it's just string substitution.

> John,
> what can you do to solve this? kill the rules?

Well, jeeze, you could give me a chance to look at them first... :)

You're always welcome to zero the scores of these rules if their filtering
performance for your mail stream doesn't justify their overhead. If there
is community consensus that detecting fill-in-the-form spams isn't
valuable enough to justify the overhead of running these (admittedly
complex) rules, then I can disable them in my sandbox; but I'd like
a consensus before doing that.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin [at] impsec FALaholic #11174 pgpk -a jhardin [at] impsec
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The difference is that Unix has had thirty years of technical
types demanding basic functionality of it. And the Macintosh has
had fifteen years of interface fascist users shaping its progress.
Windows has the hairpin turns of the Microsoft marketing machine
and that's all. -- Red Drag Diva
-----------------------------------------------------------------------
14 days until Albert Einstein's 133rd Birthday

axb.lists at gmail

Feb 28, 2012, 9:38 AM

Post #10 of 11 (344 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On 02/28/2012 03:27 PM, John Hardin wrote:
> On Tue, 28 Feb 2012, Axb wrote:
>
>>> T __FILL_THIS_FORM_SHORT2 5.9169 5.9169 1
>>> T __FILL_THIS_FORM_LONG2 5.9041 5.9041 1
>>> T RAZOR2_CF_RANGE_E8_51_100 5.0010 5.0010 1
>>> T __FILL_THIS_FORM_FRAUD_PHISH1 2.1362 2.1362 1
>>>
>>> T __FILL_THIS_FORM_SHORT2 6.9687 6.9687 1
>>> T __FILL_THIS_FORM_LONG2 6.8891 6.8891 1
>>> T __FILL_THIS_FORM_FRAUD_PHISH1 2.5187 2.5187 1
>>> T __FILL_THIS_FORM_LOAN1 1.6947 1.6947 1
>>
>> THANKS! you made my day.
>>
>> now: is this ReplaceTags slowness or .....
>
> Replacetags by itself is lightweight, it's just string substitution.
>
>> John,
>> what can you do to solve this? kill the rules?
>
> Well, jeeze, you could give me a chance to look at them first... :)
>
> You're always welcome to zero the scores of these rules if their
> filtering performance for your mail stream doesn't justify their
> overhead. If there is community consensus that detecting
> fill-in-the-form spams isn't valuable enough to justify the overhead of
> running these (admittedly complex) rules, then I can disable them in my
> sandbox; but I'd like a consensus before doing that.

ruleqa's overlaps should give us more data.

If we can agree a small group to review hits, I'm pretty sure we'd find
lots of cases where rules are redundant, merely by autopromoting similar
rules from various sandboxes.

To be honest with you, I'd rather we put emphasis in frequent auto rule
generation than collecting heaps of low scoring rules/metas which aren't
doing speed a favour.

jhardin at impsec

Feb 28, 2012, 12:13 PM

Post #11 of 11 (339 views)
Permalink

*Re: __FILL_THIS_FORM_SHORT rule slowness** [In reply to]

On Tue, 28 Feb 2012, Axb wrote:

> On 02/28/2012 03:27 PM, John Hardin wrote:
>
>> If there is community consensus that detecting fill-in-the-form spams
>> isn't valuable enough to justify the overhead of running these
>> (admittedly complex) rules, then I can disable them in my sandbox; but
>> I'd like a consensus before doing that.
>
> ruleqa's overlaps should give us more data.
>
> If we can agree a small group to review hits, I'm pretty sure we'd find lots
> of cases where rules are redundant, merely by autopromoting similar rules
> from various sandboxes.

Oh, definitely. I have noticed that before but haven't put a lot of effort
into reducing dupes.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin [at] impsec FALaholic #11174 pgpk -a jhardin [at] impsec
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
A superior gunman is one who uses his superior judgment to keep
himself out of situations that would require the use of his
superior skills.
-----------------------------------------------------------------------
14 days until Albert Einstein's 133rd Birthday

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads