Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: SpamAssassin: devel

Rule updates are too old

 

 

First page Previous page 1 2 3 4 Next page Last page  View All SpamAssassin devel RSS feed   Index | Next | Previous | View Threaded


darxus at chaosreigns

Aug 19, 2012, 9:00 AM

Post #76 of 85 (416 views)
Permalink
Rule updates are too old [In reply to]

SpamAssassin version 3.3.0 has not had a rule update since 2012-08-18.
SpamAssassin version 3.3.1 has not had a rule update since 2012-08-18.
SpamAssassin version 3.3.2 has not had a rule update since 2012-08-18.

20120818: Spam: 596883, Ham: 209684


KMcGrail at PCCC

Aug 19, 2012, 9:02 AM

Post #77 of 85 (415 views)
Permalink
Re: Rule updates are too old [In reply to]

This might be a bug. I think we have published. Are you checking with dns or how are you checking?
Regards,
KAM

darxus [at] chaosreigns wrote:

SpamAssassin version 3.3.0 has not had a rule update since 2012-08-18.
SpamAssassin version 3.3.1 has not had a rule update since 2012-08-18.
SpamAssassin version 3.3.2 has not had a rule update since 2012-08-18.

20120818: Spam: 596883, Ham: 209684


darxus at chaosreigns

Aug 31, 2012, 7:57 AM

Post #78 of 85 (385 views)
Permalink
Re: Rule updates are too old [In reply to]

Any chance the recent lack of updates while we've apparently had enough
corpora are related to this rsync permissions problem you found not long
ago? Did the scripts get updated to fix it automatically?

On 08/14, Kevin A. McGrail wrote:
> On 8/14/2012 12:07 PM, darxus [at] chaosreigns wrote:
> >On 08/14, Axb wrote:
> >>On 08/14/2012 06:00 PM, darxus [at] chaosreigns wrote:
> >>>SpamAssassin version 3.3.0 has not had a rule update since 2012-08-13.
> >>>20120813: Spam: 748887, Ham: 185653
> >>We have +150k ham and still not happy?
> >Yeah, that's weird. What, other than minimum corpora thresholds, prevented
> >it from updating? (Or is there a weird problem with my alerting script?)
>
> Your script is very useful. And I fill my mailbox with cron
> reports right now from SA.
>
> Here's the issue:
>
> rsync: send_files failed to open "/export/home/corpus-rsync/corpus/spam-bpoliakoff.log": Permission denied (13)
> spam-danmcdonald.log
> spam-darxus.log
> spam-grenier.log
> spam-jarif.log
> spam-kpg-gah.log
> spam-llanga.log
> spam-mas-mas.log
>
> sent 1595302563 bytes received 980 bytes 25524856.69 bytes/sec
> total size is 7336218720 speedup is 4.60
> rsync error: some files could not be transferred (code 23) at main.c(692)
> Exit Status 23 is not zero for do-nightly-rescore-example
>
> I fixed with a chmod and re-ran but need to update the scripts as
> well to fix this automatically.
>
> regards,
> KAM
>

--
"Think, or I will set you on fire."
http://www.ChaosReigns.com


KMcGrail at PCCC

Aug 31, 2012, 1:43 PM

Post #79 of 85 (376 views)
Permalink
Re: Rule updates are too old [In reply to]

On 8/31/2012 10:57 AM, darxus [at] chaosreigns wrote:
> Any chance the recent lack of updates while we've apparently had enough
> corpora are related to this rsync permissions problem you found not long
> ago? Did the scripts get updated to fix it automatically?
No, I'm virtually positive it has to do with the missed corpora for
weekends. For reasons I don't know, the mass check algorithms use both
daily and weekly together for every single build. If the weekend builds
fail to get enough corpora, it's definitely another week until we
generate rules.

Regards,
KAM


darxus at chaosreigns

Aug 31, 2012, 1:47 PM

Post #80 of 85 (371 views)
Permalink
Re: Rule updates are too old [In reply to]

On 08/31, Kevin A. McGrail wrote:
> On 8/31/2012 10:57 AM, darxus [at] chaosreigns wrote:
> >Any chance the recent lack of updates while we've apparently had enough
> >corpora are related to this rsync permissions problem you found not long
> >ago? Did the scripts get updated to fix it automatically?
> No, I'm virtually positive it has to do with the missed corpora for
> weekends. For reasons I don't know, the mass check algorithms use
> both daily and weekly together for every single build. If the
> weekend builds fail to get enough corpora, it's definitely another
> week until we generate rules.

Ah. I used to think that, but then there was a time it worked anyway....
Huh.

--
"Life is either a daring adventure or it is nothing at all."
- Helen Keller
http://www.ChaosReigns.com


axb.lists at gmail

Aug 31, 2012, 1:56 PM

Post #81 of 85 (369 views)
Permalink
Re: Rule updates are too old [In reply to]

On 08/31/2012 10:43 PM, Kevin A. McGrail wrote:
> On 8/31/2012 10:57 AM, darxus [at] chaosreigns wrote:
>> Any chance the recent lack of updates while we've apparently had enough
>> corpora are related to this rsync permissions problem you found not long
>> ago? Did the scripts get updated to fix it automatically?
> No, I'm virtually positive it has to do with the missed corpora for
> weekends. For reasons I don't know, the mass check algorithms use both
> daily and weekly together for every single build. If the weekend builds
> fail to get enough corpora, it's definitely another week until we
> generate rules.

Within tomorrow's masscheck window, I'll be replacing some backbone
network components but hope to make it on time to run a --net masscheck
with a smaller corpus.


KMcGrail at PCCC

Aug 31, 2012, 3:57 PM

Post #82 of 85 (372 views)
Permalink
Re: Rule updates are too old [In reply to]

On 8/31/2012 4:47 PM, darxus [at] chaosreigns wrote:
> On 08/31, Kevin A. McGrail wrote:
>> On 8/31/2012 10:57 AM, darxus [at] chaosreigns wrote:
>>> Any chance the recent lack of updates while we've apparently had enough
>>> corpora are related to this rsync permissions problem you found not long
>>> ago? Did the scripts get updated to fix it automatically?
>> No, I'm virtually positive it has to do with the missed corpora for
>> weekends. For reasons I don't know, the mass check algorithms use
>> both daily and weekly together for every single build. If the
>> weekend builds fail to get enough corpora, it's definitely another
>> week until we generate rules.
> Ah. I used to think that, but then there was a time it worked anyway....
> Huh.

Based on

do-nightly-rescore-example...

This logic leads me to believe that the weekly and daily scores are used every single day. Only once a week they change orders for which one is generated first.


if [ `date +%w` -eq 0 ]; then
./generate-new-scores 0
./generate-new-scores 1
REVISION=`grep "revision .*" scores-set1 | cut -d" " -f9`
else
./generate-new-scores 1
./generate-new-scores 0
REVISION=`grep "revision .*" scores-set0 | cut -d" " -f9`
fi

A failure on either weekly or the daily exits. So the current failure
for the weekly scan fails every daily until the next week.

I could be wrong but that's how I read it and the stuff I added to the
cron output leads me to believe that my hypothesis is accurate.

regards,
KAM


KMcGrail at PCCC

Aug 31, 2012, 3:57 PM

Post #83 of 85 (375 views)
Permalink
Re: Rule updates are too old [In reply to]

On 8/31/2012 4:56 PM, Axb wrote:
> Within tomorrow's masscheck window, I'll be replacing some backbone
> network components but hope to make it on time to run a --net
> masscheck with a smaller corpus.
Excellent. That will help a lot I'm sure!


darxus at chaosreigns

Sep 1, 2012, 8:44 AM

Post #84 of 85 (361 views)
Permalink
Re: Rule updates are too old [In reply to]

This was when an update got generated when the last weekly set appeared to
below the ham threshold. So I removed the check on the weekly set.
That weekly set (20120623) still says 119126 hams (below threshold of
150000), if you click "source details" (you'll need to ctrl-f search
the page for that).

So, why did we get an update on 20120629 when the last weekly update was
below threshold?

2012-06-27 3.3.0 1352457
2012-06-27 3.3.1 1352457
2012-06-27 3.3.2 1352457
2012-06-28 3.3.0 1354373
2012-06-28 3.3.1 1354373
2012-06-28 3.3.2 1354373
2012-06-29 3.3.0 1354859
2012-06-29 3.3.1 1354859
2012-06-29 3.3.2 1354859
2012-06-30 3.3.0 1355272
2012-06-30 3.3.1 1355272
2012-06-30 3.3.2 1355272

Requiring both the current daily and latest weekly to be above the
threshold makes sense to me, and as I said, it's what I previously believed
was required. I just can't explain why it worked in this case, if that's
required.

Any ideas? Should I re-enable the latest weekly check in my script?

On 06/30, darxus [at] chaosreigns wrote:
> On 06/30, darxus [at] chaosreigns wrote:
> > 20120629: Spam: 260720, Ham: 219330
>
> Woo.
>
> > 20120623: Spam or ham is below threshold of 150,000: http://ruleqa.spamassassin.org/?daterev=20120623
> > 20120623: Spam: 246602, Ham: 119126
>
> Removed this threshold test for the last Saturday / net run, since it
> doesn't appear to be relevant.
>
> --
> "It is better to die on your feet than to live on your knees."
> - Emiliano Zapata, Mexican Revolution Leader
> http://www.ChaosReigns.com
>

--
"For every battle there is a price to pay. Now pick up your teeth and
go home." - no fear
http://www.ChaosReigns.com


KMcGrail at PCCC

Sep 3, 2012, 8:58 PM

Post #85 of 85 (358 views)
Permalink
Re: Rule updates are too old [In reply to]

On 9/1/2012 11:44 AM, darxus [at] chaosreigns wrote:
> This was when an update got generated when the last weekly set appeared to
> below the ham threshold. So I removed the check on the weekly set.
> That weekly set (20120623) still says 119126 hams (below threshold of
> 150000), if you click "source details" (you'll need to ctrl-f search
> the page for that).
> ...
>
> Requiring both the current daily and latest weekly to be above the
> threshold makes sense to me, and as I said, it's what I previously believed
> was required. I just can't explain why it worked in this case, if that's
> required.
Digging into why it worked over a month ago with the number of changes
that have been made in the meantime is just too hard. These scripts are
complex and intertwined plus require distributed submissions so I'm
doing my best to keep a slow and steady progress forward
> Any ideas? Should I re-enable the latest weekly check in my script?
Personally, I'm a fan of your scripts and I think that more transparency
on these processes is good. Send them to ruleqa perhaps?

I thin kwe had two issues though. One a lack of corpora for the weekend
set that bombed things for a week and then the eval code the blew up
things since.

Regards,
KAM

First page Previous page 1 2 3 4 Next page Last page  View All SpamAssassin devel RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.