Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: SpamAssassin: users

URIBL lookup count

 

 

SpamAssassin users RSS feed   Index | Next | Previous | View Threaded


martin at gregorie

Oct 18, 2011, 4:51 AM

Post #1 of 6 (218 views)
Permalink
URIBL lookup count

I've just been thinking about URIBL lookups, etc and realising that I
don't know how many of these an SA configuration does or how to estimate
it.

Is it correct to assume that every configured URIBL is sent a single
lookup request for every message that SA scans?


Martin


guenther at rudersport

Oct 18, 2011, 10:22 AM

Post #2 of 6 (207 views)
Permalink
Re: URIBL lookup count [In reply to]

On Tue, 2011-10-18 at 12:51 +0100, Martin Gregorie wrote:
> I've just been thinking about URIBL lookups, etc and realising that I
> don't know how many of these an SA configuration does or how to estimate
> it.
>
> Is it correct to assume that every configured URIBL is sent a single
> lookup request for every message that SA scans?

No. This message has no URIs, thus no lookup.

Furthermore, since the BL lookups are DNS, each URI needs a lookup of
its own. The only thing safe to assume is, that with multiple aggregated
lists (or listings) per URI DNSBL, there is one DNS lookup per URI and
DNSBL -- e.g. SURBL (multiple lists) or URIBL (multiple listings).


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}


martin at gregorie

Oct 18, 2011, 3:52 PM

Post #3 of 6 (207 views)
Permalink
Re: URIBL lookup count [In reply to]

On Tue, 2011-10-18 at 19:22 +0200, Karsten Bräckelmann wrote:
> On Tue, 2011-10-18 at 12:51 +0100, Martin Gregorie wrote:
> > I've just been thinking about URIBL lookups, etc and realising that I
> > don't know how many of these an SA configuration does or how to estimate
> > it.
> >
> > Is it correct to assume that every configured URIBL is sent a single
> > lookup request for every message that SA scans?
>
> No. This message has no URIs, thus no lookup.
>
> Furthermore, since the BL lookups are DNS, each URI needs a lookup of
> its own. The only thing safe to assume is, that with multiple aggregated
> lists (or listings) per URI DNSBL, there is one DNS lookup per URI and
> DNSBL -- e.g. SURBL (multiple lists) or URIBL (multiple listings).
>
>
OK, so the answer is not straight forward: thanks for confirming it.

OTOH, a fairly regular occurrence on this list is thread from people who
get problems from hitting usage limits set by various BL lookups. So, I
wonder if it would be useful for SA to log the number of BL lookups it
does: as it need only involve of writing a log message every hour or day
giving the accumulated count for the period, its performance hit should
be tiny and, of course, it should have an enable/disable configuration
parameter. Output would be a single log message containing a total for
all BL lookups or (deluxe version) a total for each configured BL.

It goes without saying that, if this is incompatible with the SA
internal structure, then forget about it.


Martin


guenther at rudersport

Oct 18, 2011, 4:29 PM

Post #4 of 6 (201 views)
Permalink
Re: URIBL lookup count [In reply to]

On Tue, 2011-10-18 at 23:52 +0100, Martin Gregorie wrote:
> On Tue, 2011-10-18 at 19:22 +0200, Karsten Bräckelmann wrote:

> > [...] there is one DNS lookup per URI and
> > DNSBL -- e.g. SURBL (multiple lists) or URIBL (multiple listings).
>
> OK, so the answer is not straight forward: thanks for confirming it.
>
> OTOH, a fairly regular occurrence on this list is thread from people who
> get problems from hitting usage limits set by various BL lookups. So, I

>From memory, these are typically "no hits" issues by private or SOHO
users, who aren't anywhere close to the free usage limits -- but using
the DNS by their (large) ISP, instead of running their own caching
nameserver. The ISPs DNS is blocked, or in one recent occurrence the
Google DNS.

I cannot remember any large system being mentioned here, whose admins
did not know they exceed the free usage limit. (The DNSBLs do tell the
admins, the limits aren't strictly enforced AFAIK, and most importantly
only the most aggressive abusers will ever get anything worse than no
responses.)

> wonder if it would be useful for SA to log the number of BL lookups it
> does: as it need only involve of writing a log message every hour or day
> giving the accumulated count for the period, its performance hit should
> be tiny and, of course, it should have an enable/disable configuration
> parameter. Output would be a single log message containing a total for
> all BL lookups or (deluxe version) a total for each configured BL.

Wouldn't grepping the DNS logs already tell the admin about it?

Keep in mind, the actual number of queries isn't relevant unless you're
at least in the general ball-park of 100,000 messages a day.


--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}


guenther at rudersport

Oct 18, 2011, 4:49 PM

Post #5 of 6 (201 views)
Permalink
Re: URIBL lookup count [In reply to]

On Wed, 2011-10-19 at 01:29 +0200, Karsten Bräckelmann wrote:
> On Tue, 2011-10-18 at 23:52 +0100, Martin Gregorie wrote:
> > On Tue, 2011-10-18 at 19:22 +0200, Karsten Bräckelmann wrote:

> > wonder if it would be useful for SA to log the number of BL lookups it
> > does: as it need only involve of writing a log message every hour or day
> > giving the accumulated count for the period, its performance hit should
> > be tiny and, of course, it should have an enable/disable configuration
> > parameter. Output would be a single log message containing a total for
> > all BL lookups or (deluxe version) a total for each configured BL.

Oh, and of course, caching applies here, too.

The number of queries SA performs does NOT tell you the number of
queries actually hitting the URI DNSBL's infrastructure. SURBL has a TTL
of 3 minutes, URIBL even uses 30 minutes.

Thus, a spam run targeting lots of your users within a short time period
will result in more queries (sent by SA) than actually ending up at the
DNSBL's mirrors.

Similar for negative caching and not-blacklisted domains frequently
observed in your mail stream.


I probably should stop replying to self, though. ;)


> Wouldn't grepping the DNS logs already tell the admin about it?
>
> Keep in mind, the actual number of queries isn't relevant unless you're
> at least in the general ball-park of 100,000 messages a day.

--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}


martin at gregorie

Oct 18, 2011, 5:48 PM

Post #6 of 6 (200 views)
Permalink
Re: URIBL lookup count [In reply to]

On Wed, 2011-10-19 at 01:29 +0200, Karsten Bräckelmann wrote:

> Keep in mind, the actual number of queries isn't relevant unless you're
> at least in the general ball-park of 100,000 messages a day.
>
Indeed: I'm not remotely near that. It was just an idea that I thought
might be useful provided it was a no-brainer to implement.

I like your log analysis though: should have thought of it myself. That
would take just a few lines of awk to implement or a little more Perl to
make it a logwatch plugin.


Martin

SpamAssassin users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.