Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: exim: users

How to force exim to load file with whitelist IPs into memory

 

 

exim users RSS feed   Index | Next | Previous | View Threaded


no.spa at o2

Oct 30, 2011, 2:51 AM

Post #1 of 9 (775 views)
Permalink
How to force exim to load file with whitelist IPs into memory

Hello

I would like to ask, if this is possible to force exim to load files with
whitelist IPs into memory.

Currently I've setup

hosts my_whitelist = net-iplsearch;/etc/my_whitelist

and I access this via

!hosts = +my_witelist

but when I check stat of file /etc/my_whitelist I get information that this
file was accessed when exim checked for data in this file.

Because this file is 10kb and its size is growing up, I would like to know
if this is possible to somehow load this file into exim memory and then exim
will check for IPs in this file without access to filesystem ? This will
reduce I/O wait on my server.

Any help would be appreciated.

Thank you
Mike Nospa


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


wbh at conducive

Oct 31, 2011, 12:09 AM

Post #2 of 9 (767 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

no.spa [at] o2 wrote:
> Hello
>
> I would like to ask, if this is possible to force exim to load files with
> whitelist IPs into memory.
>
> Currently I've setup
>
> hosts my_whitelist = net-iplsearch;/etc/my_whitelist
>
> and I access this via
>
> !hosts = +my_witelist
>
> but when I check stat of file /etc/my_whitelist I get information that this
> file was accessed when exim checked for data in this file.
>
> Because this file is 10kb and its size is growing up, I would like to know
> if this is possible to somehow load this file into exim memory and then
> exim
> will check for IPs in this file without access to filesystem ? This will
> reduce I/O wait on my server.
>
> Any help would be appreciated.
>
> Thank you
> Mike Nospa
>
>

Not enough info.

That the file was 'accessed' does not necessarily mean that a disk-read
was involved. You'd have to look to your OS and its VM methodology,
toolsets, and stats to find that out with greater certainty.

Ordinarily, Exim, as with any other app, relies on the OS for such
housekeeping, so having the OS put the file into RAMdisk is a
possibility IF it is not already VM-cached and IF it actally represents
a significant load.

I use an ignorant flat-file that is around 23-25K for LBL, search it
many times from different acl phases... and it is such a low part of
Exim's workload that I can't be bothered even optimizing it into a CDB
or such.

YMMV

Bill
--
韓家標

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


exim-users at spodhuis

Oct 31, 2011, 12:41 AM

Post #3 of 9 (752 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

On 2011-10-30 at 10:51 +0100, no.spa [at] o2 wrote:
> but when I check stat of file /etc/my_whitelist I get information that this
> file was accessed when exim checked for data in this file.

You've mounted the filesystem with "atime" support, which reports last
access time. While useful, for busy file-systems this default support
in Unix has proven to be a historical mistake (in my opinion).

The only reason the disk is being affected here is because the act of
reading the file is updating the inode with a new atime, and this needs
to be written back to the disk. Otherwise, assuming local disk and not
NFS, the file would sit in buffer cache and all new reads would never go
to disk, because the cache would still be valid.

Unless your system has so little RAM and the file is accessed so
infrequently that it doesn't remain in cache. But in that case the
extra delay loading the file in should be insignificant in the scheme of
things.

Mount the filesystem noatime or move the file to a filesystem which is
mounted noatime and see how that affects performance.

If a linear scan of a small (<200kB) file which is rarely modified and
so sits permanently is cache is too slow, there are file-formats for
rarely-modified data which reduce the amount of the "disk" file to be
read (so reduce the amount of RAM scanned through when using the cache).
CDB is probably the way to go then.

-Phil

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


wbh at conducive

Oct 31, 2011, 3:40 AM

Post #4 of 9 (753 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

no.spa [at] o2 wrote:
> I thought that it could be possible to load lsearch file into memory
> like exim.conf etc is loaded into memory.
>
> So I see that it is not possible, if I properly understand your statement.
>
> Mike.
>

Quite the reverse.

It is not only POSSIBLE, it is probably *automatic* and already being done.

But it is up to the Operating System's Virtual Memeory implementation -
which will load, unload, and reload it (think 'swap', but not only that)
as required to honour such priorities as may arise.

Not just Exim's, but Exim AND all others.

Per Phil Pennock's note, what is confusing the issue is that you are
looking at the file *access* time. That is a 'stamp', an update to the
inode associated with such things.

It does not necessarily mean that the file itself, or even one byte of
it, was loaded or unloaded from/to disk. Primary OR swap.

You only need to put a file into RAMdisk to force it to stay in one
place IF you are convnced you know more about such things than the file
system and Virtual Memory designers.

Trust the experts - it is less work to just mount 'noatime'.

And even LESS work to simply not bother yourself with it at all.

The *queue* is generally your only/most significant resource-sensitive
portion of the file system.

And even that JFW in nearly all cases. By the time one has enough load
to stress the queue, load balancing to a second box is a better idea
than fiddling, simply so you can keep half your users off the phone when
a CPU fan or such quits.

If you want to make a difference, simply insure your WL and BL are as
terse as practical, checked only when they must be, and that searches
utilize as clean and 'light' a syntax as will get the job done.

If you - and your correspondents - are doing everything ELSE right, you
shouldn't need even 50 IP's in a WL anyway.

I've never needed but half that, and usually a quarter or less.

My BL, OTOH, is large primarily so as to NOT have to make a *remote*
callout to published RBL's on always-known-bad offenders. Unless
recently cached, those always take far longer than a stroll through
local RAM.

Also 25K 'large' because I only clean it up about once in every six to
ten years...

There are many OTHER things that will give you a better payback for time
invested.

Bill
--
韓家標

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


jh at plonk

Nov 2, 2011, 5:14 AM

Post #5 of 9 (750 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

Phil Pennock, 2011-10-31 08:41:

> You've mounted the filesystem with "atime" support, which reports last
> access time. While useful, for busy file-systems this default support
> in Unix has proven to be a historical mistake (in my opinion).
>
> The only reason the disk is being affected here is because the act of
> reading the file is updating the inode with a new atime, and this needs
> to be written back to the disk. Otherwise, assuming local disk and not
> NFS, the file would sit in buffer cache and all new reads would never go
> to disk, because the cache would still be valid.

I second your opinion. atime is kind of a broken concept. OTOH, atime
updates are cached (AFAIK), so the impact should only be visible on very
busy systems with little RAM.

> Mount the filesystem noatime or move the file to a filesystem which is
> mounted noatime and see how that affects performance.

btw, Linux uses relatime by default since 2.6.30.

> CDB is probably the way to go then.

CDB (and DBM etc.) cannot be used with iplsearch. We would need a trie
data structure for that. Don't know if there are widely used standard
formats and tools for that...

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


savage at savage

Nov 2, 2011, 5:21 AM

Post #6 of 9 (747 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

Put the text file on a small md partition? :)


On Wed, Nov 2, 2011 at 2:14 PM, Jakob Hirsch <jh [at] plonk> wrote:
> Phil Pennock, 2011-10-31 08:41:
>
>> You've mounted the filesystem with "atime" support, which reports last
>> access time. While useful, for busy file-systems this default support
>> in Unix has proven to be a historical mistake (in my opinion).
>>
>> The only reason the disk is being affected here is because the act of
>> reading the file is updating the inode with a new atime, and this needs
>> to be written back to the disk. Otherwise, assuming local disk and not
>> NFS, the file would sit in buffer cache and all new reads would never go
>> to disk, because the cache would still be valid.
>
> I second your opinion. atime is kind of a broken concept. OTOH, atime
> updates are cached (AFAIK), so the impact should only be visible on very
> busy systems with little RAM.
>
>> Mount the filesystem noatime or move the file to a filesystem which is
>> mounted noatime and see how that affects performance.
>
> btw, Linux uses relatime by default since 2.6.30.
>
>> CDB is probably the way to go then.
>
> CDB (and DBM etc.) cannot be used with iplsearch. We would need a trie
> data structure for that. Don't know if there are widely used standard
> formats and tools for that...
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>



--

Regards,
Chris Knipe

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


exim-users at spodhuis

Nov 2, 2011, 8:31 PM

Post #7 of 9 (742 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

On 2011-11-02 at 13:14 +0100, Jakob Hirsch wrote:
> CDB (and DBM etc.) cannot be used with iplsearch. We would need a trie
> data structure for that. Don't know if there are widely used standard
> formats and tools for that...

I would be very interested in knowing of such a thing. One of the items
already on my "work on Exim at some point" list of features is:

Item:
disk stable format write-infrequent read-often (a la CDB) binary trie storage
designed to be useful for IP lookups against tables (PF style) to avoid walking lists.

Regards,
-Phil

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


jh at plonk

Nov 3, 2011, 3:35 AM

Post #8 of 9 (752 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

no.spa [at] o2, 2011-10-30 10:51:

> I would like to ask, if this is possible to force exim to load files with
> whitelist IPs into memory.

You want exim to load lookup files once at startup? That's currently not
possible. It also kind of contradicts the "dynamic" nature of lookup
files, where you can update them and let exim use the new content
without restart.

What you can do is to use Exim's .include statement, e.g.

hostlist my_whitelist = \
.include /etc/my_whitelist

The file is included as-is, so it must adhere hostlist syntax, .e.g:

10.0.0.0/8 : \
192.168.1.0/24


Apart from this: 10kb is not big and easy to keep in to OS's fs cache.
net-iplsearch costs some CPU cycles, but Exim is usually IO-bound, not
CPU bound. So I don't think this will help much with your (guessed) IO
wait problem.


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


jh at plonk

Nov 3, 2011, 5:00 AM

Post #9 of 9 (739 views)
Permalink
Re: How to force exim to load file with whitelist IPs into memory [In reply to]

Phil Pennock, 2011-11-03 04:31:

> I would be very interested in knowing of such a thing. One of the items
> already on my "work on Exim at some point" list of features is:
>
> Item:
> disk stable format write-infrequent read-often (a la CDB) binary trie storage
> designed to be useful for IP lookups against tables (PF style) to avoid walking lists.

I did not look much for it, but a quick search turned up some C libs for
that.
I guess this is something nice to have, but I would not expect much from
it. We usually don't have to look up thousands or millions of IP
addresses like a router or firewall has to.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.