Mailing List Archive: iptables: Devel

ENOBUFS in nfq under heavy load

Index | Next | Previous | View Threaded

viraptor at gmail

Jul 17, 2007, 8:46 AM

Post #1 of 6 (1031 views)
Permalink

ENOBUFS in nfq under heavy load

Hi
After running nf_queue app under heavy load I get ENOBUFS error. Ok -
maybe not that heavy - around 100 concurrent voip calls - lots of
small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
socket returns -1, errno=ENOBUFS and when I try to destroy the queue
it stops:
stacktrace with gdb breaking...

#0 0xb7f38410 in __kernel_vsyscall ()
#1 0x4d23d0e1 in recvmsg () from /lib/libc.so.6
#2 0xb7f3185b in nfnl_talk (nfnlh=0x8061080, n=0xbfc9b5ec, peer=0,
groups=0, answer=0x0, junk=0, jarg=0x0)
at ../../libnfnetlink-0.0.25/src/libnfnetlink.c:659
#3 0xb7f35055 in __build_send_cfg_msg (h=0x80611c0, command=<value
optimized out>, queuenum=<value optimized out>, pf=0)
at ../../libnetfilter_queue-0.0.13/src/libnetfilter_queue.c:112
#4 0xb7f3511f in nfq_destroy_queue (qh=0x80611f8) at
../../libnetfilter_queue-0.0.13/src/libnetfilter_queue.c:258
#5 0x08049293 in main () at main.cc:287

Any idea how to avoid / fix it? Is the problem on my side?
My bufsize is 65535 for reading from nfq socket.

Thanks
Stan

e.leblond at inl

Jul 17, 2007, 8:56 AM

Post #2 of 6 (972 views)
Permalink

Re: ENOBUFS in nfq under heavy load [In reply to]

Hi,

Le mardi 17 juillet 2007 Ã 16:46 +0100, StanisÅ‚aw Pitucha a Ã©crit :
> Hi
> After running nf_queue app under heavy load I get ENOBUFS error. Ok -
> maybe not that heavy - around 100 concurrent voip calls - lots of
> small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
> socket returns -1, errno=ENOBUFS and when I try to destroy the queue
> it stops:
> stacktrace with gdb breaking...
>
> Any idea how to avoid / fix it?

You could simply use the limit match to avoid "heavy" load.

> Is the problem on my side?

No, it's an implementation problem.

> My bufsize is 65535 for reading from nfq socket.
>
> Thanks
> Stan

BR,
--
Ã‰ric Leblond, eleblond [at] inl
TÃ©lÃ©phone : 01 44 89 46 39, Fax : 01 44 89 45 01
INL, http://www.inl.fr

Attachments:

signature.asc (0.18 KB)

kaber at trash

Jul 17, 2007, 8:56 AM

Post #3 of 6 (983 views)
Permalink

Re: ENOBUFS in nfq under heavy load [In reply to]

Stanis³aw Pitucha wrote:
> After running nf_queue app under heavy load I get ENOBUFS error. Ok -
> maybe not that heavy - around 100 concurrent voip calls - lots of
> small udp packets. Nfq socket suddenly hangs. recv() from nfnl_fd()
> socket returns -1, errno=ENOBUFS and when I try to destroy the queue
> it stops:

Which kernel version? What exactly do you mean with "it stops"?

viraptor at gmail

Jul 17, 2007, 9:13 AM

Post #4 of 6 (974 views)
Permalink

Re: ENOBUFS in nfq under heavy load [In reply to]

Sorry for private mails - my bad.

On 7/17/07, Eric Leblond <e.leblond [at] inl> wrote:
> You could simply use the limit match to avoid "heavy" load.

Not an option - I need to balance some traffic - so it will get much
more connections than that by default :/

-----

On 7/17/07, Patrick McHardy <kaber [at] trash> wrote:
> Which kernel version? What exactly do you mean with "it stops"?

2.6.21.5 generic, stock - no patches

By stops I mean that it hangs on nfq_destroy_queue() if there was
ENOBUFS before.
When I connected gdb to it I got the stacktrace from the first post.
When I break it with ctrl+c, I get:

NFNETLINK answers: No such device

on console and program quits.

kaber at trash

Jul 17, 2007, 9:14 AM

Post #5 of 6 (980 views)
Permalink

Re: ENOBUFS in nfq under heavy load [In reply to]

Stanis³aw Pitucha wrote:
>
> On 7/17/07, Patrick McHardy <kaber [at] trash> wrote:
>> Which kernel version? What exactly do you mean with "it stops"?
>
> 2.6.21.5 generic, stock - no patches
>
> By stops I mean that it hangs on nfq_destroy_queue() if there was
> ENOBUFS before.

Please try to find out what the kernel is doing using sysrq.

> When I connected gdb to it I got the stacktrace from the first post.
> When I break it with ctrl+c, I get:
>
> NFNETLINK answers: No such device
>
> on console and program quits.
>

ENODEV can happen when sending a verdict for or unbinding a non-existant
queue.

viraptor at gmail

Jul 18, 2007, 2:47 AM

Post #6 of 6 (969 views)
Permalink

Re: ENOBUFS in nfq under heavy load [In reply to]

On 7/17/07, Patrick McHardy <kaber [at] trash> wrote:
> Stanis³aw Pitucha wrote:
> > By stops I mean that it hangs on nfq_destroy_queue() if there was
> > ENOBUFS before.
>
> Please try to find out what the kernel is doing using sysrq.

Right... not the kernel really :) - libnfnetlink stops at
libnfnetlink-0.0.25/src/libnfnetlink.c:659 (in nfnl_talk()). It's just
waiting for a datagram from netfilter - recvmsg(nfnlh->fd, &msg, 0).
So kernel as far as I can tell... forgot about that socket completely
- so nothing happens. I can't see any calls to NF happening after
that. If there's any possibility to get NF in debug mode, I can
recompile and post it - is there?
But I don't know if that's recoverable at all - should I be trying to
close queue after ENOBUFS? Maybe the socket is closed by design.

> ENODEV can happen when sending a verdict for or unbinding a non-existant
> queue.

http://pastebin.ca/624472
here I get ENOBUFS on read on line 4, so it breaks out of loop and
tries to clean up.

Additionally - I tried to force opening new queue after receiving
ENOBUFS. It does one or two successful reads and throws another
ENOBUFS.

Anyone knows how to solve this? Or should I just write own module?...

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads