Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Misc

PF_RING svn r4697 lockup

 

 

NTop misc RSS feed   Index | Next | Previous | View Threaded


c.d.wakelin at reading

Jun 29, 2011, 7:37 AM

Post #1 of 2 (334 views)
Permalink
PF_RING svn r4697 lockup

Just trying the latest SVN on a test machine and it locked solid (remote
access was impossible, and we had to power-cycle it) on one tcpdump
session (others had worked OK, previously). The machine is running
Ubuntu 10.04 64-bit on a Dell PowerEdge server.

Here's the kernel log in case it means something :)

...

> Jun 29 14:31:05 vinms4 kernel: [78982.301876] BUG: soft lockup - CPU#3 stuck for 61s! [tcpdump:1720]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] Modules linked in: e1000e fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper bnx
> 2 drm pf_ring i5000_edac dell_wmi psmouse i2c_algo_bit ioatdma dcdbas edac_core lp parport i5k_amb serio_raw dca shpchp raid10 raid456 async_pq async_xor xor
> async_memcpy async_raid6_recov mptsas mptscsih raid6_pq async_tx mptbase scsi_transport_sas raid1 raid0 multipath linear [last unloaded: e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] CPU 3:
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] Modules linked in: e1000e fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper bnx
> 2 drm pf_ring i5000_edac dell_wmi psmouse i2c_algo_bit ioatdma dcdbas edac_core lp parport i5k_amb serio_raw dca shpchp raid10 raid456 async_pq async_xor xor
> async_memcpy async_raid6_recov mptsas mptscsih raid6_pq async_tx mptbase scsi_transport_sas raid1 raid0 multipath linear [last unloaded: e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] Pid: 1720, comm: tcpdump Not tainted 2.6.32-32-server #62-Ubuntu PowerEdge 1950
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] RIP: 0010:[<ffffffffa016d549>] [<ffffffffa016d549>] skb_ring_handler+0x409/0xee0 [pf_ring]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] RSP: 0018:ffff8800058c3a70 EFLAGS: 00000246
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] RAX: 0000000000000001 RBX: ffff8800058c3ce0 RCX: 000000000000002e
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800058c3c58
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] RBP: ffffffff81013cb3 R08: 7203687475610402 R09: 0000000000000000
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800058c39f0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] R13: 000000000000000e R14: ffff880129cdd500 R15: ffffffff8155f2cc
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] FS: 0000000000000000(0000) GS:ffff8800058c0000(0000) knlGS:0000000000000000
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] CR2: 00007fbaeb84d6f0 CR3: 0000000001001000 CR4: 00000000000006e0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] Call Trace:
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] <IRQ> [<ffffffff8105a2d7>] ? entity_tick+0x27/0x140
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81055cc3>] ? find_busiest_group+0x273/0xb60
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffffa030e26e>] ? e1000_clean_rx_irq+0x36e/0x440 [e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffffa030a5f8>] ? e1000_receive_skb+0xb8/0xf0 [e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffffa030e18b>] ? e1000_clean_rx_irq+0x28b/0x440 [e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffffa030b700>] ? e1000_poll+0x240/0x560 [e1000e]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8147661f>] ? net_rx_action+0x10f/0x250
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8106f1e7>] ? __do_softirq+0xb7/0x1f0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff810c6440>] ? handle_IRQ_event+0x60/0x170
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff810142ec>] ? call_softirq+0x1c/0x30
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81015cb5>] ? do_softirq+0x65/0xa0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8106efe5>] ? irq_exit+0x85/0x90
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8155f1e5>] ? do_IRQ+0x75/0xf0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81013b13>] ? ret_from_intr+0x0/0x11
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] <EOI> [<ffffffffa0167166>] ? ring_release+0xf6/0x7d0 [pf_ring]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffffa016714f>] ? ring_release+0xdf/0x7d0 [pf_ring]
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81464809>] ? sock_release+0x29/0x90
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81464887>] ? sock_close+0x17/0x30
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8114831f>] ? __fput+0xdf/0x1f0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81148455>] ? fput+0x25/0x30
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff811445cd>] ? filp_close+0x5d/0x90
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81069d3f>] ? put_files_struct+0x7f/0xf0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff81069e04>] ? exit_files+0x54/0x70
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8106c23b>] ? do_exit+0x15b/0x390
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8106c4c5>] ? do_group_exit+0x55/0xd0
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff8106c557>] ? sys_exit_group+0x17/0x20
> Jun 29 14:31:05 vinms4 kernel: [78982.301878] [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
> Jun 29 14:32:11 vinms4 kernel: [79047.801876] BUG: soft lockup - CPU#3 stuck for 61s! [tcpdump:1720]
...

I was hoping to provide some evidence of PF_RING tcpdump picking up
random packets at the start of a session (but only with VLAN-tagged
packets, I think). I was using SVN rather than 4.7.0 as I thought the
patch in r4684 looked relevant.

Best Wishes,
Chris

--
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin, c.d.wakelin [at] reading
IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908
Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


branchnetconsulting at gmail

Jun 29, 2011, 1:09 PM

Post #2 of 2 (331 views)
Permalink
Re: PF_RING svn r4697 lockup [In reply to]

Chris,

This looks much like the bug I registered recently. See it at:
https://www.ntop.org/bugzilla3/show_bug.cgi?id=116

I was also running the latest SVN of PF_RING on Ubuntu 10.04 64-bit on a
Dell PowerEdge, though I've seen the same issue with 64-bit CentOS 5.6.
Your kernel log is more detailed that what I've been able to come up
with so far. Maybe it will help. I took the liberty to add your
comments to BugZilla.

Kevin


On 6/29/2011 10:37 AM, Chris Wakelin wrote:
> Just trying the latest SVN on a test machine and it locked solid (remote
> access was impossible, and we had to power-cycle it) on one tcpdump
> session (others had worked OK, previously). The machine is running
> Ubuntu 10.04 64-bit on a Dell PowerEdge server.
>
> Here's the kernel log in case it means something :)
>

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

NTop misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.