Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NTop: Misc

PF_RING packets order problem

 

 

NTop misc RSS feed   Index | Next | Previous | View Threaded


wangsy at networkbench

May 3, 2012, 5:29 AM

Post #1 of 3 (488 views)
Permalink
PF_RING packets order problem

Hi,

In our project, we are using the PF_RING to capture packets from 82576-based adapters.
First, we use pfring_open_multichannel function, then call pfring_recv function.
For we use 4 threads to deal with 8 queues, each thread has to process 2 queues.
But we met with a problem: the packets that each thread received from the 2 queues are disordered.

For example, thread1 receive packets from queue1 and queue2.
There are 5 packets and the adapter received order is packet1, packet2, packet3, packet4 and packet5.
And packet1, packet2, packet3 are received by queue1, while packet4 and packet5 are receive by queue2.

The thread1 calls the pfring_recv(non-block, wait_for_incoming_packet is 0) to receive from queue1 and queue2,
and the logic is like the code below:
while(true)
{
if(isup)
{
pfring_r = pfring_recv(_pfring1, &recv_buf, 0, &pfring_pkthdr, 0);
isup = false;
}
else
{
pfring_r1 = pfring_recv(_pfring2, &recv_buf, 0, &pfring_pkthdr, 0);
isup = true;
}
}

Sometimes, the order of packets received by calling pfring_recv is packet1, packet4, packet2, packet5 and packet3.

In our standing, each queue use one kernel PF_RING,
and packet1, packet2, packet3 are in one PF_RING,
and packet4 packet5 are in another PF_RING.
We suppose that packet1, packet2, packet3, packet4 and packet5 are inserted into the PF_RING almost at the same time
though the 5 five packets are not received by adapter at the same time.

From igb-3.0.22 README, we know that the max of InterruptThrottleRate is 100,000,
but the rate of packets captured by us is 1,000,000 per second.
So at least 10 packets are sent to PF_RING at one time.
But in order to get the correct the order of the packet, we want just one packet sent to PF_RING at one time.

Does PF_RING support our requirement(1,000,000 packets per second received by adapter and just one packet sent to PF_RING at one time)?

Thank you for your any suggestions to us.

Best Regards,
Shengyong Wang


deri at ntop

May 3, 2012, 5:48 AM

Post #2 of 3 (458 views)
Permalink
Re: PF_RING packets order problem [In reply to]

Hi,
if you have enabled multi-RX queues in the adapter, NAPI will shuffle your packets. The solution is to use a single queue, and have 1 dispatcher thread to pass packet to thread coherently.

Luca


On May 3, 2012, at 2:29 PM, wangsy wrote:

> Hi,
>
> In our project, we are using the PF_RING to capture packets from 82576-based adapters.
> First, we use pfring_open_multichannel function, then call pfring_recv function.
> For we use 4 threads to deal with 8 queues, each thread has to process 2 queues.
> But we met with a problem: the packets that each thread received from the 2 queues are disordered.
>
> For example, thread1 receive packets from queue1 and queue2.
> There are 5 packets and the adapter received order is packet1, packet2, packet3, packet4 and packet5.
> And packet1, packet2, packet3 are received by queue1, while packet4 and packet5 are receive by queue2.
>
> The thread1 calls the pfring_recv(non-block, wait_for_incoming_packet is 0) to receive from queue1 and queue2,
> and the logic is like the code below:
> while(true)
> {
> if(isup)
> {
> pfring_r = pfring_recv(_pfring1, &recv_buf, 0, &pfring_pkthdr, 0);
> isup = false;
> }
> else
> {
> pfring_r1 = pfring_recv(_pfring2, &recv_buf, 0, &pfring_pkthdr, 0);
> isup = true;
> }
> }
>
> Sometimes, the order of packets received by calling pfring_recv is packet1, packet4, packet2, packet5 and packet3.
>
> In our standing, each queue use one kernel PF_RING,
> and packet1, packet2, packet3 are in one PF_RING,
> and packet4 packet5 are in another PF_RING.
> We suppose that packet1, packet2, packet3, packet4 and packet5 are inserted into the PF_RING almost at the same time
> though the 5 five packets are not received by adapter at the same time.
>
> From igb-3.0.22 README, we know that the max of InterruptThrottleRate is 100,000,
> but the rate of packets captured by us is 1,000,000 per second.
> So at least 10 packets are sent to PF_RING at one time.
> But in order to get the correct the order of the packet, we want just one packet sent to PF_RING at one time.
>
> Does PF_RING support our requirement(1,000,000 packets per second received by adapter and just one packet sent to PF_RING at one time)?
>
> Thank you for your any suggestions to us.
>
> Best Regards,
> Shengyong Wang
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


wangsy at networkbench

May 4, 2012, 2:44 AM

Post #3 of 3 (479 views)
Permalink
Re: PF_RING packets order problem [In reply to]

Hi,

Thank you for your explanation.
But in our project, the 8 queues of 82576-based adapters(we use 2 Ethernet port) are divided into 4 pairs,
and each pair has its special filter rules.
The 2 queues in each pair have different function: one for uplink packet(use one Ethernet port) and the other for downlink(use the other Ethernet port).
I'm sorry for not explaining this in the mail before.

Now, I'm trying to use cluster to share the PR_RING.
But the results are not my expected.
My code is like below:

device = "eth4";
pfring *ring_downlink[64] = { NULL };
pfring *ring_uplink[64] = { NULL };
int promisc = 1;
int snaplen = 128;
u_int clusterId = 0;
num_channels = pfring_open_multichannel(device, promisc, snaplen, 1, ring_downlink);
int i = 0;
for (i = 0; i < num_channels; ++i){
clusterId = i + 1;
if (pfring_set_cluster(ring_downlink[i], clusterId , cluster_per_flow) != 0)
{
return(-1);
}
pfring_enable_ring(ring_downlink[i]);
}
// the eth4 and eth5 have the same multi-queue
device = "eth5";
num_channels = pfring_open_multichannel(device, promisc, snaplen, 1, ring_uplink);
for (i = 0; i < num_channels; ++i)
{
clusterId = i + 1;
if (pfring_set_cluster(ring_uplink[i], clusterId , cluster_per_flow) != 0)
{
return(-1);
}
pfring_enable_ring(ring_uplink[i]);
}

//receive package
u_int8_t wait_for_packet = 0;
while(true) {
int ret;
u_char *buffer = NULL;
struct pfring_pkthdr hdr;
for (int i = 0; i < num_channels; ++i){
ret = pfring_recv(ring_downlink[i], &buffer, 0, &hdr, wait_for_packet);
//parse the buffer and hdr content
ret = pfring_recv(ring_uplink[i], &buffer, 0, &hdr, wait_for_packet);
//parse the buffer and hdr content
}
}

Now, we cannot receive all the packages we sent.
For example, we sent 40 packages, but only received 21 packages.
I don't know the reason.
In addition, can I use just one PF_RING handle in the cluster to receive all the packages in the cluster (the code is like below)?

//receive package
u_int8_t wait_for_packet = 0;
while(true) {
int ret;
u_char *buffer = NULL;
struct pfring_pkthdr hdr;
for (int i = 0; i < num_channels; ++i){
ret = pfring_recv(ring_downlink[i], &buffer, 0, &hdr, wait_for_packet);
//parse the buffer and hdr content
}
}

Thank you very much!


Best Regards,
Shengyong Wang

发件人: Luca Deri
发送时间: 2012-05-03 20:48
收件人: ntop-misc; wangsy
抄送: chenbing
主题: Re: [Ntop-misc] PF_RING packets order problem
Hi,
if you have enabled multi-RX queues in the adapter, NAPI will shuffle your packets. The solution is to use a single queue, and have 1 dispatcher thread to pass packet to thread coherently.


Luca




On May 3, 2012, at 2:29 PM, wangsy wrote:


Hi,

In our project, we are using the PF_RING to capture packets from 82576-based adapters.
First, we use pfring_open_multichannel function, then call pfring_recv function.
For we use 4 threads to deal with 8 queues, each thread has to process 2 queues.
But we met with a problem: the packets that each thread received from the 2 queues are disordered.

For example, thread1 receive packets from queue1 and queue2.
There are 5 packets and the adapter received order is packet1, packet2, packet3, packet4 and packet5.
And packet1, packet2, packet3 are received by queue1, while packet4 and packet5 are receive by queue2.

The thread1 calls the pfring_recv(non-block, wait_for_incoming_packet is 0) to receive from queue1 and queue2,
and the logic is like the code below:
while(true)
{
if(isup)
{
pfring_r = pfring_recv(_pfring1, &recv_buf, 0, &pfring_pkthdr, 0);
isup = false;
}
else
{
pfring_r1 = pfring_recv(_pfring2, &recv_buf, 0, &pfring_pkthdr, 0);
isup = true;
}
}

Sometimes, the order of packets received by calling pfring_recv is packet1, packet4, packet2, packet5 and packet3.

In our standing, each queue use one kernel PF_RING,
and packet1, packet2, packet3 are in one PF_RING,
and packet4 packet5 are in another PF_RING.
We suppose that packet1, packet2, packet3, packet4 and packet5 are inserted into the PF_RING almost at the same time
though the 5 five packets are not received by adapter at the same time.

From igb-3.0.22 README, we know that the max of InterruptThrottleRate is 100,000,
but the rate of packets captured by us is 1,000,000 per second.
So at least 10 packets are sent to PF_RING at one time.
But in order to get the correct the order of the packet, we want just one packet sent to PF_RING at one time.

Does PF_RING support our requirement(1,000,000 packets per second received by adapter and just one packet sent to PF_RING at one time)?

Thank you for your any suggestions to us.

Best Regards,
Shengyong Wang


_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

NTop misc RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.