Mailing List Archive: NTop: Misc

Problem about the TNAPI multi-channel

Index | Next | Previous | View Threaded

baimingmail at yahoo

Dec 21, 2009, 2:55 AM

Post #1 of 5 (1635 views)
Permalink

Problem about the TNAPI multi-channel

Luca,

I am testing the “TNAPI+PF_RING” to improve the performance of server that is used to real-time collect internet data. I want that TCP packets belong to a same TCP session are processed by a same CPU core.

I had compiled and installed (“insmod ./igb.ko RSS=4,4”, “insmod ./pf_ring.ko transparent_mode=1”) the TNAPI and PF_RING successfully.

Then I do following:
1. Stop “irqbalance”.
2. Bind “eth4-TxRx-0/1/2/3” to PCU core 0/1/2/3.
3. Send the TCP packets belong to a same session into the server’s “eth4”. These TCP packets (total 47 packets): “A->B” 30 packets and “B->A” 17 packets.
4. Receive the TCP packets separately on CPU core0/1/2/3 of the server.

Here the problem occurs:
1. The “A->B” 30 packets are received on the CPU core 1, it is OK.
2. But the “B->A” 17 packets are received on the CPU core 2, it is not what I want. I want that all 47 packets are processed by a same CPU core, because of they belong with a same TCP session. Moreover, these TCP packets’s source IP/Port and destination IP/Port are changed.

I don't understand this problem, please help me!

-------------------------
My test environment:

1. Our server’s basic information:
(1) Computer: HP DL380 G6.
(2) Net Card: E1G42ET (Intel 82576 chip).
(3) CPUs: 8 cores.
(4) OS: Linux 2.6.32.

2. The information of the PF_RING and TNAPI:
(1) PF_RING: 4.1.0 (“[PF_RING] Welcome to PF_RING 4.1.0 ($Revision: 4012 $)” from “/var/log/messages”).
(2) TNAPI: igb-2.0.6_011109.tgz.
-------------------------

Bai Ming
2009-12-21

___________________________________________________________
好玩贺卡等你发，邮箱贺卡全新上线！
http://card.mail.cn.yahoo.com/
_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

deri at ntop

Dec 21, 2009, 3:05 AM

Post #2 of 5 (1592 views)
Permalink

Re: Problem about the TNAPI multi-channel [In reply to]

Bai
unfortunately Intel RSS hashing function is non by-directional. This
means that A->B packets go to queue X, and B->A go to queue Y, where X
!= Y. So TNAPI works as designed.

Luca

On 12/21/2009 11:55 AM, 明白 wrote:
> Luca,
>
> I am testing the “TNAPI+PF_RING” to improve the performance of server that is used to real-time collect internet data. I want that TCP packets belong to a same TCP session are processed by a same CPU core.
>
> I had compiled and installed (“insmod ./igb.ko RSS=4,4”, “insmod ./pf_ring.ko transparent_mode=1”) the TNAPI and PF_RING successfully.
>
> Then I do following:
> 1. Stop “irqbalance”.
> 2. Bind “eth4-TxRx-0/1/2/3” to PCU core 0/1/2/3.
> 3. Send the TCP packets belong to a same session into the server’s “eth4”. These TCP packets (total 47 packets): “A->B” 30 packets and “B->A” 17 packets.
> 4. Receive the TCP packets separately on CPU core0/1/2/3 of the server.
>
> Here the problem occurs:
> 1. The “A->B” 30 packets are received on the CPU core 1, it is OK.
> 2. But the “B->A” 17 packets are received on the CPU core 2, it is not what I want. I want that all 47 packets are processed by a same CPU core, because of they belong with a same TCP session. Moreover, these TCP packets’s source IP/Port and destination IP/Port are changed.
>
> I don't understand this problem, please help me!
>
> -------------------------
> My test environment:
>
> 1. Our server’s basic information:
> (1) Computer: HP DL380 G6.
> (2) Net Card: E1G42ET (Intel 82576 chip).
> (3) CPUs: 8 cores.
> (4) OS: Linux 2.6.32.
>
> 2. The information of the PF_RING and TNAPI:
> (1) PF_RING: 4.1.0 (“[PF_RING] Welcome to PF_RING 4.1.0 ($Revision: 4012 $)” from “/var/log/messages”).
> (2) TNAPI: igb-2.0.6_011109.tgz.
> -------------------------
>
> Bai Ming
> 2009-12-21
>
>
>
> ___________________________________________________________
> 好玩贺卡等你发，邮箱贺卡全新上线！
> http://card.mail.cn.yahoo.com/
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

jnebrera at eneotecnologia

Dec 21, 2009, 3:19 AM

Post #3 of 5 (1568 views)
Permalink

Re: Problem about the TNAPI multi-channel [In reply to]

Hi Luca,

Just one question regarding this "bidirectional stuff".

I believe (but I'm not sure) in a specific scenario its possible to
make RSS work bidirectional.

I think that in case the traffic comes and leaves always through a
particular pair of ports it could be made to be bidirectional. Of
course, this limits quite a lot its usage, but for the typical 2 port
bridge (QoS, NetFlow) it could work.

You could just configure port A to use sIPdIP pair, and port B to use
the opposite. As the hash is only compited for traffic coming in, this
could work.

What do you think?

El lun, 21-12-2009 a las 12:05 +0100, Luca Deri escribi贸:
> Bai
> unfortunately Intel RSS hashing function is non by-directional. This
> means that A->B packets go to queue X, and B->A go to queue Y, where X
> != Y. So TNAPI works as designed.
>
> Luca
>
> On 12/21/2009 11:55 AM, 鏄� 鐧� wrote:
> > Luca,
> >
> > I am testing the 鈥淭NAPI+PF_RING鈥� to improve the performance of server that is used to real-time collect internet data. I want that TCP packets belong to a same TCP session are processed by a same CPU core.
> >
> > I had compiled and installed (鈥渋nsmod ./igb.ko RSS=4,4鈥�, 鈥渋nsmod ./pf_ring.ko transparent_mode=1鈥�) the TNAPI and PF_RING successfully.
> >
> > Then I do following:
> > 1. Stop 鈥渋rqbalance鈥�.
> > 2. Bind 鈥渆th4-TxRx-0/1/2/3鈥� to PCU core 0/1/2/3.
> > 3. Send the TCP packets belong to a same session into the server鈥檚鈥渆th4鈥�. These TCP packets (total 47 packets): 鈥淎->B鈥� 30 packets and 鈥淏->A鈥� 17 packets.
> > 4. Receive the TCP packets separately on CPU core0/1/2/3 of the server.
> >
> > Here the problem occurs:
> > 1. The 鈥淎->B鈥� 30 packets are received on the CPU core 1, it is OK.
> > 2. But the 鈥淏->A鈥� 17 packets are received on the CPU core 2, it is not what I want. I want that all 47 packets are processed by a same CPU core, because of they belong with a same TCP session. Moreover, these TCP packets鈥檚 source IP/Port and destination IP/Port are changed.
> >
> > I don't understand this problem, please help me!
> >
> > -------------------------
> > My test environment:
> >
> > 1. Our server鈥檚 basic information:
> > (1) Computer: HP DL380 G6.
> > (2) Net Card: E1G42ET (Intel 82576 chip).
> > (3) CPUs: 8 cores.
> > (4) OS: Linux 2.6.32.
> >
> > 2. The information of the PF_RING and TNAPI:
> > (1) PF_RING: 4.1.0 (鈥淸PF_RING] Welcome to PF_RING 4.1.0 ($Revision: 4012 $)鈥� from 鈥�/var/log/messages鈥�).
> > (2) TNAPI: igb-2.0.6_011109.tgz.
> > -------------------------
> >
> > Bai Ming
> > 2009-12-21
> >
> >
> >
> > ___________________________________________________________
> > 濂界帺璐哄崱绛変綘鍙戯紝閭璐哄崱鍏ㄦ柊涓婄嚎锛�
> > http://card.mail.cn.yahoo.com/
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc [at] listgateway
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> >
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
--
Jaime Nebrera - jnebrera [at] eneotecnologia
Consultor TI - ENEO Tecnologia SL
Pol. PISA - C/ Manufactura 6, P1, 3B
Mairena del Aljarafe - 41927 - Sevilla
Telf.- 955 60 11 60 / 619 04 55 18

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

deri at ntop

Dec 21, 2009, 12:27 PM

Post #4 of 5 (1560 views)
Permalink

Re: Problem about the TNAPI multi-channel [In reply to]

Jaime
the answer is that depends on the NIC. For 1G and 10G with 82598 the hash is computed according to the protocol. So if you use TCP/UDP, the hash is computed also on IP and ports. So if you have a few (2, 4 etc) flows, you can force (this means that you have to manually hack the driver) RSS to be by-directional, but as in general you have many more flows this is practically impossible. With the new 82599 chip you can instrument the Intel Flow Director to compute the hash the way you like (see http://download.intel.com/design/network/datashts/82599_datasheet.pdf section 8.2.3.7.12, 'RSS Field Enable ').

Luca

On Dec 21, 2009, at 12:19 PM, Jaime Nebrera wrote:

> Hi Luca,
>
> Just one question regarding this "bidirectional stuff".
>
> I believe (but I'm not sure) in a specific scenario its possible to
> make RSS work bidirectional.
>
> I think that in case the traffic comes and leaves always through a
> particular pair of ports it could be made to be bidirectional. Of
> course, this limits quite a lot its usage, but for the typical 2 port
> bridge (QoS, NetFlow) it could work.
>
> You could just configure port A to use sIPdIP pair, and port B to use
> the opposite. As the hash is only compited for traffic coming in, this
> could work.
>
> What do you think?
>
>
>
>
> El lun, 21-12-2009 a las 12:05 +0100, Luca Deri escribi贸:
>> Bai
>> unfortunately Intel RSS hashing function is non by-directional. This
>> means that A->B packets go to queue X, and B->A go to queue Y, where X
>> != Y. So TNAPI works as designed.
>>
>> Luca
>>
>> On 12/21/2009 11:55 AM, 鏄� 鐧� wrote:
>>> Luca,
>>>
>>> I am testing the 鈥淭NAPI+PF_RING鈥� to improve the performance of server that is used to real-time collect internet data. I want that TCP packets belong to a same TCP session are processed by a same CPU core.
>>>
>>> I had compiled and installed (鈥渋nsmod ./igb.ko RSS=4,4鈥�, 鈥渋nsmod ./pf_ring.ko transparent_mode=1鈥�) the TNAPI and PF_RING successfully.
>>>
>>> Then I do following:
>>> 1. Stop 鈥渋rqbalance鈥�.
>>> 2. Bind 鈥渆th4-TxRx-0/1/2/3鈥� to PCU core 0/1/2/3.
>>> 3. Send the TCP packets belong to a same session into the server鈥檚鈥渆th4鈥�. These TCP packets (total 47 packets): 鈥淎->B鈥� 30 packets and 鈥淏->A鈥� 17 packets.
>>> 4. Receive the TCP packets separately on CPU core0/1/2/3 of the server.
>>>
>>> Here the problem occurs:
>>> 1. The 鈥淎->B鈥� 30 packets are received on the CPU core 1, it is OK.
>>> 2. But the 鈥淏->A鈥� 17 packets are received on the CPU core 2, it is not what I want. I want that all 47 packets are processed by a same CPU core, because of they belong with a same TCP session. Moreover, these TCP packets鈥檚 source IP/Port and destination IP/Port are changed.
>>>
>>> I don't understand this problem, please help me!
>>>
>>> -------------------------
>>> My test environment:
>>>
>>> 1. Our server鈥檚 basic information:
>>> (1) Computer: HP DL380 G6.
>>> (2) Net Card: E1G42ET (Intel 82576 chip).
>>> (3) CPUs: 8 cores.
>>> (4) OS: Linux 2.6.32.
>>>
>>> 2. The information of the PF_RING and TNAPI:
>>> (1) PF_RING: 4.1.0 (鈥淸PF_RING] Welcome to PF_RING 4.1.0 ($Revision: 4012 $)鈥� from 鈥�/var/log/messages鈥�).
>>> (2) TNAPI: igb-2.0.6_011109.tgz.
>>> -------------------------
>>>
>>> Bai Ming
>>> 2009-12-21
>>>
>>>
>>>
>>> ___________________________________________________________
>>> 濂界帺璐哄崱绛変綘鍙戯紝閭璐哄崱鍏ㄦ柊涓婄嚎锛�
>>> http://card.mail.cn.yahoo.com/
>>> _______________________________________________
>>> Ntop-misc mailing list
>>> Ntop-misc [at] listgateway
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc [at] listgateway
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> --
> Jaime Nebrera - jnebrera [at] eneotecnologia
> Consultor TI - ENEO Tecnologia SL
> Pol. PISA - C/ Manufactura 6, P1, 3B
> Mairena del Aljarafe - 41927 - Sevilla
> Telf.- 955 60 11 60 / 619 04 55 18
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc [at] listgateway
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

jnebrera at eneotecnologia

Dec 21, 2009, 1:33 PM

Post #5 of 5 (1564 views)
Permalink

Re: Problem about the TNAPI multi-channel [In reply to]

Hi Luca,

> the answer is that depends on the NIC. For 1G and 10G with 82598
> the hash is computed according to the protocol. So if you use
> TCP/UDP, the hash is computed also on IP and ports. So if you have
> a few (2, 4 etc) flows, you can force (this means that you have to
> manually hack the driver) RSS to be by-directional, but as in general
> you have many more flows this is practically impossible. With the new
> 82599 chip you can instrument the Intel Flow Director to compute the
> hash the way you like (see
> http://download.intel.com/design/network/datashts/82599_datasheet.pdf
> section 8.2.3.7.12, 'RSS Field Enable ').

Then I guess would depend on specific chipset. Reading Emulex card
docs I understood it was possible to instruct the chipset what to send
to you with certain fields. It seemed it was possible to change the
order in one of the ports so the source was the changed with the
destination and the final hash was the same. But again, I'm not sure at
all of this.

I'm aware 82599 precisely tried to solve this :)

--
Jaime Nebrera - jnebrera [at] eneotecnologia
Consultor TI - ENEO Tecnologia SL
Pol. PISA - C/ Manufactura 6, P1, 3B
Mairena del Aljarafe - 41927 - Sevilla
Telf.- 955 60 11 60 / 619 04 55 18

_______________________________________________
Ntop-misc mailing list
Ntop-misc [at] listgateway
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads