Mailing List Archive: DRBD: Users

Infiniband

Index | Next | Previous | View Threaded

artur at eboundhost

Nov 20, 2008, 9:08 AM

Post #1 of 9 (1988 views)
Permalink

Infiniband

Wanted to get some advice, considering infiniband for replication. Any pitfalls? Reasons to stay away?

Best Regards,

Artur
eBoundHost
http://www.eboundhost.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

reg at elconas

Nov 20, 2008, 10:32 AM

Post #2 of 9 (1915 views)
Permalink

Re: Infiniband [In reply to]

No experience with infiniband on my side so far,

however LinBit and Dolphin Networks joined a partnership some time ago,
so this may be also interestang as a "fully supported" solution.

I was talking to a Dolphin guy at a MySQL conference and he told me that
it is quite cheap and has lower latency than infiniband. So if you need
to buy new equipment anyhow this may be an option.

Regards,
Robert

Artur (eBoundHost) schrieb:
> Wanted to get some advice, considering infiniband for replication. Any pitfalls? Reasons to stay away?
>
> Best Regards,
>
> Artur
> eBoundHost
> http://www.eboundhost.com
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

artur at eboundhost

Nov 20, 2008, 10:51 AM

Post #3 of 9 (1906 views)
Permalink

Re: Infiniband [In reply to]

There are other considerations for infiniband. I want to have a unified architecture between serves and clients.

Can dolphin do nfs without major massaging?
------Original Message------
From: Robert
To: artur [at] eboundhost
Cc: drbd-user [at] lists
ReplyTo: reg [at] elconas
Subject: Re: [DRBD-user] Infiniband
Sent: Nov 20, 2008 12:32

No experience with infiniband on my side so far,

however LinBit and Dolphin Networks joined a partnership some time ago,
so this may be also interestang as a "fully supported" solution.

I was talking to a Dolphin guy at a MySQL conference and he told me that
it is quite cheap and has lower latency than infiniband. So if you need
to buy new equipment anyhow this may be an option.

Regards,
Robert

Artur (eBoundHost) schrieb:
> Wanted to get some advice, considering infiniband for replication. Any pitfalls? Reasons to stay away?
>
> Best Regards,
>
> Artur
> eBoundHost
> http://www.eboundhost.com
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>

Best Regards,

Artur
eBoundHost
http://www.eboundhost.com

igor at 3gnt

Nov 20, 2008, 10:55 AM

Post #4 of 9 (1885 views)
Permalink

Re: Infiniband [In reply to]

HI,

I would bet in infiniband.

Why:
- It's widely used and tested.
- In linux you have IP over Infiniband module, that it's working for so
many people in so many scenarios.

I bet it will work very good with drbd, even knowing this, everything
should be tested before going to production.

Sorry, i have no experience (yet) with infiniband. But i would love too.

Good luck.

Artur (eBoundHost) wrote:
> There are other considerations for infiniband. I want to have a unified architecture between serves and clients.
>
> Can dolphin do nfs without major massaging?
> ------Original Message------
> From: Robert
> To: artur [at] eboundhost
> Cc: drbd-user [at] lists
> ReplyTo: reg [at] elconas
> Subject: Re: [DRBD-user] Infiniband
> Sent: Nov 20, 2008 12:32
>
> No experience with infiniband on my side so far,
>
> however LinBit and Dolphin Networks joined a partnership some time ago,
> so this may be also interestang as a "fully supported" solution.
>
> I was talking to a Dolphin guy at a MySQL conference and he told me that
> it is quite cheap and has lower latency than infiniband. So if you need
> to buy new equipment anyhow this may be an option.
>
> Regards,
> Robert
>
> Artur (eBoundHost) schrieb:
>
>> Wanted to get some advice, considering infiniband for replication. Any pitfalls? Reasons to stay away?
>>
>> Best Regards,
>>
>> Artur
>> eBoundHost
>> http://www.eboundhost.com
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user [at] lists
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
>>
>
>
>
>
> Best Regards,
>
> Artur
> eBoundHost
> http://www.eboundhost.com
> ------------------------------------------------------------------------
>
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
>

--
_
�v� Igor Neves <igor.neves [at] 3gnt>
/(_)\ 3GNTW - Tecnologias de Informa��o, Lda
^ ^

SIP/JID/MSN: igor [at] 3gnt
Mobile Phone: 00351914503611
Local Phone: 00351252377120

nathan at robotics

Nov 20, 2008, 2:50 PM

Post #5 of 9 (1887 views)
Permalink

Re: Infiniband [In reply to]

On Thu, 20 Nov 2008, Artur (eBoundHost) wrote:

> Wanted to get some advice, considering infiniband for replication. Any pitfalls? Reasons to stay away?

I love it, using it on two 10 TB systems. We picked up all the gear off
ebay for an avg of less then $100 for cards and $1100 for switches.

><>
Nathan Stratton CTO, BlinkMind, Inc.
nathan at robotics.net nathan at blinkmind.com
http://www.robotics.net http://www.blinkmind.com
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

igor at 3gnt

Nov 21, 2008, 3:34 AM

Post #6 of 9 (1869 views)
Permalink

Re: Infiniband [In reply to]

Hi,

It's possible to connect to infiniband cards back2back/crosscable?

Nathan Stratton wrote:
> On Thu, 20 Nov 2008, Artur (eBoundHost) wrote:
>
>> Wanted to get some advice, considering infiniband for replication.
>> Any pitfalls? Reasons to stay away?
>
> I love it, using it on two 10 TB systems. We picked up all the gear
> off ebay for an avg of less then $100 for cards and $1100 for switches.
>
>> <>
> Nathan Stratton CTO, BlinkMind, Inc.
> nathan at robotics.net nathan at blinkmind.com
> http://www.robotics.net http://www.blinkmind.com
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user

--
_
�v� Igor Neves <igor.neves [at] 3gnt>
/(_)\ 3GNTW - Tecnologias de Informa��o, Lda
^ ^

SIP/JID/MSN: igor [at] 3gnt
Mobile Phone: 00351914503611
Local Phone: 00351252377120

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

nathan at robotics

Nov 21, 2008, 5:56 AM

Post #7 of 9 (1880 views)
Permalink

Re: Infiniband [In reply to]

On Fri, 21 Nov 2008, Igor Neves wrote:

> Hi,
>
> It's possible to connect to infiniband cards back2back/crosscable?

I don't think so, or at least I have not found a way.

-Nathan
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

volker.jaenisch at inqbus

Nov 22, 2008, 6:29 AM

Post #8 of 9 (1860 views)
Permalink

Re: Infiniband [In reply to]

Hi Igor!

We run a 1.5 TB storage array replicated with DRBD over infiniband in
production.
It runs very stable and very fast. Some weeks ago one of the
DRBD-machines died due to a defective power-supply.
Nobody (we runs several productive XEN-VMs on this storage) except our
Nagios noticed the failover. It took some days to
get the power supply repaired. When the repaired machine comes up again the
resynchronisation was done in 12 seconds.

By the way. A great thank to the DRBD makers and the community for this
fine piece of software !

Igor Neves schrieb:
> It's possible to connect to infiniband cards back2back/crosscable?
Yes. We use Mellanox cards with a normal infiniband cable as "cross over cable". You need no special cable nor a switch.

The only problem going back2back is the network logic. Most infiniband switches act as network controllers (aka subnet managers) wich controll the IB routing.

So if you like to work back2back without an infiniband switch you need to establish a subnet manager by software with e.g. opensm

https://wiki.openfabrics.org/tiki-index.php?page=OpenSM

at one of your two DRBD nodes.

With drbd over tcp over infinband we have a tcp tranfer rate of 2-3 GBit/sec full duplex . This is far below the bandwith of native infiniband (we messure 18.6GBit/sec back2back). Since drbd runs over tcp, only, the infiniband has to run in tcp-over-infiniband emulation mode wich slows down due to the protocol stack overhead.

I am looking foreward to a DRBD solution wich works over RDMA. :-)

But also in emulation mode the DRBD performance is astonishing. We meassure a continuosly resync rate of 100MByte/sec which is the write limit of the underlying disc-Arrays.

The latency of the tcp over infiniband line we messure is 4 times shorter than with expensive 1GB-Eth-NICs (back2back). As you all know - latency is the common bottleneck for replication systems as DRBD.
I have heard rumors that 10GBit-Eth will have also low latency - but I have not seen it with my own eyes.

An other advantage is that you can deliver your DRBD-Array via iSCSI over Infiniband (using full speed RDMA) to the application servers.

Best Regards

Volker

--
====================================================
inqbus it-consulting +49 ( 341 ) 5643800
Dr. Volker Jaenisch http://www.inqbus.de
Herlo�sohnstr. 12 0 4 1 5 5 Leipzig
N O T - F � L L E +49 ( 170 ) 3113748
====================================================

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

florian.haas at linbit

Sep 14, 2009, 12:28 AM

Post #9 of 9 (1250 views)
Permalink

Re: Infiniband [In reply to]

On 2009-09-11 06:17, Nathan Stratton wrote:
>
> How does one configure infiniband with the new DRBD?

Will be documented in the User's Guide once 8.3.3 is out. For the time
being, man drbd.conf from -rc2 is your friend (look for "address").

> P.S. Anyone know where infiniband went in 2.6.31? It was in 2.6.31-rc9
> and then it vanished.

Dunno, what DRBD ties in with is SDP from OFED 1.4.2 and above, and
AFAICS that isn't upstream just yet. If I'm not misinformed, then OFED
is in RHEL 5.4 and SLES 11 though.

Cheers,
Florian

Attachments:

signature.asc (0.25 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads