Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DRBD: Users

ib or 10gbe?

 

 

DRBD users RSS feed   Index | Next | Previous | View Threaded


james.harper at bendigoit

Apr 30, 2012, 4:11 AM

Post #1 of 4 (843 views)
Permalink
ib or 10gbe?

I'm considering configurations for a pair of new servers - a 2 node Xen cluster with shared storage.

It looks like I can build a HP server with direct connected 10gbe or ib for approximately the same price. Given the choice, what is the preference these days? The link will be dedicated to DRBD so other communications over the link are unimportant.

And are the HP ib cards well supported under Linux? Anecdotal reports appreciated!

Thanks

James
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


florian at hastexo

Apr 30, 2012, 7:29 AM

Post #2 of 4 (827 views)
Permalink
Re: ib or 10gbe? [In reply to]

Hi James,

On Mon, Apr 30, 2012 at 1:11 PM, James Harper
<james.harper [at] bendigoit> wrote:
> I'm considering configurations for a pair of new servers - a 2 node Xen cluster with shared storage.
>
> It looks like I can build a HP server with direct connected 10gbe or ib for approximately the same price. Given the choice, what is the preference these days? The link will be dedicated to DRBD so other communications over the link are unimportant.
>
> And are the HP ib cards well supported under Linux? Anecdotal reports appreciated!

We serve customers that use both, and in general recent distributions
support both OFED (for IB) and 10 GbE quite well. If your main pain
point is latency, you'll want to go with IB; if it's throughput,
you're essentially free to pick and choose -- although of course _not_
having to install any of the OFED libraries may be a plus for 10 GbE.
Cost of switches is usually not much of a factor in the decision, as
most people tend to wire their DRBD clusters back-to-back, but if
you're planning on a switched topology you may have to factor that in,
also.

Both IB and 10 GbE do require a fair amount of kernel and DRBD tuning
so that DRBD can actually max them out. Don't expect to be able to use
your distro's standard set of sysctls, and default DRBD config, and
then everything magically goes a million times faster.

Generally speaking, also don't expect too much of a performance boost
when using SDP (Sockets Direct Protocol) over IB. In general, we've
found that the performance effect in comparison to IPoIB is negligable
or even negative, but that's fine -- chances are you'll likely max out
your underlying storage hardware with IPoIB anyhow. :) SDP is also
currently suffering from a module refcount issue that is fixed in git
(http://git.drbd.org/gitweb.cgi?p=drbd-8.3.git;a=commit;h=c2c2067c661c7cba213b0301e2b39f17c1419e51)
but as yet unreleased, so that's a bit of an SDP show-stopper too...
but as pointed out, IPoIB does do the trick nicely.

Hope this helps.
Cheers,
Florian

--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


shadowsor at gmail

Apr 30, 2012, 8:04 AM

Post #3 of 4 (829 views)
Permalink
Re: ib or 10gbe? [In reply to]

If the IB is Mellanox, the newer ConnectX (2 and 3) can be run as
either a native 10Gbit ethernet or Infiniband. It's just a matter if
which driver you load. That may allow you to experiment and decide
which is best.

On Mon, Apr 30, 2012 at 8:29 AM, Florian Haas <florian [at] hastexo> wrote:
> Hi James,
>
> On Mon, Apr 30, 2012 at 1:11 PM, James Harper
> <james.harper [at] bendigoit> wrote:
>> I'm considering configurations for a pair of new servers - a 2 node Xen cluster with shared storage.
>>
>> It looks like I can build a HP server with direct connected 10gbe or ib for approximately the same price. Given the choice, what is the preference these days? The link will be dedicated to DRBD so other communications over the link are unimportant.
>>
>> And are the HP ib cards well supported under Linux? Anecdotal reports appreciated!
>
> We serve customers that use both, and in general recent distributions
> support both OFED (for IB) and 10 GbE quite well. If your main pain
> point is latency, you'll want to go with IB; if it's throughput,
> you're essentially free to pick and choose -- although of course _not_
> having to install any of the OFED libraries may be a plus for 10 GbE.
> Cost of switches is usually not much of a factor in the decision, as
> most people tend to wire their DRBD clusters back-to-back, but if
> you're planning on a switched topology you may have to factor that in,
> also.
>
> Both IB and 10 GbE do require a fair amount of kernel and DRBD tuning
> so that DRBD can actually max them out. Don't expect to be able to use
> your distro's standard set of sysctls, and default DRBD config, and
> then everything magically goes a million times faster.
>
> Generally speaking, also don't expect too much of a performance boost
> when using SDP (Sockets Direct Protocol) over IB. In general, we've
> found that the performance effect in comparison to IPoIB is negligable
> or even negative, but that's fine -- chances are you'll likely max out
> your underlying storage hardware with IPoIB anyhow. :) SDP is also
> currently suffering from a module refcount issue that is fixed in git
> (http://git.drbd.org/gitweb.cgi?p=drbd-8.3.git;a=commit;h=c2c2067c661c7cba213b0301e2b39f17c1419e51)
> but as yet unreleased, so that's a bit of an SDP show-stopper too...
> but as pointed out, IPoIB does do the trick nicely.
>
> Hope this helps.
> Cheers,
> Florian
>
> --
> Need help with High Availability?
> http://www.hastexo.com/now
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user
_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user


kavan at linbit

Apr 30, 2012, 4:31 PM

Post #4 of 4 (860 views)
Permalink
Re: ib or 10gbe? [In reply to]

Hi James,

It all depends on what you are looking to accomplish and what hardware
you are comparing.

Newer Infiniband QDR cards are rated at 40 Gbit/s which are a bit
quicker than the 10GbE alternative.

Today, IPoIB provides great performance, but would be better with native
RDMA support. 10GbE is a great solution, but you really miss out on low
latency high bandwidth capabilities that Infiniband brings to the table.
Right now, IPoIB is the best solution with DRBD if you want to exceed
current 10GbE benchmarks.

We have a tech guide that will be announced this week, but since you
asked so kindly, please check this out at your leisure:

http://www.linbit.com/en/education/tech-guides/infiniband-and-drbd-technical-guide/

Also...stay tuned... :)

In regards to how DRBD is going to support this in the future:

LINBIT is working to develop native RDMA support for Infiniband (this
will make DRBD on Infiniband much much quicker), but we still need
assistance from the community to make this feature-set possible.

HA and DRBD experts. We could always use the help! -
feedback [at] linbit if you would like to assist in developing or
sponsoring this feature for the DRBD Community. Don't just claim you're
an expert, show it! :)

Best,
-Kavan

On 04/30/2012 04:11 AM, James Harper wrote:
> I'm considering configurations for a pair of new servers - a 2 node Xen cluster with shared storage.
>
> It looks like I can build a HP server with direct connected 10gbe or ib for approximately the same price. Given the choice, what is the preference these days? The link will be dedicated to DRBD so other communications over the link are unimportant.
>
> And are the HP ib cards well supported under Linux? Anecdotal reports appreciated!
>
> Thanks
>
> James
> _______________________________________________
> drbd-user mailing list
> drbd-user [at] lists
> http://lists.linbit.com/mailman/listinfo/drbd-user


--
: Kavan Smith
: LINBIT | "Your Way to High Availability"
: 1-503-573-1262 x201 | 1-877-4-LINBIT
: Web: http://www.linbit.com
:
: Twitter: http://www.linbit.com/en/twitter
: Facebook: http://www.linbit.com/en/facebook

_______________________________________________
drbd-user mailing list
drbd-user [at] lists
http://lists.linbit.com/mailman/listinfo/drbd-user

DRBD users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.