Mailing List Archive: MythTV: Users

Samba 3 better than NFS 3 for recording over Ethernet

1 2

View All

Index | Next | Previous | View Threaded

ylee at pobox

Dec 28, 2005, 6:31 PM

Post #1 of 32 (20751 views)
Permalink

Samba 3 better than NFS 3 for recording over Ethernet

Buechler, Mark R <Mark_R_Buechler [at] bausch> says:
> My recordings directory is a raid5 set which isn't the fastest in the world

[...]

> Another thing I'm considering is HD. If I attempt to vew HD content from my
> DVB card while streaming it to the raid5 array it can't really handle it too
> well - whereas with a separate live tv buffer it wasn't a problem.

I'm going to blatantly hijack this thread to record for posterity
that, at least with my setup (streaming HDTV recordings to a software
RAID 5+LVM2 array over Gigabit Ethernet with Fedora Core and the
ATrpms version of MythTV 18.1), Samba 3 is rock solid while NFS 3 is
pretty guaranteed to generate near-endless IOBOUND error messages in
mythbackend.log. More detail when I have the time and inclination to
write up my experiences, but I hope this helps others in my situation
in the meanwhile; a month of IOBOUND error messages despite endless
tuning attempts.

--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US

goliver at cistera

Dec 28, 2005, 7:16 PM

Post #2 of 32 (20545 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

What mount option are you passing to the nfs share.. SMB versus nfs and
smb being faster is impossible IMO.

On Wed, 2005-12-28 at 18:31 -0800, Yeechang Lee wrote:
> Buechler, Mark R <Mark_R_Buechler [at] bausch> says:
> > My recordings directory is a raid5 set which isn't the fastest in the world
>
> [...]
>
> > Another thing I'm considering is HD. If I attempt to vew HD content from my
> > DVB card while streaming it to the raid5 array it can't really handle it too
> > well - whereas with a separate live tv buffer it wasn't a problem.
>
> I'm going to blatantly hijack this thread to record for posterity
> that, at least with my setup (streaming HDTV recordings to a software
> RAID 5+LVM2 array over Gigabit Ethernet with Fedora Core and the
> ATrpms version of MythTV 18.1), Samba 3 is rock solid while NFS 3 is
> pretty guaranteed to generate near-endless IOBOUND error messages in
> mythbackend.log. More detail when I have the time and inclination to
> write up my experiences, but I hope this helps others in my situation
> in the meanwhile; a month of IOBOUND error messages despite endless
> tuning attempts.
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jarod at wilsonet

Dec 28, 2005, 8:00 PM

Post #3 of 32 (20524 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Wednesday 28 December 2005 18:31, Yeechang Lee wrote:
> Buechler, Mark R <Mark_R_Buechler [at] bausch> says:
> > My recordings directory is a raid5 set which isn't the fastest in the
> > world
>
> [...]
>
> > Another thing I'm considering is HD. If I attempt to vew HD content from
> > my DVB card while streaming it to the raid5 array it can't really handle
> > it too well - whereas with a separate live tv buffer it wasn't a problem.
>
> I'm going to blatantly hijack this thread to record for posterity
> that, at least with my setup (streaming HDTV recordings to a software
> RAID 5+LVM2 array over Gigabit Ethernet with Fedora Core and the
> ATrpms version of MythTV 18.1), Samba 3 is rock solid while NFS 3 is
> pretty guaranteed to generate near-endless IOBOUND error messages in
> mythbackend.log. More detail when I have the time and inclination to
> write up my experiences, but I hope this helps others in my situation
> in the meanwhile; a month of IOBOUND error messages despite endless
> tuning attempts.

Something odd going on there, I have an almost identical setup for my master
backend, and NFS works great at handling multiple HDTV streams (both my HD
capture devices are in slave backends that write to the master via NFS),
along with the occasional SD stream thrown in as well.

--
Jarod Wilson
jarod [at] wilsonet

mfrisch at isurfer

Dec 28, 2005, 8:43 PM

Post #4 of 32 (20515 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 28-Dec-05, at 10:16 PM, Greg Oliver wrote:

> What mount option are you passing to the nfs share.. SMB versus
> nfs and
> smb being faster is impossible IMO.
>

Agreed.

At least the following NFS mount options should be used if not already:

mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
local/mountpoint

The "tcp" option may not be necessary but I haven't taken the time to
benchmark between TCP and UDP.

Mike.

jtidmore at gmail

Dec 28, 2005, 9:29 PM

Post #5 of 32 (20518 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
>
>
> The "tcp" option may not be necessary but I haven't taken the time to
> benchmark between TCP and UDP.

For gigabit ethernet, the nfs people recommend TCP.

>From the man page for nfs (5):

WARNINGS
Using NFS over UDP on high-speed links such as Gigabit can cause
silent data corruption.

The problem can be triggered at high loads, and is caused by
problems in IP fragment reassembly. NFS
read and writes typically transmit UDP packets of 4 Kilobytes or
more, which have to be broken up into
several fragments in order to be sent over the Ethernet link,
which limits packets to 1500 bytes by
default. This process happens at the IP network layer and is called
fragmentation.

They go into greater detail about this issue. For info use 'man 5 nfs'

-J

goliver at cistera

Dec 28, 2005, 9:32 PM

Post #6 of 32 (20526 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Jumbo frames are your friend

On Wed, 2005-12-28 at 21:29 -0800, Jonathan Tidmore wrote:
> On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
>
> The "tcp" option may not be necessary but I haven't taken the
> time to
> benchmark between TCP and UDP.
>
>
> For gigabit ethernet, the nfs people recommend TCP.
>
> From the man page for nfs (5):
>
> WARNINGS
> Using NFS over UDP on high-speed links such as Gigabit can
> cause silent data corruption.
>
> The problem can be triggered at high loads, and is caused
> by problems in IP fragment reassembly. NFS
> read and writes typically transmit UDP packets of 4 Kilobytes
> or more, which have to be broken up into
> several fragments in order to be sent over the Ethernet
> link, which limits packets to 1500 bytes by
> default. This process happens at the IP network layer and is
> called fragmentation.
>
> They go into greater detail about this issue. For info use 'man 5
> nfs'
>
>
> -J
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

rbsteffes at gmail

Dec 28, 2005, 9:49 PM

Post #7 of 32 (20516 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/29/05, Greg Oliver <goliver [at] cistera> wrote:
>
> Jumbo frames are your friend
>
> On Wed, 2005-12-28 at 21:29 -0800, Jonathan Tidmore wrote:
> > On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
> >
> > The "tcp" option may not be necessary but I haven't taken the
> > time to
> > benchmark between TCP and UDP.
> >
> >
> > For gigabit ethernet, the nfs people recommend TCP.
> >
> > From the man page for nfs (5):
> >
> > WARNINGS
> > Using NFS over UDP on high-speed links such as Gigabit can
> > cause silent data corruption.
> >
> > The problem can be triggered at high loads, and is caused
> > by problems in IP fragment reassembly. NFS
> > read and writes typically transmit UDP packets of 4 Kilobytes
> > or more, which have to be broken up into
> > several fragments in order to be sent over the Ethernet
> > link, which limits packets to 1500 bytes by
> > default. This process happens at the IP network layer and is
> > called fragmentation.
> >
> > They go into greater detail about this issue. For info use 'man 5
> > nfs'
> >
> >
> > -J
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users [at] mythtv
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

Hmm, I wonder if this may be related to why nfs apparently locks my machine
in heavy use. It started when I switched to gigabit nics and I was blaming
the cards, until I realized I could do high data transfers with the computer
as long as I don't use nfs.

Ryan

ylee at pobox

Dec 28, 2005, 10:12 PM

Post #8 of 32 (20523 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Greg Oliver <goliver [at] cistera> says:
> What mount option are you passing to the nfs share..

I've been using the nfsvers=3, tcp, and 32K read/write buffers options
someone else mentioned in my home network NFS setup since long before
taking up MythTV. Believe me, I'd still much rather use NFS--I'm a
*lot* more comfortable with fiddling with NFS export and mount options
than with smb.conf--but Samba works and works well.

> SMB versus nfs and smb being faster is impossible IMO.

Again, speaking as someone who's had a lot more hands-on experience
with NFS than Samba, I know very well that modern Samba is as fast as,
if not faster than (and certainly a lot more interoperable across
platforms) than NFS. Many found NFS pre-v3 to be noticeably slower
than Samba, for example, and although as I've mentioned I've always
used NFS v3 the notion that today's Samba is faster--or at least more
efficient in some way I haven't figured out yet on my particular
setup--than NFS is hardly a strange one.

N.B. - To be completely accurate, I'm actually using cifs, as smbfs has
a 2GB file limit which I only discovered after mythbackend kept
crashing with no error messages whatsoever after a certain point in my
HDTV recordings.
--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US

mrmagoo at mrmagoo

Dec 28, 2005, 10:15 PM

Post #9 of 32 (20535 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/28/05, Yeechang Lee <ylee [at] pobox> wrote:
> Buechler, Mark R <Mark_R_Buechler [at] bausch> says:
> > My recordings directory is a raid5 set which isn't the fastest in the world
>
> [...]
>
> > Another thing I'm considering is HD. If I attempt to vew HD content from my
> > DVB card while streaming it to the raid5 array it can't really handle it too
> > well - whereas with a separate live tv buffer it wasn't a problem.
>
> I'm going to blatantly hijack this thread to record for posterity
> that, at least with my setup (streaming HDTV recordings to a software
> RAID 5+LVM2 array over Gigabit Ethernet with Fedora Core and the
> ATrpms version of MythTV 18.1), Samba 3 is rock solid while NFS 3 is
> pretty guaranteed to generate near-endless IOBOUND error messages in
> mythbackend.log. More detail when I have the time and inclination to
> write up my experiences, but I hope this helps others in my situation
> in the meanwhile; a month of IOBOUND error messages despite endless
> tuning attempts.
>
> --
> Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US
>
>

Since you brought it up...

Is gigabit necessary? I have 2 Myth systems connected via 100mbit
using CIFS/samba with relatively default settings - I searched google
I really couldn't find any options that would help aside from the
default socket options. One machine is the BE/FE with PVR350+Air2PC
and the other a database/FE/RAID5 file server. I don't have any major
problems with the SD recordings, although I have seen skips maybe once
per 3-4 hours of recorded content. HD recordings regularly have tons
of IOBOUND that make shows unwatchable. It seems like the 1080i
channels do this more than 720, although I thought 720p and 1080i were
effectively the same bandwidth.
I tried NFS, but I could never get it to work right and had VERY
bad lag problems that would cause the kernel to become more or less
unresponsive until I would force an unmount of the share.
I didn't think that recording only 1HD/1SD stream could have
problems with 100mbit full duplex, but I'm at a loss as to what other
settings I should be looking at. I'm hoping there are some buffer
settings with the new HD recorder code in SVN [that patch made some
shows watchable] but I'm not sure where to look for that, or if that
would even help me.
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

ylee at pobox

Dec 28, 2005, 10:30 PM

Post #10 of 32 (20541 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Jarod Wilson <jarod [at] wilsonet> says:
> Something odd going on there, I have an almost identical setup for
> my master backend, and NFS works great at handling multiple HDTV
> streams (both my HD capture devices are in slave backends that write
> to the master via NFS), along with the occasional SD stream thrown
> in as well.

To make a long story short, my 2.8TB JFS-on-LVM2-on-software RAID 5
array has certain mysterious performance bottlenecks that result in
7.5-10MB/s local write speeds, despite eight spindles and two 3Ware
7506 controller cards in JBOD mode and on separate buses and with
iowait never going above a few percentage points at the very
most. Over the network that's even slower, of course.

I've going to be rebuilding the array after backing it up to an
Infrant 600 (the clear winner, in my mind, among the recent spate of
"NAS in a shoebox" offerings popping up all over the place) with four
500GB drives I'll be temporarily using in RAID 0 for the time being,
and am looking forward to finding out whether the bottleneck is in
mdadm, lvm2 (the likeliest suspect in my mind; none of the several
academic self-assembled terabyte-RAID 5 array setups I've read
benchmarks for online seems to use it, although otherwise essentially
identical), JFS, or elsewhere. I would be surprised if, once I reach
write performance on the array closer to theoretical potential, I can
use NFS to stream recordings to it without any trouble. But in the
meanwhile I'm glad Samba lets me stream a HDTV recording while also
playing back a HDTV stream with no IOBOUND error messages (except the
very occasional one or two if I'm doing something disk- but non-MythTV
related on the frontend/backend MythTV box); after a month of butting
my head against seemingly-endless IOBOUND messages with NFS v3 that's
a good thing, believe me.

--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US

myth at dermanouelian

Dec 28, 2005, 10:37 PM

Post #11 of 32 (20558 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Dec 28, 2005, at 10:30 PM, Yeechang Lee wrote:

> To make a long story short, my 2.8TB JFS-on-LVM2-on-software RAID 5
> array has certain mysterious performance bottlenecks that result in
> 7.5-10MB/s local write speeds, despite eight spindles and two 3Ware
> 7506 controller cards in JBOD mode and on separate buses and with
> iowait never going above a few percentage points at the very
> most. Over the network that's even slower, of course.

This actually sounds pretty typical of RAID 5 to me. Every write
results in 2 simultaneous writes. Even slower when you're doing it in
software. Great for redundancy, terrible for throughput. That's the
trade-off.

jarod at wilsonet

Dec 28, 2005, 10:51 PM

Post #12 of 32 (20546 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Wednesday 28 December 2005 22:37, Brad DerManouelian wrote:
> On Dec 28, 2005, at 10:30 PM, Yeechang Lee wrote:
> > To make a long story short, my 2.8TB JFS-on-LVM2-on-software RAID 5
> > array has certain mysterious performance bottlenecks that result in
> > 7.5-10MB/s local write speeds, despite eight spindles and two 3Ware
> > 7506 controller cards in JBOD mode and on separate buses and with
> > iowait never going above a few percentage points at the very
> > most. Over the network that's even slower, of course.
>
> This actually sounds pretty typical of RAID 5 to me. Every write
> results in 2 simultaneous writes. Even slower when you're doing it in
> software. Great for redundancy, terrible for throughput. That's the
> trade-off.

Erm, software RAID 5 isn't *that* bad. I've got a sw RAID 5 array that only
consists of four ATA drives, and can write at a sustained rate of 30MB/s
without a problem. I've got a 5-drive U320 software RAID 5 array in another
machine that does ~80MB/s without a problem.

--
Jarod Wilson
jarod [at] wilsonet

jsamyth at gmail

Dec 29, 2005, 12:19 AM

Post #13 of 32 (20527 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:

> At least the following NFS mount options should be used if not already:
>
> mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
> local/mountpoint

On my slave I mount nfs with read/write sizes of 8192, which I think I got
from the mythtv docs at
http://www.mythtv.org/docs/mythtv-HOWTO-23.html#ss23.10

Would you expect see any improvement with larger sizes on a 100meg
network?

--
----------JSA---------
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

simon.lundell at his

Dec 29, 2005, 1:17 AM

Post #14 of 32 (20534 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Ryan Steffes wrote:

>
>
> On 12/29/05, *Greg Oliver* <goliver [at] cistera
> <mailto:goliver [at] cistera>> wrote:
>
> Jumbo frames are your friend
>
> On Wed, 2005-12-28 at 21:29 -0800, Jonathan Tidmore wrote:
> > On 12/28/05, Mike Frisch <mfrisch [at] isurfer
> <mailto:mfrisch [at] isurfer>> wrote:
> >
> > The "tcp" option may not be necessary but I haven't
> taken the
> > time to
> > benchmark between TCP and UDP.
> >
> >
> > For gigabit ethernet, the nfs people recommend TCP.
> >
> > From the man page for nfs (5):
> >
> > WARNINGS
> > Using NFS over UDP on high-speed links such as Gigabit can
> > cause silent data corruption.
> >
> > The problem can be triggered at high loads, and is caused
> > by problems in IP fragment reassembly. NFS
> > read and writes typically transmit UDP packets of 4 Kilobytes
> > or more, which have to be broken up into
> > several fragments in order to be sent over the Ethernet
> > link, which limits packets to 1500 bytes by
> > default. This process happens at the IP network layer and is
> > called fragmentation.
> >
> > They go into greater detail about this issue. For info use 'man 5
> > nfs'
> >
> >
> > -J
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users [at] mythtv <mailto:mythtv-users [at] mythtv>
> > http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
>
>
> Hmm, I wonder if this may be related to why nfs apparently locks my
> machine in heavy use. It started when I switched to gigabit nics and
> I was blaming the cards, until I realized I could do high data
> transfers with the computer as long as I don't use nfs.
>
> Ryan
>
>------------------------------------------------------------------------
>
>_______________________________________________
>mythtv-users mailing list
>mythtv-users [at] mythtv
>http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
>
I have had the same exact problem. The machine hardlocks, and a cold
boot is necissary. I eventually changed nics to a 100Mbit in the frontend.

//Simon
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mythtv at edsons

Dec 29, 2005, 1:37 AM

Post #15 of 32 (20514 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Mike Frisch wrote:

>
> On 28-Dec-05, at 10:16 PM, Greg Oliver wrote:
>
>> What mount option are you passing to the nfs share.. SMB versus nfs
>> and
>> smb being faster is impossible IMO.
>>
>
> Agreed.
>
> At least the following NFS mount options should be used if not already:
>
> mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
> local/mountpoint
>
> The "tcp" option may not be necessary but I haven't taken the time to
> benchmark between TCP and UDP.
>
> Mike.
>
The tcp option is very much needed in case of a mixed speed network
(mixed 100M and 1G segments, or mixed 100M and WiFi segments)

Rudy
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

ylee at pobox

Dec 29, 2005, 4:44 AM

Post #16 of 32 (20526 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Ryan Steffes wrote:
> Hmm, I wonder if this may be related to why nfs apparently locks my
> machine in heavy use. It started when I switched to gigabit nics
> and I was blaming the cards, until I realized I could do high data
> transfers with the computer as long as I don't use nfs.

Simon Lundell <simon.lundell [at] his> says:
> I have had the same exact problem. The machine hardlocks, and a cold
> = boot is necissary. I eventually changed nics to a 100Mbit in the
> frontend.

Might the two of you be using Marvell-based gigabit Ethernet cards
and/or SysKonnect drivers? After I moved from the built-in
Intel-based 100Mbps Ethernet jack on my MythTV frontend/backend to the
D-Link DGE-560T--a PCI Express 1x gigabit Ethernet card, for which I
had to download SysKonnect's own version of sk98lin--I found that I
could consistently crash the system hard by using dd or mv to max out
the traffic to my RAID 5 file server (which remained stable, and has
always had gigabit Ethernet), regardless of jumbo frames. I have not
seen this issue while simultaneously recording and playing back HD
streams, but might well do so once I start recording more than one HD
stream at once. I look forward to the core Linux kernel's own sk98lin
or skge drivers supporting my card, hopefully soon.

--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US

mfrisch at isurfer

Dec 29, 2005, 6:40 AM

Post #17 of 32 (20513 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 29-Dec-05, at 3:19 AM, John Andersen wrote:

> On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
>
>> At least the following NFS mount options should be used if not
>> already:
>>
>> mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
>> local/mountpoint
>
>
> On my slave I mount nfs with read/write sizes of 8192, which I
> think I got
> from the mythtv docs at
> http://www.mythtv.org/docs/mythtv-HOWTO-23.html#ss23.10
>
> Would you expect see any improvement with larger sizes on a 100meg
> network?

Yes.

mtdean at thirdcontact

Dec 29, 2005, 7:50 AM

Post #18 of 32 (20514 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Mike Frisch wrote:

> On 29-Dec-05, at 3:19 AM, John Andersen wrote:
>
>> On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
>>
>>> At least the following NFS mount options should be used if not
>>> already:
>>>
>>> mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
>>> local/mountpoint
>>
>> On my slave I mount nfs with read/write sizes of 8192, which I think
>> I got
>> from the mythtv docs at
>> http://www.mythtv.org/docs/mythtv-HOWTO-23.html#ss23.10
>>
>> Would you expect see any improvement with larger sizes on a 100meg
>> network?
>
> Yes.

If supported by your NFS server's kernel (and NFS version--but that
shouldn't be a problem)...

The maximum theoretical block size for NFSv2 is 8KiB (but you want
NFSv3), and Linux 2.4 kernels without support for NFS over TCP (pre
2.4.19) use 8KiB, while those 2.4 kernels that didn't support NFS over
TCP but were patched with the NFS over TCP patch use 32KiB and those
that support NFS over TCP without patching use 8KiB (the "traditional"
default was kept even after the TCP patch was included in the tree).
Linux 2.6 kernels support 32KiB block sizes by default. However, your
distribution vendor may have modified the kernel defaults. The maximum
theoretical sizes for NFSv3 are 56KiB over UDP and depends on the
implementation with TCP (but typically won't be greater than 32KiB).

You can check your kernel by looking at the value for NFSSVC_MAXBLKSIZE
in the include/linux/nfsd/const.h file in your kernel source directory.
You can also modify the value and recompile your kernel to get better
performance.

See http://tldp.org/HOWTO/NFS-HOWTO/performance.html#BLOCKSIZES and
http://nfs.sourceforge.net/ for more.

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

mtdean at thirdcontact

Dec 29, 2005, 8:00 AM

Post #19 of 32 (20518 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Greg Oliver wrote:

>SMB versus nfs and smb being faster is impossible IMO.
>
>
This is one case where you shouldn't fall for the "invented by
Microsoft" trap. Actually, performance on a properly tuned SMB share
(both client and server tuned) will quite possibly be greater than on a
properly tuned NFS share (client/server tuned). /However/, properly
tuning an SMB share is extremely difficult compared to tuning an NFS
share, so in the real world, you're likely to never see a properly tuned
SMB share (at least when either the client or the server is not
Microsoft-based, and with both client and server running MS, other
issues--like differences in cp/copy--are likely to obscure testing
results)--meaning SMB will typically appear less performant than NFS

Note, also, that a server tuned for NFS will likely have poor SMB
performance and vice-versa because the required options are very different.

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

jarod at wilsonet

Dec 29, 2005, 8:31 AM

Post #20 of 32 (20491 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Thursday 29 December 2005 04:44, Yeechang Lee wrote:
> Ryan Steffes wrote:
> > Hmm, I wonder if this may be related to why nfs apparently locks my
> > machine in heavy use. It started when I switched to gigabit nics
> > and I was blaming the cards, until I realized I could do high data
> > transfers with the computer as long as I don't use nfs.
>
> Simon Lundell <simon.lundell [at] his> says:
> > I have had the same exact problem. The machine hardlocks, and a cold
> > = boot is necissary. I eventually changed nics to a 100Mbit in the
> > frontend.
>
> Might the two of you be using Marvell-based gigabit Ethernet cards
> and/or SysKonnect drivers? After I moved from the built-in
> Intel-based 100Mbps Ethernet jack on my MythTV frontend/backend to the
> D-Link DGE-560T--a PCI Express 1x gigabit Ethernet card, for which I
> had to download SysKonnect's own version of sk98lin--I found that I
> could consistently crash the system hard by using dd or mv to max out
> the traffic to my RAID 5 file server (which remained stable, and has
> always had gigabit Ethernet), regardless of jumbo frames. I have not
> seen this issue while simultaneously recording and playing back HD
> streams, but might well do so once I start recording more than one HD
> stream at once. I look forward to the core Linux kernel's own sk98lin
> or skge drivers supporting my card, hopefully soon.

Hrm, I have a bit of experience with SK cards lately... I've found that while
(at least some of) the SK cards have hardware tcp segmentation offload
support, it doesn't work worth spit. In particular, this was on SysKonnect
SK9Sxx PCI-X dual GbE cards, and the problems happen under high load. You
might check if you have it on, and if so, turn it off, let the kernel do it
instead. Could help...

Check with:
# ethtool -k ethX:
Offload parameters for ethX:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

Turn it off with:
# ethtool -K ethX tso off

--
Jarod Wilson
jarod [at] wilsonet

myth at dermanouelian

Dec 29, 2005, 8:32 AM

Post #21 of 32 (20502 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Dec 28, 2005, at 10:51 PM, Jarod Wilson wrote:

> On Wednesday 28 December 2005 22:37, Brad DerManouelian wrote:
>> On Dec 28, 2005, at 10:30 PM, Yeechang Lee wrote:
>>> To make a long story short, my 2.8TB JFS-on-LVM2-on-software RAID 5
>>> array has certain mysterious performance bottlenecks that result in
>>> 7.5-10MB/s local write speeds, despite eight spindles and two 3Ware
>>> 7506 controller cards in JBOD mode and on separate buses and with
>>> iowait never going above a few percentage points at the very
>>> most. Over the network that's even slower, of course.
>>
>> This actually sounds pretty typical of RAID 5 to me. Every write
>> results in 2 simultaneous writes. Even slower when you're doing it in
>> software. Great for redundancy, terrible for throughput. That's the
>> trade-off.
>
> Erm, software RAID 5 isn't *that* bad. I've got a sw RAID 5 array
> that only
> consists of four ATA drives, and can write at a sustained rate of
> 30MB/s
> without a problem. I've got a 5-drive U320 software RAID 5 array in
> another
> machine that does ~80MB/s without a problem.

My point is that RAID 5 is *MUCH* slower than just about any other
RAID config and you should expect your max speed to be half of your
fastest drive. Put those U320's on RAID 0+1 and you'll likely double
your speed (and half your recoverability). That being said, the
original poster obviously hasn't got that kind of throughput and RAID
5 is cutting whatever he has in half.

Yeechang, what kind of drives are in your RAID 5? ATA-133? 7200 RPM?
8MB Cache?

-Brad

anaerin at gmail

Dec 29, 2005, 9:16 AM

Post #22 of 32 (20514 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/29/05, Brad DerManouelian <myth [at] dermanouelian> wrote:
> My point is that RAID 5 is *MUCH* slower than just about any other
> RAID config and you should expect your max speed to be half of your
> fastest drive. Put those U320's on RAID 0+1 and you'll likely double
> your speed (and half your recoverability). That being said, the
> original poster obviously hasn't got that kind of throughput and RAID
> 5 is cutting whatever he has in half.

Okay, with properly run hardware RAID, RAID 1 is the slowest (Data's
written to both drives at the speed of the slowest drive), RAID 5 is
next fastest (As the data spans the drives, so it's 1/3 faster than a
single drive or RAID 1), and RAID 0 is the very fastest (Data's
written to both drives simultaneously).

If you have the drives to spare (4 drives), and you don't mind
"Losing" half the capacity for the backup, then RAID 0+1 is the best
combination, giving high speed AND data redundancy (50% capacity, 200%
preformance).

If you don't have the drives (3 drives), or you wish to get more
capacity from the drives you have, then RAID 5 is best, as it gives
you 66% of the capacity of the drives and 133% of the performance.

If you have only 2 drives, you can opt to use them as RAID 1 (Which
will give you 50% capacity and 100% performance) or RAID 0 (Which
gives you 100% capacity (And 0% redundancy), and 200% performance).

This is all assuming you're using hardware RAID, which is not reliant
on CPU overheads or anything like that. Software RAID changes this, as
it uses the CPU and the regular IDE controller, but can't (generally)
write to multiple drives simultaneously.
--
Robert "Anaerin" Johnston
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

ylee at pobox

Dec 29, 2005, 12:33 PM

Post #23 of 32 (20502 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

Brad DerManouelian <myth [at] dermanouelian> says:
> Yeechang, what kind of drives are in your RAID 5? ATA-133? 7200 RPM? =
> 8MB Cache?

They are Seagate Barracuda.8 400GB PATA drives (I got a fantastic
price on 'em a year ago, the month after their debut). Not far from
the state of the art for drives even today, and needless to say each
with 7200RPM and 8MB cache. I'd be shocked if having gone SATA (and
paying $1000 more for the privilege; like I said, it was a fantastic
price) or another make/model would have made a difference, especiallly
since the 3ware cards virtualize the computer-drive connection into
its own SCSI-like interface regardless of drive. It's not a case of
one drive subtly slowing things down for everything else because a)
iostate measures for all drives are always fairly consistent and the
dd read benchmarks I've done for each drive directly (sd[a-h]) are
consistent. The dd read benchmarks I've done for /dev/md0
interestingly show about twice the performance of
/dev/VolGroup0x/LogVol0x, indicating to me that lvm2 does appear to be
a chokepoint, if not perhaps the only one; rebuilding the array and
doing empirical tests of different RAID/LVM/filesystem configurations
will presumably tell me more.
--
Yeechang Lee <ylee [at] pobox> | +1 650 776 7763 | San Francisco CA US

bigwavedave at gmail

Dec 29, 2005, 12:43 PM

Post #24 of 32 (20502 views)
Permalink

Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On 12/29/05, Yeechang Lee <ylee [at] pobox> wrote:
> Brad DerManouelian <myth [at] dermanouelian> says:
> > Yeechang, what kind of drives are in your RAID 5? ATA-133? 7200 RPM? =
> > 8MB Cache?
>
> They are Seagate Barracuda.8 400GB PATA drives (I got a fantastic
> price on 'em a year ago, the month after their debut). Not far from
> the state of the art for drives even today, and needless to say each
> with 7200RPM and 8MB cache. I'd be shocked if having gone SATA (and
> paying $1000 more for the privilege; like I said, it was a fantastic
> price) or another make/model would have made a difference, especiallly
> since the 3ware cards virtualize the computer-drive connection into
> its own SCSI-like interface regardless of drive. It's not a case of
> one drive subtly slowing things down for everything else because a)
> iostate measures for all drives are always fairly consistent and the
> dd read benchmarks I've done for each drive directly (sd[a-h]) are
> consistent. The dd read benchmarks I've done for /dev/md0
> interestingly show about twice the performance of
> /dev/VolGroup0x/LogVol0x, indicating to me that lvm2 does appear to be
> a chokepoint, if not perhaps the only one; rebuilding the array and
> doing empirical tests of different RAID/LVM/filesystem configurations
> will presumably tell me more.
So the drives are all connected to a 3ware card? This should be
running hardware RAID, making a significant improvement in
performance.

Dave

----------------------------------------------------------
Are Your Friends Lemmings?
-- http://www.lemmingshirts.com
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

adeffs at gmail

Dec 29, 2005, 7:32 PM

Post #25 of 32 (20505 views)
Permalink

Re: Re: Samba 3 better than NFS 3 for recording over Ethernet [In reply to]

On Thursday 29 December 2005 10:50, Michael T. Dean wrote:
> Mike Frisch wrote:
> > On 29-Dec-05, at 3:19 AM, John Andersen wrote:
> >> On 12/28/05, Mike Frisch <mfrisch [at] isurfer> wrote:
> >>> At least the following NFS mount options should be used if not
> >>> already:
> >>>
> >>> mount -o rsize=32768,wsize=32768,nfsvers=3,tcp server:/nfs/export /
> >>> local/mountpoint
> >>
> >> On my slave I mount nfs with read/write sizes of 8192, which I think
> >> I got
> >> from the mythtv docs at
> >> http://www.mythtv.org/docs/mythtv-HOWTO-23.html#ss23.10
> >>
> >> Would you expect see any improvement with larger sizes on a 100meg
> >> network?
> >
> > Yes.
>
> If supported by your NFS server's kernel (and NFS version--but that
> shouldn't be a problem)...
>
> The maximum theoretical block size for NFSv2 is 8KiB (but you want
> NFSv3), and Linux 2.4 kernels without support for NFS over TCP (pre
> 2.4.19) use 8KiB, while those 2.4 kernels that didn't support NFS over
> TCP but were patched with the NFS over TCP patch use 32KiB and those
> that support NFS over TCP without patching use 8KiB (the "traditional"
> default was kept even after the TCP patch was included in the tree).
> Linux 2.6 kernels support 32KiB block sizes by default. However, your
> distribution vendor may have modified the kernel defaults. The maximum
> theoretical sizes for NFSv3 are 56KiB over UDP and depends on the
> implementation with TCP (but typically won't be greater than 32KiB).
>
> You can check your kernel by looking at the value for NFSSVC_MAXBLKSIZE
> in the include/linux/nfsd/const.h file in your kernel source directory.
> You can also modify the value and recompile your kernel to get better
> performance.
>
> See http://tldp.org/HOWTO/NFS-HOWTO/performance.html#BLOCKSIZES and
> http://nfs.sourceforge.net/ for more.
>
> Mike

so for those of you with successful NFS mounts running over 100bT and 1000bT,
what mount settings are you using for your NFS mounts?

Also, someone mentioned TCP being best for mix speed networks, are these
settings for how the drive is mounted or by how NFS shares the drive? Would I
want the same mount settings on a 100bT wired computer as a 802.11G wireless?

I have to see what I ended up with as my settings, I'm hoping to get some
better performance out of my NFS shares, and solve any issues I might be
having with a mixed speed network.

share the love ;-)
--
Steve
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads