Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Users

Changing filesystems?

 

 

MythTV users RSS feed   Index | Next | Previous | View Threaded


mythtv at maniclogic

Feb 4, 2004, 9:56 PM

Post #1 of 22 (41064 views)
Permalink
Changing filesystems?

All,

Got a question hopefully someone can answer. Since upgrading to 14, it seems
that whenever I delete a recording it takes *FOREVER* to delete it and come
back to the UI. This seems to be especially bad when the backend is recording
and commflag'ing at the same time as the delete.

I've got reiserfs on my LVM where I store my recordings and I'm assuming that
all that time to delete is because of the huge number of inodes used for big
files.

Other particulars: there are two drives, 60G and 120G of which the LVM has
space on both; both are on one IDE channel (probably a bad thing); and the
journaling (or something) seems to be just eating up available space.

The question is: how can I change the reiserfs to a ext3 fs to take advantage
of the largefile inode support or will this make any difference at all.

TIA
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


jcw at wilsonet

Feb 5, 2004, 1:04 AM

Post #2 of 22 (40403 views)
Permalink
Re: Changing filesystems? [In reply to]

On Feb 4, 2004, at 20:56, Jeff wrote:

> Got a question hopefully someone can answer. Since upgrading to 14,
> it seems
> that whenever I delete a recording it takes *FOREVER* to delete it and
> come
> back to the UI. This seems to be especially bad when the backend is
> recording
> and commflag'ing at the same time as the delete.
>
> I've got reiserfs on my LVM where I store my recordings and I'm
> assuming that
> all that time to delete is because of the huge number of inodes used
> for big
> files.
>
> Other particulars: there are two drives, 60G and 120G of which the LVM
> has
> space on both; both are on one IDE channel (probably a bad thing); and
> the
> journaling (or something) seems to be just eating up available space.
>
> The question is: how can I change the reiserfs to a ext3 fs to take
> advantage
> of the largefile inode support or will this make any difference at all.

Ack. Most people have said that Reiser performs better than ext3 for
Myth, so I don't know that I'd recommend going that route. I've had
excellent results with XFS myself though.

--
Jarod C. Wilson, RHCE

Got a question? Read this first...
http://catb.org/~esr/faqs/smart-questions.html
MythTV, Fedora Core & ATrpms documentation:
http://wilsonet.com/mythtv/
MythTV Searchable Mailing List Archive
http://www.gossamer-threads.com/archive/MythTV_C2/
Attachments: PGP.sig (0.18 KB)


alden at math

Feb 5, 2004, 5:53 AM

Post #3 of 22 (40379 views)
Permalink
Re: Changing filesystems? [In reply to]

Hi,

On Thu, Feb 05, 2004 at 12:04:53AM -0800, Jarod C. Wilson wrote:
> On Feb 4, 2004, at 20:56, Jeff wrote:
>
> >Got a question hopefully someone can answer. Since upgrading to 14,
> >it seems
> >that whenever I delete a recording it takes *FOREVER* to delete it and
> >come
> >back to the UI. This seems to be especially bad when the backend is
> >recording
> >and commflag'ing at the same time as the delete.
> >
> >I've got reiserfs on my LVM where I store my recordings and I'm
> >assuming that
> >all that time to delete is because of the huge number of inodes used
> >for big
> >files.
> >
> >Other particulars: there are two drives, 60G and 120G of which the LVM
> >has
> >space on both; both are on one IDE channel (probably a bad thing); and
> >the
> >journaling (or something) seems to be just eating up available space.
> >
> >The question is: how can I change the reiserfs to a ext3 fs to take
> >advantage
> >of the largefile inode support or will this make any difference at all.
>
> Ack. Most people have said that Reiser performs better than ext3 for
> Myth, so I don't know that I'd recommend going that route. I've had
> excellent results with XFS myself though.

I used to run with ReiserFS, but I had the same problem -- deleteing a
large file took several seconds. I recently switched over to XFS and
have been very happy with it. Note that there is no easy way to switch
filesystems -- you basically need to backup the data, reformat and then
restore the data.

...dave


mythtv at maniclogic

Feb 5, 2004, 9:10 AM

Post #4 of 22 (40346 views)
Permalink
Re: Changing filesystems? [In reply to]

Quoting Dave Alden <alden [at] math>:

> Hi,
>
> On Thu, Feb 05, 2004 at 12:04:53AM -0800, Jarod C. Wilson wrote:
> > On Feb 4, 2004, at 20:56, Jeff wrote:
> >
> > >Got a question hopefully someone can answer. Since upgrading to 14,
> > >it seems
> > >that whenever I delete a recording it takes *FOREVER* to delete it and
> > >come
> > >back to the UI. This seems to be especially bad when the backend is
> > >recording
> > >and commflag'ing at the same time as the delete.
> > >
> > >I've got reiserfs on my LVM where I store my recordings and I'm
> > >assuming that
> > >all that time to delete is because of the huge number of inodes used
> > >for big
> > >files.
> > >
> > >Other particulars: there are two drives, 60G and 120G of which the LVM
> > >has
> > >space on both; both are on one IDE channel (probably a bad thing); and
> > >the
> > >journaling (or something) seems to be just eating up available space.
> > >
> > >The question is: how can I change the reiserfs to a ext3 fs to take
> > >advantage
> > >of the largefile inode support or will this make any difference at all.
> >
> > Ack. Most people have said that Reiser performs better than ext3 for
> > Myth, so I don't know that I'd recommend going that route. I've had
> > excellent results with XFS myself though.
>
> I used to run with ReiserFS, but I had the same problem -- deleteing a
> large file took several seconds. I recently switched over to XFS and
> have been very happy with it. Note that there is no easy way to switch
> filesystems -- you basically need to backup the data, reformat and then
> restore the data.
>
> ...dave
>

Jarod & Dave

Thanks for the replies - sadly that was the answer I suspected. Perhaps an
ignorant question but can LVM use XFS as the file system? I really like the
ability to add on space easily.




_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


myth at brieck

Feb 5, 2004, 9:35 AM

Post #5 of 22 (40358 views)
Permalink
Re: Changing filesystems? [In reply to]

Jeff wrote:

>
> Thanks for the replies - sadly that was the answer I suspected. Perhaps an
> ignorant question but can LVM use XFS as the file system? I really like the
> ability to add on space easily.
>

I use XFS and LVM, however I was unable to resize. I'm not sure if I was
doing something wrong or not, but I got the impression that you couldn't
use the advanced resizing features of LVM with XFS.


jcaputo1 at comcast

Feb 5, 2004, 10:38 AM

Post #6 of 22 (40355 views)
Permalink
Re: Changing filesystems? [In reply to]

On Thursday 05 February 2004 11:35, David Brieck Jr. wrote:
> Jeff wrote:
> > Thanks for the replies - sadly that was the answer I suspected.
> > Perhaps an ignorant question but can LVM use XFS as the file
> > system? I really like the ability to add on space easily.
>
> I use XFS and LVM, however I was unable to resize. I'm not sure if I
> was doing something wrong or not, but I got the impression that you
> couldn't use the advanced resizing features of LVM with XFS.

LVM doesn't have any "advanced resizing" features per se. LVM is simply
a way of grouping a number of physical devices into a single logical
device. The "advanced resizing" is nothing more than whatever tools
are available for the filesystem you choose to put on your LVM volume.
So, if you choose an ext2/ext3 filesystem, you can use extresize (or
whatever it's called) to grow or shrink you filesystem to occupy more/
less of the logical volume. In order for you to be able to do this
with XFS, you need a tool that can resize an XFS filesystem (I don't
know whether one exists or not). LVM does not really affect whether or
not you can do this.

-JAC

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


sbower at cisco

Feb 5, 2004, 11:00 AM

Post #7 of 22 (40351 views)
Permalink
Re: Changing filesystems? [In reply to]

>>>>> "Jeff" == Jeff <mythtv [at] maniclogic> writes:

Jeff> Got a question hopefully someone can answer. Since upgrading to
Jeff> 14, it seems that whenever I delete a recording it takes *FOREVER*
Jeff> to delete it and come back to the UI. This seems to be especially
Jeff> bad when the backend is recording and commflag'ing at the same
Jeff> time as the delete.

Jeff> I've got reiserfs on my LVM where I store my recordings and I'm
Jeff> assuming that all that time to delete is because of the huge
Jeff> number of inodes used for big files.

Misconception alert: one file uses exactly one inode, no matter how big
it is.

The delay likely comes from tracking the free data blocks in the
filesystem -- the more blocks are freed, the more updates to the free
list/bitmap are required.

Jeff> Other particulars: there are two drives, 60G and 120G of which the
Jeff> LVM has space on both; both are on one IDE channel (probably a bad
Jeff> thing); and the journaling (or something) seems to be just eating
Jeff> up available space.

Jeff> The question is: how can I change the reiserfs to a ext3 fs to
Jeff> take advantage of the largefile inode support or will this make
Jeff> any difference at all.

I've found ext3 to be very slow at deleting large files; note that the
"largefile" options only change the number of inodes in the filesystem,
which is useful for space savings but has nothing to do with file
deletion speed.

I've been using jfs on my recordings directory (it came with RH9), and
it's been MUCH faster at deleting files. AAMOF, deleting my 45G HDTV
superbowl recording took about 5 seconds; deleting a test 10G file on an
ext3 filesystem took 17 seconds.

Steve.


bluesguy_1 at yahoo

Feb 5, 2004, 7:50 PM

Post #8 of 22 (40322 views)
Permalink
Re: Changing filesystems? [In reply to]

--- "Joseph A. Caputo" <jcaputo1 [at] comcast> wrote:
> On Thursday 05 February 2004 11:35, David Brieck Jr.
> wrote:
> > Jeff wrote:
> > > Thanks for the replies - sadly that was the
> answer I suspected.
> > > Perhaps an ignorant question but can LVM use XFS
> as the file
> > > system? I really like the ability to add on
> space easily.
> >
> > I use XFS and LVM, however I was unable to resize.
> I'm not sure if I
> > was doing something wrong or not, but I got the
> impression that you
> > couldn't use the advanced resizing features of LVM
> with XFS.
>

I don't use XFS, but it does have tools for this.
From xfsprogs in Axel's repository:

[root [at] mythbo root]# rpm -ql
xfsprogs-2.6.2-0_7.rhfc1.at

/lib
/lib/libhandle.so.1
/lib/libhandle.so.1.0.2
/sbin
/sbin/fsck.xfs
/sbin/mkfs.xfs
/sbin/xfs_repair
/usr/sbin
/usr/sbin/xfs_admin
/usr/sbin/xfs_bmap
/usr/sbin/xfs_check
/usr/sbin/xfs_copy
/usr/sbin/xfs_db
/usr/sbin/xfs_freeze
/usr/sbin/xfs_growfs
/usr/sbin/xfs_info
/usr/sbin/xfs_io
/usr/sbin/xfs_logprint
/usr/sbin/xfs_mkfile
/usr/sbin/xfs_ncheck
/usr/sbin/xfs_rtcp
<snip>

xfs_growfs is the tool for what you're talking about...

__________________________________
Do you Yahoo!?
Yahoo! Finance: Get your refund fast by filing online.
http://taxes.yahoo.com/filing.html


bluesguy_1 at yahoo

Feb 6, 2004, 7:05 AM

Post #9 of 22 (40316 views)
Permalink
Re: Changing filesystems? [In reply to]

Ok, I promise I'm not going to make a habit out of
replying to myself, but I wanted to clear up something
Steve said in the post above me, because it's
misleading and I couldn't let it slide...

He says, "one file uses exactly one inode, no matter
how big it is.", which isn't true for EXT3. This is
*mostly* true for a filesystem like XFS, that uses
dynamic inode allocations and a btree free list, but
EXT3 *DOES* use inodes per block size, dictated by the
args when you create the filesystem; thus the reason
you really want to use the "-T largefile4" option when
you create it. This reduces the inode count on the
disk, and causes less disk thrashing and I/O problems
when you delete or create large files. (e.g. 2GB+)

FYI - The -T argument effectively sets the bytes/inode
ratio as you would with the -i argument...

On the other hand, XFS is a magic in and of itself,
and that's a story for some other time (because I
don't use it...) :-)


Greg


__________________________________
Do you Yahoo!?
Yahoo! Finance: Get your refund fast by filing online.
http://taxes.yahoo.com/filing.html


sbower at cisco

Feb 6, 2004, 12:23 PM

Post #10 of 22 (40343 views)
Permalink
Re: Changing filesystems? [In reply to]

>>>>> "Greg" == Blues Guy <bluesguy_1 [at] yahoo> writes:

Greg> Ok, I promise I'm not going to make a habit out of replying to
Greg> myself, but I wanted to clear up something Steve said in the post
Greg> above me, because it's misleading and I couldn't let it slide...

Likewise! :-)

Greg> He says, "one file uses exactly one inode, no matter how big it
Greg> is.", which isn't true for EXT3. This is *mostly* true for a
Greg> filesystem like XFS, that uses dynamic inode allocations and a
Greg> btree free list, but EXT3 *DOES* use inodes per block size,
Greg> dictated by the args when you create the filesystem; thus the
Greg> reason you really want to use the "-T largefile4" option when you
Greg> create it. This reduces the inode count on the disk, and causes
Greg> less disk thrashing and I/O problems when you delete or create
Greg> large files. (e.g. 2GB+)

Greg, when you say "EXT3 *DOES* use inodes per block size", you're
really referring to the filesystem creation process -- it *creates* so
many inodes per block size, and that's all the inodes you'll ever have
in that filesystem.

When I say "one file uses exactly one inode, no matter how big it is",
I'm referring to operational activities on individual files within the
filesystem, _after_ the filesystem has been created and mounted.

Using the "-T largefile*" option does reduce the number of inodes that
are created when you build the filesystem, allowing more of the disk
space to be used for actual file data, rather than storing the inodes
themselves.

However, when you create a file (NOT a filesystem) and write data to it,
that file is defined by exactly one inode. Every ext2/3 architecture
document states this.

As the file grows, more and more data blocks are allocated to it; as the
blocks are allocated, they are marked as "in use" in the block bitmaps.

No additional inodes are allocated to a single file as it grows -- all
of the data block pointers exist in the inode, progressing to indirect
blocks (data blocks that contain pointers to other datablocks),
double-indirect and triple-indirect blocks.

If a file did get allocated more inodes as it got bigger, how would
creating a filesystem with fewer inodes help? Wouldn't you run out of
inodes even with relatively few large files?

Anyways, when a file gets deleted, all of the data blocks that had been
allocated to it must be marked as free. This occurs in the block-group
block bitmaps (not in the data blocks themselves), and reading/writing
the bitmaps across all of the block groups is what I believe makes the
deletion process take longer for larger files.

Greg> FYI - The -T argument effectively sets the bytes/inode ratio as
Greg> you would with the -i argument...

Yes.

Greg> On the other hand, XFS is a magic in and of itself, and that's a
Greg> story for some other time (because I don't use it...) :-)

Agreed! :-)

Steve.

P.S. We can discuss this off-list if you like, and report back after we
come to an agreement! :-)


mythtv at maniclogic

Feb 6, 2004, 1:49 PM

Post #11 of 22 (40375 views)
Permalink
Re: Changing filesystems? [In reply to]

Quoting Steve Bower <sbower [at] cisco>:

> >>>>> "Greg" == Blues Guy <bluesguy_1 [at] yahoo> writes:
>
> Greg> Ok, I promise I'm not going to make a habit out of replying to
> Greg> myself, but I wanted to clear up something Steve said in the post
> Greg> above me, because it's misleading and I couldn't let it slide...
>
> Likewise! :-)
>
> Greg> He says, "one file uses exactly one inode, no matter how big it
> Greg> is.", which isn't true for EXT3. This is *mostly* true for a
> Greg> filesystem like XFS, that uses dynamic inode allocations and a
> Greg> btree free list, but EXT3 *DOES* use inodes per block size,
> Greg> dictated by the args when you create the filesystem; thus the
> Greg> reason you really want to use the "-T largefile4" option when you
> Greg> create it. This reduces the inode count on the disk, and causes
> Greg> less disk thrashing and I/O problems when you delete or create
> Greg> large files. (e.g. 2GB+)
>
> Greg, when you say "EXT3 *DOES* use inodes per block size", you're
> really referring to the filesystem creation process -- it *creates* so
> many inodes per block size, and that's all the inodes you'll ever have
> in that filesystem.
>
> When I say "one file uses exactly one inode, no matter how big it is",
> I'm referring to operational activities on individual files within the
> filesystem, _after_ the filesystem has been created and mounted.
>
> Using the "-T largefile*" option does reduce the number of inodes that
> are created when you build the filesystem, allowing more of the disk
> space to be used for actual file data, rather than storing the inodes
> themselves.
>
> However, when you create a file (NOT a filesystem) and write data to it,
> that file is defined by exactly one inode. Every ext2/3 architecture
> document states this.
>
> As the file grows, more and more data blocks are allocated to it; as the
> blocks are allocated, they are marked as "in use" in the block bitmaps.
>
> No additional inodes are allocated to a single file as it grows -- all
> of the data block pointers exist in the inode, progressing to indirect
> blocks (data blocks that contain pointers to other datablocks),
> double-indirect and triple-indirect blocks.
>
> If a file did get allocated more inodes as it got bigger, how would
> creating a filesystem with fewer inodes help? Wouldn't you run out of
> inodes even with relatively few large files?
>
> Anyways, when a file gets deleted, all of the data blocks that had been
> allocated to it must be marked as free. This occurs in the block-group
> block bitmaps (not in the data blocks themselves), and reading/writing
> the bitmaps across all of the block groups is what I believe makes the
> deletion process take longer for larger files.
>
> Greg> FYI - The -T argument effectively sets the bytes/inode ratio as
> Greg> you would with the -i argument...
>
> Yes.
>
> Greg> On the other hand, XFS is a magic in and of itself, and that's a
> Greg> story for some other time (because I don't use it...) :-)
>
> Agreed! :-)
>
> Steve.
>
> P.S. We can discuss this off-list if you like, and report back after we
> come to an agreement! :-)
>

Seeing as how I started this...my original post was flawed as pointed out by
Steve, I said used a lot of inodes when I really meant block pointers (was just
typing away). Anyway, thanks everyone for the info, my greatest fears were
realized <g> and I'll probably have to find room on another box to copy off my
70G worth of recordings and create the new file system. Was hoping to skate by
and was looking for an easy way to change fs to no avail.

Jeff

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


wepprop at sbcglobal

Feb 6, 2004, 3:47 PM

Post #12 of 22 (40306 views)
Permalink
Re: Changing filesystems? [In reply to]

Jeff wrote:
>
> Seeing as how I started this...my original post was flawed as pointed out by
> Steve, I said used a lot of inodes when I really meant block pointers (was just
> typing away). Anyway, thanks everyone for the info, my greatest fears were
> realized <g> and I'll probably have to find room on another box to copy off my
> 70G worth of recordings and create the new file system. Was hoping to skate by
> and was looking for an easy way to change fs to no avail.
>
> Jeff

FWIW, since this thread started, and since I too have have experienced
problems (dropped frames) due to file deletions, I did some online
research on web regarding linux filesystem performance. Most of the
published benchmarks are designed to measure raw i/o performance and,
from those benchmarks, one would be be inclined to select ReiserFS or
XFS. However, the Mongo benchmark from Namesys does measure file
deletions as one of it's performance sub-categories and in that category
(and that category alone), JFS was the clear hands-down winner.

Based on that, I ran an experiment to compare the deletion time of a
single 10G file on an ext3 partition and on a JFS partition. The
results were 10.2 seconds to delete the file on the ext3 partition and
0.9 seconds to delete the file on the JFS partition.

Note that the time was obtained by using the following command for the
file delete: "hwclock;rm -f bigfile;hwclock" and then subtracting the
two times, so there is undoubtedly a certain amount of system overhead
in there, but it still seems clear that there is at least an order of
magnitude improvement with JFS relative to ext3.

Since file deletion appears to be a limiting condition for a myth video
partition, and raw i/o performance does not, I have converted my video
partition to JFS. Anyone who has experienced dropped frames or other
problems due to file deletes might want to at least try using JFS in
that role and see if it helps them.

Bill


mythtv at lds

Feb 6, 2004, 4:49 PM

Post #13 of 22 (40315 views)
Permalink
Re: Changing filesystems? [In reply to]

----- Original Message -----
From: "William Powers" <wepprop [at] sbcglobal>
To: "Discussion about mythtv" <mythtv-users [at] mythtv>
Sent: Friday, February 06, 2004 4:47 PM
Subject: Re: [mythtv-users] Changing filesystems?


> Based on that, I ran an experiment to compare the deletion time of a
> single 10G file on an ext3 partition and on a JFS partition. The
> results were 10.2 seconds to delete the file on the ext3 partition and
> 0.9 seconds to delete the file on the JFS partition.

You didn't by any chance do the same test with ReiserFS did you?
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


wepprop at sbcglobal

Feb 7, 2004, 12:51 PM

Post #14 of 22 (40348 views)
Permalink
Re: Changing filesystems? [In reply to]

>> Based on that, I ran an experiment to compare the deletion time of a
>> single 10G file on an ext3 partition and on a JFS partition. The
>> results were 10.2 seconds to delete the file on the ext3 partition
>> and 0.9 seconds to delete the file on the JFS partition.

> You didn't by any chance do the same test with ReiserFS did you?

I did not test anything except JFS and Ext3 the first time around, but
I have since tried to do a slightly more comprehensive test by
attempting the single 10G file delete test for all five of the relevant
file systems. However, I had some issues and wasn't able to complete
the full test as I had planned, this on an RHFC1 system with the Myth
components installed via ATRpm's.

JFS: Repeated the 10G file delete and got 0.9 seconds again

ReiserFS: Got 6.2 seconds for the 10G file delete

Ext2: Got 1.6 seconds for the 10G file delete

Ext3: For some unknown reason, I was not able to (re)format the
partition using 'mkfs.ext3 -T largefile4 -m 0 /dev/hdb4' as it was
originally without encountering a floating point exception. After
formatting the partition using 'mkfs.ext3 -m 0 /dev/hdb4', I repeated
the 10G file delete and got 2.3 seconds. At this point, I have no idea
why I was unable to use '-T largefile4', nor do I know whether the
apparent difference in times between partitions formatted with and
without 'largefile4' is significant.

XFS: For some reason, attempting to write a 10G file to the XFS
formatted partition locked up the PC (hard) at about the 4G point, each
and every time. So I am unable to perform the 10G delete test with XFS
unless and until I find out what my 4G problem is.

So, that is where things stand at the moment...

Bill


alden at math

Feb 7, 2004, 1:58 PM

Post #15 of 22 (40320 views)
Permalink
Re: Changing filesystems? [In reply to]

Hi,
Sorry -- I just reread this and realized that I wrote "XFS". I'm actually running
JFS, not XFS. I was considering both, but the in the end I decided to go with JFS
since it comes with Fedora.
...dave

On Thu, Feb 05, 2004 at 08:10:54AM -0800, Jeff wrote:
> Quoting Dave Alden <alden [at] math>:
>
> > Hi,
> >
> > On Thu, Feb 05, 2004 at 12:04:53AM -0800, Jarod C. Wilson wrote:
> > > On Feb 4, 2004, at 20:56, Jeff wrote:
> > >
> > > >Got a question hopefully someone can answer. Since upgrading to 14,
> > > >it seems
> > > >that whenever I delete a recording it takes *FOREVER* to delete it and
> > > >come
> > > >back to the UI. This seems to be especially bad when the backend is
> > > >recording
> > > >and commflag'ing at the same time as the delete.
> > > >
> > > >I've got reiserfs on my LVM where I store my recordings and I'm
> > > >assuming that
> > > >all that time to delete is because of the huge number of inodes used
> > > >for big
> > > >files.
> > > >
> > > >Other particulars: there are two drives, 60G and 120G of which the LVM
> > > >has
> > > >space on both; both are on one IDE channel (probably a bad thing); and
> > > >the
> > > >journaling (or something) seems to be just eating up available space.
> > > >
> > > >The question is: how can I change the reiserfs to a ext3 fs to take
> > > >advantage
> > > >of the largefile inode support or will this make any difference at all.
> > >
> > > Ack. Most people have said that Reiser performs better than ext3 for
> > > Myth, so I don't know that I'd recommend going that route. I've had
> > > excellent results with XFS myself though.
> >
> > I used to run with ReiserFS, but I had the same problem -- deleteing a
> > large file took several seconds. I recently switched over to XFS and
> > have been very happy with it. Note that there is no easy way to switch
> > filesystems -- you basically need to backup the data, reformat and then
> > restore the data.
> >
> > ...dave
> >
>
> Jarod & Dave
>
> Thanks for the replies - sadly that was the answer I suspected. Perhaps an
> ignorant question but can LVM use XFS as the file system? I really like the
> ability to add on space easily.
>
>
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


bluesguy at stny

Feb 7, 2004, 3:17 PM

Post #16 of 22 (40302 views)
Permalink
Re: Changing filesystems? [In reply to]

William Powers wrote:

> > You didn't by any chance do the same test with ReiserFS did you?
>
> <Bill talks about filesystem performance metrics...>


Most of your results are in sync with the test done here..
http://aurora.zemris.fer.hr/filesystems/

specifically the big_file test results here:
http://aurora.zemris.fer.hr/filesystems/big.html

Judging from these results, xfs would be the best overall filesystem for
Myth boxes, as it only lags slightly behind jfs in deletes, and is
faster in writes... This is probably the reason so many people swear by
it... duh! 8-)

And reiserfs looks like it scores pretty bad in deletes. This test is
with an older kernel, so things may have changed slightly, but I'd wager
it's still pretty accurate.


wepprop at sbcglobal

Feb 7, 2004, 5:21 PM

Post #17 of 22 (40376 views)
Permalink
Re: Changing filesystems? [In reply to]

Ok, now I'm replying to myself...

The mkfs.ext3 problem was caused by some peculiarity in the partition
table. Deleting and recreating the partition fixed it. It did not,
unfortunately, fix my XFS 4G problem...

With the mkfs.ext3 problem fixed, I was able to confirm that it does
take longer to delete a file on an ext3 filesystem when that filesystem
was formatted -T largefile4. Two instances of deleting a 10G file on a
filesystem formatted 'mkfs.ext3 -m 0' resulted in times of 1.4 and 2.3
seconds. Two attempts at deleting a 10G file on a filesystem formatted
'mkfs.ext3 -T largefile4 -m 0' took 5.9 and 10.2 seconds.

From this I would personally conclude that, if you want to use ext3 for
your video partition, the extra storage space I get by formatting '-T
largefile4' (about 2G more available space on a 100G partition) is not
worth the substantial decrease in file deletion performance.

Now, if I could just figure out my XFS problem...

Bill

William Powers wrote:
> >> Based on that, I ran an experiment to compare the deletion time of a
> >> single 10G file on an ext3 partition and on a JFS partition. The
> >> results were 10.2 seconds to delete the file on the ext3 partition
> >> and 0.9 seconds to delete the file on the JFS partition.
>
> > You didn't by any chance do the same test with ReiserFS did you?
>
> I did not test anything except JFS and Ext3 the first time around, but
> I have since tried to do a slightly more comprehensive test by
> attempting the single 10G file delete test for all five of the relevant
> file systems. However, I had some issues and wasn't able to complete
> the full test as I had planned, this on an RHFC1 system with the Myth
> components installed via ATRpm's.
>
> JFS: Repeated the 10G file delete and got 0.9 seconds again
>
> ReiserFS: Got 6.2 seconds for the 10G file delete
>
> Ext2: Got 1.6 seconds for the 10G file delete
>
> Ext3: For some unknown reason, I was not able to (re)format the
> partition using 'mkfs.ext3 -T largefile4 -m 0 /dev/hdb4' as it was
> originally without encountering a floating point exception. After
> formatting the partition using 'mkfs.ext3 -m 0 /dev/hdb4', I repeated
> the 10G file delete and got 2.3 seconds. At this point, I have no idea
> why I was unable to use '-T largefile4', nor do I know whether the
> apparent difference in times between partitions formatted with and
> without 'largefile4' is significant.
>
> XFS: For some reason, attempting to write a 10G file to the XFS
> formatted partition locked up the PC (hard) at about the 4G point, each
> and every time. So I am unable to perform the 10G delete test with XFS
> unless and until I find out what my 4G problem is.
>
> So, that is where things stand at the moment...
>
> Bill
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


wepprop at sbcglobal

Feb 7, 2004, 10:06 PM

Post #18 of 22 (41710 views)
Permalink
Re: Changing filesystems? [In reply to]

My last reply to myself. Based on a Googled reference, I was able to
break my XFS 4G file size barrier by formatting the partition 'mkfs.xfs
-dagsize=4g'. So, here are the complete results:

Time to delete a 10G file, fastest to slowest:

JFS: 0.9s, 0.9s
XFS: 1.3s
EXT3: 1.4s, 2.3s
EXT2: 1.6s
REISERFS: 6.2s
EXT3 -T largefile4: 5.9s, 10.2s

After running the XFS test, there didn't seem to be any point in
reformatting the partition again, so I left it on XFS, but I think I
would be happy with JFS, XFS, or EXT3 w/o '-T largefile4'.

Bill

William Powers wrote:
> Ok, now I'm replying to myself...
>
> The mkfs.ext3 problem was caused by some peculiarity in the partition
> table. Deleting and recreating the partition fixed it. It did not,
> unfortunately, fix my XFS 4G problem...
>
> With the mkfs.ext3 problem fixed, I was able to confirm that it does
> take longer to delete a file on an ext3 filesystem when that filesystem
> was formatted -T largefile4. Two instances of deleting a 10G file on a
> filesystem formatted 'mkfs.ext3 -m 0' resulted in times of 1.4 and 2.3
> seconds. Two attempts at deleting a 10G file on a filesystem formatted
> 'mkfs.ext3 -T largefile4 -m 0' took 5.9 and 10.2 seconds.
>
> From this I would personally conclude that, if you want to use ext3 for
> your video partition, the extra storage space I get by formatting '-T
> largefile4' (about 2G more available space on a 100G partition) is not
> worth the substantial decrease in file deletion performance.
>
> Now, if I could just figure out my XFS problem...
>
> Bill
>
> William Powers wrote:
>
>> >> Based on that, I ran an experiment to compare the deletion time of a
>> >> single 10G file on an ext3 partition and on a JFS partition. The
>> >> results were 10.2 seconds to delete the file on the ext3 partition
>> >> and 0.9 seconds to delete the file on the JFS partition.
>>
>> > You didn't by any chance do the same test with ReiserFS did you?
>>
>> I did not test anything except JFS and Ext3 the first time around,
>> but I have since tried to do a slightly more comprehensive test by
>> attempting the single 10G file delete test for all five of the
>> relevant file systems. However, I had some issues and wasn't able to
>> complete the full test as I had planned, this on an RHFC1 system with
>> the Myth components installed via ATRpm's.
>>
>> JFS: Repeated the 10G file delete and got 0.9 seconds again
>>
>> ReiserFS: Got 6.2 seconds for the 10G file delete
>>
>> Ext2: Got 1.6 seconds for the 10G file delete
>>
>> Ext3: For some unknown reason, I was not able to (re)format the
>> partition using 'mkfs.ext3 -T largefile4 -m 0 /dev/hdb4' as it was
>> originally without encountering a floating point exception. After
>> formatting the partition using 'mkfs.ext3 -m 0 /dev/hdb4', I repeated
>> the 10G file delete and got 2.3 seconds. At this point, I have no
>> idea why I was unable to use '-T largefile4', nor do I know whether
>> the apparent difference in times between partitions formatted with and
>> without 'largefile4' is significant.
>>
>> XFS: For some reason, attempting to write a 10G file to the XFS
>> formatted partition locked up the PC (hard) at about the 4G point,
>> each and every time. So I am unable to perform the 10G delete test
>> with XFS unless and until I find out what my 4G problem is.
>>
>> So, that is where things stand at the moment...
>>
>> Bill
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> mythtv-users mailing list
>> mythtv-users [at] mythtv
>> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users [at] mythtv
> http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


mythtv at lds

Feb 7, 2004, 10:29 PM

Post #19 of 22 (40350 views)
Permalink
Re: Changing filesystems? [In reply to]

On Saturday 07 February 2004 11:06 pm, William Powers wrote:
> My last reply to myself. Based on a Googled reference, I was able to
> break my XFS 4G file size barrier by formatting the partition 'mkfs.xfs
> -dagsize=4g'. So, here are the complete results:
>
> Time to delete a 10G file, fastest to slowest:
>
> JFS: 0.9s, 0.9s
> XFS: 1.3s
> EXT3: 1.4s, 2.3s
> EXT2: 1.6s
> REISERFS: 6.2s
> EXT3 -T largefile4: 5.9s, 10.2s

Very cool. I was using ReiserFS since I'd read on this mailing list that it
was much faster for larger files? When my new drive comes in I think I'll
just go back to ext3 which is what I usually use.

Thanks
Malcolm

_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users


bob at smalltime

Feb 7, 2004, 11:06 PM

Post #20 of 22 (40405 views)
Permalink
Re: Changing filesystems? [In reply to]

> My last reply to myself. Based on a Googled reference, I was able to
> break my XFS 4G file size barrier by formatting the partition 'mkfs.xfs
> -dagsize=4g'. So, here are the complete results:
>
> Time to delete a 10G file, fastest to slowest:
>
> JFS: 0.9s, 0.9s
> XFS: 1.3s
> EXT3: 1.4s, 2.3s
> EXT2: 1.6s
> REISERFS: 6.2s
> EXT3 -T largefile4: 5.9s, 10.2s
>
> After running the XFS test, there didn't seem to be any point in
> reformatting the partition again, so I left it on XFS, but I think I
> would be happy with JFS, XFS, or EXT3 w/o '-T largefile4'.

Interesting. If others care to weigh in, I can either re-write the
"Advanced Partitioning" section in the HOWTO, or whack it completely.

William, can you give some background on the hardware used for your
tests? I'd be curious if this data holds up across various drive types,
LVM, etc. (Without trying to exhaustively test all the possibilities,
that is)


wepprop at sbcglobal

Feb 8, 2004, 2:33 AM

Post #21 of 22 (40356 views)
Permalink
Re: Changing filesystems? [In reply to]

Robert Kulagowski wrote:

> Interesting. If others care to weigh in, I can either re-write the
> "Advanced Partitioning" section in the HOWTO, or whack it completely.
>
> William, can you give some background on the hardware used for your
> tests? I'd be curious if this data holds up across various drive types,
> LVM, etc. (Without trying to exhaustively test all the possibilities,
> that is)

Athlon XP 1800+ on an NForce2 motherboard w/ 256M. The system is RHFC1
with Myth, ivtv, alsa, etc., installed via ATrpms, and it resides on an
ext3 partition on a Maxtor 20G, 7200 rpm drive. The video partition is
on a separate Western Digital 120G, 7200 rpm drive. I've got the
acoustic management of both drives on quiet settings for the obvious
reason. I'm not using LVM. To run the tests, I deleted enough video
files that I could move the remaining ones over to the system drive,
leaving the entire video partition free to be reformatted for the tests.

I don't claim the tests are all that precise, because I didn't hack the
kernel or do anything fancy in order to obtain extremely accurate times.
Nor did I repeat the tests enough times to really understand how the
results might vary, mostly because it takes so darn long to write the
10G file. I was really just curious to see if any of the various
filesystem's delete performance was substantially better, or worse, than
any of the others.

It appears, based on my personal experience alone, that file deletes are
the only system operations that can stress the hard drive enough to
produce dropped frames. Unfortunately, as others have pointed out,
recordings and deletions go together in Myth. So, unusual as it may be,
it does make at least some sense to take file deletion performance into
account when deciding which filesystem to use for a video partition,
especially for people with multiple tuners.

The really ironic result from my personal perspective is that it would
appear that using the '-T largefile4' setting for ext3, which I was so
pleased with because it give me an extra 2G of storage, may well have
been responsible for all those recordings I had ruined by frame drops.

Assuming it works out, though, I could really get to like this XFS
filesystem because it appears to give me slightly more storage space
than ext3 w/ '-T largefile4' did and it has pretty fast deletes as well.

:)

Bill


linux at johannes-niess

Feb 8, 2004, 3:20 AM

Post #22 of 22 (40384 views)
Permalink
Re: Changing filesystems? [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am Sonntag, 8. Februar 2004 10:33 schrieb William Powers:
> It appears, based on my personal experience alone, that file deletes are
> the only system operations that can stress the hard drive enough to
> produce dropped frames. Unfortunately, as others have pointed out,
> recordings and deletions go together in Myth. So, unusual as it may be,
> it does make at least some sense to take file deletion performance into
> account when deciding which filesystem to use for a video partition,
> especially for people with multiple tuners.

When I think about it, we don't need to delete files when expiring but can
write over them:

1) mv expired.file new.file
2) open new.file
A) maybe truncate (so FS info is correct)
4) write
B) maybe truncate to finished size

We just need one of A or B. Option B avoids freeing sector for the OS but
leaves us with a undefined end of file, so jumping forward too much ends up
in content of expired.file. Just the size difference needs to be freed (at
end of recording) or claimed (as needed). Option A might result in much of
the effects of deletion, but the OS could free the sectors asyncronously,
after returning from the truncation call. Timing results of "echo >
10gb.file" would be interesting.

Johannes Nieß
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAJg2EZ86b9aw2E+0RAp0XAJ0fdUoDT/rt2fyurLXxXhwheKLGsQCgn3YJ
flaMjtpglC9cbpk8a8l+TfY=
=P1RI
-----END PGP SIGNATURE-----
_______________________________________________
mythtv-users mailing list
mythtv-users [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-users

MythTV users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.