Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: MythTV: Dev

Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean

  

 
MythTV dev RSS feed   Index | Next | Previous | View Threaded


simon at koala

Feb 2, 2010, 12:45 AM

Post #1 of 14 (2219 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean
On 02/02/2010 04:07, mythtv [at] cvs wrote:
> Author: mdean
> Date: 2010-02-02 04:07:41 +0000 (Tue, 02 Feb 2010)
> New Revision: 23435
> Changeset: http://svn.mythtv.org/trac/changeset/23435
>
> Modified:
>
> trunk/mythtv/libs/libmythtv/tv_rec.cpp
> trunk/mythtv/programs/mythbackend/mainserver.cpp
>
> Log:
>
> Restore the minimum truncate rate that was removed in [12167] to shorten truncate times on systems with few capture cards, and update the max bitrate used for calculating truncate rate to better describe some seen-in-the-wild bitrates.
>
> Before this change, a system with only one configured capture card and with "Delete files slowly" enabled could take as long as 7min 24sec per GiB to delete a file. After this change, the slowest the backend will delete files is 2min 8sec per GiB (for all systems with 1 or 2 capture cards defined). With 3 or more cards (or virtual cards, as used for multirec) defined, truncation will occur at 26.64Mbits * number of cards per second. Or, put another way, truncation will occur at about 3.17MiB/sec/capture card or 8MiB/sec, whichever is higher.
>
i was wondering if slowly deleting a file was a good idea?

i'm thinking of the situation of another recording being in progress?
depending on the file system being used, the end result could be a lot
of disk fragmentation
--
simon
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


danielk at cuymedia

Feb 2, 2010, 6:13 AM

Post #2 of 14 (2154 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On Tue, 2010-02-02 at 08:45 +0000, Simon Kenyon wrote:
> On 02/02/2010 04:07, mythtv [at] cvs wrote:

> i was wondering if slowly deleting a file was a good idea?
>
> i'm thinking of the situation of another recording being in progress?
> depending on the file system being used, the end result could be a lot
> of disk fragmentation

The way to avoid that is to set the reserved size to a larger value. By
default it is 1GB, but if you want to use ext3 or another file system
which requires slow deletes you should set it at 20-30 GB. XFS is still
the best choice for MythTV media storage as it is fast, doesn't require
slow deletes and has an online defragmenter you can run nightly. VFS
is still good, and when properly configured, so is ext4.

-- Daniel


_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


mtdean at thirdcontact

Feb 2, 2010, 8:38 AM

Post #3 of 14 (2145 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On 02/02/2010 09:13 AM, Daniel Kristjansson wrote:
> On Tue, 2010-02-02 at 08:45 +0000, Simon Kenyon wrote:
>> i was wondering if slowly deleting a file was a good idea?
>>
>> i'm thinking of the situation of another recording being in progress?
>> depending on the file system being used, the end result could be a lot
>> of disk fragmentation
>>
> The way to avoid that is to set the reserved size to a larger value. By
> default it is 1GB, but if you want to use ext3 or another file system
> which requires slow deletes you should set it at 20-30 GB. XFS is still
> the best choice for MythTV media storage as it is fast, doesn't require
> slow deletes and has an online defragmenter you can run nightly. VFS
> is still good, and when properly configured, so is ext4.
>

Or don't enable the setting you have to set to cause slow deletes:

Delete files slowly
Some filesystems use a lot of resources when deleting large recording
files. This option makes Myth delete the file slowly on this backend to
lessen the impact.

which is disabled by default.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


simon at koala

Feb 2, 2010, 12:36 PM

Post #4 of 14 (2133 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
Daniel Kristjansson wrote:
> On Tue, 2010-02-02 at 08:45 +0000, Simon Kenyon wrote:
>> On 02/02/2010 04:07, mythtv [at] cvs wrote:
>
>> i was wondering if slowly deleting a file was a good idea?
>>
>> i'm thinking of the situation of another recording being in progress?
>> depending on the file system being used, the end result could be a lot
>> of disk fragmentation
>
> The way to avoid that is to set the reserved size to a larger value. By
> default it is 1GB, but if you want to use ext3 or another file system
> which requires slow deletes you should set it at 20-30 GB. XFS is still
> the best choice for MythTV media storage as it is fast, doesn't require
> slow deletes and has an online defragmenter you can run nightly. VFS
> is still good, and when properly configured, so is ext4.

i am in the process of moving to ext4
you say properly configured
i will check on the web, but do you have a url which contains
recommendations?
--
simon
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


lsorense at csclub

Feb 3, 2010, 8:24 AM

Post #5 of 14 (2102 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On Tue, Feb 02, 2010 at 09:13:28AM -0500, Daniel Kristjansson wrote:
> On Tue, 2010-02-02 at 08:45 +0000, Simon Kenyon wrote:
> > On 02/02/2010 04:07, mythtv [at] cvs wrote:
>
> > i was wondering if slowly deleting a file was a good idea?
> >
> > i'm thinking of the situation of another recording being in progress?
> > depending on the file system being used, the end result could be a lot
> > of disk fragmentation
>
> The way to avoid that is to set the reserved size to a larger value. By
> default it is 1GB, but if you want to use ext3 or another file system
> which requires slow deletes you should set it at 20-30 GB. XFS is still
> the best choice for MythTV media storage as it is fast, doesn't require
> slow deletes and has an online defragmenter you can run nightly. VFS
> is still good, and when properly configured, so is ext4.

Does ext3 actually still require slow delete? I get the impression from
the developers that it does not.

At least my mythtv box runs with slow delete disabled and I use ext3
and it seems a lot better than it did with slow delete enabled.

That wasn't the case long ago with older kernels of course.

--
Len Sorensen
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


danielk at cuymedia

Feb 3, 2010, 9:35 AM

Post #6 of 14 (2100 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On Wed, 2010-02-03 at 11:24 -0500, Lennart Sorensen wrote:
> Does ext3 actually still require slow delete? I get the impression from
> the developers that it does not.
>
> At least my mythtv box runs with slow delete disabled and I use ext3
> and it seems a lot better than it did with slow delete enabled.
>
> That wasn't the case long ago with older kernels of course.

Dunno about the latest and greatest kernels, but it here
it takes 2.283 seconds to delete a 1 hr video file on ext3
with Linux 2.6.28 (10 GB). We want deletes to take less
than 0.01 seconds but can tolerate 0.04 to 0.21 seconds
depending on other factors.

-- Daniel

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


david at istwok

Feb 3, 2010, 5:21 PM

Post #7 of 14 (2091 views)
Permalink
Re: mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On Wed, Feb 03, 2010 at 12:35:42PM -0500, Daniel Kristjansson wrote:
> On Wed, 2010-02-03 at 11:24 -0500, Lennart Sorensen wrote:
> > Does ext3 actually still require slow delete? I get the impression from
> > the developers that it does not.
> >
> > At least my mythtv box runs with slow delete disabled and I use ext3
> > and it seems a lot better than it did with slow delete enabled.
> >
> > That wasn't the case long ago with older kernels of course.
>
> Dunno about the latest and greatest kernels, but it here
> it takes 2.283 seconds to delete a 1 hr video file on ext3
> with Linux 2.6.28 (10 GB). We want deletes to take less
> than 0.01 seconds but can tolerate 0.04 to 0.21 seconds
> depending on other factors.

The long delete problem with ext3 does appear to have been addressed
in recent kernels. I did some very unscientific tests on two
different systems running 2.6.31.12 and was able to delete 10 GiB
files in roughly 0.20 to 0.30 seconds. That's still not as good as
XFS and others, but it's a lot better than before.

David
--
David Engel
david [at] istwok
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


ylee at pobox

Feb 4, 2010, 8:32 AM

Post #8 of 14 (2034 views)
Permalink
Re: mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
David Engel <david [at] istwok> says:
> The long delete problem with ext3 does appear to have been addressed
> in recent kernels. I did some very unscientific tests on two
> different systems running 2.6.31.12 and was able to delete 10 GiB
> files in roughly 0.20 to 0.30 seconds. That's still not as good as
> XFS and others, but it's a lot better than before.

I agree 0.2-0.3 seconds per 10GB is much better than the bad old days,
but the times are *so* improved that I suspect what you are seeing is
something else, like caching. Daniel's 2.283 seconds/10GB is much more
realistic but still improved.

(In any case, even achieving Daniel's times is unlikely anytime soon
for people like me using older, EXT3-based NAS appliances. There's a
reason why I never considered going with EXT3 when building my current
MythTV arrays.)

The real test will come this Sunday, actually, from the Super Bowl; an
MPEG-2 HD recording of one from a couple of years back was 45GB.

--
Frontend/backend: P4 3.0GHz, 1.5TB software RAID 5 array
Backend: Quad-core Xeon 1.6GHz, 6.6TB sw RAID 6
Video inputs: Four high-definition over FireWire/OTA
Accessories: 47" 1080p LCD, 5.1 digital, and MX-600
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


ylee at pobox

Feb 4, 2010, 8:46 AM

Post #9 of 14 (2036 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
Daniel Kristjansson <danielk [at] cuymedia> says:
> We want deletes to take less than 0.01 seconds but can tolerate 0.04
> to 0.21 seconds depending on other factors.

May I again suggest, then, considering adjusting upward the stock
ThreadedBufferWriter buffer sizes? Or (WARNING: Configuration setting
feature request incoming) permitting their adjustment outside
compilation?

While the 0.22 writer's performance is *greatly* improved, I still
find that something like

-const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 2*1024*1024;
-const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 4;
-const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 32;
+const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 64*1024*1024;
+const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 32;
+const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 256;

is required to ensure that FireWire (not ATSC) recordings' lengths are
consistently (recording time - 5 seconds ± 1 second) versus (recording
time - 30 seconds ± 15 seconds) without the patch.

(I have empirically tested 64 to see whether it is the optimal value
for my JFS-based RAID arrays, but a larger value than 2 definitely
is. From 0.18 to 0.21 I had to use something akin to

+const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 256*1024*1024;
+const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 128;
+const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 1024;

for the same results, and the writer was still not nearly as resilient
as it is now.)

--
Frontend/backend: P4 3.0GHz, 1.5TB software RAID 5 array
Backend: Quad-core Xeon 1.6GHz, 6.6TB sw RAID 6
Video inputs: Four high-definition over FireWire/OTA
Accessories: 47" 1080p LCD, 5.1 digital, and MX-600
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


david at istwok

Feb 4, 2010, 9:45 AM

Post #10 of 14 (2032 views)
Permalink
Re: mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On Thu, Feb 04, 2010 at 08:32:17AM -0800, Yeechang Lee wrote:
> David Engel <david [at] istwok> says:
> > The long delete problem with ext3 does appear to have been addressed
> > in recent kernels. I did some very unscientific tests on two
> > different systems running 2.6.31.12 and was able to delete 10 GiB
> > files in roughly 0.20 to 0.30 seconds. That's still not as good as
> > XFS and others, but it's a lot better than before.
>
> I agree 0.2-0.3 seconds per 10GB is much better than the bad old days,
> but the times are *so* improved that I suspect what you are seeing is
> something else, like caching. Daniel's 2.283 seconds/10GB is much more
> realistic but still improved.

Caching might have played a part in my tests. After the first tests
looked so good, I specifically tried to flush the disk buffers on
subsequent tests and the results remained consistent. As I said, my
testing was very unscientific, though, so my simple attempts to flush
the buffers might not have really done so.

David
--
David Engel
david [at] istwok
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


stuarta at squashedfrog

Feb 5, 2010, 8:59 AM

Post #11 of 14 (1971 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On 04/02/2010 16:46, Yeechang Lee wrote:
> Daniel Kristjansson<danielk [at] cuymedia> says:
>> We want deletes to take less than 0.01 seconds but can tolerate 0.04
>> to 0.21 seconds depending on other factors.
>
> May I again suggest, then, considering adjusting upward the stock
> ThreadedBufferWriter buffer sizes? Or (WARNING: Configuration setting
> feature request incoming) permitting their adjustment outside
> compilation?
>
> While the 0.22 writer's performance is *greatly* improved, I still
> find that something like
>
> -const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 2*1024*1024;
> -const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 4;
> -const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 32;
> +const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 64*1024*1024;
> +const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 32;
> +const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 256;
>
> is required to ensure that FireWire (not ATSC) recordings' lengths are
> consistently (recording time - 5 seconds ± 1 second) versus (recording
> time - 30 seconds ± 15 seconds) without the patch.
>
> (I have empirically tested 64 to see whether it is the optimal value
> for my JFS-based RAID arrays, but a larger value than 2 definitely
> is. From 0.18 to 0.21 I had to use something akin to
>
> +const uint ThreadedFileWriter::TFW_DEF_BUF_SIZE = 256*1024*1024;
> +const uint ThreadedFileWriter::TFW_MAX_WRITE_SIZE = TFW_DEF_BUF_SIZE / 128;
> +const uint ThreadedFileWriter::TFW_MIN_WRITE_SIZE = TFW_DEF_BUF_SIZE / 1024;
>
> for the same results, and the writer was still not nearly as resilient
> as it is now.)
>

I'd be curious to see how the new lockless threaded file writer would
work for you. I intend to put it in for 0.24...

http://svn.mythtv.org/trac/ticket/6330

Stuart

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


redpepperracing at gmail

Feb 5, 2010, 9:05 AM

Post #12 of 14 (1974 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
>
> I'd be curious to see how the new lockless threaded file writer would
> work for you. I intend to put it in for 0.24...
>
> http://svn.mythtv.org/trac/ticket/6330
>
> Stuart

Does the last patch from 5 months ago still apply to trunk? I am
willing to test it, if so.

Tom
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


stuarta at squashedfrog

Feb 5, 2010, 2:07 PM

Post #13 of 14 (1948 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
On 05/02/2010 17:05, Tom Lichti wrote:
>>
>> I'd be curious to see how the new lockless threaded file writer would
>> work for you. I intend to put it in for 0.24...
>>
>> http://svn.mythtv.org/trac/ticket/6330
>>
>> Stuart
>
> Does the last patch from 5 months ago still apply to trunk? I am
> willing to test it, if so.
>

I've not tried, but i'd expect it to, as it is primarily
a new implementation in its own files.

Stuart
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev


ylee at pobox

Feb 6, 2010, 11:06 AM

Post #14 of 14 (1878 views)
Permalink
Re: [mythtv-commits] mythtv commit: r23435 - in trunk/mythtv by mdean [In reply to]
Stuart Auchterlonie <stuarta [at] squashedfrog> says:
> I'd be curious to see how the new lockless threaded file writer would
> work for you. I intend to put it in for 0.24...
>
> http://svn.mythtv.org/trac/ticket/6330

After massaging Matthias' patch to apply to current -fixes, early
testing with the new file writer still indicates more loss than the
(elapsed time - 5 seconds ± 2 seconds) with FireWire recordings I get
with a buffer-boosed stock writer. I'll continue to test the new
writer unmodified for a few days before boosting its buffer a bit,
too.

--
Frontend/backend: P4 3.0GHz, 1.5TB software RAID 5 array
Backend: Quad-core Xeon 1.6GHz, 6.6TB sw RAID 6
Video inputs: Four high-definition over FireWire/OTA
Accessories: 47" 1080p LCD, 5.1 digital, and MX-600
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

MythTV dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.