Mailing List Archive: MythTV: Dev

Re: [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option

Index | Next | Previous | View Threaded

mtdean at thirdcontact

Jul 15, 2009, 9:11 AM

Post #1 of 4 (1474 views)
Permalink

Re: [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option

On 07/14/2009 11:05 PM, MythTV wrote:
> #6726: Patch mythrename.pl to add '--hardlink' option
>
> This is a patch to mythrename.pl to add a '--hardlink' option. When
> scanning an NFS share over the network with a device such as a Popcorn
> Hour, symbolic linked files will list in the directory contents, but can't
> be played (retrieved) due to the way NFS handles (or doesn't handle)
> symbolic links. The patch adds a new option '--hardlink' to the
> mythrename.pl script that works exactly the same as the '--link' option,
> only it creates/deletes hard links instead of symbolic links.

This behavior seems like it would be very dangerous.

The first issue with it is that hard links can only be created on the
same device as the source file. So, if you use multiple filesystems via
Storage Groups (even if they're all physically on the same host), hard
linking to all recordings would be impossible.

The second--and /much/ larger issue--is that hard links provide /no/
information about the relationship between source and destination
files. Therefore, when you delete a recording in Myth, Myth would be
unaware that the hard linked filename exists and would be unable to
delete the additional filename--even if you enable "Follow symbolic
links when deleting files" (as it's /not/ a symbolic link and can /not/
be followed). Therefore, someone would have to manually go through and
delete the hard linked filename for the filesystem space to actually be
freed.

Currently, mythrename.pl /only/ deletes "files" in the pretty link dir
if they're symlinks--a safety feature done to minimize the chance of its
deleting content it didn't create. If changed to delete all files, it
could delete something it shouldn't.

And, if nothing else, on some filesystems (like ext3) where deletes can
take a long time, if myth had deleted the original filename (which would
happen in almost no time if you do not enable, "Delete files slowly,"
since the file still exists/inodes are still in use), mythrename.pl's
deleting the hard-linked filename would take a long time (potentially
problematic if mythrename.pl is run frequently in a cron job). However,
if you enable, "Delete files slowly" (as most ext3 users would), Myth
would go through and truncate the file until it was less than about 4MB
before it deletes the original filename. Therefore, the hard-linked
filename would reference the remaining (less than) 4MB file, which would
leave little bits of garbage in the filesystem. Even if you checked
filesize before deleting, deleting a non-link file may allow the script
to delete files it shouldn't.

And, there's a simple workaround to "the way NFS handles (or doesn't
handle) symbolic links"--simply have the filesystem layout mirrored on
all systems. And, since that's the recommended approach for using NFS
filesystems with Myth, you probably should do that, anyway. Because
mythrename.pl creates links with absolute paths, this means that the
entire path must be mirrored on all systems. Here's a nice description
of how to use symlinks with NFS (thanks, Allan Stirling, for digging it
up):
http://ou800doc.caldera.com/en/SDK_sysprog/_Using_Symbolic_Links_with_NFS.html

For those users using CIFS/Samba, if you want the server to "follow"
symlinks for the client, simply disable Unix extensions. So, add the line:

unix extensions = No
follow symlinks = Yes
wide links = Yes

to the "[global]" section of your smb.conf. (Thanks to Raymond Wagner
for tracking down the current settings.)

If you don't want to mirror the absolute path structure on all hosts,
then mythrename.pl's insistence on creating absolute symlinks may be the
issue. If so, it may make more sense to instead patch mythrename.pl to
allow creation of directory-relative symlinks. For example, an argument:

--relative-links <base dir>

Create symbolic links relative to the specified <base dir> instead of
using the recording file's absolute path. <base dir> may contain
path-relative specifiers, such as "." and "..". Note that using
--relative-links will create symlinks that only work from certain
directories, so may not be usable with certain applications, including
media players, or my only work if starting those applications from an
appropriate directory.

would allow you to create symlinks that would work even if the absolute
paths differ across hosts.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

simon at koala

Jul 15, 2009, 9:24 AM

Post #2 of 4 (1386 views)
Permalink

Re: [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option [In reply to]

Michael T. Dean wrote:
> all systems. And, since that's the recommended approach for using NFS
> filesystems with Myth, you probably should do that, anyway. Because
> mythrename.pl creates links with absolute paths, this means that the
> entire path must be mirrored on all systems. Here's a nice description
> of how to use symlinks with NFS (thanks, Allan Stirling, for digging it
> up):
> http://ou800doc.caldera.com/en/SDK_sysprog/_Using_Symbolic_Links_with_NFS.html
>
genuine question.
where is it "recommended"?
i had a look in the documentation and on the wiki

regards
--
simon
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

mythlists at littlefish

Jul 20, 2009, 6:34 AM

Post #3 of 4 (1325 views)
Permalink

Re: [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option [In reply to]

Sorry for the late reply. It occured to me the other day that my response was
directed to the mythtv-dev list, but I wasn't subscribed to the list.

Comments inline
On Wed, 15 Jul 2009 12:11:54 -0400, Michael T. Dean wrote
> On 07/14/2009 11:05 PM, MythTV wrote:
> > #6726: Patch mythrename.pl to add '--hardlink' option
> >
> > This is a patch to mythrename.pl to add a '--hardlink' option. When
> > scanning an NFS share over the network with a device such as a Popcorn
> > Hour, symbolic linked files will list in the directory contents, but can't
> > be played (retrieved) due to the way NFS handles (or doesn't handle)
> > symbolic links. The patch adds a new option '--hardlink' to the
> > mythrename.pl script that works exactly the same as the '--link' option,
> > only it creates/deletes hard links instead of symbolic links.
>
> This behavior seems like it would be very dangerous.
>
> The first issue with it is that hard links can only be created on
> the same device as the source file. So, if you use multiple
> filesystems via Storage Groups (even if they're all physically on
> the same host), hard linking to all recordings would be impossible.

Yes. This is a limitation of hardlinks. It's not an option for every case.

> The second--and /much/ larger issue--is that hard links provide /no/
> information about the relationship between source and destination
> files. Therefore, when you delete a recording in Myth, Myth would
> be unaware that the hard linked filename exists and would be unable
> to delete the additional filename--even if you enable "Follow
> symbolic links when deleting files" (as it's /not/ a symbolic link
> and can /not/ be followed). Therefore, someone would have to
> manually go through and delete the hard linked filename for the
> filesystem space to actually be freed.

OK, first of all, if you're using mythrename.pl with a '--link' option (or now
'--hardlink') you're specifically telling the script to make it's own link to
the original Myth files. There's no connection between the linked files and
Myth what so ever, and there is no implied relation that Myth is responsible.
Hardlinked or symlinked, if Myth decides to delete a recording, the link is
broken and dead. Myth does not need to know that the file was hardlinked else
where. As far as Myth cares, the file is deleted and no longer visible to it
(even if the physical data still exists under a different directory). The
only '/much/ larger' issue here is that Myth deletes the file, the file link
count goes down to 1 (becomes regular file, no more links) and mythrename.pl
no ignores it. Again, it's not myth's job to go chasing down external file
links. It never was.

> Currently, mythrename.pl /only/ deletes "files" in the pretty link
> dir if they're symlinks--a safety feature done to minimize the
> chance of its deleting content it didn't create. If changed to
> delete all files, it could delete something it shouldn't.

Maybe you didn't look at the code closely? My mythrename.pl patch only
deletes files that have a link count greater than 1 (hard links). It can't
delete something it shouldn't.

> And, if nothing else, on some filesystems (like ext3) where deletes
> can take a long time, if myth had deleted the original filename
> (which would happen in almost no time if you do not enable, "Delete
> files slowly," since the file still exists/inodes are still in use),
> mythrename.pl's deleting the hard-linked filename would take a long
> time (potentially problematic if mythrename.pl is run frequently in
> a cron job). However, if you enable, "Delete files slowly" (as most
> ext3 users would), Myth would go through and truncate the file until
> it was less than about 4MB before it deletes the original filename.
> Therefore, the hard-linked filename would reference the remaining
> (less than) 4MB file, which would leave little bits of garbage in
> the filesystem. Even if you checked filesize before deleting,
> deleting a non-link file may allow the script to delete files it shouldn't.

And how the heck is that any different then creating a symlink? As I
previously explained, the script doesn't delete files, it deletes (hard)links
to files.

> And, there's a simple workaround to "the way NFS handles (or doesn't
> handle) symbolic links"--simply have the filesystem layout mirrored
> on all systems. And, since that's the recommended approach for

<snip>

OK, maybe you just don't understand the way media boxes like Popcorn hour
work. These are appliance devices, there's no control over things like
file systems. They're not user configurable computers that you can just setup
anyway you like. You plug it into your network, and you browse for network
shares then play the files you find. If I want to watch Myth recordings on my
popcorn hour box from an NFS share, the files have to exist, either physically
or as hard links. My hardlinks patch isn't for everyone, but I guess that's
why it's a command line option and not a default setting no?

Scott

--
Registered Linux user #395249, http://counter.li.org
Nothing goes to waste when Little Fish are near!
(http://www.littlefish.ca)

_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

mtdean at thirdcontact

Jul 20, 2009, 12:33 PM

Post #4 of 4 (1313 views)
Permalink

Re: [mythtv-commits] Ticket #6726: Patch mythrename.pl to add '--hardlink' option [In reply to]

On 07/20/2009 09:34 AM, Scott Mazur wrote:
> On Wed, 15 Jul 2009 12:11:54 -0400, Michael T. Dean wrote
>
>> On 07/14/2009 11:05 PM, MythTV wrote:
>>
>>> #6726: Patch mythrename.pl to add '--hardlink' option
>>>
>>> This is a patch to mythrename.pl to add a '--hardlink' option. When
>>> scanning an NFS share over the network with a device such as a Popcorn
>>> Hour, symbolic linked files will list in the directory contents, but can't
>>> be played (retrieved) due to the way NFS handles (or doesn't handle)
>>> symbolic links. The patch adds a new option '--hardlink' to the
>>> mythrename.pl script that works exactly the same as the '--link' option,
>>> only it creates/deletes hard links instead of symbolic links.
>>>
>> This behavior seems like it would be very dangerous.
>>
>> The first issue with it is that hard links can only be created on
>> the same device as the source file. So, if you use multiple
>> filesystems via Storage Groups (even if they're all physically on
>> the same host), hard linking to all recordings would be impossible.
>>
> Yes. This is a limitation of hardlinks.

Yes, that's what I was trying to say. It's not a problem with your
script, but a problem with the idea of using hard links.

> It's not an option for every case.
>

Yeah, seems it would only be an option for users with a single
device/filesystem for all recordings (i.e. those still using LVM/RAID
multi-disk configurations). If nothing else, this should be mentioned
in the --help output.

>> The second--and /much/ larger issue--is that hard links provide /no/
>> information about the relationship between source and destination
>> files. Therefore, when you delete a recording in Myth, Myth would
>> be unaware that the hard linked filename exists and would be unable
>> to delete the additional filename--even if you enable "Follow
>> symbolic links when deleting files" (as it's /not/ a symbolic link
>> and can /not/ be followed). Therefore, someone would have to
>> manually go through and delete the hard linked filename for the
>> filesystem space to actually be freed.
>>
> OK, first of all, if you're using mythrename.pl with a '--link' option (or now
> '--hardlink') you're specifically telling the script to make it's own link to
> the original Myth files. There's no connection between the linked files and
> Myth what so ever, and there is no implied relation that Myth is responsible.
> Hardlinked or symlinked, if Myth decides to delete a recording, the link is
> broken and dead. Myth does not need to know that the file was hardlinked else
> where. As far as Myth cares, the file is deleted and no longer visible to it
> (even if the physical data still exists under a different directory). The
> only '/much/ larger' issue here is that Myth deletes the file, the file link
> count goes down to 1 (becomes regular file, no more links) and mythrename.pl
> no ignores it. Again, it's not myth's job to go chasing down external file
> links. It never was.
>

No, but it shouldn't be the user's job, either. If an app (in this case
a script) creates data, that app should manage it.

>> Currently, mythrename.pl /only/ deletes "files" in the pretty link
>> dir if they're symlinks--a safety feature done to minimize the
>> chance of its deleting content it didn't create. If changed to
>> delete all files, it could delete something it shouldn't.
>>
> Maybe you didn't look at the code closely?

Actually, didn't look at the code at all. It's not the code I have an
issue with--it's the functionality.

> My mythrename.pl patch only
> deletes files that have a link count greater than 1 (hard links). It can't
> delete something it shouldn't.
>

OK, so now we have safety, but that means that it will only delete
hard-linked filenames for recordings *that still exist*. Therefore,
when the user watches a show in Myth and deletes it (or when Myth
autoexpires it or ...), the link count on the hard-linked filename goes
to 1. Then, the script will no longer delete it. That means it will
leave garbage data on the filesystem (either a multi-gigabyte recording
file or a <4MB "scrap" after truncation if slow deletes are enabled).

And, speaking of which, since Myth's deleting the original filename will
have no effect on filesystem usage, creating hard links to the recording
files completely breaks autoexpire.

Both of these issues would definitely need to be mentioned in the --help
text, too.

>> And, if nothing else, on some filesystems (like ext3) where deletes
>> can take a long time, if myth had deleted the original filename
>> (which would happen in almost no time if you do not enable, "Delete
>> files slowly," since the file still exists/inodes are still in use),
>> mythrename.pl's deleting the hard-linked filename would take a long
>> time (potentially problematic if mythrename.pl is run frequently in
>> a cron job). However, if you enable, "Delete files slowly" (as most
>> ext3 users would), Myth would go through and truncate the file until
>> it was less than about 4MB before it deletes the original filename.
>> Therefore, the hard-linked filename would reference the remaining
>> (less than) 4MB file, which would leave little bits of garbage in
>> the filesystem. Even if you checked filesize before deleting,
>> deleting a non-link file may allow the script to delete files it shouldn't.
>>
> And how the heck is that any different then creating a symlink? As I
> previously explained, the script doesn't delete files, it deletes (hard)links
> to files.
>

OK, see above...

>> And, there's a simple workaround to "the way NFS handles (or doesn't
>> handle) symbolic links"--simply have the filesystem layout mirrored
>> on all systems. And, since that's the recommended approach for
>>
> <snip>
>
> OK, maybe you just don't understand the way media boxes like Popcorn hour
> work.

I definitely don't--don't have any of them.

> These are appliance devices, there's no control over things like
> file systems. They're not user configurable computers that you can just setup
> anyway you like. You plug it into your network, and you browse for network
> shares then play the files you find. If I want to watch Myth recordings on my
> popcorn hour box from an NFS share, the files have to exist, either physically
> or as hard links. My hardlinks patch isn't for everyone, but I guess that's
> why it's a command line option and not a default setting no?

If your device is really that limited, then just choose a filesystem
that allows server-side symlink resolution--i.e. CIFS with unix
extensions disabled--or continue to use your modified version of
mythrename.pl. However, I suggest that the change not be made in the
contrib script distributed with MythTV as it's likely to cause more
people more problems than it will solve (especially since you can still
get usable symlinks on any device that allows CIFS shares, and can do so
without modifying mythrename.pl).

That said, I do appreciate your working to improve Myth and making your
changes available to the project (as the license says you should). I
just don't think the project should choose to use your changes.
However, I won't be the one making the decision, so the changes may well
be accepted/incorporated into mythrename.pl.

Thanks,
Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev [at] mythtv
http://mythtv.org/cgi-bin/mailman/listinfo/mythtv-dev

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads