Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux: Kernel

[PATCH] fs: Introducing Lanyard Filesystem

 

 

First page Previous page 1 2 Next page Last page  View All Linux kernel RSS feed   Index | Next | Previous | View Threaded


tytso at mit

Aug 20, 2012, 6:21 AM

Post #26 of 33 (126 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

On Mon, Aug 20, 2012 at 11:12:07AM +0200, Alexander Thomas wrote:
>
> Flash drives are getting faster as well. Copying an 8GB file to/from a
> USB drive is not excruciatingly slow and may be quicker and more
> certain than figuring out how to get a working network connection in
> some random place, if possible at all. If it is some lousy WiFi with
> the base station at a distance, a flash drive will be faster. And
> sometimes people just want to be sure that their data will be at a
> certain place at a certain time without having to rely on a network
> that may go down due to external reasons.

OK --- and how many of these situations will you be using such a
stripped down operating system that you can't afford to implement ntfs
or ext4? Samsung is considering an Android-power point-and-shoot
digital camera[1]. And mobile phones are using Android phones where
implementation of ext4 is a worked example (and not particularly
difficult, either).

If this file system had gotten implemented for Arduino first (and
someone actually had a worked example of why you would need > 4GB
vfiles for an Arduino device --- if I were going to be implementing
something with video I'd probably be using a full Linux kernel), the
claimed use case might be more compelling. It would also demonstrate
that in fact this was a decent file system for an Arduino device (and
to demonstrate the infinite stack recursion problem wouldn't be a
gaping security hole problem for them either).

I used to think that we would need an IP unencumbered file system,
given issues around TomTom and Microsoft, but these days, given how
quickly Linux has taken over the embedded and mobile landscape[2] for
all but the most tiniest of devices, I don't think that's as important
of an issue, since we can just simply use a native linux file system.
In the time that it would take to get some other new file system
adopted across the industry, it's likely Linux will have enough market
share to perhaps compel the other OS vendors to provide
interoperability solutions. (Just as the BSD folks have implemented
ext2 support; Linux hasn't bothered to implement FFS2 support....)

- Ted

[1] http://www.engadget.com/2012/03/14/samsung-researching-android-based-digital-camera/

[2] http://money.cnn.com/2012/08/08/technology/smartphone-market-share/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


mail at danrl

Aug 20, 2012, 10:36 AM

Post #27 of 33 (131 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

On Sun, 2012-08-19 at 16:12 +0100, Al Viro wrote:
> Conversions *in* *place* are bad. [+explanation]
I think I got it now.

On Sun, 2012-08-19 at 17:24 +0200, Marco Stornelli wrote:
> [vmtruncate, dio_wait, locking, d_delete]
Noted!

Thanks both of you!

Regards
Dan

--
Dan Luedtke
http://www.danrl.de

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


mail at danrl

Aug 20, 2012, 10:48 AM

Post #28 of 33 (128 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

On Sun, 2012-08-19 at 17:04 -0400, Theodore Ts'o wrote:
> I also seriously question the niche of people who want to use a thumb
> drive to transfer > 4GB files. Try it sometime and see what a painful
> user experience it is....
I don't know if LanyFS will it ever make, and to be honest there are
some corporations _not_ interested in its success, but if it makes it,
this process might be less painful in the future. At least I have to
try, and if we have a better solution in a few years, I'd be happy to
drop the idea and go back to watching my movies again.

However, I watch this thread and I take all the answers and mails into
consideration. You are the experts, and I appreciate you take the time
to discuss the general problem of compatibility and LanyFS in
particular.

I don't want to go to deep into the network vs. removable storage
discussion, although it is an interesting one. For some reason this
issue made a lot of noise, and I have to concentrate on replying to
various mails and modifying a lot of code eventually.

Thanks all of you for your comments.

Regards
Dan


--
Dan Luedtke
http://www.danrl.de

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


slava at dubeyko

Aug 20, 2012, 11:09 PM

Post #29 of 33 (133 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

Hi,

On Sun, 2012-08-19 at 01:38 +0200, Dan Luedtke wrote:
> This patch introduces the Lanyard Filesystem (LanyFS), a filesystem
> for highly mobile and removable storage devices.
>

Did you have any performance comparison of your file system with others?
Have you any benchmark results? I think that simplicity can be a
valuable thing but performance is a key factor, especially for business
guys.

I think that maybe compression or/and encryption support can be a
valuable feature for such niche of file system that you declared.
Efficient compression support is very important feature for embedded
solutions. Moreover, using of hardware opportunity in the field of
compression or encryption can keep driver code simple.

By the way, what about fault-tolerance of your file system? I don't dive
deeply in documentation of your file systems. But, I think that for USB
sticks or removable storages it is very common situation of sudden
switch off. So, it is very important for your file system to be a very
tolerant to such use-cases. How can you estimate tolerance of your file
system architecture for failure as normal situation?

Moreover, I think that simplicity and strong tolerance to file system
corruption can be a feature. I mean that if you have simple on-disk
layout then, maybe, it is possible to try working in very corrupted
environment also. For the end user, from my point of view, possibility
to work in the case of file system corruption can be very precious
feature.

I think that also for your file system such feature as easy
recoverability of user information in the case of complete corruption of
file system can be very useful thing for an end user. Such easy
recoverability can be achieved by means of on-disk layout and file
system driver's techniques, I think. So, simplicity and easy
recoverability of user data can be a valuable feature also.

With the best regards,
Vyacheslav Dubeyko.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


arnd at arndb

Aug 22, 2012, 1:38 AM

Post #30 of 33 (125 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

On Monday 20 August 2012, Theodore Ts'o wrote:
> On Mon, Aug 20, 2012 at 11:12:07AM +0200, Alexander Thomas wrote:
> >
> > Flash drives are getting faster as well. Copying an 8GB file to/from a
> > USB drive is not excruciatingly slow and may be quicker and more
> > certain than figuring out how to get a working network connection in
> > some random place, if possible at all. If it is some lousy WiFi with
> > the base station at a distance, a flash drive will be faster. And
> > sometimes people just want to be sure that their data will be at a
> > certain place at a certain time without having to rely on a network
> > that may go down due to external reasons.

Actually, flash drives are not really getting faster as much as you
think, at least not any more. The main source for performance improvements
over the last few years was from increasing the flash page and block
sizes. Writing single bits may get slightly faster over time, but
we are writing more of them at the same time now, which leads to
significant problems:

* The page size is now effectively 16kb on most flash drives, and
writing in 4kb alignments as practically all our file systems do
(including the lanyfs proposal) means we're actually slower now
than we would be with smaller pages, unless you get into the
special case of writing large chunks of data.

* The erase block size is increasing insanely. We're seeing 12MB
and 16MB block size devices now, and our simulations have shown that
our file systems that are not aware of this block size are hitting
a wall at around 4MB.

> I used to think that we would need an IP unencumbered file system,
> given issues around TomTom and Microsoft, but these days, given how
> quickly Linux has taken over the embedded and mobile landscape[2] for
> all but the most tiniest of devices, I don't think that's as important
> of an issue, since we can just simply use a native linux file system.
> In the time that it would take to get some other new file system
> adopted across the industry, it's likely Linux will have enough market
> share to perhaps compel the other OS vendors to provide
> interoperability solutions. (Just as the BSD folks have implemented
> ext2 support; Linux hasn't bothered to implement FFS2 support....)

There will be patches very soon for a new file system from a major
flash vendor that I'm cooperating with. I haven't seen the patches
myself yet, but the design is similar to a prototype that was done
as a thesis I supervised [1]. I hope that the new implementation is
similarly simple to this design, and also able to provide optimum
performance on most flash media.

Arnd

[1] https://wiki.linaro.org/WorkingGroups/Kernel/Specs/flash-file-system-prototype
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


jengelh at inai

Aug 22, 2012, 2:53 AM

Post #31 of 33 (126 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

On Sunday 2012-08-19 15:34, Dan Luedtke wrote:

>I analyzed about 600k file stored on various removable storage devices.
>80 volunteers sent in data about their devices, generated by a program
>(windows) and scripts (linux, bsd, osx) I wrote for that purpose. The
>data shows that people use more complex filesystems as soon as they are
>confronted with problems (mostly the 4GB limit). After that they have
>problems getting their data accessed by other systems. I derived from
>that, that we need a filesystem that is so simple that even unpopular
>operating systems can implement it without having their business plan
>explode.

Those unpopular OSes do not care about *any other* filesystem at all.

The only way to get a filesystem accepted is by making an ISO standard
out of it - or something in that direction - *and* use it thoroughly
in products, like UDF. Even then, support for new revisions of UDF
has been rather slow-coming.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


ebiederm at xmission

Aug 23, 2012, 10:29 AM

Post #32 of 33 (125 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

Dan Luedtke <mail [at] danrl> writes:

> +/**
> + * struct lanyfs_opts - mount options
> + * @uid: userid of all files and directories
> + * @gid: grouid of all files and direcotries
> + * @dmask: directory mask
> + * @fmask: file mask
> + * @discard: issue discard requests on block freeing
> + * @flush: force instant writing of changed data
> + */
> +struct lanyfs_opts {
> + uid_t uid;
> + gid_t gid;
A small nit. Those above two lines should be:
kuid_t uid;
kgid_t gid;
> + unsigned int dmask;
> + unsigned int fmask;
> + unsigned int discard:1,
> + flush:1;
> +};

Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/


kerneldev100 at gmail

Aug 24, 2012, 4:50 AM

Post #33 of 33 (124 views)
Permalink
Re: [PATCH] fs: Introducing Lanyard Filesystem [In reply to]

Hi,

On Sun, Aug 19, 2012 at 5:08 AM, Dan Luedtke <mail [at] danrl> wrote:

> +
> + /* allocate filesystem private data */
> + fsi = kzalloc(sizeof(*fsi), GFP_KERNEL);
> + if (!fsi)
> + return -ENOMEM;
> + spin_lock_init(&fsi->lock);
> + sb->s_fs_info = fsi;
> +
> + /* set blocksize to minimum size for fetching superblock */
> + if (!sb_set_blocksize(sb, 1 << LANYFS_MIN_BLOCKSIZE)) {
> + if (!silent)
> + lanyfs_err(sb, "error setting blocksize to %d bytes",
> + 1 << LANYFS_MIN_BLOCKSIZE);
> + return -EIO;
> + }
> +
> + /* fetch superblock */
> + bh = sb_bread(sb, LANYFS_SUPERBLOCK);
> + if (!bh) {
> + if (!silent)
> + lanyfs_err(sb, "error reading superblock");
> + return -EIO;
> + }
> + lanysb = (struct lanyfs_sb *) bh->b_data;
> +
> + /* check magic */
> + if (lanysb->magic != cpu_to_le32(LANYFS_SUPER_MAGIC)) {
> + if (!silent)
> + lanyfs_info(sb, "bad magic 0x%x",
> + lanysb->magic);
> + goto exit_invalid;
> + }
> + sb->s_magic = LANYFS_SUPER_MAGIC;
> +
> + /* check block type */
> + if (lanysb->type != LANYFS_TYPE_SB) {
> + if (!silent)
> + lanyfs_err(sb, "bad block type 0x%x", lanysb->type);
> + goto exit_invalid;
> + }
> +
> + /* check version */
> + if (lanysb->major > LANYFS_MAJOR_VERSION) {
> + if (!silent)
> + lanyfs_err(sb, "major version mismatch");
> + goto exit_invalid;
> + }
> +
> + /* check address length */
> + if (lanysb->addrlen < LANYFS_MIN_ADDRLEN ||
> + lanysb->addrlen > LANYFS_MAX_ADDRLEN) {
> + if (!silent)
> + lanyfs_err(sb, "unsupported address length");
> + goto exit_invalid;
> + }
> + fsi->addrlen = lanysb->addrlen;
> +
> + /* check blocksize */
> + if (lanysb->blocksize < LANYFS_MIN_BLOCKSIZE ||
> + lanysb->blocksize > LANYFS_MAX_BLOCKSIZE) {
> + if (!silent)
> + lanyfs_err(sb, "unsupported blocksize");
> + goto exit_invalid;
> + }
> + fsi->blocksize = lanysb->blocksize;
> +

> +
> + /* release block buffer */
> + brelse(bh);
> +
> + /* parse mount options */
> + save_mount_options(sb, options);
> + err = lanyfs_super_options(sb, (char *) options, silent);
> + if (err)
> + return err;
> +
> + /* set blocksize to correct size */
> + if (!sb_set_blocksize(sb, 1 << fsi->blocksize)) {
> + if (!silent)
> + lanyfs_err(sb, "error setting blocksize to %d bytes",
> + 1 << fsi->blocksize);
> + return -EIO;
> + }
> + /* default flags */
> + sb->s_maxbytes = 0xffffffff; /* TODO: hmmmmm */
> + sb->s_op = &lanyfs_super_operations;
> + sb->s_time_gran = 1;
> + sb->s_flags = MS_NOSUID | MS_NOATIME | MS_NODIRATIME;
> +
> + /* make root directory */
> + inode = lanyfs_iget(sb, fsi->rootdir);
> + if (!inode)
> + return -ENOMEM;
> +
> + sb->s_root = d_make_root(inode);
> + if (!sb->s_root) {
> + iput(inode);
> + return -ENOMEM;
> + }
> + return 0;
> +
> +exit_invalid:
> + brelse(bh);
> + if (!silent)
> + lanyfs_info(sb, "no valid lanyard filesystem found");
> + return -EINVAL;
> +}

You should free the memory for "fsi" on error.

Regards.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

First page Previous page 1 2 Next page Last page  View All Linux kernel RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.