Mailing List Archive: Linux: Kernel

CDROM jukebox filesystem using autofs

Index | Next | Previous | View Threaded

aaronp at ofb

Apr 29, 1998, 6:36 PM

Post #1 of 16 (196 views)
Permalink

CDROM jukebox filesystem using autofs

I am writing a Linux filesystem / driver for a CDROM jukebox. This
Jukebox has hundreds of disks and multiple drives. It controls the robot using
a serial interface and has a SCSI interface to the CDROM drives. I have used
autofs as a starting point for the driver. I'm doing all communications with
the robot from the user-space daemon when mount requests come from the kernel.
So far, I have made it move and mount disks on command and unmount and put away
disks in an LRU fashion. For example, if I do an ls of /autofs/1, it'll get
the disk out of slot 1 and put it in one of the drives, mount it and give me
the ls.
There are several problems I am running into with this approach. First
of all, what do I do if I have 4 drives, all mounted and busy, and I get a
request for a fifth disk? I can't open a drive and swap disks since they are
all mounted and I can't unmount any of them since they are busy. I really want
to remove a disk from a drive, block all processes requesting data off of that
disk, and put the other disk into the drive to complete the request for the
fifth disk. This, of course, could make a disk swapping nightmare but at least
it would work.
What's the cleanest way to implement this functionality in the kernel?
My first thought is to make block device multiplexer. This device would be
attached to a real block device and would provide any number of multiplexed
block devices to the user. The multiplexed block device can either forward
requests to the real device or block them. This way I could have 2 CDs mounted
on /dev/scd0 through the multiplexer and switch between them as needed. This
sceme, however, does not allow swapping the disk back into a different drive
which would be a very useful feature. Any suggestions?
The second big problem is persistent caching of data and meta-data on
the CDs. It would be nice to be able to do an 'ls -lR' of the directory tree
of the jukebox and not thrash the robot around for an hour swapping disks in
and out of the drives. With proper caching, the efficiency of the system would
be increased dramatically and wear on the robot would be much reduced. It
seems like this persistent caching should be implemented as a separate entity
in the kernel. This same persistent caching would also be very useful to NFS
and other slow file systems and is already implemented in Coda. Has anyone
done any work on making a CacheFS or possibly persistent caching at the block
level? Is Coda a reasonable starting point to make a generic persistent
caching file system / driver?
Thanks for your help,
Aaron
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

ddickey at wamnet

Apr 30, 1998, 8:10 AM

Post #2 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

Aaron Passey wrote:
> I am writing a Linux filesystem / driver for a CDROM jukebox. This
> Jukebox has hundreds of disks and multiple drives. It controls the robot using
> a serial interface and has a SCSI interface to the CDROM drives. I have used
> autofs as a starting point for the driver. I'm doing all communications with
> the robot from the user-space daemon when mount requests come from the kernel.
> So far, I have made it move and mount disks on command and unmount and put away
> disks in an LRU fashion. For example, if I do an ls of /autofs/1, it'll get
> the disk out of slot 1 and put it in one of the drives, mount it and give me
> the ls.
Sounds interesting.
>
>
> There are several problems I am running into with this approach. First
> of all, what do I do if I have 4 drives, all mounted and busy, and I get a
> request for a fifth disk?
return EBUSY.
> I can't open a drive and swap disks since they are
> all mounted and I can't unmount any of them since they are busy. I really want
> to remove a disk from a drive, block all processes requesting data off of that
> disk, and put the other disk into the drive to complete the request for the
> fifth disk. This, of course, could make a disk swapping nightmare but at least
> it would work.
It seems like you'd have an awful lot of information to keep track of.First just
try not granting the request (EBUSY), and then add this feature
enhancement if you insist.
I think you may be shooting yourself in the foot doing this. In fact,
it reminds me of a tape driver I wrote once that allowed seeking....
I wanted to be able to mount a tape device as a filesystem. :) I
decided after writing it that that is not what tapes were meant for,
though it was an interesting exercise. Use Occam's (sp?) razor - simple is
better.
-Dan
----
Dan A. Dickey
ddickey [at] wamnet
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

aaronp at ofb

Apr 30, 1998, 11:16 AM

Post #3 of 16 (197 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Thu, 30 Apr 1998 10:10:57 -0500, Dan A. Dickey <ddickey [at] wamnet> wrote:
>Aaron Passey wrote:
>> There are several problems I am running into with this approach.
>> First of all, what do I do if I have 4 drives, all mounted and busy, and I
>> get a request for a fifth disk?
>
>return EBUSY.
Are you suggesting that when disks 1-4 are mounted and busy and I do an
ls on /autofs/5, the ls gets an EBUSY? This doesn't make much sense. Normal
small CDROM jukeboxes I have seen working under Linux allow you to have all of
the disks mounted and they swap disks as necessary. These are done by
multiplexing at the block device level so you get a /dev/scd0 - /dev/scd5 for a
6 disk changer. The jukebox I am writing a driver for has hundreds of disks so
it seems impractical to do this -- hence my autofs approach. Using the autofs
model, the only choices I have are to grant a request from the kernel or say it
doesn't exist (the process gets a File not Found and a negative dentry is
created). I can modify autofs to also support sending an EBUSY but is this
desired?
>> I can't open a drive and swap disks since they are all mounted and I can't
>> unmount any of them since they are busy. I really want to remove a disk
>> from a drive, block all processes requesting data off of that disk, and put
>> the other disk into the drive to complete the request for the fifth disk.
>> This, of course, could make a disk swapping nightmare but at least it would
>> work.
>It seems like you'd have an awful lot of information to keep track of.First
>just try not granting the request (EBUSY), and then add this feature
>enhancement if you insist.
This doesn't seem like an optional feature to me. The whole goal is to
make the jukebox look like one huge filesystem with all of the CDs always
available.
Aaron
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

H.H.vanRiel at phys

Apr 30, 1998, 1:29 PM

Post #4 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Thu, 30 Apr 1998, Dan A. Dickey wrote:
> Aaron Passey wrote:
> > So far, I have made it move and mount disks on command and unmount and put away
> > disks in an LRU fashion. For example, if I do an ls of /autofs/1, it'll get
> > the disk out of slot 1 and put it in one of the drives, mount it and give me
> > the ls.
> >
> > There are several problems I am running into with this approach. First
> > of all, what do I do if I have 4 drives, all mounted and busy, and I get a
> > request for a fifth disk?
>
> return EBUSY.
Or you could:
- let the user of the first started request wait, and switch
drives again after XXX kilobytes (making sure that throughput
isn't hurt too much, latency will be bad anyway)
- have a caching HD partition and read the biggest file requested
for the next X MB, then quickly switch disks to the requested
one
> though it was an interesting exercise. Use Occam's (sp?) razor - simple is
> better.
For truly high-latency apps this might not be the case.
Simply make sure that your readahead (on large files) is
big enough (and maybe HD cached) so that no more than
50% of the time/bandwidth is spent switching the disks.
Rik.
+-------------------------------------------+--------------------------+
| Linux: - LinuxHQ MM-patches page | Scouting webmaster |
| - kswapd ask-him & complain-to guy | Vries cubscout leader |
| http://www.phys.uu.nl/~riel/ | <H.H.vanRiel [at] phys> |
+-------------------------------------------+--------------------------+
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

hpa at transmeta

Apr 30, 1998, 3:14 PM

Post #5 of 16 (198 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

Followup to: <slrn6khg06.8pc.aaronp [at] ofb>
By author: aaronp [at] ofb (Aaron Passey)
In newsgroup: linux.dev.kernel
>
> On Thu, 30 Apr 1998 10:10:57 -0500, Dan A. Dickey <ddickey [at] wamnet> wrote:
> >Aaron Passey wrote:
> >> There are several problems I am running into with this approach.
> >> First of all, what do I do if I have 4 drives, all mounted and busy, and I
> >> get a request for a fifth disk?
> >
> >return EBUSY.
>
> Are you suggesting that when disks 1-4 are mounted and busy
> and I do an ls on /autofs/5, the ls gets an EBUSY? This doesn't
> make much sense. Normal small CDROM jukeboxes I have seen working
> under Linux allow you to have all of the disks mounted and they swap
> disks as necessary. These are done by multiplexing at the block
> device level so you get a /dev/scd0 - /dev/scd5 for a 6 disk
> changer. The jukebox I am writing a driver for has hundreds of
> disks so it seems impractical to do this -- hence my autofs
> approach. Using the autofs model, the only choices I have are to
> grant a request from the kernel or say it doesn't exist (the process
> gets a File not Found and a negative dentry is created). I can
> modify autofs to also support sending an EBUSY but is this desired?
I think the way to do this is going to involve some amount of disk
caching if you want it to work sanely.
-hpa
--
PGP: 2047/2A960705 BA 03 D3 2C 14 A8 A8 BD 1E DF FE 69 EE 35 BD 74
See http://www.zytor.com/~hpa/ for web page and full PGP public key
I am Bah�'� -- ask me about it or see http://www.bahai.org/
"To love another person is to see the face of God." -- Les Mis�rables
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

aaronp at ofb

Apr 30, 1998, 4:11 PM

Post #6 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

Rik van Riel <H.H.vanRiel [at] phys> wrote:
>Or you could:
>- let the user of the first started request wait, and switch
> drives again after XXX kilobytes (making sure that throughput
> isn't hurt too much, latency will be bad anyway)
>- have a caching HD partition and read the biggest file requested
> for the next X MB, then quickly switch disks to the requested
> one
I'm sure I could come up with a disk swapping policy that's somewhat
reasonable. The question is where in the kernel is that multiplexing and
caching most appropriate to do? It seems like the multiplexing could be done
at the block level but it's not clear to me how this device would manifest
itself. Take the case of 2 drives and 3 users. When the third request comes
in, I want to choose one of the 2 mounted disks to eject and leave it in a wait
queue with all operations blocked. When it's time to swap the other disk back
in, ideally it should be able to go into either drive -- not only the one it
was originally removed from. Is this reasonable?
The caching, it seems, should be implemented at the filesystem level
since we want to be able to control the caching of data and meta-data
separately. It would be nice, for example to cache all of the meta-data
permanently and the data on an LRU basis.
>For truly high-latency apps this might not be the case.
>Simply make sure that your readahead (on large files) is
>big enough (and maybe HD cached) so that no more than
>50% of the time/bandwidth is spent switching the disks.
Well, it seems like often users would only request one or two files off
of a disk which would work very well with a little bit of read-ahead caching.
I imagine the heavily used disks (like the Redhat disk while a company is
upgrading machines) would have almost all of their contents cached on harddisk.
Just using the Linux built in page-cache (although would help things a lot) is
not sufficient for this application.
Aaron
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

kernel at eiterra

Apr 30, 1998, 5:31 PM

Post #7 of 16 (194 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Thu, 30 Apr 1998, Rik van Riel wrote:
#On Thu, 30 Apr 1998, Dan A. Dickey wrote:
#> Aaron Passey wrote:
#
#> > So far, I have made it move and mount disks on command and unmount and put away
#> > disks in an LRU fashion. For example, if I do an ls of /autofs/1, it'll get
#> > the disk out of slot 1 and put it in one of the drives, mount it and give me
#> > the ls.
#> >
#> > There are several problems I am running into with this approach. First
#> > of all, what do I do if I have 4 drives, all mounted and busy, and I get a
#> > request for a fifth disk?
#>
#> return EBUSY.
#
#Or you could:
#- let the user of the first started request wait, and switch
# drives again after XXX kilobytes (making sure that throughput
# isn't hurt too much, latency will be bad anyway)
#- have a caching HD partition and read the biggest file requested
# for the next X MB, then quickly switch disks to the requested
# one
Or, do it the way I've done it with a large quantity (12) of Pioneer 6
disc SCSI changers...
User1->Request File /cd3/images/boot.img
(1 Second)
User2->Request File /cd3/images/boot.img
(3 Seconds)
Server->Send File /cd3/images/boot.img /cdcache
(1 Second)
Server->Send File->User1
Server->Send File->User2
This is a very effecient way that I've found to deal with CD changers and
jukeboxes when there are files that are frequently accessed.
User1->Request File /cd16/system/XFree86-All.tgz
(1 Second)
User2->Request File /cd17/misc/games/quake-tf.tgz
(3 Seconds)
Server->Send File /cd16/system/XFree86-All.tgz /cdcache
(Wait for file to copy)
Server->Send File /cdcache/XFree86-All.tgz->User1
Server->Change CD17
(3 Seconds)
Server->Send File /cd17/misc/games/quake-tf.tgz->User2
This allows for multiple connection, with minimal slowdown and wait in a
changer environment.
Now for the fun part, a jukebox environment. My example will use a file
queueing system I used before with a machine that had 6 SCSI CD-ROMs, and
nothing more. Boot disk was IDE. You know the routine. Ran Linux. :)
User1->Request File /cd5/thisfile.tgz
User2->Request File /cd2/thatfile.txt
User5->Request File /cd1/allfiles.txt
User3->Request File /cd6/303wavs.gz
Server->Process Requests, sort by filesize
(File Sizes:
thisfile.tgz 53K
thatfile.txt 32K
allfiles.txt 10K
303wavz.gz 23M;)
Server->Send File->User5
(Wait for 10% of send before starting next)
Server->Send File->User2
(Wait again)
Server->Send File->User1
(Wait)
Server->Send File->User3
User4->Request File /cd3/bob/thepipes.tgz (33M)
Server->Send File->User4
User2->Request File /cd3/bob/thefrop.txt (60K)
Server->Communicate->User2 "Server is busy, your request has been queued"
(Wait till User3's file is 40% done)
Server->Send File->User2
In a high use environment, the above principle would apply as well. With
some modifications, of course. With the current technology, that kind of
machine would have 6 CD-ROMs on an UltraWide PCI, and two 6.4G UW drives.
One for the OS, the other for caching exclusively. It may sound a little
big, but YMMV, as always. :)
#For truly high-latency apps this might not be the case.
#Simply make sure that your readahead (on large files) is
#big enough (and maybe HD cached) so that no more than
#50% of the time/bandwidth is spent switching the disks.
The easiest way to catch this is user limits, and using the principle
outlined above, where it waits for a certain percentage of the file to be
sent. Another issue to consider is processor use for sorting files. In a
Perfect Environment(tm), you wouldn't need to worry about it, and could be
picky as all hell, down to a single bit of difference in size. In reality,
you're probably running on a single P200 or dual PPro180, which means you
do have limited processing power, and you need to keep it low. The easiest
way to do this is to classify filesizes in blocks. 1bit to 4M, 4.001M to
10M, and so on. This lets the machine run through a small list of sizes,
decide where the file fits, and throw it in there. (At this point, a
coworker distracted me to fix his bloody AIX box.) (And again, to fix
permissions. And again, to find X. And again, to fix something else) (Now
I try to find train of thought again after about an hour.) By using
filesize blocks, the machine can easily categorize. Utilize caching as
well, and it speeds up a lot. Either way, jukebox will always be faster
than changer, and I give up. I can't remember what I was going to say. :)
-Phil R. Jaenke (kernel [at] nls / prj [at] nls)
TheGuyInCharge(tm), Ketyra Designs - We get paid to break stuff :)
Linux pkrea.ketyra.INT 2.0.33 #15 Sat Apr 18 00:40:21 EDT 1998 i586
Linux eiterra.nls.net 2.0.33 #15 Fri Apr 17 00:22:13 EDT 1998 i586
- Linus says for 'brave people only.' I say 'keep a backup.' - :)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

alan at lxorguk

May 1, 1998, 1:21 AM

Post #8 of 16 (194 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

> > in, ideally it should be able to go into either drive -- not only the one it
> > was originally removed from. Is this reasonable?
> Probably, it would be best to make a driver (similar to loop) with a
> number of minors equal to the number of CDs you will support. Make each
> CD it's own device, and then the user can mount them normaly.
Remember that each CD might have partitions. With a 4x4 SCSI changer on
a Mac you might have 32 file systems to mount. Any encoding needs to take
account of this in its minor numbers
Alan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

aaronp at ofb

May 1, 1998, 10:26 AM

Post #9 of 16 (197 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Thu, 30 Apr 1998, James Mastros <root [at] jennifer-unix> wrote:
>Probably, it would be best to make a driver (similar to loop) with a
>number of minors equal to the number of CDs you will support. Make each
>CD it's own device, and then the user can mount them normaly.
Well, the problem with this scheme is that the jukebox I am working on
has hundreds of CDs so taking into account partitions on CDs would mean
thousands of minors. This is not reasonable with our current dev_t. It could
work with devfs or a larger dev_t, I guess. It still seems like representing
this jukebox as thousands of files in /dev is messy at best.
>Perhaps you should mount each CD, then make a httpfs (/w userfs, probably)
>that reads through a squid cache? That way, you can easily (?) steal
>(umm... leverage) existing cache logic, and, additionaly, make the
>cd jukebox and the caching code independant of each other.
Jeez, I hope you're kidding. :)
>It might be if you remove the code to not swap out data that we can
>re-read from disk. Also, in your example, you would probably be able to
>keep the RedHat CD in one drive -- if everybody's upgrading, they probably
>aren't going to be playing mp3s off of the NFS server. <G>
For performance reasons, we would probably want to cache the disk even
if we have a free drive.
Aaron
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

root at jennifer-unix

May 1, 1998, 6:59 PM

Post #10 of 16 (196 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On 1 May 1998, Aaron Passey wrote:
> On Thu, 30 Apr 1998, James Mastros <root [at] jennifer-unix> wrote:
> >Probably, it would be best to make a driver (similar to loop) with a
> >number of minors equal to the number of CDs you will support. Make each
> >CD it's own device, and then the user can mount them normaly.
>
> Well, the problem with this scheme is that the jukebox I am working on
> has hundreds of CDs so taking into account partitions on CDs would mean
> thousands of minors. This is not reasonable with our current dev_t. It could
> work with devfs or a larger dev_t, I guess. It still seems like representing
> this jukebox as thousands of files in /dev is messy at best.
Hmmm... I'd tend to maintain that keeping each disk a sepperate device
would be cleaner then the other way around. However, your right that for
a huge disk array (well, huge to me -- I don't think I've ever used a
machine /w more then one CDROM drive though) you would use huge amounts of
minors (though you could pre-scan for multiple partitions, instead of
allowing room for 16 for each CD). However, I think one device, one
medium is a good rule -- what if sombody wants to cat out the whole CD?
(Or do "cdid /dev/cdjukebox115" to see what CD #115 has on it?)
> >Perhaps you should mount each CD, then make a httpfs (/w userfs, probably)
> >that reads through a squid cache? That way, you can easily (?) steal
> >(umm... leverage) existing cache logic, and, additionaly, make the
> >cd jukebox and the caching code independant of each other.
>
> Jeez, I hope you're kidding. :)
Mostly. But I do think that the agressive caching code should be
independant from the jukebox code -- after all, they don't need to be
conceptiuly related. You would want to do the same thing with
slow NFS or NBD, or tapes, or floppies, or ZIPs, or pretty much any type
of slow storage that is used on machines with plentiful memory.
> >It might be if you remove the code to not swap out data that we can
> >re-read from disk. Also, in your example, you would probably be able to
> >keep the RedHat CD in one drive -- if everybody's upgrading, they probably
> >aren't going to be playing mp3s off of the NFS server. <G>
>
> For performance reasons, we would probably want to cache the disk even
> if we have a free drive.
I supose -- though the page cache is probably enough for this case anyway.
With enough memory, the page cache will cache tons. The problem here is
mantaining control of when we switch CDs, and optmising the CD<->drive
mapping.
-=- James Mastros
--
True mastery is knowing enough to bullshit the rest.
-=- Me
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

kjahds at kjahds

May 1, 1998, 10:31 PM

Post #11 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On 30 Apr 1998, Aaron Passey wrote:
>
> I am writing a Linux filesystem / driver for a CDROM jukebox.
[...]
> The second big problem is persistent caching of data and meta-data on
> the CDs. It would be nice to be able to do an 'ls -lR' of the directory tree
> of the jukebox and not thrash the robot around for an hour swapping disks in
> and out of the drives. With proper caching, the efficiency of the system would
> be increased dramatically and wear on the robot would be much reduced.
I've been musing on this issue (ever since I saw a $50 4x4 IDE CDROM
drive). My thought is that while caching is certainly a good idea, a
simple hysteresis technique could at least ameleoriate the thrashing
problem. My basic idea is to only allow disks to be switched (and really
this technique could apply to any jukebox media, not just CDROM drives)
after X amount of data has been transferred, or Y amount of time has
expired -- and every time a switch happens at the X or Y limit, double
that limit. When X or Y passes without a switch request, halve them (back
to the original value as a minimum.) This would mean that tasks competing
for the same resources would quickly cause the multiplexer to give more
and more time/data to each process in turn, thus reducing jukebox
manuevers, saving time & wear, at the cost of more bursty data retrieval.
Obviously this would need some substantial tuning to be safe and
effective. However, it wouldn't help at all with interactive response for
things like file directories, which would seem to be very much worth
caching, even if nothing else is cached. If Coda provides a useful caching
mechanism, it would seem worth considering it as a general mechanism for
the entire project.
--
Kenneth Albanowski (kjahds [at] kjahds, CIS: 70705,126)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

rhw at bigfoot

May 2, 1998, 10:19 AM

Post #12 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

Hi James, Aaron.
>>> Probably, it would be best to make a driver (similar to loop)
>>> with a number of minors equal to the number of CDs you will
>>> support. Make each CD it's own device, and then the user can
>>> mount them normaly.
>> Well, the problem with this scheme is that the jukebox I am
>> working on has hundreds of CDs so taking into account partitions
>> on CDs would mean thousands of minors. This is not reasonable with
>> our current dev_t. It could work with devfs or a larger dev_t, I
>> guess. It still seems like representing this jukebox as thousands
>> of files in /dev is messy at best.
> Hmmm... I'd tend to maintain that keeping each disk a sepperate
> device would be cleaner then the other way around. However, your
> right that for a huge disk array (well, huge to me -- I don't think
> I've ever used a machine /w more then one CDROM drive though) you
> would use huge amounts of minors (though you could pre-scan for
> multiple partitions, instead of allowing room for 16 for each CD).
> However, I think one device, one medium is a good rule -- what if
> sombody wants to cat out the whole CD? (Or do "cdid
> /dev/cdjukebox115" to see what CD #115 has on it?)
A variant of that might make more sense here - perhaps have the
/dev/jukebox (or whatever) as the device number for the entire jukebox
array, similar to the RAID devices. One could then have that as a
pseudo-directory that has a separate [directory] entry for each
available CD similar to the virtual /proc file system's method of
'storing' files...
This would mean that a system with two 3-changer drives would appear
something like the following listing:
Q> $ mount /dev/jukebox /mnt
Q> $ ls -la /mnt | tr -s ' ' | cut -d ' ' -f 1,9-
Q> dr-xr-xr-x A0
Q> dr-xr-xr-x A1
Q> dr-xr-xr-x A2
Q> dr-xr-xr-x B0
Q> dr-xr-xr-x B1
Q> dr-xr-xr-x B2
Q> $ umount /mnt
The letter would be used internally to identify which drive, and the
number that followed would say which disc from that drive...one could
easily support MILLIONS of disks that way...
Best wishes from Riley.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

root at jennifer-unix

May 2, 1998, 10:35 AM

Post #13 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Sat, 2 May 1998, Riley Williams wrote:
RW> A variant of that might make more sense here - perhaps have the
RW> /dev/jukebox (or whatever) as the device number for the entire jukebox
RW> array, similar to the RAID devices. One could then have that as a
RW> pseudo-directory that has a separate [directory] entry for each
RW> available CD similar to the virtual /proc file system's method of
RW> 'storing' files...
RW>
RW> This would mean that a system with two 3-changer drives would appear
RW> something like the following listing:
RW>
[...]
RW> The letter would be used internally to identify which drive, and the
RW> number that followed would say which disc from that drive...one could
RW> easily support MILLIONS of disks that way...
RW>
RW> Best wishes from Riley.
Hmm... This, I assume, is nearly what the original idea was -- only without
the letter -- the whole point is that we want to be independent of what
drive holds what CD, so we can optmise use of the drives (IE each drive can
hold any CD that you have, so if we have 10 drives, the users can use any 10
of the 1,000 avaible CDs).
However, this dosn't take into account my problem at all -- that being that
this dosn't allow you to do IO (I, rather, these are CD-_RO_Ms) on the raw
disk -- only to mount it. Instead, how about making the jukeboxfs layout be
either of n files (where n=the number of avaible CDs), with names of 1 to n
(well, 1 to n, followed by a letter from 'a' to 'q' indicating the
partition), which you could mount with loopback. Or, you could have a set
of files like the ones in the last sugestion, only precided by 'D' (for
device), and internaly "mount" them into directories with the same name,
only with D replaced by F (filesystem).
-=- James Mastros
--
True mastery is knowing enough to bullshit the rest.
-=- Me
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

linker at nightshade

May 2, 1998, 2:24 PM

Post #14 of 16 (201 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

On Fri, 1 May 1998, James Mastros wrote:
[snip talk about hugh cd jukebox]
> Hmmm... I'd tend to maintain that keeping each disk a sepperate device
> would be cleaner then the other way around. However, your right that for
> a huge disk array (well, huge to me -- I don't think I've ever used a
> machine /w more then one CDROM drive though) you would use huge amounts of
> minors (though you could pre-scan for multiple partitions, instead of
^^^^^^^^ Whats this?? I dont know about where you are.. but in the US we
have Child Labor laws! Anyways, why would they need them.. They have a
robot arm to autoload! :) :) :)
> allowing room for 16 for each CD). However, I think one device, one
> medium is a good rule -- what if sombody wants to cat out the whole CD?
> (Or do "cdid /dev/cdjukebox115" to see what CD #115 has on it?)
[snip]
Actually, here is how it IMHO 'should' be done:
1)
Interface to the real cdroms through the normal scsi crap.. (there were 4
of them as I recall)..
2)
Write a program that acts as a NFS server. It stores files such:
/usr/local/cdthingy/data/index #The merged and sorted directory structure
of the entire array.
/usr/local/cdthingy/data/cache/ #A cache spool
When mounted this thing should provide a single tree for the array..
When the user reads a file, it figures out what disk it's on and loads the
disk (finding a drive described below), then it copys the file (and
perhaps some other files from that disk) to the cache. If the cache is
more then x% of it's maximum defined size then it deletes some of the last
used non open file to bring it down to a low water mark.
If all drives are full it find the drive that hasn't had files copyed for
the longest time and unloads it..
I dont know about this array physiclly, but perhaps the robot could resort
it as it swaps the disks in and out of the drives, thus moving the most
used CDs close to the drives..
If this is acceptible then this thread can move elseware.. This doesn't
need to be in the kernel, this is too array dependent to warrent inclusion
anyways..
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

rhw at bigfoot

May 3, 1998, 3:38 AM

Post #15 of 16 (194 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

Hi James.
>> A variant of that might make more sense here - perhaps have the
>> /dev/jukebox (or whatever) as the device number for the entire
>> jukebox array, similar to the RAID devices. One could then have
>> that as a pseudo-directory that has a separate [directory] entry
>> for each available CD similar to the virtual /proc file system's
>> method of 'storing' files...
>> This would mean that a system with two 3-changer drives would
>> appear something like the following listing:
>> The letter would be used internally to identify which drive, and
>> the number that followed would say which disc from that drive: one
>> could easily support MILLIONS of disks that way...
> Hmm... This, I assume, is nearly what the original idea was -- only
> without the letter -- the whole point is that we want to be
> independent of what drive holds what CD, so we can optmise use of
> the drives (IE each drive can hold any CD that you have, so if we
> have 10 drives, the users can use any 10 of the 1,000 avaible CDs).
Not quite what I intended to say then - I would envisage such a driver
being able to handle MULTIPLE multichanger arrays, with each array
having a set of CD's available to that array only.
I don't know what particular make of jukebox you're using, so for
argument's sake, let's assume it's a SUPADUPA A1 unit. Imagine that
you have FOUR such units, all connected to the same computer, each
with its collection of CD's available. It stands to reason that one
would NOT be able to take a CD stored in the first unit and mount it
in the drive in the fourth unit, or so I would assume...
Indeed, the driver would probably make sense implemented as some sort
of superdriver like the current RAID driver is, so it could be told to
take charge of not only the jukebox you envision, but also other
multi-changers, and even CD images stored on the hard disk, or the
like. This would result in TWO drivers, one a fairly simple one that
took direct control of an individual jukebox unit, and mounted a
particular disk in a particular drive when told to do so, the other a
far more 'intelligent' one that kept track of what CD's were mounted
and implemented the mount optimisations...
> However, this dosn't take into account my problem at all -- that
> being that this dosn't allow you to do IO (I, rather, these are
> CD-_RO_Ms) on the raw disk -- only to mount it.
> Instead, how about making the jukeboxfs layout be either of n files
> (where n=the number of avaible CDs), with names of 1 to n (well, 1
> to n, followed by a letter from 'a' to 'q' indicating the
> partition), which you could mount with loopback. Or, you could have
> a set of files like the ones in the last sugestion, only precided
> by 'D' (for device), and internaly "mount" them into directories
> with the same name, only with D replaced by F (filesystem).
Why not use the format used with the hard drives at the moment. Using
the idea of two multichangers with three disks each, and each with two
partitions, with only disk number 1 in the first unit and disk 0 in
the second unit having been mounted so far, the following would appear
reasonable:
Q> $ mount /jukebox
Q> $ ls -lad /jukebox/* | tr -s ' ' | cut -d ' ' -f 1,9-
Q> dr-xr-xr-x a0
Q> dr-xr-xr-x a1
Q> dr-xr-xr-x a1a
Q> dr-xr-xr-x a1b
Q> dr-xr-xr-x a2
Q> dr-xr-xr-x b0
Q> dr-xr-xr-x b0a
Q> dr-xr-xr-x b0b
Q> dr-xr-xr-x b1
Q> dr-xr-xr-x b2
Basically, each available disk would appear with its number when the
unit starts up, but without the subpartition entries, and those would
be added when each disk is first mounted. Also, if any of the drives
are free for any length of time, the driver could mount and unmount
previously unmounted CD's so as to add this information ready for if
that CD gets referenced in the future...
Alternatively, since that would inevitably produce an enormous
directory listing, perhaps the listing could be split up with two or
more levels of directories? Do CD's have volume labels? If so, maybe
those could be made use of, producing something like the following for
the above:
Q> $ mount /jukebox
Q> cd /jukebox
Q> $ ls -lad */* | tr -s ' ' | cut -d ' ' -f 1,9-
Q> lrwxrwxrwx 1/1001_interesting_facts_about_space -> ../cd/b0a
Q> lrwxrwxrwx E/Encyclopaedia_of_Science -> ../cd/a1a
Q> lrwxrwxrwx NA/b0b -> ../b0b
Q> lrwxrwxrwx T/Thesaurus_of_Science -> ../cd/a1b
Q> dr-xr-xr-x cd/a0
Q> dr-xr-xr-x cd/a1
Q> dr-xr-xr-x cd/a1a
Q> dr-xr-xr-x cd/a1b
Q> dr-xr-xr-x cd/a2
Q> dr-xr-xr-x cd/b0
Q> dr-xr-xr-x cd/b0a
Q> dr-xr-xr-x cd/b0b
Q> dr-xr-xr-x cd/b1
Q> dr-xr-xr-x cd/b2
There would be the /jukebox/cd directory which contained a list of the
valid cd's and the identified partitions thereon, and in addition,
there would be a set of directories relating to the volume labels on
the disks that have been mounted, with the name of the directory being
the upper case form of the first letter in the volume name, as shown
above, or the character itself if not a letter, thus reducing the
number of suchlike names to be searched in any individual directory.
For CD's without volume labels, the NA (for "Not Available") directory
would be used for the link, and directories would only appear if there
was at least one volume label in them...
In the above listing, I've assumed long volume labels (probably
unrealistic), and that spaces in the volume label would be translated
to underscores, but that isn't absolutely necessary either...
Best wishes from Riley.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

stevem at chiark

May 3, 1998, 9:07 AM

Post #16 of 16 (195 views)
Permalink

Re: CDROM jukebox filesystem using autofs [In reply to]

H. Peter Anvin writes:
>
>> Are you suggesting that when disks 1-4 are mounted and busy
>> and I do an ls on /autofs/5, the ls gets an EBUSY? This doesn't
>> make much sense. Normal small CDROM jukeboxes I have seen working
>> under Linux allow you to have all of the disks mounted and they swap
>> disks as necessary. These are done by multiplexing at the block
>> device level so you get a /dev/scd0 - /dev/scd5 for a 6 disk
>> changer. The jukebox I am writing a driver for has hundreds of
>> disks so it seems impractical to do this -- hence my autofs
>> approach. Using the autofs model, the only choices I have are to
>> grant a request from the kernel or say it doesn't exist (the process
>> gets a File not Found and a negative dentry is created). I can
>> modify autofs to also support sending an EBUSY but is this desired?
>
>I think the way to do this is going to involve some amount of disk
>caching if you want it to work sanely.
Yes, very much so. In fact at this point it's probably worth looking at
some of the commercial implementations of this kind of thing. For example,
I know that Plasmon ship copies of Manager, a jukebox control/interface
product, with most of their jukeboxes. This makes the jukebox itself
appear as a single filesystem, with subdirectories below that for each
disk in the system. Add persistent caching of file and meta data...
Steve, not speaking for Plasmon even if he does work for them...
--
Steve McIntyre, CURS CCE, Cambridge, UK. stevem [at] chiark
<a href=http://www.chiark.greenend.org.uk/~stevem/>My home page
"Can't keep my eyes from the circling sky, +------------------
"Tongue-tied & twisted, Just an earth-bound misfit, I..." |Finger for PGP key
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo [at] vger

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads