Mailing List Archive: Catalyst: Users

File downloads - Plugin?

Index | Next | Previous | View Threaded

tjcatwintrmute.net

Aug 11, 2005, 1:25 PM

Post #1 of 17 (430 views)
Permalink

File downloads - Plugin?

Hi,
I'm in a position where I'd like to serve file downloads, out of database
BLOBs, via Catalyst.

These files are potentially very large, and so I would rather not suck the
entire lot of data out of the DB and into Catalyst's $c->response->output()
variable. I'd rather have some kind of solution whereby I pass a
IO::Handle, and then at output time this is used to write directly out to
Apache.

Is there any plugin that will help me here, or would I need to write this
myself?

Am I barking up the wrong tree with this approach?

Thanks,
Toby

--
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key 897E5FF3)

dbix-classattrout.me.uk

Aug 11, 2005, 9:52 PM

Post #2 of 17 (436 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 11, 2005 at 12:27:29PM +0100, Toby Corkindale wrote:
> Hi,
> I'm in a position where I'd like to serve file downloads, out of database
> BLOBs, via Catalyst.
>
> These files are potentially very large, and so I would rather not suck the
> entire lot of data out of the DB and into Catalyst's $c->response->output()
> variable. I'd rather have some kind of solution whereby I pass a
> IO::Handle, and then at output time this is used to write directly out to
> Apache.
>
> Is there any plugin that will help me here, or would I need to write this
> myself?

I think the refactor/ branch in svn and the HTTP-Body package (also in svn)
are working towards this ability but not quite here yet. Maybe you could have
a go at helping out with that.

--
Matt S Trout Website: http://www.shadowcatsystems.co.uk
Technical Director E-mail: mst (at) shadowcatsystems.co.uk
Shadowcat Systems Ltd.

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +

sriatoook.de

Aug 12, 2005, 6:25 AM

Post #3 of 17 (435 views)
Permalink

File downloads - Plugin? [In reply to]

Am 11.08.2005 um 22:03 schrieb Matt S Trout:

> On Thu, Aug 11, 2005 at 12:27:29PM +0100, Toby Corkindale wrote:
>
>> Hi,
>> I'm in a position where I'd like to serve file downloads, out of
>> database
>> BLOBs, via Catalyst.
>>
>> These files are potentially very large, and so I would rather not
>> suck the
>> entire lot of data out of the DB and into Catalyst's $c->response-
>> >output()
>> variable. I'd rather have some kind of solution whereby I pass a
>> IO::Handle, and then at output time this is used to write directly
>> out to
>> Apache.
>>
>> Is there any plugin that will help me here, or would I need to
>> write this
>> myself?
>>
>
> I think the refactor/ branch in svn and the HTTP-Body package (also
> in svn)
> are working towards this ability but not quite here yet. Maybe you
> could have
> a go at helping out with that.

Well, we are working on uploads actually, handles for output were
planned for a long time but so far we havn't agreed on how to
implement them (without breaking backwards compatibility).

--
sebastian

tjcatwintrmute.net

Aug 12, 2005, 11:24 AM

Post #4 of 17 (438 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 11, 2005 at 09:03:04PM +0100, Matt S Trout wrote:
> On Thu, Aug 11, 2005 at 12:27:29PM +0100, Toby Corkindale wrote:
> > Hi,
> > I'm in a position where I'd like to serve file downloads, out of database
> > BLOBs, via Catalyst.
> >
> > These files are potentially very large, and so I would rather not suck the
> > entire lot of data out of the DB and into Catalyst's $c->response->output()
> > variable. I'd rather have some kind of solution whereby I pass a
> > IO::Handle, and then at output time this is used to write directly out to
> > Apache.
> >
> > Is there any plugin that will help me here, or would I need to write this
> > myself?
>
> I think the refactor/ branch in svn and the HTTP-Body package (also in svn)
> are working towards this ability but not quite here yet. Maybe you could have
> a go at helping out with that.

Sure, I'm happy to help out. I've checked out HTTP-Body from the trunk, but
Body.pm is rather POD and comment deficient.. Does Christian have any pages on
the wiki or somewhere about what it's supposed to do?

Thanks for the pointers,
Toby

--
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key 897E5FF3)

sriatoook.de

Aug 12, 2005, 11:29 AM

Post #5 of 17 (438 views)
Permalink

File downloads - Plugin? [In reply to]

Am 12.08.2005 um 11:26 schrieb Toby Corkindale:
> Sure, I'm happy to help out. I've checked out HTTP-Body from the
> trunk, but
> Body.pm is rather POD and comment deficient.. Does Christian have
> any pages on
> the wiki or somewhere about what it's supposed to do?

No (as usual). :(

--
sebastian

samwyseatgmail.com

Aug 16, 2005, 1:49 PM

Post #6 of 17 (437 views)
Permalink

File downloads - Plugin? [In reply to]

On 8/11/05, Toby Corkindale <tjc [at] wintrmute> wrote:
> Hi,
> I'm in a position where I'd like to serve file downloads, out of database
> BLOBs, via Catalyst.
>
> These files are potentially very large, and so I would rather not suck the
> entire lot of data out of the DB and into Catalyst's $c->response->output()
> variable.

If you want to serve things via Catalyst, I'd expect the data to pass
through Catalyst. If you want to get Catalyst out of the way, then
I'd look towards alternate methods of serving. In other words, handle
it in much the same way that static data is handled, because that's
what it is: static data that just happens to be kept in a database
instead of a filesystem.

clacoatchrislaco.com

Aug 16, 2005, 2:41 PM

Post #7 of 17 (431 views)
Permalink

File downloads - Plugin? [In reply to]

samwyse wrote:
> On 8/11/05, Toby Corkindale <tjc [at] wintrmute> wrote:
>
>>Hi,
>>I'm in a position where I'd like to serve file downloads, out of database
>>BLOBs, via Catalyst.
>>
>>These files are potentially very large, and so I would rather not suck the
>>entire lot of data out of the DB and into Catalyst's $c->response->output()
>>variable.
>
>
> If you want to serve things via Catalyst, I'd expect the data to pass
> through Catalyst. If you want to get Catalyst out of the way, then
> I'd look towards alternate methods of serving. In other words, handle
> it in much the same way that static data is handled, because that's
> what it is: static data that just happens to be kept in a database
> instead of a filesystem.
>

I've been thinking about this lately. Shouldn't it be possible to have
Catalysts MP engine add the urls it handle to apache config at runtime?

Then, anything Catalyst handles is forwarded to set-handler to itself,
and everything else Just Works as real Apache requests?

This suedo code of course...I've not yet put Catalyst under MP...but you
get the idea....

MyApp::C::Foo;

sub doit : Global {

};

Then, when Catalyst loads into MP, it does:

<Location /doit>
set-handler perl-scripts
Catalyst::Engine
</Location>

From there on out, all request to things like /css/style.cc, etc Just
Work, while Catalyst still does it's thing on methods it knows... like /doit

-=Chris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3178 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.rawmode.org/pipermail/catalyst/attachments/20050816/dbbe8a1e/smime.bin

mcummingsatgentoo.org

Aug 16, 2005, 5:59 PM

Post #8 of 17 (435 views)
Permalink

File downloads - Plugin? [In reply to]

On Tue, 16 Aug 2005 08:44:22 -0400
"Christopher H. Laco" <claco [at] chrislaco> wrote:

> I've been thinking about this lately. Shouldn't it be possible to have

> Catalysts MP engine add the urls it handle to apache config at
runtime?

Sounds along the same vein as:

http://enemy-combatant.com/abstracting-the-apache-configuration-api.pdf

(gleamed from a conversation in #modperl a few day
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.rawmode.org/pipermail/catalyst/attachments/20050816/2198fc8e/attachment.pgp

clacoatchrislaco.com

Aug 16, 2005, 6:11 PM

Post #9 of 17 (436 views)
Permalink

File downloads - Plugin? [In reply to]

Michael Cummings wrote:
> On Tue, 16 Aug 2005 08:44:22 -0400
> "Christopher H. Laco" <claco [at] chrislaco> wrote:
>
>
>>I've been thinking about this lately. Shouldn't it be possible to have
>
>
>>Catalysts MP engine add the urls it handle to apache config at
>
> runtime?
>
> Sounds along the same vein as:
>
> http://enemy-combatant.com/abstracting-the-apache-configuration-api.pdf
>
> (gleamed from a conversation in #modperl a few days ago by dorian)
>
>

Interesting. Assuming it could be done with the MP13, MP19 and MP20
engines, would the same be possible with the HTTP/FCGI engines?

Personally, I think having to load and configure
Catalyst::Plugin::Static to serve static content or tweak httpd.conf to
add new static directories is somewhat backwards. Why does Catalyst need
to handle every request when it clearly knows what public urls are
available [based onthe debug output]?

Time to start a new thread.
-=Chris

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3178 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.rawmode.org/pipermail/catalyst/attachments/20050816/80ba8915/smime-0001.bin

tjcatwintrmute.net

Aug 18, 2005, 3:55 PM

Post #10 of 17 (437 views)
Permalink

File downloads - Plugin? [In reply to]

On Tue, Aug 16, 2005 at 06:52:32AM -0500, samwyse wrote:
> On 8/11/05, Toby Corkindale <tjc [at] wintrmute> wrote:
> > Hi,
> > I'm in a position where I'd like to serve file downloads, out of database
> > BLOBs, via Catalyst.
> >
> > These files are potentially very large, and so I would rather not suck the
> > entire lot of data out of the DB and into Catalyst's $c->response->output()
> > variable.
>
> If you want to serve things via Catalyst, I'd expect the data to pass
> through Catalyst. If you want to get Catalyst out of the way, then
> I'd look towards alternate methods of serving. In other words, handle
> it in much the same way that static data is handled, because that's
> what it is: static data that just happens to be kept in a database
> instead of a filesystem.

Ah, the thing is, I'd like to continue to use Catalyst for the authentication
and session management in regards to retrieving these files - plus there is
metadata associated with the files-in-database that is accessed/modified when
you go to download the file.

It'd just be.. nice.. if I could do it thru Catalyst.

tjc

--
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key 897E5FF3)

dbix-classattrout.me.uk

Aug 18, 2005, 4:44 PM

Post #11 of 17 (437 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 18, 2005 at 02:55:58PM +0100, Toby Corkindale wrote:
> Ah, the thing is, I'd like to continue to use Catalyst for the authentication
> and session management in regards to retrieving these files - plus there is
> metadata associated with the files-in-database that is accessed/modified when
> you go to download the file.
>
> It'd just be.. nice.. if I could do it thru Catalyst.

What you *could* do is mirror them to disk and then use the latest version
of AndyG's Static::Simple plugin, which under mod_perl will return DECLINED
- at which point apache will serve them, but you still go through Catalyst
in the process.

--
Matt S Trout Website: http://www.shadowcatsystems.co.uk
Technical Director E-mail: mst (at) shadowcatsystems.co.uk
Shadowcat Systems Ltd.

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +

tjcatwintrmute.net

Aug 18, 2005, 5:30 PM

Post #12 of 17 (437 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 18, 2005 at 03:55:51PM +0100, Matt S Trout wrote:
> On Thu, Aug 18, 2005 at 02:55:58PM +0100, Toby Corkindale wrote:
> > Ah, the thing is, I'd like to continue to use Catalyst for the authentication
> > and session management in regards to retrieving these files - plus there is
> > metadata associated with the files-in-database that is accessed/modified when
> > you go to download the file.
> >
> > It'd just be.. nice.. if I could do it thru Catalyst.
>
> What you *could* do is mirror them to disk and then use the latest version
> of AndyG's Static::Simple plugin, which under mod_perl will return DECLINED
> - at which point apache will serve them, but you still go through Catalyst
> in the process.

Hmm, so upon the request I dump the blob to a temporary disk file, then do
a redirect? yeah, that could work.. Is there any danger of the user being able
to manually specify a different filename and accessing the other users' data?

tjc

--
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key 897E5FF3)

dbix-classattrout.me.uk

Aug 18, 2005, 7:24 PM

Post #13 of 17 (434 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 18, 2005 at 04:32:32PM +0100, Toby Corkindale wrote:
> On Thu, Aug 18, 2005 at 03:55:51PM +0100, Matt S Trout wrote:
> > On Thu, Aug 18, 2005 at 02:55:58PM +0100, Toby Corkindale wrote:
> > > Ah, the thing is, I'd like to continue to use Catalyst for the authentication
> > > and session management in regards to retrieving these files - plus there is
> > > metadata associated with the files-in-database that is accessed/modified when
> > > you go to download the file.
> > >
> > > It'd just be.. nice.. if I could do it thru Catalyst.
> >
> > What you *could* do is mirror them to disk and then use the latest version
> > of AndyG's Static::Simple plugin, which under mod_perl will return DECLINED
> > - at which point apache will serve them, but you still go through Catalyst
> > in the process.
>
> Hmm, so upon the request I dump the blob to a temporary disk file, then do
> a redirect? yeah, that could work.. Is there any danger of the user being able
> to manually specify a different filename and accessing the other users' data?

No no no, static returns DECLINED which makes apache try the file *for the
same request*. No redirect required, so no danger - every request still goes
through Catalyst; that way it can return DECLINED to apache if they're allowed
to see the file, or send an error page from your Catalyst app as normal.
the file,

--
Matt S Trout Website: http://www.shadowcatsystems.co.uk
Technical Director E-mail: mst (at) shadowcatsystems.co.uk
Shadowcat Systems Ltd.

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +

andyathybridized.org

Aug 18, 2005, 7:42 PM

Post #14 of 17 (432 views)
Permalink

File downloads - Plugin? [In reply to]

Matt S Trout wrote:
> On Thu, Aug 18, 2005 at 04:32:32PM +0100, Toby Corkindale wrote:
>
>>On Thu, Aug 18, 2005 at 03:55:51PM +0100, Matt S Trout wrote:
>>
>>>On Thu, Aug 18, 2005 at 02:55:58PM +0100, Toby Corkindale wrote:
>>>
>>>>Ah, the thing is, I'd like to continue to use Catalyst for the authentication
>>>>and session management in regards to retrieving these files - plus there is
>>>>metadata associated with the files-in-database that is accessed/modified when
>>>>you go to download the file.
>>>>
>>>>It'd just be.. nice.. if I could do it thru Catalyst.
>>>
>>>What you *could* do is mirror them to disk and then use the latest version
>>>of AndyG's Static::Simple plugin, which under mod_perl will return DECLINED
>>>- at which point apache will serve them, but you still go through Catalyst
>>>in the process.
>>
>>Hmm, so upon the request I dump the blob to a temporary disk file, then do
>>a redirect? yeah, that could work.. Is there any danger of the user being able
>>to manually specify a different filename and accessing the other users' data?
>
>
> No no no, static returns DECLINED which makes apache try the file *for the
> same request*. No redirect required, so no danger - every request still goes
> through Catalyst; that way it can return DECLINED to apache if they're allowed
> to see the file, or send an error page from your Catalyst app as normal.
> the file,
>

I'm not sure this will work, though. Static::Simple short-circuits the
dispatch phase when it finds a valid static file (by checking if the
request path is a valid file on the filesystem with an extension). Once
this comes back as true, it sends the static file right away. There
isn't really a chance to put any kind of security logic in place. Maybe
this would be better done with the original Static plugin, where you can
put that logic in your Static controller.

-Andy

dbix-classattrout.me.uk

Aug 18, 2005, 7:48 PM

Post #15 of 17 (431 views)
Permalink

File downloads - Plugin? [In reply to]

On Thu, Aug 18, 2005 at 01:45:37PM -0400, Andy Grundman wrote:
> >>Hmm, so upon the request I dump the blob to a temporary disk file, then do
> >>a redirect? yeah, that could work.. Is there any danger of the user being
> >>able
> >>to manually specify a different filename and accessing the other users'
> >>data?
> >
> >
> >No no no, static returns DECLINED which makes apache try the file *for the
> >same request*. No redirect required, so no danger - every request still
> >goes
> >through Catalyst; that way it can return DECLINED to apache if they're
> >allowed
> >to see the file, or send an error page from your Catalyst app as normal.
> >the file,
> >
>
> I'm not sure this will work, though. Static::Simple short-circuits the
> dispatch phase when it finds a valid static file (by checking if the
> request path is a valid file on the filesystem with an extension). Once
> this comes back as true, it sends the static file right away. There
> isn't really a chance to put any kind of security logic in place. Maybe
> this would be better done with the original Static plugin, where you can
> put that logic in your Static controller.

Assuming Static can be modified to also send DECLINED (should this maybe be
the same codebase, and static controller vs. automatic operation two modes?)
then yes, that makes more sense. I'd misunderstood how soon Static::Simple
interrupted the processing, sorry.

--
Matt S Trout Website: http://www.shadowcatsystems.co.uk
Technical Director E-mail: mst (at) shadowcatsystems.co.uk
Shadowcat Systems Ltd.

+ Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +

samwyseatgmail.com

Aug 20, 2005, 9:35 AM

Post #16 of 17 (439 views)
Permalink

File downloads - Plugin? [In reply to]

tjcatwintrmute.net

Aug 22, 2005, 7:38 PM

Post #17 of 17 (428 views)
Permalink

File downloads - Plugin? [In reply to]

On Fri, Aug 12, 2005 at 08:14:04AM -0500, samwyse wrote:
> On 8/11/05, Toby Corkindale <tjc [at] wintrmute> wrote:
> > Hi,
> > I'm in a position where I'd like to serve file downloads, out of database
> > BLOBs, via Catalyst.
> >
> > These files are potentially very large, and so I would rather not suck the
> > entire lot of data out of the DB and into Catalyst's $c->response->output()
> > variable.
>
> If you want to serve things via Catalyst, I'd expect the data to pass
> through Catalyst. If you want to get Catalyst out of the way, then
> I'd look towards alternate methods of serving. In other words, handle
> it in much the same way that static data is handled, because that's
> what it is: static data that just happens to be kept in a database
> instead of a filesystem.

Yes, but static data that is potentially larger than available system ram.
Even if it wasn't, then it would be a poor use of it to fill with once-off
data and totally hose the buffercache.

The data isn't quite static - it is produced on demand by the user, but takes
a while to be generated, and is large. So we send the user off to a 'Pending'
page in the meantime. Once the data is ready, they can collect it.
We want access control to be via the standard, existing Catalyst setup,
because that just makes sense. We must have access control on the data, and
security through obscurity (eg. putting a randomly-named file into an insecure
directory, then redirecting user) is not an option. Unfortunately, doing
$c->response->body($query->blob_read(...)); is not an option either.

I'm looking at the Static plugin currently though.

-Toby

--
Turning and turning in the widening gyre/The falcon cannot hear the falconer;
Things fall apart, the centre cannot hold/Mere anarchy is loosed upon the world
(gpg --keyserver www.co.uk.pgp.net --recv-key 897E5FF3)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads