Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Zope: Dev

zope.filerepresentation

 

 

Zope dev RSS feed   Index | Next | Previous | View Threaded


optilude+lists at gmail

Sep 30, 2009, 8:00 AM

Post #1 of 7 (678 views)
Permalink
zope.filerepresentation

Hi,

I'm doing some work with WebDAV representations in Zope 2, among other
things, and I'm trying to see if we should use zope.filerepresentation
as the central abstraction for file read/write operations.

However, I find myself lacking a couple of things:

1) A way for an IReadFile to return the file's MIME (content) type
(for the Content-Type, for example) and an optional filename (for the
Conent-Disposition header, for example)

2) A way for an IWriteFile to be told about the MIME type and encoding
of the data it is consuming. I know that IFileFactory() contains this,
but data is not always going to be written to a new file, so setting
this from the factory won't work.

3) I see this comment:

# TODO: We will add ILargeReadFile and ILargeWriteFile to efficiently
# handle large data.

But it seems this was never implemented. I need a way for the
IWriteFile.write() to be passed a file-like object (in fact, there's not
really any clear notion of what the 'data' parameter is, but I assume
it's meant to be a str object), and for IReadFile.read() to return a
file stream iterator of some kind instead of the full raw data.

I see two options here:

- add a new plone.filerepresentation that extends these interfaces
- commit some changes to zope.filerepresentation and make a new
release of this

Since zope.filerepresentation is really *just* interfaces, #2 seems
best. Any objections?

Here's an initial take on the interfaces:

class IReadFile(Interface):
"""Provide read access to file data
"""

def read():
"""Return the file data as a str
"""

def size():
"""Return the data length
"""

class ILargeReadFile(IReadFile):
"""Provide efficient read access to file data
"""

def getContentType():
"""Get the content/MIME type of the file as a string in the form
'major/minor'. Return None if this is not known.
"""

def getFilename():
"""Get the intended file name for this file. Return None if this
is not known.
"""


def getIterator():
"""Return an iterable of the file data
"""

class IWriteFile(Interface):

def write(data):
"""Update the file data
"""

class ILargeWriteFile(IWriteFile):

def writeFile(data, contentType=None, encoding=None):
"""Update the file data.

data is a file-like object containing input data

contentType is the content/MIME type of the data as a string in
the form 'major/minor'. It may be None if unknown.

encoding is the encoding of the string. It may be None
if unknown.
"""

This should be 100% backwards compatible. The downside is that people
will need to implement both write() and writeFile() / read() and
getIterator() to support both str based and stream based reading and
writing, but that's easily addressed with a StringIO.

And of course, nothing in the ZTK will actually use these new
interfaces, but that could come later. ;)

Martin


--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


hanno at hannosch

Sep 30, 2009, 8:50 AM

Post #2 of 7 (645 views)
Permalink
Re: zope.filerepresentation [In reply to]

On Wed, Sep 30, 2009 at 5:00 PM, Martin Aspeli <optilude+lists [at] gmail> wrote:
> Here's an initial take on the interfaces:
>
> class IReadFile(Interface):
>     """Provide read access to file data
>     """
>
>     def read():
>         """Return the file data as a str
>         """
>
>     def size():
>         """Return the data length
>         """
>
> class ILargeReadFile(IReadFile):
>     """Provide efficient read access to file data
>     """
>
>     def getContentType():
>         """Get the content/MIME type of the file as a string in the form
>         'major/minor'. Return None if this is not known.
>         """
>
>     def getFilename():
>         """Get the intended file name for this file. Return None if this
>         is not known.
>         """
>
>
>     def getIterator():
>         """Return an iterable of the file data
>         """
>
> class IWriteFile(Interface):
>
>     def write(data):
>         """Update the file data
>         """
>
> class ILargeWriteFile(IWriteFile):
>
>     def writeFile(data, contentType=None, encoding=None):
>         """Update the file data.
>
>         data is a file-like object containing input data
>
>         contentType is the content/MIME type of the data as a string in
>         the form 'major/minor'. It may be None if unknown.
>
>         encoding is the encoding of the string. It may be None
>         if unknown.
>         """

Is there any reason to invent a new API and not just use Python's file API?

So .name instead of getFilename, iterator per __iter__ and next?

I'm not sure about the different read/write methods on file-objects,
but I think the standard ones do cover all the same use-cases.

The standard file implementation has no knowledge of its size, as this
is sometimes impossible to get, when dealing with stream based
file-like objects. Do we really need to have files to know their size?

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


optilude+lists at gmail

Sep 30, 2009, 5:13 PM

Post #3 of 7 (642 views)
Permalink
Re: zope.filerepresentation [In reply to]

Hanno Schlichting wrote:

> Is there any reason to invent a new API and not just use Python's file API?

I don't know. IReadFile and IWriteFile have been around for ever and are
used by a number of things in Zope. They have read(), write() and
size(). The first two are on file, the last one isn't. I'd like to be
able to use this API as a base, since it's used for things like
z3c.blobfile already, and is documented as the way to do this kind of
thing in Philipp's book.

> So .name instead of getFilename, iterator per __iter__ and next?

Yeah, that's a better idea.

> I'm not sure about the different read/write methods on file-objects,
> but I think the standard ones do cover all the same use-cases.
>
> The standard file implementation has no knowledge of its size, as this
> is sometimes impossible to get, when dealing with stream based
> file-like objects. Do we really need to have files to know their size?

Well, for the writeFile() stuff maybe we don't. Thinking through my use
cases again, I can't see a need for passing the content type in, and
encoding can be set if we support setting the '.encoding' property.

It's kind of important to be able to indicate the size and MIME type for
a read operation, though. In this case, I want to be able to put that
information into the Content-Type and Content-Length headers. The
IStreamData stuff in Zope 2 also wants to know the length before it
starts streaming.

In some cases, it may not be possible to know, in which case we can fall
back on len(data), but in other cases, the length is known
(plone.namedfile and z3c.blobfile keep track of it, for example), in
which case having a way to ask the adapter means it can be done much
more efficiently.

So, how about this:

class IReadFile(Interface):
"""Provide read access to file data
"""

def read():
"""Return the file data as a str
"""

def size():
"""Return the data length
"""

class ILargeReadFile(IReadFile):
"""Provide efficient read access to file data
"""

def getContentType():
"""Get the content/MIME type of the file as a string in the form
'major/minor'. Return None if this is not known or undefined.
"""

name = schema.TextLine(title=u"Filename", readonly=True)
encoding = schema.ASCIILine(title=u"Encoding", readonly=True)

def __iter__():
"""Get an iterator"""

def next():
"""See file"""

def seek():
"""See file"""

def tell():
"""See file"""

class IWriteFile(Interface):

def write(data):
"""Update the file data
"""

class ILargeWriteFile(IWriteFile):

def write(data):
"""Write a chunk of data
"""

name = schema.TextLine(title=u"Filename", readonly=False)
encoding = schema.ASCIILine(title=u"Encoding", readonly=False)

def tell():
"""See file"""

def close():
"""See file""

It may be worth supporting the rest of the functions (readlines,
writelines, truncate, readline, isatty, fileno, flush) so that
ILargeReadFile + ILargeWriteFile are equivalent to file.

I've also allowed here for the name and encoding to be set on an
ILargeWriteFile, with the caveat that this has to be done before the
writing.

In this approach, the caller is responsible for writing large streams in
chunks and then calling close, i.e. the iterator isn't passed to the
ILargeWriteFile. I think that's OK, though.

I've still got the content type as a method (maybe make it a read-only
property?) that may return None. That could be in a separate adapter,
but that feels like overkill to me. :)

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


hanno at hannosch

Oct 1, 2009, 4:34 AM

Post #4 of 7 (646 views)
Permalink
Re: zope.filerepresentation [In reply to]

On Thu, Oct 1, 2009 at 2:13 AM, Martin Aspeli <optilude+lists [at] gmail> wrote:
> Hanno Schlichting wrote:
>
>> Is there any reason to invent a new API and not just use Python's file API?
>
> I don't know. IReadFile and IWriteFile have been around for ever and are
> used by a number of things in Zope. They have read(), write() and
> size(). The first two are on file, the last one isn't. I'd like to be
> able to use this API as a base, since it's used for things like
> z3c.blobfile already, and is documented as the way to do this kind of
> thing in Philipp's book.

Ok. I have a feeling that Zope3 at various times invented a new
Java-ish API instead of using standard Python protocols. I'd just like
to avoid going down that way even more.

> class IReadFile(Interface):
>     """Provide read access to file data
>     """
>
>     def read():
>         """Return the file data as a str
>         """
>
>     def size():
>         """Return the data length
>         """

This needs a clarification on what kind of length this is. Is it the
length of the binary data? Since these interfaces also know about
encoding, it's otherwise not clear if the size for textual data might
be its Unicode length.

> class ILargeReadFile(IReadFile):
>     """Provide efficient read access to file data
>     """
>
>     def getContentType():
>         """Get the content/MIME type of the file as a string in the form
>         'major/minor'. Return None if this is not known or undefined.
>         """
>
>     name = schema.TextLine(title=u"Filename", readonly=True)
>     encoding = schema.ASCIILine(title=u"Encoding", readonly=True)

encoding only makes sense for text/*, so maybe some small hint at that?

>     def __iter__():
>         """Get an iterator"""
>
>     def next():
>         """See file"""
>
>     def seek():
>         """See file"""
>
>     def tell():
>         """See file"""
>
> class IWriteFile(Interface):
>
>     def write(data):
>         """Update the file data
>         """
>
> class ILargeWriteFile(IWriteFile):
>
>     def write(data):
>         """Write a chunk of data
>         """
>
>     name = schema.TextLine(title=u"Filename", readonly=False)
>     encoding = schema.ASCIILine(title=u"Encoding", readonly=False)
>
>     def tell():
>         """See file"""
>
>     def close():
>         """See file""
>
> I've still got the content type as a method (maybe make it a read-only
> property?) that may return None. That could be in a separate adapter,
> but that feels like overkill to me. :)

There's no standard way to spell "content type", so I don't really
care. On the pure file level it's even called mime type and only
internet data handling uses "content type". So you have to look things
up anyways. Compared to the other attributes/methods "getContentType"
looks like a Java-intruder in otherwise Python naming conventions,
though.

Hanno
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


optilude+lists at gmail

Oct 1, 2009, 6:51 AM

Post #5 of 7 (646 views)
Permalink
Re: zope.filerepresentation [In reply to]

Hanno Schlichting wrote:
> On Thu, Oct 1, 2009 at 2:13 AM, Martin Aspeli <optilude+lists [at] gmail> wrote:
>> Hanno Schlichting wrote:
>>
>>> Is there any reason to invent a new API and not just use Python's file API?
>> I don't know. IReadFile and IWriteFile have been around for ever and are
>> used by a number of things in Zope. They have read(), write() and
>> size(). The first two are on file, the last one isn't. I'd like to be
>> able to use this API as a base, since it's used for things like
>> z3c.blobfile already, and is documented as the way to do this kind of
>> thing in Philipp's book.
>
> Ok. I have a feeling that Zope3 at various times invented a new
> Java-ish API instead of using standard Python protocols. I'd just like
> to avoid going down that way even more.
>
>> class IReadFile(Interface):
>> """Provide read access to file data
>> """
>>
>> def read():
>> """Return the file data as a str
>> """
>>
>> def size():
>> """Return the data length
>> """
>
> This needs a clarification on what kind of length this is. Is it the
> length of the binary data? Since these interfaces also know about
> encoding, it's otherwise not clear if the size for textual data might
> be its Unicode length.

I assume so. Note that this interface already exists, so I'm not
proposing to change it. ;-)

>> class ILargeReadFile(IReadFile):
>> """Provide efficient read access to file data
>> """
>>
>> def getContentType():
>> """Get the content/MIME type of the file as a string in the form
>> 'major/minor'. Return None if this is not known or undefined.
>> """
>>
>> name = schema.TextLine(title=u"Filename", readonly=True)
>> encoding = schema.ASCIILine(title=u"Encoding", readonly=True)
>
> encoding only makes sense for text/*, so maybe some small hint at that?

Sure, yeah.

>> def __iter__():
>> """Get an iterator"""
>>
>> def next():
>> """See file"""
>>
>> def seek():
>> """See file"""
>>
>> def tell():
>> """See file"""
>>
>> class IWriteFile(Interface):
>>
>> def write(data):
>> """Update the file data
>> """
>>
>> class ILargeWriteFile(IWriteFile):
>>
>> def write(data):
>> """Write a chunk of data
>> """
>>
>> name = schema.TextLine(title=u"Filename", readonly=False)
>> encoding = schema.ASCIILine(title=u"Encoding", readonly=False)
>>
>> def tell():
>> """See file"""
>>
>> def close():
>> """See file""
>>
>> I've still got the content type as a method (maybe make it a read-only
>> property?) that may return None. That could be in a separate adapter,
>> but that feels like overkill to me. :)
>
> There's no standard way to spell "content type", so I don't really
> care. On the pure file level it's even called mime type and only
> internet data handling uses "content type". So you have to look things
> up anyways. Compared to the other attributes/methods "getContentType"
> looks like a Java-intruder in otherwise Python naming conventions,
> though.

Haha. Maybe I'll have a property mimeType instead? That looks better, I
guess.

But we *are* in a Zope package, so Zope naming conventions probably apply.

Ok, I'm going to do this shortly, unless anyone objects.

Martin

--
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


charlie.clark at clark-consulting

Oct 1, 2009, 7:02 AM

Post #6 of 7 (646 views)
Permalink
Re: zope.filerepresentation [In reply to]

Am 01.10.2009, 15:51 Uhr, schrieb Martin Aspeli <optilude+lists [at] gmail>:

> Haha. Maybe I'll have a property mimeType instead? That looks better, I
> guess.

+1 to use a property rather than a getter. Not sure about content-type vs.
mime-type. If this is used only in an HTTP environment then I'd prefer to
stick with content-type otherwise I'd suggest the email module's
content-type handling.

> But we *are* in a Zope package, so Zope naming conventions probably
> apply.

I go with Pythonic conventions wherever possible.

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Helmholtztstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


l at lrowe

Oct 1, 2009, 8:25 AM

Post #7 of 7 (634 views)
Permalink
Re: zope.filerepresentation [In reply to]

2009/10/1 Martin Aspeli <optilude+lists [at] gmail>:
> Hanno Schlichting wrote:
>> The standard file implementation has no knowledge of its size, as this
>> is sometimes impossible to get, when dealing with stream based
>> file-like objects. Do we really need to have files to know their size?
>
> Well, for the writeFile() stuff maybe we don't. Thinking through my use
> cases again, I can't see a need for passing the content type in, and
> encoding can be set if we support setting the '.encoding' property.
>
> It's kind of important to be able to indicate the size and MIME type for
> a read operation, though. In this case, I want to be able to put that
> information into the Content-Type and Content-Length headers. The
> IStreamData stuff in Zope 2 also wants to know the length before it
> starts streaming.
>
> In some cases, it may not be possible to know, in which case we can fall
> back on len(data), but in other cases, the length is known
> (plone.namedfile and z3c.blobfile keep track of it, for example), in
> which case having a way to ask the adapter means it can be done much
> more efficiently.

To find the length of a file:

f.seek(0,2) # 0 bytes from the end of the file
size = f.tell() # position in file.

Laurence
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )

Zope dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.