Mailing List Archive: Python: Dev

new httplib?

Index | Next | Previous | View Threaded

gstein at lyra

Feb 16, 2000, 5:03 PM

Post #1 of 10 (160 views)
Permalink

new httplib?

Hi all,

I've been using my HTTP/1.1 httplib for quite a while now. It seems quite
stable in everything that I've tried. I need to port over the recent
changes for SSL stuff, but it is ready to go.

The library contains a deprecated, backwards-compat class (HTTP), and its
replacement: HTTPConnection. Responses are now handled through
HTTPResponse class.

Doc for the new class will also be needed, but (of course) the old doc
still applies.

Any comments on the module and/or its inclusion into 1.6?

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

guido at python

Feb 16, 2000, 5:59 PM

Post #2 of 10 (156 views)
Permalink

Re: new httplib? [In reply to]

> I've been using my HTTP/1.1 httplib for quite a while now. It seems quite
> stable in everything that I've tried. I need to port over the recent
> changes for SSL stuff, but it is ready to go.
>
> The library contains a deprecated, backwards-compat class (HTTP), and its
> replacement: HTTPConnection. Responses are now handled through
> HTTPResponse class.
>
> Doc for the new class will also be needed, but (of course) the old doc
> still applies.
>
> Any comments on the module and/or its inclusion into 1.6?

Doesn't Jeremy have one too?

I would suggest putting the new classes in a new module,
e.g. httplib2.py, and leaving the old httplib unchanged.

BTW, where can we look at this code?

--Guido van Rossum (home page: http://www.python.org/~guido/)

gstein at lyra

Feb 17, 2000, 1:05 AM

Post #3 of 10 (156 views)
Permalink

Re: new httplib? [In reply to]

On Wed, 16 Feb 2000, Guido van Rossum wrote:
> > I've been using my HTTP/1.1 httplib for quite a while now. It seems quite
> > stable in everything that I've tried. I need to port over the recent
> > changes for SSL stuff, but it is ready to go.
> >
> > The library contains a deprecated, backwards-compat class (HTTP), and its
> > replacement: HTTPConnection. Responses are now handled through
> > HTTPResponse class.
> >
> > Doc for the new class will also be needed, but (of course) the old doc
> > still applies.
> >
> > Any comments on the module and/or its inclusion into 1.6?
>
> Doesn't Jeremy have one too?

Yes, we talked about it at IPC7. As I recall, Jeremy wasn't happy with the
result... it was request/response based with a lot of helper classes. My
httplib is connection-oriented (like the current httplib), and I think
Jeremy had said that was good (empirically, after his prototype).

But that's just let my memory... Jeremy?

> I would suggest putting the new classes in a new module,
> e.g. httplib2.py, and leaving the old httplib unchanged.

Why? The compatibility class exports the same attributes, the same
methods, the same constructor, etc. Creating a new module is needless
duplication and confusing for users ("which one do I use?" "how is that
one different?" "do I have to switch modules to use HTTP/1.1?")

A single module that reuses code is going to be much more maintainable
than two, possibly diverging, modules. I also feel having one module will
help to gently prod people towards using HTTP/1.1 and the new
HTTPConnection class (which I think is *subjectively* a nice goal).

> BTW, where can we look at this code?

Oops. Sorry about that... As usual, it is with the rest of my Python code
at http://www.lyra.org/greg/python/

(the davlib is there, too)

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

jeremy at cnri

Feb 17, 2000, 11:57 AM

Post #4 of 10 (156 views)
Permalink

Re: new httplib? [In reply to]

>>>>> "GS" == Greg Stein <gstein [at] lyra> writes:

>> Doesn't Jeremy have one too?

GS> Yes, we talked about it at IPC7. As I recall, Jeremy wasn't
GS> happy with the result... it was request/response based with a
GS> lot of helper classes. My httplib is connection-oriented (like
GS> the current httplib), and I think Jeremy had said that was good
GS> (empirically, after his prototype).

GS> But that's just let my memory... Jeremy?

Sounds right to me. I haven't worked on http code in years. I seem to
recall liking the basic architecture, but not the implementation.
That said, I have no objections to including your modifications: It
preserves the current interface and improves its support for http/1.1.

I still think the httplib interface could be improved, but I don't
have any time to work on it.

The one thing I don't see how to do with the new code is pipeline
requests. Is that possible? It's not a big deal if pipelining isn't
support, since the current implementation doesn't support it at all.

Jeremy

gstein at lyra

Feb 17, 2000, 12:40 PM

Post #5 of 10 (156 views)
Permalink

Re: new httplib? [In reply to]

On Thu, 17 Feb 2000, Jeremy Hylton wrote:
>...
> The one thing I don't see how to do with the new code is pipeline
> requests. Is that possible? It's not a big deal if pipelining isn't
> support, since the current implementation doesn't support it at all.

Pipelining is very important... it is once of the big advances in HTTP/1.1
(well, the advance is really the "chunked" transfer encoding which
*allows* pipelining to occur more often).

To do pipelining, just keep feeding requests into the thing:

-------------------------
conn = httplib.HTTPConnection('www.python.org')

conn.request('GET', '/')
code, msg, response = conn.getreply()
print response.read()

conn.request('GET', '/sigs/')
code, msg, response = conn.getreply()
print response.read()
-------------------------

Note that the code requires you to consume the response before issuing
another request (it will raise an error otherwise).

The connection object will connect automatically connect to the host. It
will keep the connection open per HTTP/1.1 guidelines (or close it when
told to). If an "broken pipe" occurs during the sending of the request
(the other end closed the socket), then it will close, reconnect, and
re-issue the request (just once). The broken pipe typically occurs if you
wait too long between requests. The client can force a connect or a close
thru the connection object API.

In addition to the simple request() method above, there is the standard
putrequest, putheader, endheader style.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

fdrake at acm

Feb 17, 2000, 12:50 PM

Post #6 of 10 (159 views)
Permalink

Re: new httplib? [In reply to]

Greg Stein writes:
> Pipelining is very important... it is once of the big advances in HTTP/1.1
> (well, the advance is really the "chunked" transfer encoding which
> *allows* pipelining to occur more often).

You should be able to pipeline requests even if responses don't use
chunking; if I parse the first HTML page while still receiving, I
should be able to request non-cached images or other dependencies
before the first HTML is completely received. This allows the server
to locate the resources while still waiting for me finish pulling the
first response off the network. It improves the possibility of
scheduling resource location at the server (possibly important if
there's a dynamic backend) as well as avoiding establishing new TCP
connections.
(Yes, some of this can be done with keep-alive stuff, but I don't
know that httplib supports that at all.)

-Fred

--
Fred L. Drake, Jr. <fdrake at acm.org>
Corporation for National Research Initiatives

gstein at lyra

Feb 17, 2000, 1:01 PM

Post #7 of 10 (154 views)
Permalink

Re: new httplib? [In reply to]

On Thu, 17 Feb 2000, Fred L. Drake, Jr. wrote:
> Greg Stein writes:
> > Pipelining is very important... it is once of the big advances in HTTP/1.1
> > (well, the advance is really the "chunked" transfer encoding which
> > *allows* pipelining to occur more often).
>
> You should be able to pipeline requests even if responses don't use
> chunking;

Not in all cases. You can pipeline only if the server sends a
Content-Length: header or uses chunking. There are a few other conditions
(look at the httplib.HTTPResponse constructor in my httplib), but the
Content-Length/chunking is the major point.

> if I parse the first HTML page while still receiving, I
> should be able to request non-cached images or other dependencies
> before the first HTML is completely received. This allows the server
> to locate the resources while still waiting for me finish pulling the
> first response off the network. It improves the possibility of
> scheduling resource location at the server (possibly important if
> there's a dynamic backend) as well as avoiding establishing new TCP
> connections.

This would be nice, but my httplib currently enforces consumption before a
new request. I do not recall the specific reasons why, but I imagine it
would be possible to change it.

[. today: you could certainly use a second connection object to improve
client response, with the corresponding impact on the server ]

> (Yes, some of this can be done with keep-alive stuff, but I don't
> know that httplib supports that at all.)

My httplib deals with the keep-alive header, when applicable (you are
supposed to use Connection: in HTTP/1.1; it only looks at keep-alive if
you aren't using 1.1).

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

jeremy at cnri

Feb 17, 2000, 1:04 PM

Post #8 of 10 (155 views)
Permalink

Re: new httplib? [In reply to]

This sounds good enough for me! I endorse it for inclusion as long as
their is accompanying documentation <0.25 wink>.

Jeremy

gstein at lyra

Feb 17, 2000, 1:33 PM

Post #9 of 10 (156 views)
Permalink

Re: new httplib? [In reply to]

On Thu, 17 Feb 2000, Jeremy Hylton wrote:
> This sounds good enough for me! I endorse it for inclusion as long as
> their is accompanying documentation <0.25 wink>.

Nowadays, I consider it a prerequisite, along with an updated test suite.
In the past, I've submitted stuff and the doc stuff has hung over my head
for a long while... I'd rather get it done up front :-)

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

fdrake at acm

Feb 17, 2000, 1:40 PM

Post #10 of 10 (159 views)
Permalink

Re: new httplib? [In reply to]

Greg Stein writes:
> Nowadays, I consider it a prerequisite, along with an updated test suite.
> In the past, I've submitted stuff and the doc stuff has hung over my head
> for a long while... I'd rather get it done up front :-)

And Greg knows how heavy it can hang! ;) Thanks for the buffer
documentation, Greg!

-Fred

--
Fred L. Drake, Jr. <fdrake at acm.org>
Corporation for National Research Initiatives

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads