Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

python 2.7 + https + urlopen = ?

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


urllib3 at yahoo

Aug 12, 2012, 6:28 PM

Post #1 of 5 (379 views)
Permalink
python 2.7 + https + urlopen = ?

I am a python 2.7.x user and am hoping to reach with this email some python developers who would be sympathetic to this scenario (And I understand that you might not, which is perfectly fine -- I've already requested one developer not to reply ) :

How would you feel, if you issued :

import urllib
urlopen("""https://server.domain.com""").read()

and the command got you data from some other URL without telling you! You use firefox, and the site is different than the data you got! Same with chrome. Safari. Even IE !
Cheated? (Well I was mad -- after IE worked).

Then, you dig a little and say, hey there are bugs in networks/code, lets try the other tools that are available on python 2.x, who uses urlopen from urllib in 2012. There are tons, right?

urllib2, urllib, urllib3, requests, twisted.getPage, ...

None of them worked! Wow. Then you wonder, whats going on. You poke one of the server administrator, and he sends you the logs, and you see the problem. The keyword being "SNI". Now you start googling. First read about SNI perhaps. Here is a 2 line summary:

SNI is a server side "feature" that extends SSL and TLS protocols to let you talk to a https server which is on an IP that serves multiple certificates for multiple https servers. SNI was first used in 2004 and OpenSSL started support in 2006. In 2007, it was backported to OpenSSL 0.9.x. In 2009 there was a bug filed with python-devs for fixing this in 2.6. The feature enhancement (or "bug fix") eventually happened -- for 3.2+. (http://en.wikipedia.org/wiki/Server_Name_Indication)

Then you google more and you land up on this page: http://bugs.python.org/issue5639

which shows you that 2.6 has a patch. Then you wonder, why wasn't it included in 2.7 -- and you read -- AP : "No, Python 2 only receives bug fixes.". You instantly hate the guy. Sorry AP, nothing personal, but please do not reply to this post. I think I know what your reply will be.  

After a lot of pain, I got myself out of this trouble, and my code now works correctly on 2.7.x (thanks to Jean-Paul Calderone's pyopenssl). But do "you" think this is a "feature" and not a "bug"? -- And do you think debating on this, killing time on the debate, and letting all python 2.x users suffer sooner or later is right --. Something as basic as urlopen! 

Thanks for your time and I wish good luck to most python users.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


urllib3 at yahoo

Aug 12, 2012, 6:39 PM

Post #2 of 5 (380 views)
Permalink
python 2.7 + https + urlopen = ? [In reply to]

I am a python 2.7.x user and am hoping to reach with this email some python developers who would be sympathetic to this scenario (And I understand that you might not, which is perfectly fine -- I've already requested one developer not to reply ) :

How would you feel, if you issued :

import urllib
urlopen("""https://server.domain.com""").read()

and the command got you data from some other URL without telling you! You use firefox, and the site is different than the data you got! Same with chrome. Safari. Even IE !
Cheated? (Well I was mad -- after IE worked).

Then, you dig a little and say, hey there are bugs in networks/code, lets try the other tools that are available on python 2.x, who uses urlopen from urllib in 2012. There are tons, right?

urllib2, urllib, urllib3, requests, twisted.getPage, ...

None of them worked! Wow. Then you wonder, whats going on. You poke one of the server administrator, and he sends you the logs, and you see the problem. The keyword being "SNI". Now you start googling. First read about SNI perhaps. Here is a 2 line summary:

SNI is a server side "feature" that extends SSL and TLS protocols to let you talk to a https server which is on an IP that serves multiple certificates for multiple https servers. SNI was first used in 2004 and OpenSSL started support in 2006. In 2007, it was backported to OpenSSL 0.9.x. In 2009 there was a bug filed with python-devs for fixing this in 2.6. The feature enhancement (or "bug fix") eventually happened -- for 3.2+. (http://en.wikipedia.org/wiki/Server_Name_Indication)

Then you google more and you land up on this page: http://bugs.python.org/issue5639

which shows you that 2.6 has a patch. Then you wonder, why wasn't it included in 2.7 -- and you read -- AP : "No, Python 2 only receives bug fixes.". You instantly hate the guy. Sorry AP, nothing personal, but please do not reply to this post. I think I know what your reply will be.  

After a lot of pain, I got myself out of this trouble, and my code now works correctly on 2.7.x (thanks to Jean-Paul Calderone's pyopenssl). But do "you" think this is a "feature" and not a "bug"? -- And do you think debating on this, killing time on the debate, and letting all python 2.x users suffer sooner or later is right --. Something as basic as urlopen! 

Thanks for your time and I wish good luck to most python users.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


ncoghlan at gmail

Aug 12, 2012, 7:07 PM

Post #3 of 5 (385 views)
Permalink
Re: python 2.7 + https + urlopen = ? [In reply to]

On Mon, Aug 13, 2012 at 11:28 AM, Python Urlopen <urllib3 [at] yahoo> wrote:
> which shows you that 2.6 has a patch. Then you wonder, why wasn't it included in 2.7 -- and you read -- AP : "No, Python 2 only receives bug fixes.". You instantly hate the guy. Sorry AP, nothing personal, but please do not reply to this post. I think I know what your reply will be.

It's not merely Antoine that will give that reply.

Yes, there are many features that Python 2 is lacking relative to the
Python 3 series. That's what "maintenance mode" means. It's incredibly
frustrating when you hit one of them (for myself, I feel the pain
every time an error in an exception handler conceals the original
exception).

The available solutions are:
1. Use a third party PyPI package which offers that feature (in this
case, the requirement seems to be to use PyOpenSSL)
2. Upgrade to Python 3
3. Fork Python 2 to create a Python 2.8 which adds new backported
features from the Python 3 series

That last option has indeed been discussed by a few people at various
times, but the first option generally ends up being the preferred
choice if the second isn't yet viable due to missing dependencies.

Regards,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


python at mrabarnett

Aug 12, 2012, 7:13 PM

Post #4 of 5 (374 views)
Permalink
Re: python 2.7 + https + urlopen = ? [In reply to]

On 13/08/2012 02:39, Python Urlopen wrote:
>
[snip]

> After a lot of pain, I got myself out of this trouble, and my code
> now works correctly on 2.7.x (thanks to Jean-Paul Calderone's
> pyopenssl). But do "you" think this is a "feature" and not a "bug"?
> -- And do you think debating on this, killing time on the debate, and
> letting all python 2.x users suffer sooner or later is right --.
> Something as basic as urlopen!
>
It doesn't sound like a bug to me, more a missing feature, added in a
later version of Python. That's what upgrading is all about.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Aug 13, 2012, 3:40 AM

Post #5 of 5 (370 views)
Permalink
Re: python 2.7 + https + urlopen = ? [In reply to]

> How would you feel, if you issued :
>
> import urllib
> urlopen("""https://server.domain.com""").read()
>
> and the command got you data from some other URL without telling
> you! You use firefox, and the site is different than the data you
> got! Same with chrome. Safari. Even IE !
> Cheated? (Well I was mad -- after IE worked).
[...]
> None of them worked! Wow. Then you wonder, whats going on. You poke
> one of the server administrator, and he sends you the logs, and you
> see the problem. The keyword being "SNI".

I believe there is a bug in the HTTP server; it doesn't conform to the
HTTP/1.1 protocol. Even without the client using SNI, you should still
get the right page, since the HTTP Host: header indicates the host you
are trying to contact at this point, not SNI. The SNI is only relevant
for the certificate that the server presents.

Regards,
Martin


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.