Gossamer Forum
(Advanced Search)
for
Home : General : Perl Programming :

Using & in a cookie value

May 6, 2002, 7:42 PM
AlexJ
User (113 posts)
May 6, 2002, 7:42 PM
Post #1 of 12
Views: 8157
Shortcut
Quote Reply
Using & in a cookie value
Hi,

I am working on a program that has to emulate a browser, and log into the My UPS website. Everything was going fairly smoothly until one of the cookies it sent, contained an "&".

Specifically they send:

Code:
session_ups_com=1020737632&mycompany&0&GMi+QdG9tEbLeLGzyfBNfTTzji7x80rp;domain=.ups.com;path=/
To send it back with my next page request, I parsed out the actual cookie value and did this:
Code:
my $http = new GT::URI::HTTPS(
'request_method' => 'POST',
'headers' => { Referer => 'https://www.ups.com/servlet/login?returnto=https://www.ups.com/ups.app/iss.class',
},
'cookies' => { test_ups_com => 'init',
session_ups_com => '1020736548&mycompany&0&YK5eYeY3By+w3SFqYlqm9FdH8KEEqnaJ'
},
'parameters' => {},
'agent_name' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461)'
);
$http->fetch( $url );


And yes... as odd as it sounds... this is part of a Links SQL Plugin... so its using the GT library... So what happens here is it gets stuck... and it gives me a message saying I have to log in again.

To debug this... I did this post to my links install forcing an error, and it shows the second cookie as only being: 1020736548.... Cutting off at the &.

Obviously the & works when using the web browser.... however it seems that somewhere in one of the modules, the & cuts off the cookie before it gets sent out with the request headers.

Just in case... here is the version info on my URI::HTTPS module:
Code:
# ==================================================================
# Gossamer Threads Module Library
#
# GT::URI::HTTP
# Author : Aki Mimoto (support@gossamer-threads.com)
# CVS Info : 087,069,093,087,083
# $Id: HTTPS.pm,v 1.5 2001/02/11 01:11:24 aki Exp $
#
# Copyright (c) 2000 Gossamer Threads Inc. All Rights Reserved.
# ==================================================================
#
# Description:
# Gets HTTP data
#package GT::URI::HTTPS;
Does anyone know if this is a bug in the GT package.... or how I can get around this?



Best regards,

AlexJ
May 6, 2002, 9:55 PM
Aki
Staff / Moderator (429 posts)
May 6, 2002, 9:55 PM
Post #2 of 12
Views: 7999
Shortcut
Quote Reply
Re: [AlexJ] Using & in a cookie value In reply to
Hi Alex,

Can you do this for me?

If you look in the GT/URI/HTTPS.pm module, there should be a subroutine called "_create_request".

Can alter the end of the function so that it looks like this?

Code:

warn $get_str; # add this line
return $get_str;

}

If you're running apache, you should be able to see the raw http requests that are being sent to the remote server. Can you post up the results?

I'm not sure yet why the "&" would end up being the cut point, as I can't see anything in the code, but this should give some clues.

Cheers,
Aki
May 7, 2002, 7:04 AM
AlexJ
User (113 posts)
May 7, 2002, 7:04 AM
Post #3 of 12
Views: 7947
Shortcut
Quote Reply
Re: [Aki] Using & in a cookie value In reply to
Aki,

Thank you so much for the quick reply. I added the warning line in the "_create_request" routine as you requested (it was in the HTTP.pm module though).

What I discovered was interesting. The headers that were being sent were actually correct (however the cookie was URL encoded). Apparently the problem with the & in a cookie value is actually a problem with LSQL receiving cookies, not with the GT library sending them.

It gets even more interesting.... I edited the "_create_cookie_request" routine to take out the url encoding of the cookie, and now ups.com recognises the cookie with the & in it and lets me continue on.

I have not looked into the RFCs on this - but I am definately wondering - are cookies supposed to be url encoded? Because it definately seems like their server was not url-decoding them.

Thanks again for the help - please let me know if you find out what the official docs say on this...

Best regards,

AlexJ
May 7, 2002, 9:14 AM
Aki
Staff / Moderator (429 posts)
May 7, 2002, 9:14 AM
Post #4 of 12
Views: 7929
Shortcut
Quote Reply
Re: [AlexJ] Using & in a cookie value In reply to
Hi AlexJ,

It appears that the cookie spec is somewhat ambiguous about how to handle escaping. In all our libraries, we use the %xx to prevent special characters, but I guess it's a vendor dependant thing. I'll update the our local libraries to have it as a switch *ugh*

You're the first person to make use of that method in earnest - I wrote the method and even I haven't even used it yet ;)

Incidentally, what sort of interface are you writing for UPS? I'm in the process of putting together a shopping cart plugin for Links SQL and if it's anything to do with automatic shipping calculations, would be very interested in seeing what you've done.

Cheers,

Aki
May 7, 2002, 10:12 AM
AlexJ
User (113 posts)
May 7, 2002, 10:12 AM
Post #5 of 12
Views: 7976
Shortcut
Quote Reply
Re: [Aki] Using & in a cookie value In reply to
Aki,

I certainly didn't mean to create more work for you... sorry! Good job on the module though. I've got it working nicely for me doing a number of interesting things for our shopping cart plugin.

My current project involves creating a one click interface to printing UPS Online labels right from a view order screen. So basically when you are viewing an order, if it looks good, and you want to ship it, you just click on a link, and a pop-up window opens up with a UPS label ready to be printed... No filling anything out.. logging into their system..etc... We've already been able to get this working with FedEx Online, and the US Postal Service Delivery Confirmation service...

For automatic rates, address verification and delivery times... UPS has recently released a public XML interface... that should make it really easy to do.

As I'm replying to you, I find myself with one more (hopefully last) question. I've run into a page, when viewed by a browser, they set multiple cookies. I've gone through all the html on the page and see that none of the cookies are set through images or loaded js files.... However when I get the page with the GT library, these are the headers I get:
Code:
COOKIE = HASH(0x89a47dc)
ERROR_CODE = 200
ERROR_MESSAGE = OK
connection = close
content-length = 35609
content-type = text/html
date = Tue, 07 May 2002 16:38:54 GMT
server = Netscape-Enterprise/4.1
set-cookie = SA_Enc_iss.class=313577302
The set-cookie header is only showing one.....the "SA_Enc_iss.class" cookie and it happens to be the last of the 3 cookies that the page sets. What is the COOKIE = HASH header? Any ideas on where those other 2 cookies might be hanging out?

Best regards,

AlexJ
May 7, 2002, 11:15 AM
Aki
Staff / Moderator (429 posts)
May 7, 2002, 11:15 AM
Post #6 of 12
Views: 7923
Shortcut
Quote Reply
Re: [AlexJ] Using & in a cookie value In reply to
My goodness, you're making a complete system. I'm just looking at a mid-level solution.

But about the HTTPS, I think you've found a bug in the module Pirate

The module assumes that no header is doubled up and there's my mistake. Set-cookie headers can be numerous and the system is getting confused about it. Since it seems you're making a plugin, if you'd like, I can whip up a wrapper class (the only thing that will change is the name of the object you call) to repair the problem.

There is nothing wrong with the HASH, that just implies that you're working with a hash reference, so in the future you'd be able to do:

Code:


my $cookies = $https->resource_attrib()->{COOKIE};

use Data::Dumper;

print Dumper($cookies);


Aki
May 7, 2002, 11:31 AM
AlexJ
User (113 posts)
May 7, 2002, 11:31 AM
Post #7 of 12
Views: 7939
Shortcut
Quote Reply
Re: [Aki] Using & in a cookie value In reply to
Aki,

Yes this is a long term project, and it will be a complete system when its done. As soon as we get a little further along in it's development, we will be sharing a demo with everyone in the forums for comments/feedback/suggestions feature requests.... We have at least a few very long threads on our system in the LSQL Plugins Forum... But we've stopped discussing it in there, because everyone wants it 2 months ago and dev work always takes 4x as long as you plan... so everyone just gets frusterated, writes you off as vaporware... when in reality, its just the opposite, rock solid... because of the time we've taken writing and rewriting....testing.... then beta testing......

As far as the bug goes... Sorry! I hate to make more work for anyone... it seems I am the one around here with an endless pile.... and hate to contribute to anyone elses...

But if you could whip something together to deal with those multiple cookies, it would be great. I'm at a stand still on our UPS integration until we figure that one out....

Thanks in advance,

AlexJ
May 7, 2002, 12:11 PM
Aki
Staff / Moderator (429 posts)
May 7, 2002, 12:11 PM
Post #8 of 12
Views: 8000
Shortcut
Quote Reply
Re: [AlexJ] Using & in a cookie value In reply to
Hi AlexJ,

Good to hear :) And no worries about the work-load.

Can you send me your email address so I can send you the patch? Mine is aki@gossamer-threads.com

Cheers,

Aki
May 7, 2002, 5:09 PM
AlexJ
User (113 posts)
May 7, 2002, 5:09 PM
Post #9 of 12
Views: 7933
Shortcut
Quote Reply
Re: [Aki] Using & in a cookie value In reply to
alex@netmall.net

Thanks!
May 8, 2002, 1:31 AM
Wil
Veteran / Moderator (4108 posts)
May 8, 2002, 1:31 AM
Post #10 of 12
Views: 7886
Shortcut
Quote Reply
Re: [AlexJ] Using & in a cookie value In reply to
I think this is a good example of that 'private message' plugin that was disucssed earlier :-)

- wil
Jun 23, 2002, 10:41 PM
jykchang
New User (2 posts)
Jun 23, 2002, 10:41 PM
Post #11 of 12
Views: 7844
Shortcut
Quote Reply
Re: [Aki] Using & in a cookie value In reply to
In Reply To:
Hi Alex,

Can you do this for me?

If you look in the GT/URI/HTTPS.pm module, there should be a subroutine called "_create_request".

Can alter the end of the function so that it looks like this?

Code:

warn $get_str; # add this line
return $get_str;

}

If you're running apache, you should be able to see the raw http requests that are being sent to the remote server. Can you post up the results?

I'm not sure yet why the "&" would end up being the cut point, as I can't see anything in the code, but this should give some clues.

Cheers,
Aki
Jun 23, 2002, 11:57 PM
jykchang
New User (2 posts)
Jun 23, 2002, 11:57 PM
Post #12 of 12
Views: 7808
Shortcut
Quote Reply
Re: [jykchang] Using & in a cookie value In reply to
Hi Alex & Aki...

Sorry.. the message didn't manage to get thru..

Anyway I am currently working on a program which reads in multiple https pages and collect the cookies before submitting request to a login site. Mmm I am using the Crypt::SSLeay and together with LWP... The getting of pages is ok.. but once I try to submit a request to do a post - login.. it fail...

this is from the header log

Content-Type: text/html; charset=ISO-8859-1
Expires: Mon, 24 Jun 2002 06:29:35 GMT
Client-Date: Mon, 24 Jun 2002 06:30:39 GMT
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
Client-SSL-Cert-Subject: /C=AU/ST=Australia.......
Client-SSL-Cipher: RC4-MD5
Client-SSL-Warning: Peer certificate not verified...

Is there something wrong with this? and that is why I can't access the site??

Anyway can GT::URI::HTTPS solve the problem? MMmm I am trying to find the module in CPAN and it is no way to be found. Where can I find and download a the GT::URI module? Is it free for public usage?



Cheers

Jacky