Gossamer Forum
Home : General : Perl Programming :

download authorized pages via LWP

Quote Reply
download authorized pages via LWP
Hi everyone!
I've developed a little perl program using the module LWP to download italian stock pages,
to make real time statistics.
The program works for delayed data, but it doesn't work for real time data because of the
authentication.
I mean the authentication is made at session level, so I need to authenticate in some
way in the same program.
I've tried to use $ua->credentials and $req->authorization_basic, but without results.
I confess I'm novice in perl programming and in using LWP, and surely it's my fault.
Can anyone help me?
My simple code is:

$protected_page="http://www.24oreborsaonline.ilsole24ore.com/fc?sezId=10026&cmd=miafinanza&page=ListinoPersonale";
use LWP::UserAgent;
$ua = LWP::UserAgent->new;

#
# Authentication block ???
#

$req = HTTP::Request->new(GET => "<$protected_page>");
$res = $ua->request($req);
if ($res->is_success)
{
$out="c:/mypage.txt";
open OUT, ">$out" or die "Cannot open $out for write :$!";
print OUT $res->content;
close OUT;
}
else
{
print "Time out: fault in download \n";
}


If you would like to test if it works I can leave the logging credentials.
Site logging page: http://www.24oreborsaonline.ilsole24ore.com/...md=index&chId=43
username: yer
password: man
I thank you in advance for whatever help you can give me.
Quote Reply
Re: [yerman] download authorized pages via LWP In reply to
Hi,

You *may* be able to do;

$protected_page="http://user:password@www.24oreborsaonline.ilsole24ore.com/fc?

Hope that helps.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] download authorized pages via LWP In reply to
Dear Andy,
I already tried your suggestion.
Unfortunately it doesn't work (as you can see using user=yer and password=man)

I thank you for your help, if you have any other suggestions.
Quote Reply
Re: [yerman] download authorized pages via LWP In reply to
Mmm.. is there any reason your not using LWP::Simple?

Code:
use LWP::Simple;
my $url = qq|http://user:password@24oreborsaonline.ilsole24ore.com/fc?sezId=10026&cmd=miafinanza&page=ListinoPersonale|;
my $page = get($page)

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [yerman] download authorized pages via LWP In reply to
Andy:
The authentication scheme is form-based.

Yerman:
1. You need to use POST instead of GET.
2. I believe the URL (that I constructed from <FORM>...</FORM> on remote page) is something like:
http://du.ilsole24ore.com/...mp;chId=43&SC=PF

Not tested, but should point you in the right direction.

----
Cheers,

Dan
Founder and CEO

LionsGate Creative
GoodPassRobot
Magelln
Quote Reply
Re: [dan] download authorized pages via LWP In reply to
Andy:
The authentication scheme is form-based.

Yerman:
1. You need to use POST instead of GET.
2. I believe the URL (that I constructed from <FORM>...</FORM> on remote page) is something like:
http://du.ilsole24ore.com/...mp;chId=43&SC=PF

Not tested, but should point you in the right direction.

Andy:
Thank you again, but $page remains empty.

Dan:
I realize I should post, and I see the interactive login page is
http://du.ilsole24ore.com/du/login.aspx?

But I don't understand the meaning of your link (naturally it's empty)
Where should I put user and password?
Probably my questions are almost naive, but I'm absolutely novice in perl, and in general programming is not comfortable for me, being mainly involved in statistics and economics.
Do you think it would be hard for you dropping few lines of code for my case?
I would be ethernally grateful!
My protected url is http://www.24oreborsaonline.ilsole24ore.com/...age=ListinoPersonale
My user=yer my password=man.

In any case I thank you for your kind contribution.