Hello I would like to know where I can get a good script for harvesting news from other sites. I checked the search forum but no such luck on finding a good script for harvesting news from other sites. If anyone knows of any please let me know. Thanks
Dec 26, 1999, 9:56 PM
Veteran (1311 posts)
Dec 26, 1999, 9:56 PM
Post #2 of 18
Views: 10304
yeah, i was thinking about doing that with Yahoo, and I don't just want to have the headlines for stocks and no search, so i'll look at different sites searches and pick which one i want to do. Now i have to do something but later. What is the address you put the stockav file at?
[This message has been edited by Bmxer (edited December 27, 1999).]
[This message has been edited by Bmxer (edited December 27, 1999).]
Dec 27, 1999, 4:17 AM
Veteran (1311 posts)
Dec 27, 1999, 4:17 AM
Post #3 of 18
Views: 10317
Check out
http://www.lifelinenews.net
I have them on my site below. Its only the news thing, i did the stock thing myself.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
http://www.lifelinenews.net
I have them on my site below. Its only the news thing, i did the stock thing myself.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Dec 27, 1999, 5:10 AM
Veteran (1311 posts)
Dec 27, 1999, 5:10 AM
Post #5 of 18
Views: 10329
The Stock headline thing i use is www.imchat.com Imchats TopNews Script.
I've only modified so i can get stocks from Yahoo, Altavista, CnetInvestor, Hotbot, and I'm working on Excite. Its not hard to modify, you just change the url and change the html to parse so if you have a different sites stuff you wanna get, thats what you'd do. Here is the one i have on my site with like 8 stocks or something
http://lookhard.hypermart.net/stockav.txt
Change .txt to cgi
Oh cool, i just got this thing to work, called the wealth meter, check it out
http://lookhard.hypermart.net/...ocks/wealthmeter.cgi
It takes the wealth meter from here
http://investor.cnet.com/
and lets me call it with ssi
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
I've only modified so i can get stocks from Yahoo, Altavista, CnetInvestor, Hotbot, and I'm working on Excite. Its not hard to modify, you just change the url and change the html to parse so if you have a different sites stuff you wanna get, thats what you'd do. Here is the one i have on my site with like 8 stocks or something
http://lookhard.hypermart.net/stockav.txt
Change .txt to cgi
Oh cool, i just got this thing to work, called the wealth meter, check it out
http://lookhard.hypermart.net/...ocks/wealthmeter.cgi
It takes the wealth meter from here
http://investor.cnet.com/
and lets me call it with ssi
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Dec 27, 1999, 2:41 PM
Veteran (1311 posts)
Dec 27, 1999, 2:41 PM
Post #9 of 18
Views: 10321
Cool, i was a little too late reading this, I already modified that Altavista.cgi script to get the results. It works but since they have their urls as / instead of their full url, i'm trying to change it to their address.
From stockmaster
do you want the quotes on the frontpage, or just the search results?
[This message has been edited by Bmxer (edited December 27, 1999).]
From stockmaster
do you want the quotes on the frontpage, or just the search results?
[This message has been edited by Bmxer (edited December 27, 1999).]
Dec 27, 1999, 3:12 PM
Veteran (1311 posts)
Dec 27, 1999, 3:12 PM
Post #11 of 18
Views: 10336
Ok, i won't get to it for a while, at least an hour. But i finished the Cnet one, its very low on design but that doesn't matter now. Its at lookhard.hypermart.net/look-bin/Look/stocks/stockfind.cgi
------------------
LookHard Search
lookhard.hypermart.net
Lavon Russell
[This message has been edited by Bmxer (edited December 29, 1999).]
------------------
LookHard Search
lookhard.hypermart.net
Lavon Russell
[This message has been edited by Bmxer (edited December 29, 1999).]
Dec 27, 1999, 4:32 PM
Veteran (1311 posts)
Dec 27, 1999, 4:32 PM
Post #13 of 18
Views: 10338
um, you just look at the html code, if something is bolded like
Copy this text
If it looks like that on the page
then you would put
Copy this <b>text</b>
All it's doing is getting the page with a variable like
$results
Then it strips down stuff in the variable $results with this s/HTML here//;
// means it's gonna change that code into nothing.
with something like this
$results =~ s/.*HTML here//; That means anything from HTML here and before will be deleted. And this gets rid of anything after it
$results =~ s/HTML here+.*//;
and this gets rid of stuff in between things
like say you had HTML is here
you can just go
$results =~ s/HTML(.+?)here//;
that gets rid of HTML, is, and here.
Just examine the first script i put up. You learn alot about perl by just looking at scripts. When i started with Links in July, I knew no perl or LWP, but about a month later i made the LWP URL Checker mod.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Copy this text
If it looks like that on the page
then you would put
Copy this <b>text</b>
All it's doing is getting the page with a variable like
$results
Then it strips down stuff in the variable $results with this s/HTML here//;
// means it's gonna change that code into nothing.
with something like this
$results =~ s/.*HTML here//; That means anything from HTML here and before will be deleted. And this gets rid of anything after it
$results =~ s/HTML here+.*//;
and this gets rid of stuff in between things
like say you had HTML is here
you can just go
$results =~ s/HTML(.+?)here//;
that gets rid of HTML, is, and here.
Just examine the first script i put up. You learn alot about perl by just looking at scripts. When i started with Links in July, I knew no perl or LWP, but about a month later i made the LWP URL Checker mod.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Dec 27, 1999, 8:03 PM
Veteran (1311 posts)
Dec 27, 1999, 8:03 PM
Post #14 of 18
Views: 10335
I got my stock search script the way i want it. I'm deciding whether i wanna use templates to customize it or just edit in the cgi.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Dec 28, 1999, 5:15 AM
Novice (26 posts)
Dec 28, 1999, 5:15 AM
Post #15 of 18
Views: 10348
Getting back to your stockav.cgi script where it says
$news =~ s/.*Top Tech//s;
$news =~ s/Quotes+.*//s;
$news =~ s/<!-- begin dispatch -->+.*//s;
$news =~ s/<br>//sg;
$news =~ s/<\/font><\/b><\/td>(.+?)<\/table>//sg;
so if I wanted to start pulling something from another page I would start with Top Tech and end with <\/font><\/b><\/td> ?
$news =~ s/.*Top Tech//s;
$news =~ s/Quotes+.*//s;
$news =~ s/<!-- begin dispatch -->+.*//s;
$news =~ s/<br>//sg;
$news =~ s/<\/font><\/b><\/td>(.+?)<\/table>//sg;
so if I wanted to start pulling something from another page I would start with Top Tech and end with <\/font><\/b><\/td> ?
Dec 28, 1999, 5:17 AM
Veteran (1311 posts)
Dec 28, 1999, 5:17 AM
Post #16 of 18
Views: 10303
Yeah, or any convenient email that comes after the section of html you want.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
Dec 28, 1999, 8:57 AM
Veteran (1311 posts)
Dec 28, 1999, 8:57 AM
Post #18 of 18
Views: 10335
Yeah, i always look at other scripts when i need to get things done before i try and make a script myself. And those (TopNews, and Altavista.cgi) are two good scripts.
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell
------------------
LookHard Search
http://lookhard.hypermart.net
Lavon Russell