After installing both the altavista.cgi
and bulkload mod from the resouce center
I thought of maybe combining the 2.
Sorry if this gets a bit long winded ..
Hopefully some people will see where im going
My idea is to create a script that can be used to bulk-up your link database.
The script would have to ...
Create a page where a admin can choose which category to buildup, and whether to use
meta tags or link descriptions as keywords.
Once a admin chooses these items and submits
the page the script would search altavista
using the category title plus either that categories meta tags and or words grabbed
from the pre-existing links in that category.
What is returned from the search engine gets
placed into text fields on a results type page. This results page would allow
editing of what is returned.
Then the next step would be to have the cgi write that info to a file for bulkloading
into your link database. That is unless the data can be passed straight from the page
into the database without having to save it.
Ideally a check to see if the link is already in the database before adding it.
Now I won't comment on the legalities of
doing such a grab.
I have already altered the altavista.cgi
so that it returns the items in pipe
delimited form ready for bulkloading
but I still would have to enter the categories manually for it to be of any use.
I would prefer to use metacrawler as it
uses alot of search engines at once so
there would be no need to do a version for yahoo and webcralwer etc etc
So anyone have any thoughts on my idea ?
Summary of idea:
* Script to bulk up your own link db.
* Use categories and meta tags to figure
out what to search for.
* Return the info in a editable form so
bad links and descriptions can be removed or fixed.
* Check for duplicate links
* Bulkload the results into your link.db
* Avoid being sued
and bulkload mod from the resouce center
I thought of maybe combining the 2.
Sorry if this gets a bit long winded ..
Hopefully some people will see where im going
My idea is to create a script that can be used to bulk-up your link database.
The script would have to ...
Create a page where a admin can choose which category to buildup, and whether to use
meta tags or link descriptions as keywords.
Once a admin chooses these items and submits
the page the script would search altavista
using the category title plus either that categories meta tags and or words grabbed
from the pre-existing links in that category.
What is returned from the search engine gets
placed into text fields on a results type page. This results page would allow
editing of what is returned.
Then the next step would be to have the cgi write that info to a file for bulkloading
into your link database. That is unless the data can be passed straight from the page
into the database without having to save it.
Ideally a check to see if the link is already in the database before adding it.
Now I won't comment on the legalities of
doing such a grab.
I have already altered the altavista.cgi
so that it returns the items in pipe
delimited form ready for bulkloading
but I still would have to enter the categories manually for it to be of any use.
I would prefer to use metacrawler as it
uses alot of search engines at once so
there would be no need to do a version for yahoo and webcralwer etc etc
So anyone have any thoughts on my idea ?
Summary of idea:
* Script to bulk up your own link db.
* Use categories and meta tags to figure
out what to search for.
* Return the info in a editable form so
bad links and descriptions can be removed or fixed.
* Check for duplicate links
* Bulkload the results into your link.db
* Avoid being sued
