Hello,

I'm running a TGP site, and what i need is a spider that
scans for pictures on the submitting page. (realtime)
So it should scan the submitted page for something like:
<a href="picture1.jpg"><img scr="picture1_thumb.jpg"></a>
And then add the total amount of pictures to the database.
Is there a mod out there that does this? Or something similar?

Thanks a bunch,
Andre

LWP::Simple

OR

Socket

BTW: This virtually has nothing directly to do with LINKS 2.0 at least by reading your post, so please post general questions like this in the Perl/CGI Forum.

Regards,

Eliot Lee

Well thanks for the pointers but i don't understand much
of them...
I'm running this thing on links 2.0, and i have the 'links
spider' mod installed. What i need is a mod that uses
the spider to scan for pictures, on the submit page of
links 2.0.
Hope that clears things up.. Any ideas?

Take care,
Dre

Here's a solution using LWP to spider for number of pictures, description, author and title before they add their site:

In add.cgi replace:

if (keys %in != 0) {
&process_form;
}

With:


if (keys %in != 0) {
if ($in{'spider'}){
&spider_it;
}
else {
&process_form;
}
}

Replace:


else {
&site_html_add_form ();
}

With:

else {
&spider_site;
}


At the bottom of add.cgi add:

sub spider_it {
use LWP::Simple;
my $url = get "$in{'url'}";
@content=split(/\n/,$url);
$count = 0;
foreach $line (@content) {
chomp($line);
($title = $1) if ($line =~ m|<TITLE>([^"]+)</TITLE>|i);
($description = $1) if ($line =~ m|<meta name="Description" content="([^"]+)">|i);
($name = $1) if ($line =~ m|<meta name="Author" content="([^"]+)">|i);
$count++ if ($line =~ m|<img src|i);
&site_html_add_form ();
}

sub spider_site {
print qq|
<form action="add.cgi" METHOD=POST>
<input type="text" name="url"><input type=submit value="Spider" name="spider">
</form>
|;
}


In site_html_templates.pl replace sub site html add form with:


sub site_html_add_form {
# --------------------------------------------------------
# This routine determines how the add form page will look like.
#
&html_print_headers;

my $category = shift;
$category ?
($category = qq~$category <input type=hidden name="Category" value="$category">~) :
($category = &build_select_field ("Category", "$in{'Category'}"));

print &load_template ('add.html', {
Category => $category,
description => $description,
title => $title,
name => $name,
count => $count,
%globals
});
}

Then in add.html add a hidden field with a value <%count%> (gives the number of pictures on the page spidered) and use value = <%title%> etc.. for the title, description and contact name fields if you want to use the stuff collected from the spidered page.


Glenn

Links 2 Mods Site:
http://cgi-resource.co.uk/pages/links2mods.shtml

Should this:

my $url = get "$in{'url'}";


...not be:

my $url = get($in{'url'});


Or do both ways work?


Paul
Installations:http://wiredon.net/gt/
Support: http://wiredon.net/forum/

Yeah well spoted. Both ways work but I guess it's better as:

my $url = get($in{'url'});



Glenn

Links 2 Mods Site:
http://cgi-resource.co.uk/pages/links2mods.shtml

Glenn;

I realize that this is an old post, but I was wondering if you could explain how to get this spider mod to work. I added the code to the Add.cgi and the code to the site_html_templates.pl. I also included :

<form action="Add.cgi" METHOD=POST>
<input type="text" name="url"><input type=submit value="Spider" name="spider">
</form>
And the hidden variables and values for name, title, description, keywords to my add.html page.

This is where the problem is. I don’t know exactly how this mod works, and with a bunch of mods installed (like password auto-filler mod), what my add.htm should have on it. If it has the url input box and the hidden fields and that’s all, how does it spider and send the info to the error page to finish filling out form?

When you call Add.cgi it brings up add.html. Then you are supposed to enter a url to spider right? Then the Add.cgi is supposed to fill in the hidden fields in the add.html and take you to the add_error.html with the title, name, description, keywords fields already filled in right? If it does not take you the error page how can a user finish filling out the form?

My Add.cgi has the password username mod installed except I’m using Contact Email for username. Anyway according to instructions I added the form stuff above and the proper hidden variable tags. I tried using just the
<input type="text" name="url"> plus hidden fields, but it does not spider anything and does not take you to the error page.

Any help would be appreciated.

I will upload the Add.cgi with the site_html_templates modified subs.

Thanks,

Jake