Gossamer Forum
Home : Products : Links 2.0 : Customization :

Mod Idea - Auto Check for Duplicate URL

(Page 1 of 3)
> >
Quote Reply
Mod Idea - Auto Check for Duplicate URL
Hi,
I have been having a number of people submitting the same URL over and over to multiple categories (despite a "guidelines" page). I have Loony's multiple category mod installed, which makes it unnecessary to submit to multiple categories. However, I find it very inconvenient to have to click on "duplicate check" each time, and then have to delete the listing, which is not nice for those that submit either. An automatic duplicate check in add.cgi is my idea here (check in existing links and the validate database), to verify that the URL is not already listed, and giving an error message if it is. Any idea how this can be done? I'm not that familiar with Links at this time, even though I have written a few CGIs and have done a number of mods on my version of Links. But I'm sure this would be an interesting mod for many users, and I hope that someone who has more experience with Links can help with this. Thanks!
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
You would want to look at one of those modify.cgi security system things, but copy and paste it again to check the validate also.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
even if you managed to write such a script, it's not 100% proof. let's say i submit
htp://www.mysite.com
then i can submit http://mysite.com and you would not know it's a duplicate or
also some guys would try short url such as come.to fly.to and so on. i no longer accept such sites.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi Theguy,
Thanks for your input - well, it would then need a subroutine to check for http://www.site.com, http://site.com and http://(www.)site.com/index.htm(l) ... that part should not be so bad, and it should be possible to opt for disallowing known forwarders (I think I saw a mod for that somewhere). The main point is, it would help to reduce duplicate listings if they get stopped before the submission is even added to validate.db. Saves me from having to click on "duplicate check" for each new submission.
There will never be a 100% safe method, unless there is a way to compare the content of the page to be submitted with the contents of all pages listed in validate.db and in links.db - this would not be very practical unless it could be done in the background (via cron with notifcation to the webmaster via e-mail when duplicates are found). Not sure if this could be done and if it would be practical or too much load on the server.
As this would be much more difficult, I'd really like to see a simpler solution comparing the URL of the submission with the URLs in validate.db and links.db (with and without "www.") - I hope somebody can help me with this!
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi Bmxer,
I tried to search for that but came up with nothing that relates to my idea. If you can give me more details, I'd appreciate that. Thanks.
Greetings,
Gerrit
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
You would put something like this in
Code:
my ($key, $status, @values, $found);
local (%original);

# Make sure we have an ID, URL, and Email address for the resource to modify.
# If any are missing, return an error message to the submitter.
((!$in{'Current URL'}) or
(!$in{'LinkID'}) or
(!$in{'LinkContact'})) and &site_html_modify_failure (qq|<ul>
<li>Missing the ID, URL, or Contact Email address of the resource to modify
</ul>|) and return;

# Let's check to make sure the link we want to update is actually in the
# database and that the link ID and Contact Email address match what is
# specified in the submission form.
open (DB, "<$db_file_name") or
&cgierr("Error in validate_records. Unable to open $db_file_name. Reason: $!");
$found = 0;
LINE: while (<DB> ) {
(/^#/) and next LINE;
(/^\s*$/) and next LINE;
chomp;
@data = &split_decode($_);
# Check for matching record ID, URL, and Contact Email address.
# If everything matches, get the record, stop searching.
if (($data[0] eq $in{'LinkID'}) and
($data[$db_url] eq $in{'Current URL'}) and
($data[$db_contact_email] eq $in{'LinkContact'})) {
$in{$db_key} = $data[0];
$found = 1;
%original = &array_to_hash (0, @data);
last LINE;
}
}
close DB;
# If no matching record found, display error message to user
!$found and &site_html_modify_failure (qq|<ol>
<li>The requested Link URL was not found in the database.
<li>The requested Link URL does not match the database URL for the specified Link ID.
<li>The requested Link Contact Email address does not match the database Contact Email
address for the specified Link ID.
<li>The Link ID doesn't exist.
</ol>|) and return;

# Since we have a valid link, let's make sure the system fields are set to their
# proper values. We will simply copy over the original field values. This is to stop
# people from trying to modify system fields like number of hits, etc.
foreach $key (keys %add_system_fields) {
$in{$key} = $original{$key};
}
This is not my code.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi Bmxer,
Thanks, I'll try to adapt that to add.cgi and change it to kick out an error when the record IS found. Maybe I can get it to work.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Gerrit,
If you get it to work please post it here as I to am always getting multiple submissions in diffrent categories and it's a pain to manualy delete them.
thanks,
Lewis
www.atozfitness.com
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Awesome idea Gerrit.. Personally, im praying you pull it off.. Even if we are able to stop just a percentage of the cheaters, then its still worth it.. Its that much "less" work for us on a daily basis.

Please make sure to let us all know when its completed.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi lewisw and Conan,
Thank you for your support... I'm currently working on it.
First the good news: I have it so far to go that it checks for EXACTLY the same URL and kicks out an error if a duplicate is found (just as good as the duplicate checker included with Links) - however, I'm still working on the routine to fight the trick with including or not including www., a trailing slash, or index.htm(l) - i.e. it would be possible to submit
http://www.domain.com
http://www.domain.com/
http://www.domain.com/index.htm(l)http://domain.com
http://domain.com/
http://domain.com/index.htm(l)
I'll post the finished mod when I get done! Looks like this will be an easy one to install - only one set of code in one place!
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi everyone,
I think I may have the solution. I have tested this several times to the point where it appears to work just fine on my system. If you have any problems, let me know! This mod gives an error message if a duplicate URL is found in links.db or validate.db so we should, hopefully, no longer be flooded with duplicate listings. It not only ignores www., but also capitalization, and any index.htm or index.html, as well as trailing slash(/). More to ignore could be easily added. Here it is:

http://www.gossamer-threads.com/...rces/jump.cgi?ID=788

[to easier reflect changes and bug fixes, I now posted this on my site]

Let me know how it goes! If you have any suggestions or comments, please post them here!

[This message has been edited by Gerrit (edited November 24, 1999).]
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
let's say someone submit a site http://www.mysite.com under category a
then he comes back a minute later and submit the same site, does he get an error or not using your mod?
what i'm trying to say if the site is not validated, and he tries to submit it to adiffernt category, does he get an error or not?
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Gerrit!! "YOU ARE THE MAN"! The routine works great. We tried everything, and it stopped all.. Congrats...



By the way, since you seem to be pretty good at CGI, i need a alternative way of protecting my admin.cgi file, as htaccess does not seem to work on my server.. Any ideas?
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi TheGuy,
If someone just submitted
http://www.mysite.com
it gives an error if that person tries to submit any of these under any category:
http://www.mysite.com
http://www.mysite.com/
http://www.mysite.com/index.htm
http://www.mysite.com/index.html
http://mysite.com
http://mysite.com/
http://mysite.com/index.htm
http://mysite.com/index.html
That should cover just about all chances of duplicate URLs. What it will not do, is stop anyone adding something like
http://mysite.com/page1.html
but I don't think that should be necessary, at least I allow people to submit various sections of their sites to be submitted under various categories. I just don't want the same page to be listed more than once, and this should do it.
Greetings
Gerrit
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi Conan,
Thanks for the feedback! As far as I know, what would be needed is a password routine within the script. It won't be near as safe as .htaccess, but will reduce the chance of foul play.

I came up with something I pieced together to protect admin.cgi to a certain point - but be advised that this is a simple protection, and that I can see a way for those familiar with how Links works to circumvent this protection. I don't want to post the way to hack this here for obvious reasons.

http://204.180.41.204/links2mod2.htm

Let me know if this works for you.

Greetings
Gerrit

[This message has been edited by Gerrit (edited November 25, 1999).]
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Gerrit:

Great job, but something might be missing, or is it just an error from my side?

I've implemented the changes as you described. It works great for new submissions, but Links now returns "Duplicate URL error - already listed!" if I try to modify an existing link with the same URL as it had before.

If I change the link URL to a new, it's no problem modifying, but sometimes it might not be the URL that is changed, and then we're not able to update the link because the error message...

Any workaround for this?

Thanks!

Hugs
Nina

[This message has been edited by oktrg500 (edited November 24, 1999).]
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Thanks for pointing this out - I'll check and see if I have the same problem, and if so, I'll try to fix this and post the fix here.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Yihaa! 2 good news! :-)

1. It looks like the problem mentioned above wasn't an error from my side. First I thought that I had lost the things mentioned below during some copy/paste raids, but I checked the original package, and it looks like the same has to be added there too.

2. I think I have the solution.

I'm not using templates, and in my site_html.pl I had to do this:

In site_html_add_form
Find <input type="SUBMIT" value="Add Resource">
Change to <input type="SUBMIT" name="add_record" value="Add Resource">

In site_html_add_failure
Find <input type="SUBMIT" value="Add Resource">
Change to <input type="SUBMIT" name="add_record" value="Add Resource">

In site_html_modify_form
Find <input type="SUBMIT" value="Modify Resource">
Change to <input type="SUBMIT" name="modify_record" value="Modify Resource">

In site_html_modify_failure
Find <input type="SUBMIT" value="Modify Resource">
Change to <input type="SUBMIT" name="modify_record" value="Modify Resource">

I guess the same goes for the templates. Find the 4 add/failed, modify/failed templates and add name="add_record"/name="modify_record" to the submit button code.

Strange no one has mentioned this before, or had problems with it?

Anyway, great mod! Thanks Gerrit!
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
you are right, it's not your mistake, it's a bug. i only made changes to modify.html and modify_error.html and it worked fine for me.
i do not think you need to make changes to add.html and add_error.html. doing so may reverse what this mode does in the first place. check it out and let us know
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Great idea - however, since I (and maybe some others) have the password mod for modifying URLs, it didn't work for me - but with a few minor changes, it did:

Try this:

In site_html_add_form
Find
<input type="SUBMIT" value="Add Resource">

Change to
<input type="SUBMIT" name="add_record" value="Add Resource">

In site_html_add_failure
Find
<input type="SUBMIT" value="Add Resource">
Change to
<input type="SUBMIT" name="add_record" value="Add Resource">

and change in my mod (in site_html.pl) in the first line of the mod
unless ($rec{'modify_record'})
to
if ($rec{'add_record'})

That did the trick for me! Thanks for your help!


[This message has been edited by Gerrit (edited November 24, 1999).]
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Seems like a decent Mod, however, it does NOT work with the Add Confirmation Mod.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
----------------------


Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi Elliot,
Interesting - I wanted to add such confirmation, but since it was only for templates, what I found, I haven't taken the time to figure out how to adapt for non-templates. I'm sure there has to be a way to make it compatible - if someone could tell me how to add the confirmation (or where I can find a thread for it - search didn't help me) for non-templates, I can see about finding a way to make it compatible.
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
If you ever have the time to update the documentation for us template users I would love to install this mod Smile
Great work and thanks for the contribution.
Lewis
www.atozfitness.com
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
i really do not understand why you guys are making changes to site_html_add_form and
site_html_add_failure
you only need to make chnages to modify forms
Quote Reply
Re: Mod Idea - Auto Check for Duplicate URL In reply to
Hi TheGuy,
I know that there are two ways, and changing only the modify part would work if it wasn't for the password mod for modifying, which makes it more difficult. This seemed easier to me - if you have a better idea, let us know! Thanks!
Greetings
Gerrit

P.S.: An updated version with the changes marked in red for those that already use the mod has been posted here: http://www.gossamer-threads.com/...rces/jump.cgi?ID=788

[This message has been edited by Gerrit (edited November 25, 1999).]
> >