Gossamer Forum
Home : Products : Gossamer Links : Discussions :

dead link brainstorming

(Page 1 of 2)
> >
Quote Reply
dead link brainstorming
We (Pugdog, myself, and possibly a few others) had a discussion way back in the days of Links 1.1x about how to handle dead links. The gist was that we, as Links site owners, put too much work into collecting, organizing, and maintaining our precious links to simply delete them when they become inaccessible.

My solution was to create a 'Other/Dead Links' category and a custom error code that I use for setting aside such links with the hope that I'll eventually find an updated URL or it will become active again (happens maybe 10% of the time, which is better than nothing). Works pretty well, although a number of users still seem confused by the seemingly intuitive category title and still report those links as dead... It also doesn't preclude them from showing up in the Random link option, if used, although I imagine that could be gotten around by tinkering with the database query used there.

I haven't used Pugdog's Bad Link plugin, but I believe it handles things slightly differently by creating a second table that those links can be stored in as a holding bin of sorts. That's probably a cleaner method, although it doesn't allow users to help you update your dead links. They can't report what they can't see... I actually do get the occasional report of that sort.

I haven't switched over to v2.1.1 yet -- still working out a few issues with my test installation -- but I'll be going with the PHP front end when I do. Interfaces with the rest of my site much easier, although I'm not 100% sold on the idea of going purely dynamic. Seems to slow things down a bit, but it's probably less resource intensive overall (not building hundreds of low-traffic pages that rarely get viewed). Anyway, I digress...

A thought has been bouncing around in the back of my mind and is starting to take form. Being a dynamic setup (could probably be approached the same for page.cgi), I'm thinking something could be done similar to my current Dead Links category, but having it be a virtual category that mirrors all the other categories. That way, you wouldn't have to manually move links in and out of their regular category and the Dead Links category, and you wouldn't have to create all the sub-categories within Dead Links for keeping things organized. Simply set a flag to indicate the link is dead, and it would show up when browsing the virtual category structure. It would also be precluded from being shown in the regular directory when that flag is turned on.

Furthermore, a link could be added at the bottom of each category to send users to the corresponding virtual category. Why? Well, take marathons (I have a running directory), for example. Someone wants to find a marathon in their area but doesn't see the one they're looking for (because it's dead). Do you want them thinking you've done a poor job with your directory because you overlooked such an important event? Of course not. And I doubt even 1% of users would think to navigate on over to the Dead Links category to see if it happens to be lumped in there... Maybe if they search for it, but there's no guarantee they'll find it that way. But with the virtual category link, something like, "View Dead Links in this Category," they can see that you're really on the ball and it's the other site that's at fault...

I think the major modification this would require is to alter the category page queries to look for the 'dead' flag. That and building the virtual categories somehow. Any thoughts?

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Nice idea. Sounds pretty easy to implement tool. Just add the dead link column as you suggested, and then perhaps through the use of conditionals in tags in your templates, you could handle the problem, at least in part.


http://www.iuni.com/...tware/web/index.html
Links Plugins
Quote Reply
Re: [Ian] dead link brainstorming In reply to
>>
Nice idea. Sounds pretty easy to implement tool.
<<

lol you've got tools on the brain ;)

Last edited by:

Paul: Jul 13, 2002, 2:30 PM
Quote Reply
Re: [Ian] dead link brainstorming In reply to
Man, I go days without a reply in the Links SQL PHP forum, and you guys are all over this one within minutes!

Glad you like the idea. I agree it doesn't sound too tough on paper, but I still haven't familiarized myself with the code well enough to say that with much certainty...

One tricky thing would be to make the virtual categories navigable. Linking directly to them from their corresponding real categories should be easy, but if they mirror the entire category structure, it makes sense to be able to click from one virtual category to the next.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Mirroring could be a big use of SQL space in some cases! I would imagine.

Paul: ooops, yep.. tool (and pizza) on the brain.


http://www.iuni.com/...tware/web/index.html
Links Plugins
Quote Reply
Re: [Ian] dead link brainstorming In reply to
Quote:
Mirroring could be a big use of SQL space in some cases!
How so? There wouldn't be any added categories, just the "mirrored" set would have some signal passed in to indicate its the virtual version of the category and would only show those links flagged as dead (with a link back to the actual category). Other than checking the dead flag in the regular category query, I don't see where there would be any added overhead.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Sorry, I must have misunderstood your idea.


Edit:

Quote:


One tricky thing would be to make the virtual categories navigable
Thats where the additional table may come in handy. But yes, I agree, one would have to sit down and plan this one out...


http://www.iuni.com/...tware/web/index.html
Links Plugins

Last edited by:

Ian: Jul 13, 2002, 2:58 PM
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Some suggestions:

1) Rather than creating another category (cluttering the directory structure, IMO), I would UPDATE the Status field to whatever status is found via LWP:UserAgent.

Then you could create a template that displays all "bad" links by their error message (403, 500, 302, etc.).

2) Set up a nightly email process that would email link owners whose link "status" is NOT 200 to let them know that the link is bad. Set up another column called "contacted" (SMALLINT, NOT Null, Default = 0). Then increment the "contacted" column by the number of times they've been contacted. Set up a global "counter" setting that will check against the number of contacts, and if the status is still NOT 200 and they have been contacted the number of times in the global "counter", then the link is automatically deleted.

3) Use LWP:UserAgent within the JUMP.CGI script and if the link is NOT 200, then set-up an email routine to email the administrator to let them know the LINKID, ERROR, and even the REMOTE ADDRESS of the user who has clicked on a bad link to avoid any SPAMMING.

Now, I think some of these features are already in the BAD LINK MOD that Pugdog wrote, but some are definitely not.

I, personally, use #3 as the least intensive option since the only thing that happens is that an error message appears if the link is bad on the web page and an email is sent to me. Not totally automated, but since I run links verification on a weekly basis, it really is not necessary for me to UPDATE the status column in the LINKS table, too much CPU, IMO. But for those with dedicated servers, a live UPDATE to LINKS status would not be very intensive.
========================================
Buh Bye!

Cheers,
Me
Quote Reply
Re: [Stealth] dead link brainstorming In reply to
Hmm...

Quote:
1) Rather than creating another category
I think you misunderstood me. The basis of my idea is to avoid adding categories, not to add more of them. The virtual categories I'm talking about would be the exact same ones as the regular directory structure (same ID and everything), just accessed in dynamic fashion to show only the dead links.

Quote:
2) Set up a nightly email process that would email link owners whose link "status" is NOT 200
Yikes, not on a shared server! That would chew up an insane amount of resources for a process of arguably minimal worth. Besides, the vast majority of links in my directory were added by myself and do not have an owner.

3) I've already got a bad link reporting system, so I'm not sure the time spent programming that in would accomplish much I don't already have...

I'm curious, did you take my invitation to comment on my suggestions as meaning ignore what I was talking about and propose alternatives that accomplish a completely different set of goals? Crazy

Thanks for the input, though...

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Uh...Dan, I was trying to add to the discussion, and the goals of my statements were in-sync with what you were asking.

And "brainstorming" is not a linear process, DAN! Tongue. Just trying to help...but since you already have a plan, go with it, create a plugin and move on to the PLUGIN forum.
========================================
Buh Bye!

Cheers,
Me
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
I've given this a lot of thought over time, for exactly the reasons you give. The links are really important, and sometimes they will come back, and sometimes they are gone forever, but to simply lose them, is a wasted resource.

The way links handles things, the best way to get rid of them, is if they show up dead for a week or so, to physically move them to a holding table. Preserve the category relations (copy the cat_links to a hold_cat_links table -- remember, these ID's are unique in links, they are not reused, unless manually overridden).

Use a simple script to flick through the links, and manually check them, and if they are live again, move them back.

You could use a flag field, but that would mean updating any of the routines that access the database, to recognize that flag field.

Alex has not yet implemented a dynamic flag system, for control of links display, so to do that, you'd have to modify anywhere a database call is made, to include a "and not isDead" clause.

This gets tricky, especially as more and more plugins make use of the database, and do things on their own.

The only way to really control this, is to install the dynamic flag system, and make it a part of the Links SQL calling engine.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] dead link brainstorming In reply to
Quote:
This gets tricky, especially as more and more plugins make use of the database, and do things on their own.
Of course, my going with the PHP front end precludes me from the luxury of plugins or having to worry about coexisting with them for the time being... :)

I take it you don't make your dead links holding section visible to users in any manner, as in to allow them to report updates to you? That's the major disadvantage I see to the system you describe, although it may be a minor drawback when compared to the modifications that would have to be made and maintained over time to the various db queries for the method I described...

Has there been any serious talk of implementing a dynamic flag system?

Stealth, not to beat a dead horse, but I see nothing in your post that added to the discussion rather than steer it elsewhere. As for "create a plugin and move on to the PLUGIN forum," I would have to point out that I'm talking about the PHP front end which does not use plugins, so that still isn't exactly a relevant or helpful suggestion. :(

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Well, I guess we will have to agree to disagree. Again, I was trying to provide helpful suggestions.

And since you are developing something for PHP, then post in that forum, why don't ya? Since that forum is specifically for PHP.
========================================
Buh Bye!

Cheers,
Me
Quote Reply
Re: [Stealth] dead link brainstorming In reply to
Quote:
And since you are developing something for PHP, then post in that forum, why don't ya?
I also pointed out that the same could probably be done for page.cgi, so aside from the plugin comments, there's nothing overly PHP-specific about the topic.

The last fairly general question I asked in the PHP forum drew little interest there and the moderator said I should probably post it in this general forum, which I haven't done yet...

On a more productive note, I've started playing around with this and it is working pretty well thus far. I've added a new field to the Links database, modified the queries in generate_category_page() to grab either active or inactive links, and added to the category template to show a link to either the regular or virtual category, depending on which is currently selected. The same $cat_id variable is used, which makes things easy. I'm just using an extra 'rip=yes' variable in the query string to signify the dead links option.

The area I haven't tackled yet is making the virtual categories browsable (including page spanning). That could be a bit trickier, since functions like _title_linked() are set up a bit different with respect to globals. I don't see any reason why $IN can't be added to the globals to pick up $rip, though...

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Dang, that was easy. Almost too easy... Adding $IN to the globals for _print_cat() and _title_linked() and conditionally tacking '&rip=yes' onto $category_url and $output, respectively, seems to have done the trick. Virtual categories appear to now be fully browsable.

Can anyone think of other places that would need to be modified? I probably won't be using the Random Link option anymore, so the jump routines don't need to be touched. I'm undecided if I want to exclude dead links from searches, but that's a possible area to look into.

The obvious next question is if people are interested in seeing the specific changes I made?

If you want to see it in action, here's my test installation with a sample virtual category (the only one currently set up with any dead links moved into it):

http://run-down.com/...t_id=283&rip=yes

Keep in mind that's a temporary/test installation and may not be functioning or at that location depending on what I'm doing when you happen to visit it...

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
I'd like to see what you did.

I've been spending the past 2 days (nights) working on rewriting my plugin install system.

People (authors?) forget all the potential problems that can happen, and the plugin error reporting system is at best obscure -- at worst, not really there.

Any one ever notice how when uploading a new version of a plug in file, the automatic perl_check doesn't match (often) the clicked link perl_check?? Line numbers and some other odd info are different.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] dead link brainstorming In reply to
   

Quote:
Any one ever notice how when uploading a new version of a plug in file, the automatic perl_check doesn't match (often) the clicked link perl_check?? Line numbers and some other odd info are different.


Yes. But it has been pretty close.... I find the fact the the inbuilt editor does not have line number the biggest problem... I have to keep two versions, and cut and paste each time I am hunting for an error after a perl check.

My apologies for diverting from the thread topic Dan.


http://www.iuni.com/...tware/web/index.html
Links Plugins
Quote Reply
Re: [pugdog] dead link brainstorming In reply to
>>
Any one ever notice how when uploading a new version of a plug in file, the automatic perl_check doesn't match (often) the clicked link perl_check?? Line numbers and some other odd info are different.
<<

Im not sure about 2.1.1 but in 2.1.0 if you try to add a new file to your plugin by entering code into the textarea rather than uploading and then go to the plugin editor and view the file, the code doesn't take and your file is empty.
Quote Reply
Re: [pugdog] dead link brainstorming In reply to
I'd love to have something like this be a plugin, but fortunately, there
aren't many changes that had to be made, so future upgrades
shouldn't be too unwieldy.

Ok, here's what I've done thus far. Keep in mind it's for the PHP
front end, and I haven't compared page.php to page.cgi to see
how similar the functions are.

Step 1) Added a 'DeadLink' column to the Links table:

Type - INT
Nut Null - Yes
Default - 0
Form Type - SELECT
Form Size - 1
Form Names - 1/0 (separate lines)
Form Values - Yes/No (separate lines)
Form Regex - 1|0

Step 2) in function generate_category_page() in Page.inc.php,
changed the 'get the links' queries from:

Code:
$sth = $DB->query("SELECT COUNT(*) FROM ${PREFIX}CatLinks, ${PREFIX}Links WHERE ${PREFIX}CatLinks.LinkID = ${PREFIX}Links.ID AND ${PREFIX}Links.isValidated = 'Yes' AND ${PREFIX}CatLinks.CategoryID = '" . $category['ID'] . "'");

to:

Code:
if ($IN['rip'] == "yes") {
$vcat = 1; // inactive links (virtual category)
} else {
$vcat = 0; // active links
}
$sth = $DB->query("SELECT COUNT(*) FROM ${PREFIX}CatLinks, ${PREFIX}Links WHERE ${PREFIX}CatLinks.LinkID = ${PREFIX}Links.ID AND ${PREFIX}Links.isValidated = 'Yes' AND ${PREFIX}Links.DeadLink = '$vcat' AND ${PREFIX}CatLinks.CategoryID = '" . $category['ID'] . "'");

and right after it, from:

Code:
$query = "SELECT ${PREFIX}Links.*
FROM ${PREFIX}CatLinks, ${PREFIX}Links
WHERE ${PREFIX}CatLinks.LinkID = ${PREFIX}Links.ID
AND ${PREFIX}Links.isValidated = 'Yes'
AND ${PREFIX}CatLinks.CategoryID = '" . $category['ID'] . "'";
if (isset($CFG['build_sort_order_category'])) {
$query .= ' ORDER BY ' . $CFG['build_sort_order_category'];
}

to:

Code:
$query = "SELECT ${PREFIX}Links.*
FROM ${PREFIX}CatLinks, ${PREFIX}Links
WHERE ${PREFIX}CatLinks.LinkID = ${PREFIX}Links.ID
AND ${PREFIX}Links.isValidated = 'Yes'
AND ${PREFIX}Links.DeadLink = '$vcat'
AND ${PREFIX}CatLinks.CategoryID = '" . $category['ID'] . "'";
if (isset($CFG['build_sort_order_category'])) {
$query .= ' ORDER BY ' . $CFG['build_sort_order_category'];
}

Step 3) In Utils.inc.php, function _print_cat(), added $IN to the globals and changed the $category_url setting in the foreach() loop from:

Code:
$CFG['db_php_url'] . '/index.php?cat_id=' . $cat_r['ID'] . hidden_query();

to:

Code:
$category_url = $CFG['db_php_url'] . '/index.php?cat_id=' . $cat_r['ID'];
if ($IN['rip'] == "yes") {
$category_url .= '&rip=yes';
}
$category_url .= hidden_query();

(You might notice I have index.php in there instead of page.php. That
is something I talked to Adrian about changing the hard coded page
name default, so it will probably be a $php_self or config variable
in its place in future versions.)

Step 4) Also in Utils.inc.php, added $IN to the globals for
_title_linked() and changed $output in the for() loop of the
'Create the linked title' section from:

Code:
$output .= '<a href="' . $CFG['db_php_url'] . '/index.php?cat_id=' . $categories[$i]['ID'] . hidden_query() . '">' . $categories[$i]['Name'] . '</a> : ';

to:

Code:
$output .= '<a href="' . $CFG['db_php_url'] . '/index.php?cat_id=' . $categories[$i]['ID'];
if ($IN['rip'] == "yes") {
$output .= '&rip=yes';
}
$output .= hidden_query() . '">' . $categories[$i]['Name'] . '</a> > ';

(Also note that I have > instead of : as a separator in there.
Personal preference. I had hoped that would get put into the
language globals one of these days, but it's still a manually hack...)

I believe all that leaves is page spanning, which I'll walk through
here as I try it out myself...

Step 5) Back to Page.inc.php's generate_category_page(),
change $url in the page spanning section (bottom of the function) from:

Code:
$url = $CFG['db_php_url'] . '/index.php?cat_id=' . $cat_id . hidden_query();

to:

Code:
$url = $CFG['db_php_url'] . '/index.php?cat_id=' . $cat_id;
if ($IN['rip'] == "yes") {
$url .= '&rip=yes';
}
$url .= hidden_query();

Excellent, that worked as expected. :)

I think that's everything so far.

Quote:
I've been spending the past 2 days (nights) working on rewriting my plugin install system.
That sounds quite tedious and frustrating. I haven't read details of your
system crash that resulted in lots of work -- just references to the
recovery process -- but it sounds like a lot of valuable time was
washed down the drain.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Oops, I forgot to add the template changes... In category.html, I made two additions:

1) Above the 'links in this category' section (placement entirely optional):

Code:
<!-- Dead Link Virtual Categories -->
<?if ($rip == "yes") {?>
<p><b>Note:</b> The links in this category are known to be inactive for a variety of reasons. If you know of a new location or status update for any of them, please use the <b>Report Dead Link</b> option to inform us. Thanks!</p>
<?}?>

2) Bottom of the page (again, optional):

Code:
<!-- Dead Link Virtual Categories -->
<p align=center><b>
<?if ($rip == "yes") {?>
<a href="<?echo $CFG['db_php_url']?>/index.php?cat_id=<?echo $cat_id?>">View Active Links in this Category</a>
<? } else { ?>
<a href="<?echo $CFG['db_php_url']?>/index.php?cat_id=<?echo $cat_id?>&rip=yes">View Dead Links in this Category</a>
<?}?>
</b></p>

I believe I had previously assigned $category_id to $cat_id in category.html.
I don't remember if I had a specific reason for doing that (I use the cat# in
my footer include for linking to the add/modify forms) or if I just
wanted it to be consistent throughout the program, but it's reflected in the
above code at any rate.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
A possible extension of this hack, although I think it would be far more trouble than it's worth, is to get the actual/virtual link counts for each category (and subcategories) reflected ($Number_of_Links; haven't found where that gets assigned yet).

For example, my current setup shows all links in a category in the count, which might be seen as misleading or confusing if some of them are in the virtual category. If the count for that category showed just the active links (or just the inactive ones if browsing the virtual categories), that would be more intuitive.

Similarly, the link in category.html to jump between actual and virtual categories could have an extra conditional layer that only shows the virtual link if there are dead links in the category. It could also show the number of such links, sort of like the links to the Reviews pages.

I suspect a number of extra database queries would be required to pull it off, though.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
Quote:
($Number_of_Links; haven't found where that gets assigned yet).
Seems whenever I get temporarily stumped on where a variable comes from, it's a database entry... Number_of_Links is a Category table entry for each cat/subcat, so that would be even more difficult than I first thought to reflect the actual/virtual links dynamically for each category. Doesn't sound worth bothering with.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
What about searches?

I think I may have lost the current thread here, as to what is going on, exactly.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] dead link brainstorming In reply to
Quote:
I think I may have lost the current thread here
Which thread do you mean, this one?

For the time being at least, I'm not doing anything with searches. I figure the dead links could be excluded from searches by requiring DeadLink to equaly '0', but I'm not completely sure I want to do that. Search results show the category is in -- uhh, wait a minute... It just occurred to me that I will no longer have dead links in a separate category, so that will be quite unclear in search results...

I was also thinking showing the dead links in the search results is in the same vein as showing them in virtual categories -- it lets people know the link is there, just not active.

Looks like I'll need to either exclude them from search results or add some notation next to them in the results letting people know what their status is.

Dan
Quote Reply
Re: [Dan Kaplan] dead link brainstorming In reply to
This will probably require more in-depth testing, but a small addition to Search.inc.php seems to take care of excluding those dead links from search results (leaving them in and making a notation next to them looks to be *much* more complicated).

In _query_ni(), right after the '# Now search the links table' section that assembles the WHERE clause of the database query, add:

$query .= " AND DeadLink = '0'";

Dan
> >