Hello all.

While nph-verify.cgi is still unable to deal with big link databases in browsers (lock up after 100-200 links checked) I offer a long, but working way to get a handy htm - formatted results using Telnet.

Here is complete instructions for amateurs like me:

1. Connect your server via Telnet (SecureCRT program recommended).
2. Go to your links directory:

cd cgi-bin/links/admin (in my case)

3. Start nph-verify script and tell it to store verifying results in txt file:

./nph-verify.cgi >deadlink.txt

4. Give the script a time to finish it's work (it may be a pretty long time). It's a perfect time to drink a couple of beer.

5. Launch your FTP program and enter links/admin directory. Download report file (deadlink.txt) to your computer in ASCII format.

6. Here is the best part of the trip: Open this file with Microsift Word and make a MACROS turning a plain text report to a formatted html one.

Here is how your pain text report string will look like:

1070 - http://www.some_of_checked.com/ : 500 - Internal Error

Here is how this string have to look like after applying Word macros:

<br>1070 - <a href=http://www.some_of_checked.com target=_blank> Check it </a> <font size=-1><b> [<a href=http://www.yoursite.com/cgi-bin/links/admin/admin.cgi?db=links&amp;modify_form=1&amp;ID=1070&amp;ww=1 target=_blank> modify </a>| <a href=http://www.yoursite.com/cgi-bin/links/admin/admin.cgi?db=links&amp;delete_form=1&amp;ID=1070&amp;ww=1 target=_blank> delete </a>]</b></font> : 500 - Internal Error

Apply this macros to all text strings and save this file as deadlink.htm

7. Launch your browser and open deadlink.htm.

You're done. Enjoy.

******************

It takes a while to write such a macros, but then you can easily turn boring text report to a nice html page with possibility of one-click checking/modifying/deleting dead links. Be extremely careful recording macros and double check every character you type.



Thank you.

PS. Please, don't ask me how to record Word macros. It's boring, but simpliest procedure ever. Just place your cursor at the beginning of plain report string and make all nesessary changes. To force ID number "variavle" copy the number while recording macros and then paste it after ID= in URL's . To the right navigation through plain string use Find and enter _:_ and _-_ where _ are spaces.
Remeber that error string lenght and URL lenght are vary in each report. So use only common elements, as I said above.

Sorry for such a lamer approach, but this is the only one working for now.

All professionals, please, do something with nph-verify.cgi.

Thank you.

Or i would imagine opening with wordpad or editplus and just saving it as .html would work.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Nope. This action will just save the same text with other extension. I'm talking about real html-formatting of the plain text + automatic adding (by macros) of special links to review/modify/delete link with the single click.

Kangaroo,
Oh ok, I understand now.
oh, btw... http://lh.links247.net/...admin/nph-verify.cgi
Splits the pages, and uses span. You can set maxhits per page to any number you want under 100. Any number over 100 will default to 100.
Bye.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Hi!

is it possible to download somwhere "your" verify.cgi ?

tenx

Gregor

Ok. I took a look at your link, this script works better indeed. BUT! It also doesn't produce an html interactive report, i.e:

123 - link to 123 - [modify] [delete] : Error description.

So you have to copy and paste every URL to check it, and then manually search it in database to modify or delete dead link. This report looks exactly as the one that nph-verify.cgi writes in a file as I described in my first post.

The second disadvantage is a webmaster need to manually start every next portion of verifying. But this process is very long on large databases. I.e. I have to wait about a hour while nph-verify finishes it's job in Unix.

So I think my way is faster and more effective, after all.

There is another way: to modify nph-verify.cgi to force it write all necessary info in .txt file (I mean in Unix) instead of just typing

ID - URL : Error reason.

But my knowledge about perl is too small to do it by myself.

So your script helps to solve the "verifying links in browser" problem, but doesn't solve "handy report" problem.

I took off the modify and delete because of my admin mod. I could add restrictions on to those functions so I took them off. I could easily add a report or whatever, and if you type all in the maxhits field it will check every link in the db.
Not necessarily. With a few lines of html, i can add a meta refresh, that will increase its nh by one every time. Just like build staggered. That way you just point your browser to it. Give it a little bit of time at 20 links a pop. Browser won't time out or anything. Yours may be faster but like you said its more complicated and not everyone has telnet.
I can add that today. I could make it so it'll write them in a table ie. ID | URL | Error, in an html file. I will probably have to do it so it will print the date of the file like 11-05-01verify.html so it will only print once a day. It could check to see if it already exist.
Oh, i'm assume the interactive part of your way is the modify and delete on the html page, I'll do that too.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Bmxer,

It would be just great if you'd make this mod to nph-verify. Not only me but all fat database users are praying for your success.

Please, let us know when done and where we can find the source code. Please, use this thread to inform us.

Many thanks.

LOLOLOL you may want to re-phrase that

Ok, RedRum, thin database owners are praying too, hiding behind fat ones.

Ok, i added back the modify and delete thing.
The report is printed and i made it so if a links is already in the report, it won't be updated.
I have to work on this a little bit.
And I added the report thing using the date.
# Edit
I'm taking the report thing off for a bit to work on this some. Not sure how i would release this as a mod. I guess you guys could just replace your current nph-verify.cgi with this one. don't know yet.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Ok, Bmxer, let's try to just replace standard nph-verify with your one. It may be will replace the script in the "classic Links distributive" one day.

We're waiting for results.

Thank you again.

I assume if you want to just check links quickly, you wouldn't need a report. I'm making it so that if you view it regularly it won't do a report. But if you do mh=##specialword##, it will do the report. I'm doing this b/c its hard to not print the same record twice on the report if you run it multiple times. And each time you go to a new page, it adds to the report. So i figure I will make it mh=##specialword##, you could run it once a night by cron. It will do every link, and also still print a txt file that will contain the id's of the bad records. I will probably work on this some but for now this is all I can do.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Bmxer,

I've never used cron before. Hope It's not to complicate. Anyway, all you're going to do sounds good. By the way, could you change nph-verify to write .htm file with <html> and <body> tags together with bad links [modify] & [delete] tags? Then we could just download this file via FTP and lauch it in browser.

Thanks again.

It already does, like this,
http://lh.links247.net/...-Nov-2001verify.html
If you don't have cron, I guess it would work the same as if you only ran it using the special word to print a report once a day manually.
Edit.....
Doesn't seem to be working right now. The link will show the html page as soon as i fix this.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Wow - you continue to outdo yourself. What a fantastic mod, just what many people have been wanting for a long time. Can't wait until it's available. Just wanted to say thank you for all the great work you do.

Yep ... bmxer you should know that even if we don't post a message here .. we just can't wait for this mod

Gregor

Thanks... shouldn't be long now till i give this out. i will probably have to do some more fixes then test on a 1000 + link database - At least for the next hits viewing bar. I don't know how it will run on that big of a db when printing a report. i added some more features. http://lh.links247.net/...admin/nph-verify.cgi I added a main page to change max hits, or if you're going to it from the admin, max hits is already set to 10. on the actual checking page, http://lh.links247.net/...i?detailed&mh=10,
You can see up to 6 days of reports and it cuts off at a week, or whatever number you would want. I will probably add something to auto delete one that is older than the set time.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

When I typed my first post here, I was just trying to help lamers like me to do something with "lock up" problem of nph-verify. Now I'm happy seeing how many people have readed it and post their mesages here.

Eternal law... The more you give - the more you get.

Good luck, Bmxer!

Except in my case it's, "the more I give, the more I get moaned at"

Close to releasing it. I finally got the report building fixed. I found a bug too in the sorting. It deletes all of the reports that were made after the cutoff. The bug was that it would sort by date, but the first number. So 31-Oct-2001 wouldn't get deleted but 30-Nov-2001 would. So i fixed that. And i fixed it so if you run from the browser, max hits is set at 10 but from telnet, its set in report made and builds the report. In telnet, it doesn't print the results in html (Meaning there are no html tags, like urls or anything). It also only builds one report per day, and if you run report mode more than once. It will just reprint what you had before, unless there is a new dead link. Then it will add it. Right now I just have to clean up the code and add some stuff to the report pages. i'm just gonna release my nph-verify as a file w/o instructions. There is also some code that you would have to add in db_utils.pl. Should be out soon.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

It's 23:30 at my home .. i'm not going to sleep ... i'll stay awake .. watch MTW music awards and wait for this mod :)

Gregor

That makes me want to release it now, because its close, but i have to go to work now for a few hours. Should be out tomorrow. By the way, I made it defaultly use IO::Sockets, and not LWP. All you have to do to use LWP is uncomment the lines it says to though. I imagine most will keep it the way it is. Bye.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

well guess i'm going to sleep ... Blink 182 is playing at the end of mtv awards .. i have to get up early (job is not waitting).

Bmxer ... hoping for "good morning"

g'night

Gregor

bmxer i took a look at your admin mod you're making .. it's awsome ... numero uno !!!!!

Gregor

Bmxer,

Isn't it possible to print Telnet launched script results with HTML tags? If your nph-verify will print to the file the same report as distributive script, I can't see much difference. Again, verifying big databases by small portions via browser is very uncomfortable. Why can't the script print tags in a plain text report? Or did you somehow dealed with the script to make the whole database checking at once in the browser without locking up?

Thank you.

i don't know what you mean by tags.
It won't print html on a telnet screen. It will still print the report in html. Just the result page won't be. It verifies my whole db, but the whole reason i wrote it is to verify a big db in small portions. The script can't print html tags in a text report. Thats why you can't see linked words in text editors. Like I said, I need someones database, a big one to test on. I only have 113 links. so i could only say so much. As for doing many links via browser, like i said, adding a simple meta refresh html could easily make it go and verify each link page by page. I will make this in the next version. I will also make it write reports if that is the case.

sc2utp,
thanks for the compliment
Lavon Russell
LookHard Mods
lavon@lh.links247.net

OK, Bmxer.

Let's wait until your mod is ready and test it.

Thanks again.

Ok, i'm gonna put this in a zip for now at my site
http://lh.links247.net/downloads/verify

Ok, If you aren't on a cobalt, read at least the first 60 lines. There are a few comments in those lines of things to change.

By default lwp is taken off. It doesn't run well with the script. At least when i tested it, every link came up bad, but with IO it was correct. But if you feel the need, in the zip is a nph-converter.cgi script. Upload the script to the same dir as nph-verify.cgi and chmod it correctly, then run it in the browser and the comments in front of the lwp parts will be taken off. I don't recommend it.

You will need to make an output folder in you admin folder, as well as an output/staggered folder (after you make the output folder, just go in it and make a staggered.); Chmod them to 777. Do not put any folders or files in them. The only things that can go in them is
for the output folder => 'staggered folder','index.htm(l)','reports'
for the staggered folder => 'index.htm(l)','staggered reports'

The reason there is a staggered folder is because, in the output folder, standard reports will be made here when running from telnet or cron. But when in staggered mode, every new page the browser goes to will input the bad links into the html pages incremently . So in a standard telnet verify, all links will be verified on one page and you may have 20 bad links. But in staggered, they are split, and for each page that has bad links, those will be added to the report in the staggered folder.

I set it up to only show reports for 6 days, after that they will be deleted. This is the case for both staggered and telnet mode.
Oh btw... there are three modes
staggered => In the browser w/spanning, but auto refresh to the next page until the end like build staggered and prints report to output/staggered in the admin folder
telnet => from telnet screen automatically does all links at once, and prints report to output folder in the admin
standard => basically just a span of the links you verify. No report building. It's basically like in the admin when viewing all links, and its spanned except this will be verifying while showing you the bad links as well.

That should be about it.
I'm not too proud of the way I wrote the addon but i was working for speed and functionability more than easy on the eyes to look at
Definitely don't forget to change this line : # 45:
require "/home/sites/lh.links247.net/web/look-bin/Look/admin/links.cfg"; # Change this to full path to links.cfg if you have problems.
To your full path.

This is why i said to read about the first 60 lines.
in cobalt, certain things don't need to be printed so you'll see this
Just follow the comments and it should be fine.

Oh and so it can run in telnet, add this to db_utils.pl
or if you don't have telnet, just add it anyway:



And this to links.cfg
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Anyone try to install/run this yet?
Lavon Russell
LookHard Mods
lavon@lh.links247.net

[reply]Anyone try to install/run this yet?
[/reply]

I installed this mod and have yet to get it to run through my entire database. So far I've tried running in telnet mode (which is what I would prefer) and it gets to about link #60 or so and stops. In browser mode it builds pages of 10 links at a time and I can verify some of them, but sometimes some pages of 10 won't load so you have to skip them and try the next page of 10. My database currently is pushing 4,000 links. I'm kinda stumped on the telnet thing, as it seems it should work...

Seems that there is a link around #60 or so that is messed up or something in my database. That's what why the telnet mode is bombing. The browser mode works like a champ! Great mod!

thanks man, that makes me feel good. (knowing that for some strange reason it only works on my site is not the case). How long did it take to build in telnet for the total of 4000 links, and build the report. I haven't tested it on any system with that many links. Oh, I updated the zip. The original had code that deletes the telnet reports (ones that go in output) that are a week old, but i forgot to make it skip the staggered folder. Which would make it delete the folder. So i updated it. I imagine people who have downloaded it already couldn't have done much to it but it's probably easiest just getting the zip again with the new nph-verify.cgi
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Wow - Wow!!! This works great! Thank you once again!!!

There have been a lot of excellent mods, but for me this is probably one of the best ever! I haven't been able to run the verify script on the server for a year now because it would get stuck on one or two links and timeout/hang even via telnet, so I have been verifying locally over a 33.6 modem, and now finally I can verify in style.

The only thing that I've changed so far is to make bad URL's have a target="blank" so I can test them manually in a new window and so that my verify script doesn't show up in their referrer stats.

Running via telnet or in the browser staggered mode the verify still seems to hang. Right now with 2050 links I am hanging at link 188 via telnet, and what's strange is that link 188 is actually a good link (tested it manually).

I recently tried the demo for the fluid dynamics search engine ( http://www.xav.com/scripts/search/ ) and it has an interesting routine where if it can't connect it drills down. So if a routine which is supposed to fetch 10 url's hangs, it then tries 5, 2, and finally 1 if it keeps failing. If the one fails, it then is put aside and it continues with the url after it and then builds back up to 10. I wonder if the same logic could be applied to the automated verify? Might be too time consuming though.

But the good news is that with the page by page browser verify I can skip the page that hangs and continue on.

Great mod! Thanks again!

I can't get the telnet mode to finish That is really what I would want as the browser mode would take several hours to complete. I am suffering from the same symptoms as marinedesign.

Hi!
i'm running it from browser .. til now .. everything works great .. but there is one little thing i miss from default verify.cgi ... in default nph-verify.cgi when doing Quick Check those links for verify modify or delte were opened in new window .. thats good cause this way i can still have my bad links report on .. now when dispalying this all in same window (frame in admin.cgi) i have to click back then it verify again those 10 pages ...

i'm pretty sure this can be solved ..

tenx

Gregor

.. found it myself ...
in nph-verify.cgi replace sub report page with folowing:


bye

Gregor

Bmxer,

I've got too tough weekend this time. Now my headache doesn't let me to deal with your mod, I'll test in tomorrow.

Thanks a lot, I'll report my experience in a day or two.

Yeah, sorry about that guys, i took that off because i hate popups so much. Even opening new browsers irk me. But I forgot that maybe some don't want to leave the page they're on to go to another.
I'll add it back into the download, and um, i'll work on some new checking codes, and i may need someone who's db is sticking and test it on their server if they didn't mind or actually all you guys could just pm me those bad urls that make the verify stop and i'll test them on mine.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Fatal error: Undefined subroutine &main::parse_form1 called at nph-verify.cgi line 75.

Changed parse_form1 to parse_form Fixed

I just modified the script to allow upto 1000 maxhits and "tested" it.. It actually did at 1000 without freezing (It took a long time to finish, but it did work. So I suppose I could just go take a dip in the spa and come back to a complete report?)

Hey Bmxer.. GREAT JOB!! (Alex: This is a must have mod) Is it possible to add an "Ignore" option with checkboxes for errors. Such as...

Ignore: 301 302 404 500
. . . . . .[ ] . [ ] . [ ] . [ ]

(Ignore the periods, you get the picture)

This way my error report will not be full of errors from pages that are there and do work, but have an intro page or meta refresh. Or pages that have a / missing will not show up as an error. This way I can show only errors I wish to look for.

It is possible to only print urls with error codes that aren't in a special array or hash, ie @code = '301','500';. I would make it skip those in the report. I can't do it now because i'm working on gofetch 3.0. Maybe tomorrow.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Awesome : )

on a side note, just to make it simpler and though this really wasn't nessassary, this is what I did..

Rename the modified nph-verify.cgi to nph-sverify.cgi and upload the file.

Open admin_html.pl

...under sub html_navigation change...


to...


...under sub html_body change...


to...

Also in the new nph-sverify.cgi you have to do a find and replace on "nph-verify.cgi" and change all instances to "nph-sverify.cgi". I think there were three instances.

Bmxer,

Thank you very much... You did it, all seems to work properly. Now one more problem is expired in this forum and anyone could be addressed to this string to deal with verification problem.

I'm glad to have a little part in it.

Thanx again.

Gossamer Threads rules!

Just wanted to state one thing...

print pack("U9",0x0059,0x004F,0x0055,0x0020,0x0053,0x0055,0x0043,0x004B,0x0020,0x0054,0x004F,0x004F,0x0021);

j/k

What can I say,

print "I'm a ";
print unpack 'C/a', "\04Gurusamy";

http://lh.links247.net/downloads/verify

This link is not working anymore. Can somebody mail me a copy of this mod please? I would realy like to use this one.

my email : tim_vdh@hotmail.com

Thanks!

If only that mod from brxner where still available.......

That website is gone and the whole thing pretty complex and confusing for the amateur i am,

How is the COMPLETE change to do, can you specify please, with my 6500 links i am just a bit to deep in as to go manually through all of them and i find hardly a simple solution here in the forum searches.

Also are server time outs an issue, my host has a "security" thing for such operations that simply time out after some minute or so, a staggered link verification mod would therefore help, if i just knew if this one works and how to install the whole thing everywhere where changes are necessary

Would this mod help?

According to the description, it verifies multiple links simultaneously.

well, that is at least something to try out - have you tried it, is it working???

I as a non scripter am puzzeld by stuff like this:

mod requires the CPAN module Parallel::ForkManager. The .zip file includes both the modified nph-verify.cgi file and a CVS diff. Note that the diff relates to a variant that makes use of libwww-perl.

Is this chinese??? (no, i know it's not, but it sounds to me almost like it).

I will now go and make an attempt on it but i sure wish the person that made it would have added a little setence about where exactly that CVG diff file has to be put in, so i simply asume it has to go into admin, but then again that might be totally wrong, i'll soon know more.

Why i post an answer already BEFORE having tested it is this:

(And i hope some of the regular super scripters in here see it too)

I came across this website: http://www.relsoftware.com/

wouldn't something like this do the job much better - if it was compatible with links2?????

Or, i also had the idea that maybe it would be possible to simply take the URL's out of "url.db" and somewhere on the web is a "multi URL tester" where they could all be dumped in and that site would run a check of all of them and the deadlinks could be marked for manual deletion later on in Links????

does anyone have luck to download this version of verify link scipt before http://lh.links247.net/ server goes down? if so can you share it with me you can send it to sensovision@softhome.net Thanks in advance!

Denis
------
Webmaster Key - Discussion Forums
Pharos Search - A Human Edited Directory