Gossamer Forum: Products: Gossamer Links: Discussions: nph-verify.cgi not working?

Jul 25, 2006, 10:28 AM

afrayer

User (105 posts)

Jul 25, 2006, 10:28 AM

Post #1 of 27

Shortcut

nph-verify.cgi not working?

I'll say on the onset that this is probably not a bug, but simply something I'm doing wrong... but maybe it is a bug, and I'd like to get to the bottom of the issue in any case. I'm running 3.1.0.

I have placed the following in my host's "Scheduled jobs" feature, which appears to be a custom front-end to cron:

Code:
/hermes/web06/b2265/pow.frayernet/yourworldnews/htdocs/cgi-bin/admin/nph-verify.cgi --check_all

I have this scheduled to run at 1:15 AM daily. The intent is to verify the legitimacy of all the links each day, as most links lead to news article content that may become unavailable at any time.

No errors occur (according to the host), leading me to believe the script does run, and should check each URL each night.

Yet the next morning, when I click "Link Status", I find all of the links added since the previous day's manual "Verify Links" are still marked "New" and have not been verified.

So, have I done something wrong, or have I found a bug? Crazy

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 26, 2006, 5:11 AM

afrayer

User (105 posts)

Jul 26, 2006, 5:11 AM

Post #2 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

It appears Verify isn't the only problem. Frown

I've also placed the following line in the schedule:

Code:
/hermes/web06/b2265/pow.frayernet/yourworldnews/htdocs/cgi-bin/admin/nph-build.cgi --flags

Subsequent checking says it, too, runs, but nothing happens. The flags don't get updated unless I run a manual Build All. No error messages occur.

Could this be a permissions thing? Anyone with some suggestions?

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 26, 2006, 5:57 AM

eupos

User (339 posts)

Jul 26, 2006, 5:57 AM

Post #3 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hi Alan,

Have you tried to run both nph-builg.cgi flags from shell?

And if yes, what happened? Does gives you some error?

In the other hand if above method works. This can be permission issue, check the cron user, and the user your scripts run from.

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 26, 2006, 6:18 AM

afrayer

User (105 posts)

Jul 26, 2006, 6:18 AM

Post #4 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

In Reply To:

I'm afraid I do not have shell access to this server (shared hosting). The utilities work from the web interface, but I don't know if there is something different about using them via cron.

I'll see if there is anyway for a hosting customer to determine what rights the cron user has, recognizing they wouldn't possibly give it root access. If it WERE a rights issue, wouldn't I see error messages from the script?

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 26, 2006, 6:29 AM

eupos

User (339 posts)

Jul 26, 2006, 6:29 AM

Post #5 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hi Alan,

Depends on how hosting software setups crons, with right syntax you can tell cron to not say anything :)

But in most cases it will return error with reason of failure to root@ or some other account.

I think you must first talk with your support to check the permissions, and investigate where the problem is.

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 26, 2006, 6:53 AM

afrayer

User (105 posts)

Jul 26, 2006, 6:53 AM

Post #6 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

In Reply To:

I am presently suspecting some odd syntax required by the host, who chose not to use a simple crontab approach. I'm digging further, but no longer suspect the GT code... Thx.

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 27, 2006, 3:25 PM

afrayer

User (105 posts)

Jul 27, 2006, 3:25 PM

Post #7 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

In Reply To:

I am presently suspecting some odd syntax required by the host, who chose not to use a simple crontab approach. I'm digging further, but no longer suspect the GT code... Thx.

Was I not kidding! Shocked

This is what the host wants me to put in my "Scheduled Jobs" command line:

Code:
/usr/bin/perl 
/home/users/web/b2265/pow.frayernet/yourworldnews/htdocs/cgi-bin/admin/nph-verify.cgi 
>/home/users/web/b2265/pow.frayernet/yourworldnews/htdocs/log/nph-verify.log 
2>/home/users/web/b2265/pow.frayernet/yourworldnews/htdocs/log/nph-verify.err

Now my question is... what does that do???

It looks like they are redirecting the output of nph-verify (without the --verify_all) to a file called nph-verify.log but the last line means little to me (is 2> some kind of alternative redirect?).

The host says this works, but it doesn't look like it to me. Anyone care to help me straighten this out? Unsure

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 27, 2006, 11:47 PM

eupos

User (339 posts)

Jul 27, 2006, 11:47 PM

Post #8 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hi Alan,

This is look fine to me :)

The first log line (nph-verify.log) is stdout output from script - when everything is ok output will be written to that log.

The 2> is a stderr output ... so if something goes wrong and in execution of the script something goes wrong, it will be logged to nph-verify.err

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 28, 2006, 3:23 AM

afrayer

User (105 posts)

Jul 28, 2006, 3:23 AM

Post #9 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

I think I was thrown by the content of the first log when they ran the test... it was suspiciously like the Verify Links page, but in retrospect, without the options, that is what it should look like!

I put the options in, and will see if they work. Thanks for the consoling remarks!

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 4:35 AM

afrayer

User (105 posts)

Jul 28, 2006, 4:35 AM

Post #10 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

The mystery continues...

Apparently the script is ignoring the options; vph-verify.cgi runs (and redirects the Verify Links html page properly to a log file), but seems to ignore the --check_all option that was provided.

I have inquired with the host about passing options to the scripts, but wonder now if I have the right syntax on the GT end.

I'm using:

vph-verify.cgi --check_all

Is this correct? What are the other options?

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 4:48 AM

eupos

User (339 posts)

Jul 28, 2006, 4:48 AM

Post #11 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hi Alan,

The right syntax for the check all is: --check-all

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 28, 2006, 5:10 AM

afrayer

User (105 posts)

Jul 28, 2006, 5:10 AM

Post #12 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

Am I misunderstanding the effect of nph-verify.cgi --check-all?

After running the script via the command line, any URLs in the database that were "new" should now be some version of "checked", right? If there are still URLs in the "new" status (as shown by Link Status in Admin), then the script didn't do anything, right?

And if nph-verify.cgi is run without the option, it simply displays the page requesting one to choose a verify type and start the actual verification, right?

So if the redirect gives me a log page that shows the html for the verify type page, I am safe in saying the option is ignored, right?

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 5:54 AM

eupos

User (339 posts)

Jul 28, 2006, 5:54 AM

Post #13 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hi Alan,

Quote:

After running the script via the command line, any URLs in the database that were "new" should now be some version of "checked", right? If there are still URLs in the "new" status (as shown by Link Status in Admin), then the script didn't do anything, right?

Yes iIf all new links are checked the New must shows 0.

Quote:

And if nph-verify.cgi is run without the option, it simply displays the page requesting one to choose a verify type and start the actual verification, right?

From admin interface yes, from command line it simply displays the options you can use.

Quote:

So if the redirect gives me a log page that shows the html for the verify type page, I am safe in saying the option is ignored, right?

I'm not sure I understand you Alan, can you paste please the snippet from the verify log?

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 28, 2006, 6:06 AM

afrayer

User (105 posts)

Jul 28, 2006, 6:06 AM

Post #14 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

In Reply To:

Quote:

So if the redirect gives me a log page that shows the html for the verify type page, I am safe in saying the option is ignored, right?

I'm not sure I understand you Alan, can you paste please the snippet from the verify log?

Check out this log...

Code:
Content-type: text/html 


<html> 

<head> 
<title>Gossamer Links - Tools - Status</title> 
<base target="_self"> 
</head> 

<body bgcolor="#FFFFFF"> 

<table border="1" cellpadding="0" cellspacing="0"><tr><td> 
	<table bgColor="#ffffff" border="0" cellPadding="3" cellSpacing="3" width="500" valign="top"> 
	  <tr> 
		<td align="left" bgColor="navy"><b><font color="#ffffff" size="2" face="Tahoma,Arial,Helvetica">Link 
		  Status</font></b></td> 
	  </tr> 
	  <tr> 
		<td> 
		  <p align="center"><b><font color="#000000" size="2" face="Tahoma,Arial,Helvetica">Link 
		  Status</font></b></p> 
          <p><font size="2" face="Tahoma,Arial,Helvetica">This allows you to go 
          and check the status of links in your database. <b>Please note:</b> for very 
          large databases, you will need to run nph-verify.cgi from telnet and 
          not from the web. It is recommended that you setup your system to run 
          nph-verify.cgi nightly to automatically check links that haven't been 
          checked in 7 days.</font></td> 
	  </tr> 
	</table> 
	</td></tr> 
</table> 

<br> 
<form action="nph-verify.cgi" method="POST"> 
<table border="1" cellpadding="0" cellspacing="0"><tr><td> 
	<table bgColor="#ffffff" border="0" cellPadding="3" cellSpacing="3" width="500" valign="top"> 
	  <tr><td> 

        <table border="1" cellpadding="0" cellspacing="0" width="100%"> 
          <tr> 
            <td width="100%" colspan="4" align="center"> 

        <font size="2" face="Tahoma,Arial,Helvetica"><b>Link Status:</b></font> 
            </td> 
          </tr> 
          <tr> 
            <td width="25%"> 

        <font size="2" face="Tahoma,Arial,Helvetica"> <b><font color="#008000">&nbsp;Good: 188</font></b><font face="Tahoma,Arial,Helvetica"> &nbsp;</font></font> 
            </td> 
            <td width="25%"><font size="2" face="Tahoma,Arial,Helvetica">&nbsp;<font face="Tahoma,Arial,Helvetica"><font color="#FF0000"><b>&nbsp;Bad: 
        0</b></font></font></font> 
            </td> 
            <td width="25%"><font face="Tahoma,Arial,Helvetica" color="#000080" size="2"><b>&nbsp;New: 8</b></font></td> 
            <td width="25%"><b><font face="Tahoma,Arial,Helvetica" size="2">&nbsp;Total: 196</font></b></td> 
          </tr> 
        </table> 
        <p><font face="Tahoma,Arial,Helvetica" size="2">Verify links that:</font></p> 
        <p><font face="Tahoma,Arial,Helvetica" size="2"><input CHECKED name="method" type="radio" value="1"> 
        Haven't been checked for <input name="days" size="2" value="7"> days<br> 
        <input name="method" type="radio" value="2"> Haven't been checked between <input name="since" size="12" value="2000/01/01"> 
        and <input name="to" size="12" value="2006/07/28"><br> 
        <input name="method" type="radio" value="3"> Recheck problem links<br> 
        <input name="method" type="radio" value="4"> Unchecked links<br> 
        <input name="method" type="radio" value="6"> With status code <input name="status" size="4" value="404"><br> 
        <input name="method" type="radio" value="5"> Everything</font></p> 

		<p><center><font size="2" face="Tahoma,Arial,Helvetica"><input type="submit" value="Verify Links"></font></center></p> 
		</td></tr> 
	</table> 
	</td></tr> 
</table> 
<input type="hidden" name="do" value="check_links"> 
</form> 

</body> 
</html>

Like I was saying, by all appearances the option is being ignored (even after I fixed the syntax (--check-all)).

I'm afraid I'm about to get into a finger pointing mess, and frankly I trust the advice I get here more than I trust my host. Unimpressed

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 6:20 AM

eupos

User (339 posts)

Jul 28, 2006, 6:20 AM

Post #15 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Hummm this is weird :(

This is the output when you run nph-verify.cgi from your crontab?

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 28, 2006, 6:28 AM

afrayer

User (105 posts)

Jul 28, 2006, 6:28 AM

Post #16 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

In Reply To:

Hummm this is weird :(

This is the output when you run nph-verify.cgi from your crontab?

From my host's equivalent of the crontab. I don't know what my host is actually doing with it. I get the same thing, whether I run with or without the option. Frown

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 6:49 AM

eupos

User (339 posts)

Jul 28, 2006, 6:49 AM

Post #17 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

I think you must talk with them to investigate and fix the problem Crazy

.

If this is the standard crontab, this really shouldn't happen.

The reason html is showing is because nph-verify.cgi "think" that you start it from your web browser Pirate

.

Cheers,
Boris

Facebook, Twitter and Google+ Auth for GLinks and GCommunity | reCAPTCHA for GLinks | Free GLinks Plugins

Jul 28, 2006, 11:33 AM

afrayer

User (105 posts)

Jul 28, 2006, 11:33 AM

Post #18 of 27

Shortcut

Re: [eupos] nph-verify.cgi not working? In reply to

In Reply To:

I think you must talk with them to investigate and fix the problem Crazy

.

If this is the standard crontab, this really shouldn't happen.

The reason html is showing is because nph-verify.cgi "think" that you start it from your web browser Pirate

The good news is I've gotten them to admit the problem is on their end. Smile

The bad news is they're not ready to admit they should do anything about it. Pirate

They're not inclined to support options for scripts run from their command-line. I wonder, would anyone happen to know how to produce a wrapper script that when executed, executes the enclosed nph- script with the appropriate option? Something tells me it should be possible, but the actual syntax is beyond my experience.

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 1:08 PM

brewt

Staff (4101 posts)

Jul 28, 2006, 1:08 PM

Post #19 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Their so-called crontab isn't a real crontab, it's some web-application that is running things. All the nph-*.cgi scripts have code that detects whether it's being run from shell or the web and displays things accordingly. In your case, it's detecting that it is being run from the web, and thus the shell arguments you are passing it aren't being used. You might be able to get it to work if you pass in 'do=check_links;method=5' as the arguments to nph-verify.cgi instead of --check-all.

Adrian

Jul 28, 2006, 1:23 PM

afrayer

User (105 posts)

Jul 28, 2006, 1:23 PM

Post #20 of 27

Shortcut

Re: [brewt] nph-verify.cgi not working? In reply to

In Reply To:

Nice thought, but alas, I replaced --check-all with 'do=check_links;method=5' and with '?do=check_links;method=5' and found nothing at all happened... nothing in the feedback log or the error log, and nothing happened to the database.

I'm starting to wonder if I need to hack the nph-verify.cgi and nph-build.cgi files to versions that force specific behavior on execution.

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 1:47 PM

afrayer

User (105 posts)

Jul 28, 2006, 1:47 PM

Post #21 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

In Reply To:

Let me correct this... when I tried nph-verify.cgi?do=check_links;method=5 their system logged a CGI error stating nph-verify.cgi?do=check_links does not exist, so apparently the program can't parse out the text from the ? on.

And even if I get them to solve the options bug, it still probably can't pass the arguments I need to pass.

Editing the scripts is looking better all the time!

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 28, 2006, 2:07 PM

brewt

Staff (4101 posts)

Jul 28, 2006, 2:07 PM

Post #22 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

It should be something like:

/hermes/web06/b2265/pow.frayernet/yourworldnews/htdocs/cgi-bin/admin/nph-verify.cgi 'do=check_links;method=5'

Adrian

Jul 28, 2006, 2:14 PM

afrayer

User (105 posts)

Jul 28, 2006, 2:14 PM

Post #23 of 27

Shortcut

Re: [brewt] nph-verify.cgi not working? In reply to

In Reply To:

It should be something like:

/hermes/web06/b2265/pow.frayernet/yourworldnews/htdocs/cgi-bin/admin/nph-verify.cgi 'do=check_links;method=5'

Tried it just now ../nph-verify.cgi 'do=check_links;method=5'

The parameters were ignored entirely.

Unsure

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

Jul 31, 2006, 1:15 PM

brewt

Staff (4101 posts)

Jul 31, 2006, 1:15 PM

Post #24 of 27

Shortcut

Re: [afrayer] nph-verify.cgi not working? In reply to

Don't know what crazy things your so-called crontab is doing :) The other way around this to just write a wrapper script which has all the options you want that just calls the functions that nph-verify.cgi would do.

Adrian

Jul 31, 2006, 1:41 PM

afrayer

User (105 posts)

Jul 31, 2006, 1:41 PM

Post #25 of 27

Shortcut

Re: [brewt] nph-verify.cgi not working? In reply to

In Reply To:

It's lovely! Tongue

The host runs the Scheduled Jobs through a web-based Perl script impersonation of crontab. Because it's web-based, nph-verify.cgi doesn't expect options to be presented, and simply doesn't see them.

The fix, apparently, is simple, yet I'm reluctant to fork a copy of nph-verify.cgi, on the possibility GT has an update somewhere down the pike. All I have to do is make a duplicate of the program with another name, and set it so it doesn't look at the environment it's called from. Force it to think it's running at the command-line (since that is what the host is supposed to be pretending), and it'll take the options.

Whether nph-verify.cgi will run under another name is another question! Angelic

Alan Frayer
Don't just read the news - make the news!
Your World News - http://yourworldnews.frayernet.com

nph-verify.cgi not working?

(Page 1 of 2)