Mailing List Archive: ModPerl: ModPerl

robots.txt with httpd and root level SetHandler perl-script

Index | Next | Previous | View Threaded

a.smith at ukgrid

Mar 14, 2011, 5:09 AM

Post #1 of 8 (823 views)
Permalink

robots.txt with httpd and root level SetHandler perl-script

Hi,

I'd like to have a robots.txt on a site that has the following
apache httpd config:

<Location />
SetHandler perl-script
PerlHandler RT::Mason
</Location>

But if I install a robots.txt to the DocumentRoot and test it via wget
I just download the front page of the site, as its handled by
perl-script. It it possible to have a robots.txt in this situation?

thanks for any tips, Andy.

aw at ice-sa

Mar 14, 2011, 6:53 AM

Post #2 of 8 (801 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

a.smith [at] ukgrid wrote:
> Hi,
>
> I'd like to have a robots.txt on a site that has the following apache
> httpd config:
>
> <Location />
> SetHandler perl-script
> PerlHandler RT::Mason
> </Location>
>
> But if I install a robots.txt to the DocumentRoot and test it via wget I
> just download the front page of the site, as its handled by perl-script.
> It it possible to have a robots.txt in this situation?
>
> thanks for any tips, Andy.
>
Ideas :
1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to reset the handler
to the default Apache (that may not be so easy)
2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"

randolf at modperl

Mar 14, 2011, 7:28 AM

Post #3 of 8 (798 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

> a.smith [at] ukgrid wrote:
> > Hi,
> >
> > I'd like to have a robots.txt on a site that has the following apache
> > httpd config:
> >
> > <Location />
> > SetHandler perl-script
> > PerlHandler RT::Mason
> > </Location>
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget I
> > just download the front page of the site, as its handled by perl-script.
> > It it possible to have a robots.txt in this situation?
> >
> > thanks for any tips, Andy.
> >
> Ideas :
> 1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to reset the handler
> to the default Apache (that may not be so easy)

It may be easier to have the files match only the needed suffixes
(e.g., .pl, .perl, .html, etc.), otherwise you may need to exclude
other types in the future such as .css, .js, .png, .jpg, .jpeg, .gif,
.ico, etc.

> 2) Create a Mason handler to handle the URL "robots.txt" and return the file "as is"

Be careful here -- you may need to return it with the correct
content type of "text/plain" since "as-is" may be for raw file output
(including HTTP headers).

Randolf Richardson - randolf [at] inter-corporate
Inter-Corporate Computer & Network Services, Inc.
Vancouver, British Columbia, Canada
http://www.inter-corporate.com

torsten.foertsch at gmx

Mar 14, 2011, 7:34 AM

Post #4 of 8 (800 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

On Monday, March 14, 2011 14:53:35 André Warnier wrote:
> > I'd like to have a robots.txt on a site that has the following apache
> >
> > httpd config:
> >
> > <Location />
> > SetHandler perl-script
> > PerlHandler RT::Mason
> > </Location>
> >
> >
> > But if I install a robots.txt to the DocumentRoot and test it via wget I
> > just download the front page of the site, as its handled by perl-script.
> > It it possible to have a robots.txt in this situation?
> >
> >
> >
> > thanks for any tips, Andy.
> >
> >
>
> Ideas :
> 1) Try a <FilesMatch ^robots\.txt$> section inside the above section, to
> reset the handler to the default Apache (that may not be so easy)

SetHandler None (imho)

> 2) Create a Mason handler to handle the URL "robots.txt" and return the
> file "as is"

or instead of the Location block:

PerlMapToStorageHandler "sub { \
use Apache2::Const -compile=>DECLINED; \
use Apache2::RequestRec (); \
use Apache2::RequestUtil (); \
unless( $_[0]->uri eq '/robots.txt' ) { \
$_[0]->add_config(['SetHandler perl-script', \
'PerlHandler RT::Mason']); \
} \
return Apache2::Const::DECLINED; \
}"

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

a.smith at ukgrid

Mar 14, 2011, 10:45 AM

Post #5 of 8 (799 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

Quoting Torsten Förtsch <torsten.foertsch [at] gmx>:

> or instead of the Location block:
>
> PerlMapToStorageHandler "sub { \
> use Apache2::Const -compile=>DECLINED; \
> use Apache2::RequestRec (); \
> use Apache2::RequestUtil (); \
> unless( $_[0]->uri eq '/robots.txt' ) { \
> $_[0]->add_config(['SetHandler perl-script', \
> 'PerlHandler RT::Mason']); \
> } \
> return Apache2::Const::DECLINED; \
> }"

Thanks Torsten (and all). So in this situation do I put this code in
httpd.conf in between the <Location> </Location> lines? Just to check,
as you said "instead of the Location block"?

thanks Andy.

torsten.foertsch at gmx

Mar 14, 2011, 11:03 AM

Post #6 of 8 (798 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

On Monday, March 14, 2011 18:45:39 a.smith [at] ukgrid wrote:
> Quoting Torsten Förtsch <torsten.foertsch [at] gmx>:
> > or instead of the Location block:
> >
> > PerlMapToStorageHandler "sub { \
> >
> > use Apache2::Const -compile=>DECLINED; \
> > use Apache2::RequestRec (); \
> > use Apache2::RequestUtil (); \
> > unless( $_[0]->uri eq '/robots.txt' ) { \
> > $_[0]->add_config(['SetHandler perl-script', \
> > 'PerlHandler RT::Mason']); \
> > } \
> > return Apache2::Const::DECLINED; \
> >
> > }"
>
> Thanks Torsten (and all). So in this situation do I put this code in
> httpd.conf in between the <Location> </Location> lines? Just to check,
> as you said "instead of the Location block"?

A PerlMapToStorageHandler inside a Location block doesn't make sense.

See
http://perl.apache.org/docs/2.0/user/config/config.html#mod_perl_Directives_Argument_Types_and_Allowed_Location

BTW, it should read

use Apache2::Const -compile=>'DECLINED';

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

a.smith at ukgrid

Mar 14, 2011, 11:14 AM

Post #7 of 8 (802 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

Quoting Torsten Förtsch <torsten.foertsch [at] gmx>:

>
> A PerlMapToStorageHandler inside a Location block doesn't make sense.
>
> See
> http://perl.apache.org/docs/2.0/user/config/config.html#mod_perl_Directives_Argument_Types_and_Allowed_Location
>
> BTW, it should read
>
> use Apache2::Const -compile=>'DECLINED';
>

Hi,

so, if I have other <Location> blocks, will setting as you suggest
override all of them? That would break other parts of the httpd
virtual host on my config...

thanks Andy.

torsten.foertsch at gmx

Mar 14, 2011, 11:39 AM

Post #8 of 8 (797 views)
Permalink

Re: robots.txt with httpd and root level SetHandler perl-script [In reply to]

On Monday, March 14, 2011 19:14:35 a.smith [at] ukgrid wrote:
> if I have other <Location> blocks, will setting as you suggest
> override all of them? That would break other parts of the httpd
> virtual host on my config...

no, Location blocks are evaluated *after* a PerlMapToStorageHandler. You can
also try

<Location /robots.txt>
SetHandler None
</Location>

Torsten Förtsch

--
Need professional modperl support? Hire me! (http://foertsch.name)

Like fantasy? http://kabatinte.net

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads