Gossamer Forum
Home : Products : Gossamer Links : Discussions :

google search ranking

Quote Reply
google search ranking
I'm looking around and paying more attention to things -- like how much the Internet has changed in the past year.

One thing, is that google is not following .cgi pages, and is not ranking on their tool bar page-rank any pages with extensions it does not understand.

My directories used a rewrite rule to allow Links SQL to come up as the index page. This was giving a 'null' value on googles page rankings.

I changed that to use a

DirectoryIndex /cgi-bin/path/to/page.cgi?d=1

instead.

On a refresh, my home pages were now ranking as high as 6/10 (which is pretty damn good <G>). Hopefully on the next round of indexing I'll see even better performance.

If you wanted to use index.html with that, you could

DirectoryIndex index.html /cgi-bin/path/to/page.cgi?d=1

this would load index.html if it existed, and the dynamic site if it didn't.

This applies to *all* directories below the root.

The only "hole" I see in this, is that if your script is broken, it could potentially list the contents of the directory.

There has to be a way to make sure that indexes are off, but I have not figured how to make it work with this (I don't have my manuals here). I'd rather give a forbidden or error page, than list my directories.

Any apache guru's know for sure how to make that happen with certainty??

[edited 7/28 to fix a typo]


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

Last edited by:

pugdog: Jul 28, 2003, 10:22 AM
Quote Reply
Re: [pugdog] google search ranking In reply to
Options -Indexes
Quote Reply
Re: [pugdog] google search ranking In reply to
Quote:
My directories used a rewrite rule to allow Links SQL to come up as the index page. This was giving a 'null' value on googles page rankings.
How is that possible?
That's one reason why the rewrite rule exists...

How did your rewritten URLs look? Example?

Best regards,
Webmaster33


Paid Support
from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...
Quote Reply
Re: [webmaster33] google search ranking In reply to
The rewrite rule was:

RewriteRule ^/$ http://server_name/...LinkSQL/page.cgi?d=1 [R,L,NC]

What happens is that you get the whole URL, and Google doesn't like that.

Same as if you were browsing your dynamic directory. Google doesn't seem to like it very much.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [Paul] google search ranking In reply to
Obviously -Indexes, but according to some texts, that shuts off the DirectoryIndex as well. Or, more exactly, by using DirectoryIndex, you override -Indexes.

As I said, someone who knows for sure how these tags work. I don't like surprises if my cgi-scripts stop working for some reason.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] google search ranking In reply to
How did look that?
Quote:
RewriteRule ^/$ http://server_name/cgi-bin/LinkSQL/page.cgi?d=1 [R,L,NC]
http://server_name/page.cgi?d=1 ?

How about a rule which results this:
http://server_name/page/d1/index.html ?

This would be search engine friendly...

Best regards,
Webmaster33


Paid Support
from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...
Quote Reply
Re: [webmaster33] google search ranking In reply to
Problem is, I could probably have a rule to sent DirectoryIndex to the cgi script, and have it "look" like index.html, and then reference that.

The problem is that it could easily break, especially if a configuration changes.

One solution, maybe, would be to figure out how to have "http://sitename.com/.../page.cgi?d=1" (no other arguments) returned as

http://sitename.com/index.html

That might require some reverse proxy or a fake flag that does that with the rewrite rule.

Right now, my suggestion above *seems* to work. It bothers me that there might be a hole in it, and I'd like to know for sure if it works well with -Indexes and wouldn't bomb out at some point.

I don't care about pages deeper in my site. I'm mostly concerned with my main page. A very specific rewrite rule would work great for me :)


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] google search ranking In reply to
This is ok, but how you treat the category pages, like "g=Computers/Accessories&page=3" type dynamic pages?
It is not enough to get indexed just the home page.

Best regards,
Webmaster33


Paid Support
from Webmaster33. Expert in Perl programming & Gossamer Threads applications. (click here for prices)
Webmaster33's products (upd.2004.09.26) | Private message | Contact me | Was my post helpful? Donate my help...
Quote Reply
Re: [pugdog] google search ranking In reply to
Quote:
RewriteRule ^/$ http://server_name/cgi-bin/LinkSQL/page.cgi?d=1 [R,L,NC]

Try:

RewriteRule ^/$ http://server_name/cgi-bin/LinkSQL/page.cgi?d=1 [L,NC]

instead. The R means do a browser redirect rather than an internal redirect. Options -Indexes means Apache won't display the list of file names in a directory if there is no matching index page.

Cheers,

Alex
--
Gossamer Threads Inc.

Last edited by:

Paul: Dec 31, 2002, 4:39 PM
Quote Reply
Re: [Alex] google search ranking In reply to
>> RewriteRule ^/$ http://server_name/cgi-bin/LinkSQL/page.cgi?d=1 [L,NC]

Doesn't work. I get a page error at http://servername.com/

Even tried with the

T=application/x-httpd-cgi

option, didn't work.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] google search ranking In reply to
I'm using

RewriteRule ^/$ /cgi-bin/LinkSQL/page.cgi [PT]

which seems to be working fine - I'm not using dynamic pages but I did want the homepage to be dynamic.

Laura.
The UK High Street
Quote Reply
Re: [afinlr] google search ranking In reply to
Hi,

Currently my homepage is http://www.indigoclothing.com/index.shtml

However I would now want my homeapge to be: http://www.indigoclothing.com/...=1&page=homepage

Can a rewrite rule or something be used so that if www.indigoclothing.com is typed in, the user is taken to my dynamic page but as far as Google is concerned it does not look like a dynamic url?

Regards

Alex



Indigo Clothing is a t-shirt printing company based in the UK.
Indigo Clothing | Promotional Clothing | T-Shirt Printing | Embroidery
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
Hi,

That's exactly what the rewrite rule in the post above should do. Just change the second part to what you need so it will be something like this:

RewriteRule ^/$ /cgi-bin/page.cgi?d=1&page=homepage [PT]

Laura.
Quote Reply
Re: [afinlr] google search ranking In reply to
Laura,

I have added you line to my .htaccess file so it now reads:

RewriteEngine on
RewriteRule ^.*L([0-9]+)/?$ /printing-embroidery/Detailed/$1.html [L]
RewriteRule ^/$ /cgi-bin/page.cgi?d=1&page=homepage [PT]

The first two lines already existed. However, when I type in www.indigoclothing.com I still get my old home page. I know mod_rewrite works as I have been using the first rule succesfully.

Any clues? Many Thanks - Alex



Indigo Clothing is a t-shirt printing company based in the UK.
Indigo Clothing | Promotional Clothing | T-Shirt Printing | Embroidery
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
I'm not sure. I've just had a look at your site and the page at www.indigoclothing.com and www.indigoclothing.com/cgi-bin/page.cgi?d=1&page=homepage seem to be very similar. Can you tell me what the difference is? (Could it be that you are seeing a cached version of your old page?)
Quote Reply
Re: [afinlr] google search ranking In reply to
On the dynamic homepage there is a 'isFeatured' global so that each time the page is loaded three of the links out of the 20 or so that have been marked as featured will randomly be displayed. On the static homepage this does not happen.

I have cleared my temporary internet files folder so I don't think the cache is the problem.

Thanks for all your help so far - this is all very infuriating!



Indigo Clothing is a t-shirt printing company based in the UK.
Indigo Clothing | Promotional Clothing | T-Shirt Printing | Embroidery
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
I think you use a global to inlcude the featured links in your homepage.

This content will only rotate when you build your static pages again.

You better use a cgi script on shtml pages to random change the

featured links on each call of your homepage.

Best regards from
Bremen/Germany

Lothar
Quote Reply
Re: [eljot] google search ranking In reply to
Eljot - Not really sure what you are suggesting - do you mean I should do an SSI of the featured part dynamic page?

Thanks - Alex



Indigo Clothing is a t-shirt printing company based in the UK.
Indigo Clothing | Promotional Clothing | T-Shirt Printing | Embroidery
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
Very strange. Unfortunately I'm not an expert on rewriterules - I have just read enough to get them working how I want on my site. I am actually running my whole site dynamically now with a lot of rewrite rules but I'm using them in my httpd.conf file rather than in .htaccess. I can't believe this is the problem.

My only suggestion is maybe moving the homepage rule above the first one to see whether that makes a difference? Maybe someone else might have an idea?
Quote Reply
Re: [afinlr] google search ranking In reply to
Got teh answer form a guy at expert-exchange:

RewriteRule ^$ /cgi-bin/page.cgi?d=1&page=homepage [L,T=application/x-http-cgi]

Thanks everyone for all you help Smile

Alex



Indigo Clothing is a t-shirt printing company based in the UK.
Indigo Clothing | Promotional Clothing | T-Shirt Printing | Embroidery
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
In Reply To:
Eljot - Not really sure what you are suggesting - do you mean I should do an SSI of the featured part dynamic page?

Thanks - Alex


Hi Alex,

this is the way which I use to include random links on my homepage

and other pages of my site.

But not on dynamic, only on static pages.

Best regards from
Bremen/Germany

Lothar
Quote Reply
Re: [IndigoClothing] google search ranking In reply to
Hi!

I think it's ../page.cgi?d=1&p=homepage
not ../page.cgi?d=1&page=homepage



This one should work

RewriteRule ^$ /cgi-bin/gk/page.cgi?d=1&p=homepage [L,NC]
Quote Reply
Re: [afinlr] google search ranking In reply to
hi,

I *don't* use .htaccess files, rather I specify the rules in the httpd.conf file.

I use the httpd.conf file, rather than .htaccess files, for a couple of reasons, some of which is legacy. When I started out, I saw a problem with .htaccess files on shared servers, and avoided them. (Also, there is server overhead, which the older administrative books harped on a *LOT*.) Also, by putting all the configuration data in ONE file, you never get surprised by an .htaccess file you forgot about, or didn't check.

So, I agree with you on putting everything you can in the httpd.conf file (or use include files, one per server, or even sub-directory if you have a complex one).

I'd rather have the server choke starting up, and restart the server to make changes, than just be able to edit a little file ON the server. Paranoia?? <G> Maybe. But it also prevents you from getting lazy and sloppy, since restarting the server means logging in as root, it's something you think about doing. It's way too easy to just edit or add an .htaccess file.

But, this is personal style, and preference, not a "rule." The point is, that using httpd.conf vs .htaccess shouldn't be the problem -- unless you have another .htaccess file conflicting with the rules in the one you are trying to adjust.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] google search ranking In reply to
Hi Pugdog,

Thanks for the clarification - thats basically what I thought was going on. I still don't understand why my rewrite rule works on my site and not others Unsure but it seems to be sorted out now anyway.

Laura.