Hi,
I was reading that serving static pages from one server and running CGI from another one was a good solution for high traffic sites using LinkSQL.
Could anyone explain this in detail ?
Thanks,
Emilio

For high volume sites, it's very useful to remove CGI, and replace it with something faster. Now by high volume, I mean > 250,000 hits a day.

A common solution is to use mod_perl. mod_perl offers huge improvements in CGI. However, you will need a dedicated server in order to use it. For more info on mod_perl, please see:

http://perl.apache.org/

Hope that helps,

Alex

There are two concepts here -- one is having a CGI-server and the other is running it under mod_perl.

LinkSQL will run under mod_perl (I haven't tried it, but that's the rumor<G> ) mod_perl bloats the Apache process, so you can't handle as many processes per machine, but if the process is running a cgi program it runs a lot faster. So, the best of both worlds is to use 'lean' apache for static pages, and mod_perl apache for cgi processes, and the best way to do it is to run two machines, thus you can apportion resources more easily as to which part of the process needs beefing up. (usually the static page serve never does <G> )

So, if you have two machines, put your /cgi-bin directory on one machine, and your /html directory on the other.

You would name your machines mydomain.com and cgi.mydomain.com (for example).

All the cgi calls would go to the cgi. machine (example):

http://cgi.mydomain.com/cgi-bin/page.cgi

Your static pages are served off of:

http://www.mydomain.com/html

The cgi.mydomain.com machine is running a version of Apache with mod_perl compiled in, and it will run faster that way (as start up penalties are avoided, and processes stay "alive")

There are specifics on how to do this in the mod_perl area on http://www.apache.org

The idea is to "force" the cpu-intensive processes off the http://www.mydomain.com machine, and onto cgi.mydomain.com

You can't just alias the directory to the other machine, because you want the Apache _on_ the other machine to actually run it, so you need to invoke that with a new http://cgi.mydomain.com call.

[[NOTE: There are instructions for using the ReWrite engine to alias between two copies of Apache across two machines:


http://perl.apache.org/guide/scenario.html#One_Light_and_One_Heavy_Server_w


It's probably _not_ as simple as that, and some debugging is bound to be needed.

****>>>> Is _anyone_ running two machines like this??? Or links spread across a network?

I won't be there for a few more months... til then, this is all theoretical



------------------
POSTCARDS.COM -- Everything Postcards on the Internet www.postcards.com
LinkSQL FAQ: www.postcards.com/FAQ/LinkSQL/

You don't even need two machines. The ideal situation is:

1. Install "light" apache with mod_proxy compiled into:
/usr/local/apache
2. Install "perl" apache with mod_perl compiled into:
/usr/local/apache_perl
3. For light apache setup:
ProxyPass /perl http://localhost:81/perl
ProxyPassReverse /perl http://localhost:81/perl
4. For perl apache setup:
Listen 81

Now what you have is two copies of apache running on the same machine. The light apache handles everything, but if a request that starts with /perl comes in, it passes it to the backend mod_perl server, which runs the program and sends the results back to the light apache. The light apache then send the results back to the client.

It sounds like a lot of overhead, but it's not. It has some great advantadges:

1. The perl server is never tied up by a slow dial up connection, so you usually never need more then 5-10 children around (hence saving on memory needed).

2. The perl server only handles perl scripts, nothing else. All images and html are handled by the light apache.

3. It's totally transparent to the end user. You don't need to worry about going to different domains, or different ports. All the end user sees is the normal URL.

In my opinion, if you have a dedicated server and handle a lot of perl cgi's, you would be a lot better off using the layout described above.

Cheers,

Alex

That's how that works.... I've been going over the docs, and it's somewhat confusing.

I'll give that a try maybe this weekend, and see what happens.

Now, the Links scripts need to be moved to the /perl/ subdirectory? Or can that be aliased to the /cgi-bin/ subdirectory?

Obviously you don't want the mod_perl apache having to handle non-perl CGI programs.

So, would it work to create that alias of /perl/ /cgi-bin/

and then change the cgi directory in the Links.pm files (and anywhere it's hard coded) to be /perl/ ?

If it's done this way, any /cgi-bin/ requests that were hard coded, and missed, would be caught by the _same_ scripts running from the _same_ physical location, without generating an error message (maybe??)

Would that work?

I'm not there yet... but if it's that easy to pass the request handling to the back end, I might try it this weekend when our loads go down.

It is that easy! What I would do is:

1. Install a second apache with mod_perl. Try and keep it separate from everything else, and install it in it's own directory say:

/usr/local/apache_perl

2. Edit the httpd.conf, and change the port number it listens to to 81 (defaults to 80).

3. Install Links SQL into the servers cgi-bin directory.

4. Try out the program by going to:

http://yourmachinename:81/cgi-bin/admin/admin.cgi

and test that it works. You need the :81 to tell the script that it is going to the mod_perl server.

5. Once you are satisfied that it works, add:

ProxyPass /perl/ http://localhost:81/cgi-bin/
ProxyPassReverse /perl/ http://localhost:81/cgi-bin/

to your main httpd.conf. note: this requires that you have mod_proxy compiled into your main Apache. If you don't you will need to add that in. This is where things will get a bit risky if you are unsure of what you are doing as you will be affecting your main web server. You can tell if you have mod_proxy by typing:

/path/to/httpd -l

and you should see mod_proxy.c in the list.

6. Restart the main web server and you should now be able to go to:

http://yourserver/perl/admin/admin.cgi

You'll need to edit the Links.pm to set the URL's properly. Note: you want the /perl/ URL's, not /cgi-bin/, not port 81.

You should now be all done. I don't recommend changing /cgi-bin/ on your main server, as the mod_perl server is only useful at optimizing perl scripts. Also, not all CGI scripts will work well with mod_perl, and you may experience problems with some (like this forum, which is why it's not in /perl/).

Finally, http://perl.apache.org/guide/ is your friend.

If you have any problems or questions though, don't hesitate to ask!

Cheers,

Alex

It was _almost_ that simple

I'll post complete step by step in another thread, since it's not really "running on multiple machines."



------------------
POSTCARDS.COM -- Everything Postcards on the Internet www.postcards.com
LinkSQL FAQ: www.postcards.com/FAQ/LinkSQL/