kastner at devicen
Apr 4, 2005, 1:20 AM
Post #1 of 1
-----BEGIN PGP SIGNED MESSAGE-----
success story: www.cebit.de
Hello to all,
after 4 years of working with mod_backhand, I thought it was now the
share with you our experiences with this great piece of software.
Should there be someone not knowing anything about CeBIT, here's a
blurb from the CeBIT website:
"Once a display category within the HANNOVER FAIR - now the largest
in the world
With 6,270 exhibitors (52 percent from abroad) and a net display area of
309,000 square meters CeBIT was once again reasserting its singular role
among the world's IT and telecoms trade fairs in 2005. In just one and
decades, CeBIT has grown from its origins as part of the HANNOVER FAIR
become the world's leading event for information technology,
telecommunications, software and services."
As you can imagine, the CeBIT web site attracts a lot of visitors,
in the weeks before and during the event. At peak times, the web site
several hundred hits a second, serving millions of requests a day.
So performance is crucial. To handle the load, the CeBIT web site is
as a 3-tier cluster:
6 lean apache frontends (2 stand-by, 4 active) are hosting the
addresses (DNS round-robin). In the years before, we had 4 pairs of
each serving one address, connected via heartbeat. This year we used
wackamole to ensure high-availability of the front-end interfaces.
Compared to heartbeat, wackamole gave us higher availability with less
machines and allowed us to eliminate the timeout before takeover. In
had (and still have) 100% availability. One minor glitch: I never
get the "preferred address" directive to work, the addresses jumped
rather randomly at startup but that was not really a problem.
Behind the frontends, the cluster sports a pool of 12 full-blown backend
machines, handling all the php-scripts for page generation, session
etc. which communicate via soap with a localhost tomcat. Tomcat hosts
web services providing database communication and "business logic".
delivered by a large Oracle installation and 2 PostgreSQL instances on
dedicated servers. Additionally, there are several machines
providing fuzzy search matches. Communication with these is handled also
by the tomcat java classes.
BTW: Totally separated from the mod_backhand-managed cluster I'm
here, there is one machine handling static content (pictures, pdf files,
etc.) by a tux web server. But back to the cluster:
The load is balanced and distributed by mod_backhand (of course). This
perfectly, although not all backend servers are equally powered.
I took a screenshot of the backhand-page to give you an imagination of
setup. As you can see, the cluster was idling when I took the
However, even at highest load, the cluster was happily humming along.
problem we encountered were 2 short hickups when the frontends bailed
with lots of "mod_backhand: recursion detected bogus %p" and "failed to
establish umbilical to moderator". This situation required a reload of
frontend apaches and was triggered one time by one hanging backend and
a backend restart in the second event.
I know that Theo suggested to switch off connection pooling in this
but this caused funny behaviour in combination with keepalive-on to the
clients. The years before we had keepalive off, but recieved sporadic
of hanging connections from users. With a short keep-alive (5s) we
this problem without exhausting or servers, but we had to switch
Nevertheless, this years CeBIT web site was a great success which would
have been possible without your passion and work. So thanks to all and
especially to Theo for providing a great software! Please move on and
mod_backhand available for Apache 2. If I can help (unfortunately, I'm
coder), I would be glad to to so!
- -- Tilman Kastner abian GmbH
Tel.: (05 11) 9 29 99 66 Deisterstrasse 81
Fax: (05 11) 9 29 99 33 30449 Hannover
PGP key available Germany
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (Darwin)
-----END PGP SIGNATURE-----
backhand-users mailing list
backhand-users [at] lists