Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS with Tomcat application server

Index | Next | Previous | View Threaded

jlittle_97 at yahoo

Nov 14, 2007, 5:16 PM

Post #1 of 6 (589 views)
Permalink

[lvs-users] LVS with Tomcat application server

Hi all

I am testing LVS using a tomcat server (real server) to pass hl7 (medical) messages to an end application. The setup looks like this:
_ _ _
|_| Medical information server -> |_| LVS -> |_| tomcat -> _
| _ -> |_| end user application
| -> -> |_| tomcat ->

The load balancer is setup using direct routing. There are mulitple port numbers in the 8000 and 9000 range running across a single VIP. Approximately 50000 messages are currently processed with double that planned over the next year.

The question/problem: The tomcat server must return an acknowledgement to the medical information server. On the three ports that I am testing the ports are going into a close_wait state on the real server. I believe it is because the acknoweldgement is either not getting to the medical information server or that it is receiving it from the real ip of the LVS router.

Is this the correct way to set this up for what I want to accomplish so that the acknowledgment is properly returning to the medical information server?

Thank you for your help.

Regards,
John Little
Hendricks Regional Health
http://hendrickshospital.org

____________________________________________________________________________________
Be a better pen pal.
Text or chat with friends inside Yahoo! Mail. See how. http://overview.mail.yahoo.com/

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Nov 14, 2007, 5:49 PM

Post #2 of 6 (562 views)
Permalink

Re: [lvs-users] LVS with Tomcat application server [In reply to]

On Wed, 14 Nov 2007, John Little wrote:

> Hi all
>
> I am testing LVS using a tomcat server (real server) to
> pass hl7 (medical) messages to an end application. The
> setup looks like this:
_ _ _
|_| Medical information server -> |_| LVS -> |_| tomcat ->
_
|_ -> |_| end user application

| -> -> |_| tomcat ->

I fiddled with the ascii art but couldn't make anything
sensible of it. You have a medical information server, which
I assume is a realserver. Then you have
->tomcat->application->tomcat. Later you seem to say that
the medical server is a client. This is too many layers deep
to be sensible. Still none of it seems to bear on your
problem.

> The question/problem: The tomcat server must return an
> acknowledgement to the medical information server. On the
> three ports that I am testing the ports are going into a
> close_wait state on the real server. I believe it is
> because the acknoweldgement is either not getting to the
> medical information server or that it is receiving it from
> the real ip of the LVS router.

I assume this all works with a single standalone server.

If the client is connecting to the VIP, then the realserver
will be contacted on the VIP. If the application is replying
on the RIP, then there is something wrong with the
application. If so you aren't the first one to find this.
rsh does the same thing (see the HOWTO).

You can check for this problem with tcpdump (netstat on the
realserver will show IPs and ports, check with `netstat -p
prog_name`).

Make sure the application is not listening/running/attached
to the RIP (at least for testing that your LVS works).

If the realserver is replying from the VIP and the reply is
not getting to the client, then there is a problem with your
routing (`ipvsadm` will show InActConn without an ActConn).
You can check your routing by running telnet as your LVS'ed
service to see if this behaves.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jlittle_97 at yahoo

Nov 15, 2007, 6:16 AM

Post #3 of 6 (563 views)
Permalink

Re: [lvs-users] LVS with Tomcat application server [In reply to]

Hi Joe

Thank you for your help. Please see below..

----- Original Message ----
> From: Joseph Mack NA3T <jmack [at] wm7d>
> To: John Little <jlittle_97 [at] yahoo>; LinuxVirtualServer.org users mailing list. <lvs-users [at] linuxvirtualserver>
> Sent: Wednesday, November 14, 2007 8:49:07 PM
> Subject: Re: [lvs-users] LVS with Tomcat application server
>
> On Wed, 14 Nov 2007, John Little wrote:
>
> > Hi all
> >
> > I am testing LVS using a tomcat server (real server) to
> > pass hl7 (medical) messages to an end application. The
> > setup looks like this:
> _ _ _
> |_| Medical information server -> |_| LVS -> |_| tomcat ->
> _
> |_ -> |_| end user application
>
> | -> -> |_| tomcat ->
>
> I fiddled with the ascii art but couldn't make anything
> sensible of it.
Sorry about the ascii are. I wasn't sure how that would turn out.
>You have a medical information server, which
> I assume is a realserver. Then you have
> ->tomcat->application->tomcat. Later you seem to say that
> the medical server is a client. This is too many layers deep
> to be sensible.

Medical server ip: 128.1.6.3
LVS RIP: 128.1.247.72 eth0
LVS VIP: 128.1.247.167 eth0:1
Real Srv RIP (Tomcat): 128.1.247.51 eth0
Real Srv VIP (Tomcat): 128.1.247.167 lo0:67

Netmask is 255.255.0.0. except on lo:67 which is 255.255.255.255

Routing table:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
128.1.247.167 * 255.255.255.255 UH 0 0 0 lo
128.1.0.0 * 255.255.0.0 U 0 0 0 eth0
link-local * 255.255.0.0 U 0 0 0 eth0
loopback * 255.0.0.0 U 0 0 0 lo
default 128.1.7.7 0.0.0.0 UG 0 0 0 eth0

The setup is:

Medical info. server -> LVS -> Real Server (Tomcat) -> end user server

> Still none of it seems to bear on your
> problem.

ok

>
>
> > The question/problem: The tomcat server must return an
> > acknowledgement to the medical information server. On the
> > three ports that I am testing the ports are going into a
> > close_wait state on the real server. I believe it is
> > because the acknoweldgement is either not getting to the
> > medical information server or that it is receiving it from
> > the real ip of the LVS router.
>
> I assume this all works with a single standalone server.

Please see above.

>
> If the client is connecting to the VIP, then the realserver
> will be contacted on the VIP. If the application is replying

The medical info. server connects to the VIP on the LVS router

> on the RIP, then there is something wrong with the
> application. If so you aren't the first one to find this.
> rsh does the same thing (see the HOWTO).
>
> You can check for this problem with tcpdump (netstat on the
> realserver will show IPs and ports, check with `netstat -p
> prog_name`).

netstat output from the real server:
tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54364 CLOSE_WAIT 11968/is_domain2
tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54356 CLOSE_WAIT 11968/is_domain2
tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54348 CLOSE_WAIT 11968/is_domain2

As you can see from the above the tomcat server is connecting back to the RIP on the LVS router and going into a CLOSE_WAIT state.
I believe that this is something wrong with my setup but I'm not sure. Can you tell from the information that I've provided?

>
> Make sure the application is not listening/running/attached
> to the RIP (at least for testing that your LVS works).

It isn't.

>
> If the realserver is replying from the VIP and the reply is
> not getting to the client, then there is a problem with your
> routing (`ipvsadm` will show InActConn without an ActConn).
> You can check your routing by running telnet as your LVS'ed
> service to see if this behaves.
This works ok.
>
> Joe
>
> --
> Joseph Mack NA3T EME(B,D), FM05lw North Carolina
> jmack (at) wm7d (dot) net - azimuthal equidistant map
> generator at http://www.wm7d.net/azproj.shtml
> Homepage http://www.austintek.com/ It's GNU/Linux!
>
Thanks,
John

____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Nov 15, 2007, 8:00 AM

Post #4 of 6 (565 views)
Permalink

Re: [lvs-users] LVS with Tomcat application server [In reply to]

On Thu, 15 Nov 2007, John Little wrote:

> Sorry about the ascii are. I wasn't sure how that would turn out.

ascii art just has to have no tabs and be limited to about
60 chars wide to make it through e-mail

> Medical server ip: 128.1.6.3
> LVS RIP: 128.1.247.72 eth0

is this what we call the DIP?

> LVS VIP: 128.1.247.167 eth0:1
> Real Srv RIP (Tomcat): 128.1.247.51 eth0
> Real Srv VIP (Tomcat): 128.1.247.167 lo0:67
>
> Netmask is 255.255.0.0. except on lo:67 which is 255.255.255.255
>
> Routing table:
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use Iface
> 128.1.247.167 * 255.255.255.255 UH 0 0 0 lo
> 128.1.0.0 * 255.255.0.0 U 0 0 0 eth0
> link-local * 255.255.0.0 U 0 0 0 eth0
> loopback * 255.0.0.0 U 0 0 0 lo
> default 128.1.7.7 0.0.0.0 UG 0 0 0 eth0
>
> The setup is:
>
> Medical info. server -> LVS -> Real Server (Tomcat) -> end user server

> netstat output from the real server:
> tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54364 CLOSE_WAIT 11968/is_domain2
> tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54356 CLOSE_WAIT 11968/is_domain2
> tcp 1 0 ieng-repo-dev.hrh.:8325 128.1.247.72%720:54348 CLOSE_WAIT 11968/is_domain2
> =>snip

> As you can see from the above the tomcat server is
> connecting back to the RIP on the LVS router and going
> into a CLOSE_WAIT state.

I assume ieng-repo-dev.hrh is a realserver.

I have no idea why an application on the realserver is
connecting to the DIP. It's not part of the connection
that you've established from the client to the VIP on the
realserver.

Either something is misconfigured (LVS? application?) (most
likely) or the application is making assumptions about the
connection to the client which is invalid (less likely).

>> Make sure the application is not listening/running/attached
>> to the RIP (at least for testing that your LVS works).
>
> It isn't.

on the realserver do you see a connection from the client to
the VIP?

>> You can check your routing by running telnet as your LVS'ed
>> service to see if this behaves.

> This works ok.

you get load balancing and on the realserver you see a
connection to the VIP?

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jlittle_97 at yahoo

Nov 16, 2007, 6:29 AM

Post #5 of 6 (560 views)
Permalink

Re: [lvs-users] LVS with Tomcat application server [In reply to]

All..

---- Original Message ----
> From: Joseph Mack NA3T <jmack [at] wm7d>
> To: John Little <jlittle_97 [at] yahoo>; LinuxVirtualServer.org users mailing list. <lvs-users [at] linuxvirtualserver>
> Sent: Wednesday, November 14, 2007 8:49:07 PM
> Subject: Re: [lvs-users] LVS with Tomcat application server
>
> On Wed, 14 Nov 2007, John Little wrote:
>
> > Hi all
> >
> > I am testing LVS using a tomcat server (real server) to
> > pass hl7 (medical) messages to an end application. The
> > setup looks like this:
> _ _ _
> |_| Medical information server -> |_| LVS -> |_| tomcat ->
> _
> |_ -> |_| end user application
>
> | -> -> |_| tomcat ->
>
> I fiddled with the ascii art but couldn't make anything
> sensible of it. You have a medical information server, which
> I assume is a realserver. Then you have
> ->tomcat->application->tomcat. Later you seem to say that
> the medical server is a client. This is too many layers deep
> to be sensible. Still none of it seems to bear on your
> problem.
>
>
> > The question/problem: The tomcat server must return an
> > acknowledgement to the medical information server. On the
> > three ports that I am testing the ports are going into a
> > close_wait state on the real server. I believe it is
> > because the acknoweldgement is either not getting to the
> > medical information server or that it is receiving it from
> > the real ip of the LVS router.
>
> I assume this all works with a single standalone server.
>
> If the client is connecting to the VIP, then the realserver
> will be contacted on the VIP. If the application is replying
> on the RIP, then there is something wrong with the
> application. If so you aren't the first one to find this.
> rsh does the same thing (see the HOWTO).
>
> You can check for this problem with tcpdump (netstat on the
> realserver will show IPs and ports, check with `netstat -p
> prog_name`).
>
> Make sure the application is not listening/running/attached
> to the RIP (at least for testing that your LVS works).
>
> If the realserver is replying from the VIP and the reply is
> not getting to the client, then there is a problem with your
> routing (`ipvsadm` will show InActConn without an ActConn).
> You can check your routing by running telnet as your LVS'ed
> service to see if this behaves.
>
> Joe

Having tested this some more I have discovered that the Tomcat (real) server on startup is listening on port 8325 as it should without any connections in the close_wait state.

Immediately on starting LVS it appears that the LVS router establishes a connection to the real server using the RIP of the LVS router and the RIP of the real server. This is without any data passing through the LVS router to the real server.

Is this normal behavior or can someone tell me where to start looking for what I have setup incorrectly?

Netstat and tcpdump output is shown below.

Thanks,
John

Output of netstat:
ieng-repo-dev-lh2:/opt/JavaCAPS # netstat -aln |grep 8325
tcp 0 0 :::8325 :::* LISTEN
tcp 1 0 128.1.247.52:8325 128.1.247.72:52570 CLOSE_WAIT (128.1.247.52 =real server RIP)
tcp 1 0 128.1.247.52:8325 128.1.247.72:52562 CLOSE_WAIT (128.1.247.72 =LVS RIP)

The output of tcpdump indicates that they are communicating over this ports as well:

09:26:57.662346 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], length: 44) ieng-repo-dev-lh2.hrh.org.8325 > 128.1.247.72.54719: S [tcp sum ok] 1087972898:1087972898(0) ack 4189165685 win 5840 <mss 1460>
09:26:57.662953 IP (tos 0x0, ttl 64, id 20036, offset 0, flags [DF], length: 40) 128.1.247.72.54719 > ieng-repo-dev-lh2.hrh.org.8325: . [tcp sum ok] 1:1(0) ack 1 win 5840
09:26:57.663371 IP (tos 0x0, ttl 64, id 20038, offset 0, flags [DF], length: 40) 128.1.247.72.54719 > ieng-repo-dev-lh2.hrh.org.8325: F [tcp sum ok] 1:1(0) ack 1 win 5840
09:26:57.663790 IP (tos 0x0, ttl 64, id 61228, offset 0, flags [DF], length: 40) ieng-repo-dev-lh2.hrh.org.8325 > 128.1.247.72.54719: . [tcp sum ok] 1:1(0) ack 2 win 5840

____________________________________________________________________________________
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Nov 16, 2007, 6:48 AM

Post #6 of 6 (568 views)
Permalink

Re: [lvs-users] LVS with Tomcat application server [In reply to]

On Fri, 16 Nov 2007, John Little wrote:

> Having tested this some more I have discovered that the
> Tomcat (real) server on startup is listening on port 8325
> as it should without any connections in the close_wait
> state.
>
> Immediately on starting LVS it appears that the LVS router
> establishes a connection

the director doesn't establish connections to anything, it's
just a router with slightly different rules to a normal
router. What you're observing isn't part of LVS.

> to the real server using the RIP

do you mean DIP?

> Is this normal behavior

it's not part of LVS.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads