Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: RSyslog: users

rsyslog | VIP

 

 

RSyslog users RSS feed   Index | Next | Previous | View Threaded


Carlos.R.Martinez at schwab

Apr 19, 2012, 6:47 AM

Post #1 of 10 (242 views)
Permalink
rsyslog | VIP

I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.

It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.

Any suggestions is appreciated.


Here is my configuration:


syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1, rsyslog_server2

VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.

I prefer the load balance configuration for data center HA.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


vladg at illinois

Apr 19, 2012, 7:17 AM

Post #2 of 10 (225 views)
Permalink
Re: rsyslog | VIP [In reply to]

Well, the simplest solution would be for your clients to use RELP.

That being said, I don't have RELP forwarding for most of my clients. I have two rsyslog forwarders behind a VIP. They both use RELP to forward to the central rsyslog server. I use load balance/round-robin, and have a health check for a high TCP port that I open with imtcp. There are no logs going to that port, it's just open to SYN/ACK. The health check probes the port every second, and if it goes down, it shunts all traffic to the other system. It's not as fast as it could be, but short of using multicast, I haven't found a better solution.

--Vlad

On 4/19/12 8:47 AM, Martinez, Carlos R wrote:
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1, rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Murphy.Brandon at principal

Apr 19, 2012, 8:13 AM

Post #3 of 10 (224 views)
Permalink
Re: rsyslog | VIP [In reply to]

I've been doing something very similar by using a linux based load balancer. It wasn't the easiest thing to figure out how to configure, but it works very well. Here are couple of links that should get you started. I have heard on the IPVS mailing lists that Piranha will not supported by RedHat in the future, so you might be cautious about using that as a front end.
https://www.centos.org/docs/5/html/Virtual_Server_Administration/index.html
https://www.centos.org/docs/4/html/rh-cs-en-4/ch-lvs-piranha.html
http://kezhong.wordpress.com/2010/03/28/setup-linux-loadbalancer-with-piranha-and-lvs-on-centos-5-4/


-Brandon

-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Martinez, Carlos R
Sent: Thursday, April 19, 2012 8:48 AM
To: rsyslog [at] lists
Subject: [rsyslog] rsyslog | VIP

I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.

It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.

Any suggestions is appreciated.


Here is my configuration:


syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1, rsyslog_server2

VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.

I prefer the load balance configuration for data center HA.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


-----Message Disclaimer-----

This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect [at] principal and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.

Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.

While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Carlos.R.Martinez at schwab

Apr 19, 2012, 8:25 AM

Post #4 of 10 (220 views)
Permalink
Re: rsyslog | VIP [In reply to]

I assume the RELP configuration on the rsyslog server is easy to setup... would you happen to have a sample configuration using VIP load balancing?

I will also search the rsyslog.com site for specific documentation.

Thanks!

-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Vlad Grigorescu
Sent: Thursday, April 19, 2012 7:18 AM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog | VIP

Well, the simplest solution would be for your clients to use RELP.

That being said, I don't have RELP forwarding for most of my clients. I have two rsyslog forwarders behind a VIP. They both use RELP to forward to the central rsyslog server. I use load balance/round-robin, and have a health check for a high TCP port that I open with imtcp. There are no logs going to that port, it's just open to SYN/ACK. The health check probes the port every second, and if it goes down, it shunts all traffic to the other system. It's not as fast as it could be, but short of using multicast, I haven't found a better solution.

--Vlad

On 4/19/12 8:47 AM, Martinez, Carlos R wrote:
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1, rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Carlos.R.Martinez at Schwab

Apr 19, 2012, 11:31 AM

Post #5 of 10 (224 views)
Permalink
Re: rsyslog | VIP [In reply to]

Well after looking at the documentation for this it appears this is a client/server use case... I cannot leverage this since my devices that are syslogging to my rsyslog servers are appliance type devices where I cannot configure RELP.

Back to the drawing board...

-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Martinez, Carlos R
Sent: Thursday, April 19, 2012 8:26 AM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog | VIP

I assume the RELP configuration on the rsyslog server is easy to setup... would you happen to have a sample configuration using VIP load balancing?

I will also search the rsyslog.com site for specific documentation.

Thanks!

-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of Vlad Grigorescu
Sent: Thursday, April 19, 2012 7:18 AM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog | VIP

Well, the simplest solution would be for your clients to use RELP.

That being said, I don't have RELP forwarding for most of my clients. I have two rsyslog forwarders behind a VIP. They both use RELP to forward to the central rsyslog server. I use load balance/round-robin, and have a health check for a high TCP port that I open with imtcp. There are no logs going to that port, it's just open to SYN/ACK. The health check probes the port every second, and if it goes down, it shunts all traffic to the other system. It's not as fast as it could be, but short of using multicast, I haven't found a better solution.

--Vlad

On 4/19/12 8:47 AM, Martinez, Carlos R wrote:
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1,
> rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 19, 2012, 12:18 PM

Post #6 of 10 (219 views)
Permalink
Re: rsyslog | VIP [In reply to]

You can use heartbeat (http://linux-ha.org) running on your rsyslog
servers to manage a VIP on the servers themselves (no need for an external
load balancer device)

The hard part of the question comes when you need to determine under what
conditions you want to declare a box dead to force a failover.

From a practical point of view, rsyslog is pretty darn reliable, so if you
just run rsyslog on both systems and only configure heartbeat to failover
if a box is completely dead you will be in pretty good shape.

Hoever, this would not work in the case where rsyslog stops on a box that
is otherwise healthy. How much checking you want to do to decide if
rsyslog is healthy can be complex, various tests can include:

1. is rsyslog running (simple ps |grep check)

2. is rsyslog listening to a port (lsof check, netstat check, or probing a
port)

3. is rsyslog processing the message (sec watching for a message from
rsyslog on a file or named pipe)

4. internal status checking in rsyslog

There are also other subtle prroblems that a box can have that will make a
box still think it's healthy, but not be working properly for your logging

For example:

Routing problems could cause a box to still be up, but be unable to talk
to boxes on other networks

Disk I/O problems could cause the box to be unable to write logs, even
though it can still receive them (or if the box is a relay, a downstream
box could be not responding so it can't forward the log messages)

If a box runs out of memory, various things on the box will get killed,
which may leave a box not working the way it should

I have had about a hundred pairs of boxes running heartbeat for the last
decade (more at some points than at others, but averaging around a hundred
pairs) and I've only had about a dozen or so problems stranger than what
the simplist heartbeat configuration detects (full box down issues) during
that time, which is why I say that as a practical matter, you are probably
just fine with the simplest configuration, without trying to monitor
rsyslog explicitly.

David Lang



On Thu, 19 Apr 2012, Martinez, Carlos R wrote:

> Date: Thu, 19 Apr 2012 13:47:39 +0000
> From: "Martinez, Carlos R" <Carlos.R.Martinez [at] schwab>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: "rsyslog [at] lists" <rsyslog [at] lists>
> Subject: [rsyslog] rsyslog | VIP
>
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1, rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Carlos.R.Martinez at schwab

Apr 19, 2012, 2:22 PM

Post #7 of 10 (222 views)
Permalink
Re: rsyslog | VIP [In reply to]

Thanks David... it is looking like to reliably do this I'm going to have to leverage an HA application with static VIP but I still wonder if I can get away with iptables CLUSTERIP option.

We have our own HA application tool that we use and will need to pursue using it...


-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of david [at] lang
Sent: Thursday, April 19, 2012 12:18 PM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog | VIP

You can use heartbeat (http://linux-ha.org) running on your rsyslog servers to manage a VIP on the servers themselves (no need for an external load balancer device)

The hard part of the question comes when you need to determine under what conditions you want to declare a box dead to force a failover.

From a practical point of view, rsyslog is pretty darn reliable, so if you just run rsyslog on both systems and only configure heartbeat to failover if a box is completely dead you will be in pretty good shape.

Hoever, this would not work in the case where rsyslog stops on a box that is otherwise healthy. How much checking you want to do to decide if rsyslog is healthy can be complex, various tests can include:

1. is rsyslog running (simple ps |grep check)

2. is rsyslog listening to a port (lsof check, netstat check, or probing a
port)

3. is rsyslog processing the message (sec watching for a message from rsyslog on a file or named pipe)

4. internal status checking in rsyslog

There are also other subtle prroblems that a box can have that will make a box still think it's healthy, but not be working properly for your logging

For example:

Routing problems could cause a box to still be up, but be unable to talk to boxes on other networks

Disk I/O problems could cause the box to be unable to write logs, even though it can still receive them (or if the box is a relay, a downstream box could be not responding so it can't forward the log messages)

If a box runs out of memory, various things on the box will get killed, which may leave a box not working the way it should

I have had about a hundred pairs of boxes running heartbeat for the last decade (more at some points than at others, but averaging around a hundred
pairs) and I've only had about a dozen or so problems stranger than what the simplist heartbeat configuration detects (full box down issues) during that time, which is why I say that as a practical matter, you are probably just fine with the simplest configuration, without trying to monitor rsyslog explicitly.

David Lang



On Thu, 19 Apr 2012, Martinez, Carlos R wrote:

> Date: Thu, 19 Apr 2012 13:47:39 +0000
> From: "Martinez, Carlos R" <Carlos.R.Martinez [at] schwab>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: "rsyslog [at] lists" <rsyslog [at] lists>
> Subject: [rsyslog] rsyslog | VIP
>
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1,
> rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 19, 2012, 4:19 PM

Post #8 of 10 (225 views)
Permalink
Re: rsyslog | VIP [In reply to]

remember that CLUSTERIP is a load balancing mechinism, it doesn't do
failover.

you can configure a VIP on one of the boxes by just defining a static
address on eth0:0

where you need a HA application is to detect that one of the boxes has
died and therefor the other box should take over the VIP

With CLUSTERIP, if you configure both systems with the same IP as box 1/1,
then both will get the traffic if it's UDP (if it's TCP you will have the
same effect as a dup IP, and will probably have nothing get through). This
can be a _very_ useful feature as it allows the log message to be sent
once and received by multiple boxes, but I don't think it's what you were
after.

David Lang

On Thu, 19 Apr 2012, Martinez, Carlos R wrote:

> Thanks David... it is looking like to reliably do this I'm going to have
> to leverage an HA application with static VIP but I still wonder if I
> can get away with iptables CLUSTERIP option.
>
> We have our own HA application tool that we use and will need to pursue using it...
>
>
> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of david [at] lang
> Sent: Thursday, April 19, 2012 12:18 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog | VIP
>
> You can use heartbeat (http://linux-ha.org) running on your rsyslog servers to manage a VIP on the servers themselves (no need for an external load balancer device)
>
> The hard part of the question comes when you need to determine under what conditions you want to declare a box dead to force a failover.
>
> From a practical point of view, rsyslog is pretty darn reliable, so if you just run rsyslog on both systems and only configure heartbeat to failover if a box is completely dead you will be in pretty good shape.
>
> Hoever, this would not work in the case where rsyslog stops on a box that is otherwise healthy. How much checking you want to do to decide if rsyslog is healthy can be complex, various tests can include:
>
> 1. is rsyslog running (simple ps |grep check)
>
> 2. is rsyslog listening to a port (lsof check, netstat check, or probing a
> port)
>
> 3. is rsyslog processing the message (sec watching for a message from rsyslog on a file or named pipe)
>
> 4. internal status checking in rsyslog
>
> There are also other subtle prroblems that a box can have that will make a box still think it's healthy, but not be working properly for your logging
>
> For example:
>
> Routing problems could cause a box to still be up, but be unable to talk to boxes on other networks
>
> Disk I/O problems could cause the box to be unable to write logs, even though it can still receive them (or if the box is a relay, a downstream box could be not responding so it can't forward the log messages)
>
> If a box runs out of memory, various things on the box will get killed, which may leave a box not working the way it should
>
> I have had about a hundred pairs of boxes running heartbeat for the last decade (more at some points than at others, but averaging around a hundred
> pairs) and I've only had about a dozen or so problems stranger than what the simplist heartbeat configuration detects (full box down issues) during that time, which is why I say that as a practical matter, you are probably just fine with the simplest configuration, without trying to monitor rsyslog explicitly.
>
> David Lang
>
>
>
> On Thu, 19 Apr 2012, Martinez, Carlos R wrote:
>
>> Date: Thu, 19 Apr 2012 13:47:39 +0000
>> From: "Martinez, Carlos R" <Carlos.R.Martinez [at] schwab>
>> Reply-To: rsyslog-users <rsyslog [at] lists>
>> To: "rsyslog [at] lists" <rsyslog [at] lists>
>> Subject: [rsyslog] rsyslog | VIP
>>
>> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>>
>> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>>
>> Any suggestions is appreciated.
>>
>>
>> Here is my configuration:
>>
>>
>> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1,
>> rsyslog_server2
>>
>> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>>
>> I prefer the load balance configuration for data center HA.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


Murphy.Brandon at principal

Apr 19, 2012, 4:31 PM

Post #9 of 10 (219 views)
Permalink
Re: rsyslog | VIP [In reply to]

Out of curiosity Is heartbeat limited to only 2 nodes? Can more than one node be active? There are some of the features that I enjoy about having an external load balancer.


Brandon Murphy | IT Network Security Assoc-Senior | Information Services - Information Security | The Principal Financial Group® | ph 515.247.5161


-----Original Message-----
From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of david [at] lang
Sent: Thursday, April 19, 2012 2:18 PM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog | VIP

You can use heartbeat (http://linux-ha.org) running on your rsyslog servers to manage a VIP on the servers themselves (no need for an external load balancer device)

The hard part of the question comes when you need to determine under what conditions you want to declare a box dead to force a failover.

From a practical point of view, rsyslog is pretty darn reliable, so if you just run rsyslog on both systems and only configure heartbeat to failover if a box is completely dead you will be in pretty good shape.

Hoever, this would not work in the case where rsyslog stops on a box that is otherwise healthy. How much checking you want to do to decide if rsyslog is healthy can be complex, various tests can include:

1. is rsyslog running (simple ps |grep check)

2. is rsyslog listening to a port (lsof check, netstat check, or probing a
port)

3. is rsyslog processing the message (sec watching for a message from rsyslog on a file or named pipe)

4. internal status checking in rsyslog

There are also other subtle prroblems that a box can have that will make a box still think it's healthy, but not be working properly for your logging

For example:

Routing problems could cause a box to still be up, but be unable to talk to boxes on other networks

Disk I/O problems could cause the box to be unable to write logs, even though it can still receive them (or if the box is a relay, a downstream box could be not responding so it can't forward the log messages)

If a box runs out of memory, various things on the box will get killed, which may leave a box not working the way it should

I have had about a hundred pairs of boxes running heartbeat for the last decade (more at some points than at others, but averaging around a hundred
pairs) and I've only had about a dozen or so problems stranger than what the simplist heartbeat configuration detects (full box down issues) during that time, which is why I say that as a practical matter, you are probably just fine with the simplest configuration, without trying to monitor rsyslog explicitly.

David Lang



On Thu, 19 Apr 2012, Martinez, Carlos R wrote:

> Date: Thu, 19 Apr 2012 13:47:39 +0000
> From: "Martinez, Carlos R" <Carlos.R.Martinez [at] schwab>
> Reply-To: rsyslog-users <rsyslog [at] lists>
> To: "rsyslog [at] lists" <rsyslog [at] lists>
> Subject: [rsyslog] rsyslog | VIP
>
> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>
> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>
> Any suggestions is appreciated.
>
>
> Here is my configuration:
>
>
> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1,
> rsyslog_server2
>
> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>
> I prefer the load balance configuration for data center HA.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


-----Message Disclaimer-----

This e-mail message is intended only for the use of the individual or
entity to which it is addressed, and may contain information that is
privileged, confidential and exempt from disclosure under applicable law.
If you are not the intended recipient, any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify us immediately by
reply email to Connect [at] principal and delete or destroy all copies of
the original message and attachments thereto. Email sent to or from the
Principal Financial Group or any of its member companies may be retained
as required by law or regulation.

Nothing in this message is intended to constitute an Electronic signature
for purposes of the Uniform Electronic Transactions Act (UETA) or the
Electronic Signatures in Global and National Commerce Act ("E-Sign")
unless a specific statement to the contrary is included in this message.

While this communication may be used to promote or market a transaction
or an idea that is discussed in the publication, it is intended to provide
general information about the subject matter covered and is provided with
the understanding that The Principal is not rendering legal, accounting,
or tax advice. It is not a marketed opinion and may not be used to avoid
penalties under the Internal Revenue Code. You should consult with
appropriate counsel or other advisors on all matters pertaining to legal,
tax, or accounting obligations and requirements.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards


david at lang

Apr 19, 2012, 4:44 PM

Post #10 of 10 (224 views)
Permalink
Re: rsyslog | VIP [In reply to]

No, heartbeat can work with many nodes.

The original heartbeat v1 only worked with 2 nodes, and the config syntax
(haresources) is limited to that. As of heartbeat v2 it also supports a
xml config that allows you to have many nodes.

If you want to load balance across multiple nodes, you would have
heartbeat use CLUSTERIP to distribute the traffic across the nodes, and if
a node fails, heartbeat will redirect that traffic to another node.

By default, it avoids disrupting existing connections, so if you have
three nodes they would be 1/3, 2/3, 3/3 and then if node 3 fails you would
have 1/2 3/3, 2/3 (one box getting 2/3 of the traffic), I remember seeing
discussion of an option to redistribute the traffic so it would become
1/2, 2/2 instead, but I don't remember if that got accepted or not.

David Lang

On Thu, 19 Apr 2012, Murphy, Brandon wrote:

> Out of curiosity Is heartbeat limited to only 2 nodes? Can more than
> one node be active? There are some of the features that I enjoy about
> having an external load balancer.
>
>
> Brandon Murphy | IT Network Security Assoc-Senior | Information Services - Information Security | The Principal Financial Group? | ph 515.247.5161
>
>
> -----Original Message-----
> From: rsyslog-bounces [at] lists [mailto:rsyslog-bounces [at] lists] On Behalf Of david [at] lang
> Sent: Thursday, April 19, 2012 2:18 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] rsyslog | VIP
>
> You can use heartbeat (http://linux-ha.org) running on your rsyslog servers to manage a VIP on the servers themselves (no need for an external load balancer device)
>
> The hard part of the question comes when you need to determine under what conditions you want to declare a box dead to force a failover.
>
> From a practical point of view, rsyslog is pretty darn reliable, so if you just run rsyslog on both systems and only configure heartbeat to failover if a box is completely dead you will be in pretty good shape.
>
> Hoever, this would not work in the case where rsyslog stops on a box that is otherwise healthy. How much checking you want to do to decide if rsyslog is healthy can be complex, various tests can include:
>
> 1. is rsyslog running (simple ps |grep check)
>
> 2. is rsyslog listening to a port (lsof check, netstat check, or probing a
> port)
>
> 3. is rsyslog processing the message (sec watching for a message from rsyslog on a file or named pipe)
>
> 4. internal status checking in rsyslog
>
> There are also other subtle prroblems that a box can have that will make a box still think it's healthy, but not be working properly for your logging
>
> For example:
>
> Routing problems could cause a box to still be up, but be unable to talk to boxes on other networks
>
> Disk I/O problems could cause the box to be unable to write logs, even though it can still receive them (or if the box is a relay, a downstream box could be not responding so it can't forward the log messages)
>
> If a box runs out of memory, various things on the box will get killed, which may leave a box not working the way it should
>
> I have had about a hundred pairs of boxes running heartbeat for the last decade (more at some points than at others, but averaging around a hundred
> pairs) and I've only had about a dozen or so problems stranger than what the simplist heartbeat configuration detects (full box down issues) during that time, which is why I say that as a practical matter, you are probably just fine with the simplest configuration, without trying to monitor rsyslog explicitly.
>
> David Lang
>
>
>
> On Thu, 19 Apr 2012, Martinez, Carlos R wrote:
>
>> Date: Thu, 19 Apr 2012 13:47:39 +0000
>> From: "Martinez, Carlos R" <Carlos.R.Martinez [at] schwab>
>> Reply-To: rsyslog-users <rsyslog [at] lists>
>> To: "rsyslog [at] lists" <rsyslog [at] lists>
>> Subject: [rsyslog] rsyslog | VIP
>>
>> I'm looking for the best way to put my rsyslog servers behind a VIP for TCP & UDP traffic.
>>
>> It appears that the best solution thus far at least for TCP is fault tolerant setup with port probing. I have tried a VIP load balance/round-robin configuration but it takes a while for the syslog/rsyslog client to start logging to the surviving rsyslog server when I bring one down. As for UDP I'm not having good results at all and I'm starting to wonder if the best solution would be to have a static VIP IP where I use CLUSTERIP configuration on my systems where the failover occurs over the virtual network interface.
>>
>> Any suggestions is appreciated.
>>
>>
>> Here is my configuration:
>>
>>
>> syslog/rsyslog client ->->->-> VIP ->->->-> rsyslog_server1,
>> rsyslog_server2
>>
>> VIP options: 1) fault tolerant w/port probing 2) load balance/round robin 3) static IP (multicast) utilizing CLUSTERIP option.
>>
>> I prefer the load balance configuration for data center HA.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>
>
> -----Message Disclaimer-----
>
> This e-mail message is intended only for the use of the individual or
> entity to which it is addressed, and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If you are not the intended recipient, any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have
> received this communication in error, please notify us immediately by
> reply email to Connect [at] principal and delete or destroy all copies of
> the original message and attachments thereto. Email sent to or from the
> Principal Financial Group or any of its member companies may be retained
> as required by law or regulation.
>
> Nothing in this message is intended to constitute an Electronic signature
> for purposes of the Uniform Electronic Transactions Act (UETA) or the
> Electronic Signatures in Global and National Commerce Act ("E-Sign")
> unless a specific statement to the contrary is included in this message.
>
> While this communication may be used to promote or market a transaction
> or an idea that is discussed in the publication, it is intended to provide
> general information about the subject matter covered and is provided with
> the understanding that The Principal is not rendering legal, accounting,
> or tax advice. It is not a marketed opinion and may not be used to avoid
> penalties under the Internal Revenue Code. You should consult with
> appropriate counsel or other advisors on all matters pertaining to legal,
> tax, or accounting obligations and requirements.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
>

RSyslog users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.