Mailing List Archive: Linux Virtual Server: Users

[lvs-users] ldirectord email features

Index | Next | Previous | View Threaded

eric.robinson at psmnv

Nov 5, 2009, 4:43 AM

Post #1 of 12 (398 views)
Permalink

[lvs-users] ldirectord email features

I'm a fan of ldirectord so I am proposing a possible feature. ldirectord
sends me an email alert when services go down, which sometimes works too
well. A few months ago I was replacing some switches and make a spanning
tree mistake which caused ldirectord to be unable to reach any of my
realservers. It took me a while to find the problem, during which I made
several changes to the network infrastructure. As a result, I got about
2500 text messages on my cell phone from ldirectord. More recently, I
enabled a script that restarts certain services every night, so now I
get 250 text messages on my cell every night!

I wish there was a way to tell ldirectord stuff like this:

1. Don't send alerts during certain time windows
2. Except always send alerts for certain services or hosts, even during
excluded time windows
3. If you find that you are sending a lot of alerts in a short time
frame (say, more than X per second), wait Y seconds and send an email
digest, not separate emails for each alert

That's my Christmas wish list!

--
Eric Robinson

Disclaimer - November 5, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

malte at snapscouts

Nov 5, 2009, 5:10 AM

Post #2 of 12 (388 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Hi,

Don't you think this is more or less the meaning of having a "bigger"
monitoring infrastructur
like Nagios - to handle such things ?
For me it sounds like another approach to reinvent the wheel.
I'd suggest writing a simple nagios check would be a lot easier - there
you can even fire up
actions if necessary such as trigger a failover between your
loadbalancers or similar.

> sends me an email alert when services go down, which sometimes works too
> well. A few months ago I was replacing some switches and make a spanning
> tree mistake which caused ldirectord to be unable to reach any of my
> realservers. It took me a while to find the problem, during which I made
> several changes to the network infrastructure. As a result, I got about
> 2500 text messages on my cell phone from ldirectord. More recently, I
> enabled a script that restarts certain services every night, so now I
> get 250 text messages on my cell every night!
>
> I wish there was a way to tell ldirectord stuff like this:
>
> 1. Don't send alerts during certain time windows
> 2. Except always send alerts for certain services or hosts, even during
> excluded time windows
> 3. If you find that you are sending a lot of alerts in a short time
> frame (say, more than X per second), wait Y seconds and send an email
> digest, not separate emails for each alert
>
> That's my Christmas wish list!
>
>
The digest function is not supported in Nagios while you can define to
be only notified once
while a service is in state Critical.

kind regards,
Malte Geierhos

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

eric.robinson at psmnv

Nov 5, 2009, 5:42 AM

Post #3 of 12 (388 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

> Don't you think this is more or less the meaning of having a "bigger"
> monitoring infrastructure like Nagios - to handle such things ?

To me it seems like apple and oranges. ldirectord manages lvs and runs
right on the load balancers. It is fast, relaible, and easy for me to
maintain. I'd rather work around the alerting problem than intruduce a
whole new technology that I don't understand as well. But since you
raised the question, does Nagios manage lvs or does it just do
monitoring and alerting? (I'm sure it is possible to have Nagios control
lvs, but is the functionality already there or would I have to write
custom scripts?) Also, can Nagios be managed in turn by heartbeat?

> I'd suggest writing a simple nagios check would be a lot easier
> - there you can even fire up actions if necessary such as trigger
> a failover between your loadbalancers or similar.

My load balancer cluster is managed by heartbeat.

Best,

--
Eric Robinson

Disclaimer - November 5, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Nov 5, 2009, 5:52 AM

Post #4 of 12 (388 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

On Thu, 2009-11-05 at 04:43 -0800, Robinson, Eric wrote:
> I'm a fan of ldirectord so I am proposing a possible feature. ldirectord
> sends me an email alert when services go down, which sometimes works too
> well. A few months ago I was replacing some switches and make a spanning
> tree mistake which caused ldirectord to be unable to reach any of my
> realservers. It took me a while to find the problem, during which I made
> several changes to the network infrastructure. As a result, I got about
> 2500 text messages on my cell phone from ldirectord. More recently, I
> enabled a script that restarts certain services every night, so now I
> get 250 text messages on my cell every night!

Disregarding for a moment the time-based stuff (and exceptions to that),
what you're asking for is an implementation of "flap detection", in
essence - if event A happens more often than frequency X, suppress
notifications.

More widely - if ldirectord notices a service/host is flapping, should
it remove that host/service and lengthen the check time before testing
again? It could have a geometric growth, too.

Not sure I could code that right now but it's fairly widely applied
logic in networking monitoring.

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

malte at snapscouts

Nov 5, 2009, 5:53 AM

Post #5 of 12 (388 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Hi,
> To me it seems like apple and oranges. ldirectord manages lvs and runs
> right on the load balancers. It is fast, relaible, and easy for me to
> maintain. I'd rather work around the alerting problem than intruduce a
> whole new technology that I don't understand as well. But since you
> raised the question, does Nagios manage lvs or does it just do
> monitoring and alerting? (I'm sure it is possible to have Nagios control
> lvs, but is the functionality already there or would I have to write
> custom scripts?) Also, can Nagios be managed in turn by heartbeat?
>
no - but Nagios can monitor your Heartbeat infrastructure and alert the
right persons if something went wrong.
Heartbeat itself can only handle a absolut minimum like writing mail if
something changed (failover i.e.)
For sure typical Nagios Check interval is 3 to 10 minutes (free
configurable) which is by far slower then
using a Monitor Operation with 10s in Heartbeat.

But what i really meant is - that normally contacting the right people
in with the right media in defined times
is more or less a typical issue for Monitor & Alerting systems.

>> I'd suggest writing a simple nagios check would be a lot easier
>> - there you can even fire up actions if necessary such as trigger
>> a failover between your loadbalancers or similar.
>>
> My load balancer cluster is managed by heartbeat.
>
Of course i monitor my heartbeat cluster with Nagios...
see crm_mon -s for instance .
For sure you can break it down to groups, ressources and so on as well.
I just meant the Logic would somewhat bloat ldirectord to something
which already exists as proven solution.

kind regards,
Malte Geierhos

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

eric.robinson at psmnv

Nov 5, 2009, 7:36 AM

Post #6 of 12 (388 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

> Disregarding for a moment the time-based stuff (and
> exceptions to that), what you're asking for is an
> implementation of "flap detection"

I see what you're saying, and that would definitely be useful, but what
I'm asking for is not really flap detection. I would think of it as
"alert throttling." I currently have 300+ services that are managed by
ldirectord, so I sometimes get 300+ alerts on a SINGLE up-down event,
even without "flapping." That's what I'm trying to avoid.

--
Eric Robinson

Disclaimer - November 5, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Nov 5, 2009, 4:05 PM

Post #7 of 12 (380 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

On Thu, Nov 05, 2009 at 02:10:13PM +0100, Malte Geierhos wrote:
> Hi,
>
> Don't you think this is more or less the meaning of having a "bigger"
> monitoring infrastructur
> like Nagios - to handle such things ?
> For me it sounds like another approach to reinvent the wheel.
> I'd suggest writing a simple nagios check would be a lot easier - there
> you can even fire up
> actions if necessary such as trigger a failover between your
> loadbalancers or similar.

For what its worth, I think that these functions are probably reasonable
to include in ldirectord. Though I my initial reaction was also that
it sounds a lot like its going in the direction of a comprehensive
monitoring system.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

anthony at duck

Nov 6, 2009, 1:33 PM

Post #8 of 12 (367 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Since I was the person that contributed the patch for the email alert code back in 2006, I could make another contribution attempt without making things too complex.

1. i think that is a good idea. the option should probably be on a per virtual service basis. before it sends the email, it would just compare the current time to whats defined in the exception period or alert period depending on how its implemented

2. with #1 implemented, you would just make sure the specific service does not have any alert exceptions

3. if using multiple configuration files/ldirectord processes, you would then probably need to have something like a control file that is accessed (read/write) by any ldirectord process. That means adding a lock file and checking. I am not sure if there would be a better way to implement it.
There is the emailalertfreq option that limits how many times an email is sent on a per service basis. You can set that to a higher value to limit the amount of emails you get per service when its down.

anthony

Robinson, Eric wrote:
> I'm a fan of ldirectord so I am proposing a possible feature. ldirectord
> sends me an email alert when services go down, which sometimes works too
> well. A few months ago I was replacing some switches and make a spanning
> tree mistake which caused ldirectord to be unable to reach any of my
> realservers. It took me a while to find the problem, during which I made
> several changes to the network infrastructure. As a result, I got about
> 2500 text messages on my cell phone from ldirectord. More recently, I
> enabled a script that restarts certain services every night, so now I
> get 250 text messages on my cell every night!
>
> I wish there was a way to tell ldirectord stuff like this:
>
> 1. Don't send alerts during certain time windows
> 2. Except always send alerts for certain services or hosts, even during
> excluded time windows
> 3. If you find that you are sending a lot of alerts in a short time
> frame (say, more than X per second), wait Y seconds and send an email
> digest, not separate emails for each alert
>
> That's my Christmas wish list!
>
>
> --
> Eric Robinson
>
>
>
>
>
> Disclaimer - November 5, 2009
> This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
> This disclaimer was added by Policy Patrol: http://www.policypatrol.com/
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

horms at verge

Nov 6, 2009, 5:05 PM

Post #9 of 12 (363 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

On Fri, Nov 06, 2009 at 04:33:38PM -0500, anthony[at]duck.com wrote:
> Since I was the person that contributed the patch for the email alert code back in 2006, I could make another contribution attempt without making things too complex.
>
> 1. i think that is a good idea. the option should probably be on a per
> virtual service basis. before it sends the email, it would just compare
> the current time to whats defined in the exception period or alert period
> depending on how its implemented

Agreed, though it would probably also be worthwhile having global
directives too.

>
> 2. with #1 implemented, you would just make sure the specific service
> does not have any alert exceptions
>
> 3. if using multiple configuration files/ldirectord processes, you would
> then probably need to have something like a control file that is accessed
> (read/write) by any ldirectord process. That means adding a lock file
> and checking. I am not sure if there would be a better way to implement
> it. There is the emailalertfreq option that limits how many times an
> email is sent on a per service basis. You can set that to a higher value
> to limit the amount of emails you get per service when its down.

I'm not sure why you feel such locking would be necessary.
The configs in different config files should be independent of
each other.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

anthony.ciaravalo at on2

Nov 6, 2009, 5:57 PM

Post #10 of 12 (354 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Simon Horman wrote:
>> 3. if using multiple configuration files/ldirectord processes, you would
>> then probably need to have something like a control file that is accessed
>> (read/write) by any ldirectord process. That means adding a lock file
>> and checking. I am not sure if there would be a better way to implement
>> it. There is the emailalertfreq option that limits how many times an
>> email is sent on a per service basis. You can set that to a higher value
>> to limit the amount of emails you get per service when its down.
>
> I'm not sure why you feel such locking would be necessary.
> The configs in different config files should be independent of
> each other.

if the throttling/digest of alerts is strictly based on a per configuration file/single ldirectord process basis, then it makes sense for them to be independent of each other. if the throttling/digest of alerts is based on multiple config files and there are many of those then having a single
control file would potentially reduce the amount of simultaneous open files.

i am not sure if eric is using for example a single config file/ldirectord process with 300 services in it, or 300 config files/ldirectord processes each with a single service. i could see how the latter would cause thousands of files to be open if each ldirectord had to open all other control
files to determine throttling/digesting which was why i was suggesting the use of a single control file for all processes.

anthony

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

eric.robinson at psmnv

Nov 9, 2009, 3:12 PM

Post #11 of 12 (316 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Hi Anthony,

> i am not sure if eric is using for example a single config
> file/ldirectord process with 300 services in it, or 300
> config files/ldirectord processes each with a single service.

I just have one ldirectord process and one config file. It's currently a
little over 2100 lines long, but it still seems quite manageable because
the different services are broken into sections. I have a tomcat
section, a MySQL section, etc.

I agree with Horms that a global diretive would be very useful. In fact,
I would generally prefer it to a per-service directive (but the latter
is still necessary to create exceptions).

--
Eric Robinson

Disclaimer - November 9, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

eric.robinson at psmnv

Nov 9, 2009, 3:24 PM

Post #12 of 12 (316 views)
Permalink

Re: [lvs-users] ldirectord email features [In reply to]

Also, the digest idea is still somewhat problematic because the max
length of an SMS message is too short, so 300 cell-phone texts might get
trimmed down to ~150 texts, which is better but still a lot. It would be
great if the digest were very simplified, something like this:

From: ldirectord1[at]mydomain.com
<IP>:<PORT> DOWN
<IP>:<PORT> DOWN
<IP>:<PORT> DOWN
<IP>:<PORT> DOWN
<IP>:<PORT> DOWN

--
Eric Robinson

Disclaimer - November 9, 2009
This email and any files transmitted with it are confidential and intended solely for LinuxVirtualServer.org users mailing list.. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management and Physician's Managed Care. Warning: Although Physician Select Management and Physician's Managed Care have taken reasonable precautions to ensure no viruses are present in this email, the companies cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
This disclaimer was added by Policy Patrol: http://www.policypatrol.com/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com