Mailing List Archive: RSyslog: users

Logging silently quits working logging server reboots (on Amazon EC2)

Index | Next | Previous | View Threaded

mark at summersault

Jun 29, 2012, 12:27 PM

Post #1 of 4 (181 views)
Permalink

Logging silently quits working logging server reboots (on Amazon EC2)

Hello,

I'm using Rsyslog on Ubuntu 12.04 on Amazon's EC2. We're using rsyslog
4.2.0.

Our web servers log to local rsyslog processes, which then send the logs
on to a central logging server.

Recently, the central logging server was rebooted. After it came back
up, no more logging appeared from our application nodes, until I
rebooted rsyslog on the application nodes.

I'd like logging to automatically start appearing on the central server
after reboots in the future.

Here's what I suspect happened:

- We'd specified the central logging server so that it resolves to one
of amazon's special dual-DNS-horizon host names that are associated with
Elastic IPs. Hosts outside of Amazon see a static IP, while hosts inside
Amazon (like our app nodes) resolve the host name to a IP address which
changes when the server reboots.

It appears that rsyslog was aggressive about caching this DNS
resolution, and didn't check again to see that the host name of the
target server was resolving to a new IP.

I could update the application nodes to use the "Elastic IP" of the
central logging server. This has the advantage that it never changes,
but I think the routing may be less direct than using the dynamic
internal IPs.

What advice do you to address this situation? Is there a way to get this
version of Rsyslog to respect the TTL of the target rsyslog host?

Thanks!

Mark

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

Jun 29, 2012, 1:56 PM

Post #2 of 4 (162 views)
Permalink

Re: Logging silently quits working logging server reboots (on Amazon EC2) [In reply to]

On Fri, 29 Jun 2012, Mark Stosberg wrote:

> Hello,
>
> I'm using Rsyslog on Ubuntu 12.04 on Amazon's EC2. We're using rsyslog
> 4.2.0.
>
> Our web servers log to local rsyslog processes, which then send the logs
> on to a central logging server.
>
> Recently, the central logging server was rebooted. After it came back
> up, no more logging appeared from our application nodes, until I
> rebooted rsyslog on the application nodes.
>
> I'd like logging to automatically start appearing on the central server
> after reboots in the future.
>
> Here's what I suspect happened:
>
> - We'd specified the central logging server so that it resolves to one
> of amazon's special dual-DNS-horizon host names that are associated with
> Elastic IPs. Hosts outside of Amazon see a static IP, while hosts inside
> Amazon (like our app nodes) resolve the host name to a IP address which
> changes when the server reboots.
>
> It appears that rsyslog was aggressive about caching this DNS
> resolution, and didn't check again to see that the host name of the
> target server was resolving to a new IP.
>
> I could update the application nodes to use the "Elastic IP" of the
> central logging server. This has the advantage that it never changes,
> but I think the routing may be less direct than using the dynamic
> internal IPs.
>
> What advice do you to address this situation? Is there a way to get this
> version of Rsyslog to respect the TTL of the target rsyslog host?

Mark, I think you are better off getting a fixed IP for this. The reason
that rsyslog doesn't do frequent lookups is that the overhead of doing so
can cripple performance.

There is the option to close and re-open a connection periodically (look
for the rebind options), this may work for you (although in any case there
is going to be some delay between a system changing it's IP address and
clients finding out about the change).

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

mark at summersault

Jul 3, 2012, 12:35 PM

Post #3 of 4 (156 views)
Permalink

Re: Logging silently quits working logging server reboots (on Amazon EC2) [In reply to]

On 06/29/2012 04:56 PM, david [at] lang wrote:
> On Fri, 29 Jun 2012, Mark Stosberg wrote:
>
>> Hello,
>>
>> I'm using Rsyslog on Ubuntu 12.04 on Amazon's EC2. We're using rsyslog
>> 4.2.0.
>>
>> Our web servers log to local rsyslog processes, which then send the logs
>> on to a central logging server.
>>
>> Recently, the central logging server was rebooted. After it came back
>> up, no more logging appeared from our application nodes, until I
>> rebooted rsyslog on the application nodes.
>>
>> I'd like logging to automatically start appearing on the central server
>> after reboots in the future.
>>
>> Here's what I suspect happened:
>>
>> - We'd specified the central logging server so that it resolves to one
>> of amazon's special dual-DNS-horizon host names that are associated with
>> Elastic IPs. Hosts outside of Amazon see a static IP, while hosts inside
>> Amazon (like our app nodes) resolve the host name to a IP address which
>> changes when the server reboots.
>>
>> It appears that rsyslog was aggressive about caching this DNS
>> resolution, and didn't check again to see that the host name of the
>> target server was resolving to a new IP.
>>
>> I could update the application nodes to use the "Elastic IP" of the
>> central logging server. This has the advantage that it never changes,
>> but I think the routing may be less direct than using the dynamic
>> internal IPs.
>>
>> What advice do you to address this situation? Is there a way to get this
>> version of Rsyslog to respect the TTL of the target rsyslog host?
>
> Mark, I think you are better off getting a fixed IP for this. The reason
> that rsyslog doesn't do frequent lookups is that the overhead of doing
> so can cripple performance.
>
> There is the option to close and re-open a connection periodically (look
> for the rebind options), this may work for you (although in any case
> there is going to be some delay between a system changing it's IP
> address and clients finding out about the change).

David,

Thanks for the reply! I think I'll try using the static IP or look into
upgrading rsyslog so that we can use the "Rebind" option. I don't see
any Rebind options in the man pages or config files of 4.2.0.

Mark

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

david at lang

Jul 8, 2012, 4:05 PM

Post #4 of 4 (146 views)
Permalink

Re: Logging silently quits working logging server reboots (on Amazon EC2) [In reply to]

On Tue, 3 Jul 2012, Mark Stosberg wrote:

> On 06/29/2012 04:56 PM, david [at] lang wrote:
>>
>> Mark, I think you are better off getting a fixed IP for this. The reason
>> that rsyslog doesn't do frequent lookups is that the overhead of doing
>> so can cripple performance.
>>
>> There is the option to close and re-open a connection periodically (look
>> for the rebind options), this may work for you (although in any case
>> there is going to be some delay between a system changing it's IP
>> address and clients finding out about the change).
>
> David,
>
> Thanks for the reply! I think I'll try using the static IP or look into
> upgrading rsyslog so that we can use the "Rebind" option. I don't see
> any Rebind options in the man pages or config files of 4.2.0.

according to the page http://www.rsyslog.com/features/ these features were
added between 4.3.1 and 4.5

4.2 is _very_ old and is missing a LOT of features (and bugfixes)
Unfortunantly Red Hat doesn't update versions between major releases, so
they are shipping a version that's almost 5 years old at this point. The
current stable version is 6.2 (but a lot of people are still on 5.8)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads