mark at summersault
Jul 3, 2012, 12:35 PM
Post #3 of 4
On 06/29/2012 04:56 PM, david [at] lang wrote:
Re: Logging silently quits working logging server reboots (on Amazon EC2)
[In reply to]
> On Fri, 29 Jun 2012, Mark Stosberg wrote:
>> I'm using Rsyslog on Ubuntu 12.04 on Amazon's EC2. We're using rsyslog
>> Our web servers log to local rsyslog processes, which then send the logs
>> on to a central logging server.
>> Recently, the central logging server was rebooted. After it came back
>> up, no more logging appeared from our application nodes, until I
>> rebooted rsyslog on the application nodes.
>> I'd like logging to automatically start appearing on the central server
>> after reboots in the future.
>> Here's what I suspect happened:
>> - We'd specified the central logging server so that it resolves to one
>> of amazon's special dual-DNS-horizon host names that are associated with
>> Elastic IPs. Hosts outside of Amazon see a static IP, while hosts inside
>> Amazon (like our app nodes) resolve the host name to a IP address which
>> changes when the server reboots.
>> It appears that rsyslog was aggressive about caching this DNS
>> resolution, and didn't check again to see that the host name of the
>> target server was resolving to a new IP.
>> I could update the application nodes to use the "Elastic IP" of the
>> central logging server. This has the advantage that it never changes,
>> but I think the routing may be less direct than using the dynamic
>> internal IPs.
>> What advice do you to address this situation? Is there a way to get this
>> version of Rsyslog to respect the TTL of the target rsyslog host?
> Mark, I think you are better off getting a fixed IP for this. The reason
> that rsyslog doesn't do frequent lookups is that the overhead of doing
> so can cripple performance.
> There is the option to close and re-open a connection periodically (look
> for the rebind options), this may work for you (although in any case
> there is going to be some delay between a system changing it's IP
> address and clients finding out about the change).
Thanks for the reply! I think I'll try using the static IP or look into
upgrading rsyslog so that we can use the "Rebind" option. I don't see
any Rebind options in the man pages or config files of 4.2.0.
rsyslog mailing list
What's up with rsyslog? Follow https://twitter.com/rgerhards