Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: nsp: foundry

ServerIron dropping empty UDP DNS reply

 

 

nsp foundry RSS feed   Index | Next | Previous | View Threaded


dhubbard at dino

Jun 21, 2012, 5:37 PM

Post #1 of 7 (1045 views)
Permalink
ServerIron dropping empty UDP DNS reply

Hi all, we're using ServerIron's to load balance internal
DNS queries to a series of systems running a caching dns
server software called dnscache (part of djbdns). The
way dnscache works is it sends an empty reply if it has no
answer to a given query. Apparently the ServerIron's drop
these valid replies so the querying system has to wait for a
timeout to the failed lookup instead of knowing it failed
immediately.

This had not really been a problem in the past however RHEL 6
and most of its applications and daemons are now IPv6-aware
and will issue quad-A queries for a given name even if IPv6
is disabled on the host in question. That's causing usability
issues at the application level because every DNS lookup
causes a five second pause while the app waits for the quad-A
answer that the ServerIron has discarded.

The correct reply looks something like this; just changed
the names and query, hex is original:

20:27:41.791038 IP HOST.60850 > CACHE.domain: 54370+ AAAA?
012345.012345678901.net. (41)
0x0000: 4500 0045 90d7 4000 4011 9c57 d04d 3027
E..E..@.@..W.M0'
0x0010: cc0a 40fa edb2 0035 0031 0dbc d462 0100
..@....5.1...b..
0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
.........012345.
0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
012345678901.net
0x0040: 0000 1c00 01 .....
20:27:41.810226 IP CACHE.domain > HOST.60850: 54370 0/0/0 (41)
0x0000: 4500 0045 0000 4000 3e11 2f2f cc0a 40fa
E..E..@.>.//..@.
0x0010: d04d 3027 0035 edb2 0031 bee4 d462 8180
.M0'.5...1...b..
0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
.........012345.
0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
012345678901.net
0x0040: 0000 1c00 01 .....

The same request sent to the ServerIron makes it to one of the
servers behind the ServerIron, the same response is sent back
to the serverIron (using source-nat on the 'server real'
definitions), and then it disappears without going back to the
requesting host.

Any idea of parameters I can set to let those make it back through,
known bug, etc.?

Thanks,

David

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


bjorn at mork

Jun 22, 2012, 4:48 AM

Post #2 of 7 (997 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

David Hubbard <dhubbard [at] dino> writes:

> Hi all, we're using ServerIron's to load balance internal
> DNS queries to a series of systems running a caching dns
> server software called dnscache (part of djbdns). The
> way dnscache works is it sends an empty reply if it has no
> answer to a given query. Apparently the ServerIron's drop
> these valid replies so the querying system has to wait for a
> timeout to the failed lookup instead of knowing it failed
> immediately.
>
> This had not really been a problem in the past however RHEL 6
> and most of its applications and daemons are now IPv6-aware
> and will issue quad-A queries for a given name even if IPv6
> is disabled on the host in question. That's causing usability
> issues at the application level because every DNS lookup
> causes a five second pause while the app waits for the quad-A
> answer that the ServerIron has discarded.

How about configuring Direct Server Return? That would prevent the
ServerIron from processing the replies at all, which would give you
twice as many sessions too.


Bjørn

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


drew.weaver at thenap

Jun 22, 2012, 7:05 AM

Post #3 of 7 (1000 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

We actually just fixed this problem in our environment the issue is that RHEL 6 is sending multiple requests in the same connection and the serveriron isn't expecting this.

We had to do this to get around it.

udp-age 2
port dns udp-normal-age

it sucks because it keeps the dns connections open for 2 minutes, but it sucks less than the 10~ second delay.

Brocade really needs to shorten the minimum udp age.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck [mailto:foundry-nsp-bounces [at] puck] On Behalf Of David Hubbard
Sent: Thursday, June 21, 2012 8:38 PM
To: foundry-nsp [at] puck
Subject: [f-nsp] ServerIron dropping empty UDP DNS reply

Hi all, we're using ServerIron's to load balance internal DNS queries to a series of systems running a caching dns server software called dnscache (part of djbdns). The way dnscache works is it sends an empty reply if it has no answer to a given query. Apparently the ServerIron's drop these valid replies so the querying system has to wait for a timeout to the failed lookup instead of knowing it failed immediately.

This had not really been a problem in the past however RHEL 6 and most of its applications and daemons are now IPv6-aware and will issue quad-A queries for a given name even if IPv6 is disabled on the host in question. That's causing usability issues at the application level because every DNS lookup causes a five second pause while the app waits for the quad-A answer that the ServerIron has discarded.

The correct reply looks something like this; just changed the names and query, hex is original:

20:27:41.791038 IP HOST.60850 > CACHE.domain: 54370+ AAAA?
012345.012345678901.net. (41)
0x0000: 4500 0045 90d7 4000 4011 9c57 d04d 3027 E..E..@.@..W.M0'
0x0010: cc0a 40fa edb2 0035 0031 0dbc d462 0100 ..@....5.1...b..
0x0020: 0001 0000 0000 0000 0677 6562 3030 330c .........012345.
0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574 012345678901.net
0x0040: 0000 1c00 01 .....
20:27:41.810226 IP CACHE.domain > HOST.60850: 54370 0/0/0 (41)
0x0000: 4500 0045 0000 4000 3e11 2f2f cc0a 40fa E..E..@.>.//..@.
0x0010: d04d 3027 0035 edb2 0031 bee4 d462 8180 .M0'.5...1...b..
0x0020: 0001 0000 0000 0000 0677 6562 3030 330c .........012345.
0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574 012345678901.net
0x0040: 0000 1c00 01 .....

The same request sent to the ServerIron makes it to one of the servers behind the ServerIron, the same response is sent back to the serverIron (using source-nat on the 'server real'
definitions), and then it disappears without going back to the requesting host.

Any idea of parameters I can set to let those make it back through, known bug, etc.?

Thanks,

David

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


dhubbard at dino

Jun 22, 2012, 2:01 PM

Post #4 of 7 (997 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

Ah, yep, same exact issue. I ended up just doing
Bjrn's suggestion of direct server return, problem
is now gone and didn't have to worry about the
session table since that could have posed an issue
with several thousand servers querying the VIP.

Thanks,

Dave

> -----Original Message-----
> From: Drew Weaver [mailto:drew.weaver [at] thenap]
> Sent: Friday, June 22, 2012 10:05 AM
> To: David Hubbard; foundry-nsp [at] puck
> Subject: RE: ServerIron dropping empty UDP DNS reply
>
> We actually just fixed this problem in our environment the
> issue is that RHEL 6 is sending multiple requests in the same
> connection and the serveriron isn't expecting this.
>
> We had to do this to get around it.
>
> udp-age 2
> port dns udp-normal-age
>
> it sucks because it keeps the dns connections open for 2
> minutes, but it sucks less than the 10~ second delay.
>
> Brocade really needs to shorten the minimum udp age.
>
> Thanks,
> -Drew
>
> -----Original Message-----
> From: foundry-nsp-bounces [at] puck
> [mailto:foundry-nsp-bounces [at] puck] On Behalf Of
> David Hubbard
> Sent: Thursday, June 21, 2012 8:38 PM
> To: foundry-nsp [at] puck
> Subject: [f-nsp] ServerIron dropping empty UDP DNS reply
>
> Hi all, we're using ServerIron's to load balance internal DNS
> queries to a series of systems running a caching dns server
> software called dnscache (part of djbdns). The way dnscache
> works is it sends an empty reply if it has no answer to a
> given query. Apparently the ServerIron's drop these valid
> replies so the querying system has to wait for a timeout to
> the failed lookup instead of knowing it failed immediately.
>
> This had not really been a problem in the past however RHEL 6
> and most of its applications and daemons are now IPv6-aware
> and will issue quad-A queries for a given name even if IPv6
> is disabled on the host in question. That's causing
> usability issues at the application level because every DNS
> lookup causes a five second pause while the app waits for the
> quad-A answer that the ServerIron has discarded.
>
> The correct reply looks something like this; just changed the
> names and query, hex is original:
>
> 20:27:41.791038 IP HOST.60850 > CACHE.domain: 54370+ AAAA?
> 012345.012345678901.net. (41)
> 0x0000: 4500 0045 90d7 4000 4011 9c57 d04d 3027
> E..E..@.@..W.M0'
> 0x0010: cc0a 40fa edb2 0035 0031 0dbc d462 0100
> ..@....5.1...b..
> 0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
> .........012345.
> 0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
> 012345678901.net
> 0x0040: 0000 1c00 01 .....
> 20:27:41.810226 IP CACHE.domain > HOST.60850: 54370 0/0/0 (41)
> 0x0000: 4500 0045 0000 4000 3e11 2f2f cc0a 40fa
> E..E..@.>.//..@.
> 0x0010: d04d 3027 0035 edb2 0031 bee4 d462 8180
> .M0'.5...1...b..
> 0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
> .........012345.
> 0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
> 012345678901.net
> 0x0040: 0000 1c00 01 .....
>
> The same request sent to the ServerIron makes it to one of
> the servers behind the ServerIron, the same response is sent
> back to the serverIron (using source-nat on the 'server real'
> definitions), and then it disappears without going back to
> the requesting host.
>
> Any idea of parameters I can set to let those make it back
> through, known bug, etc.?
>
> Thanks,
>
> David
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


drew.weaver at thenap

Jun 22, 2012, 4:02 PM

Post #5 of 7 (1004 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

Okay,

Unless you have more than 4,368 requests/sec the session table shouldn't ever fill up even on a ServerIron XL which has 524,000 sessions.

But DSR works too.

Thanks,
-Drew


-----Original Message-----
From: foundry-nsp-bounces [at] puck [mailto:foundry-nsp-bounces [at] puck] On Behalf Of David Hubbard
Sent: Friday, June 22, 2012 5:02 PM
To: foundry-nsp [at] puck
Subject: Re: [f-nsp] ServerIron dropping empty UDP DNS reply

Ah, yep, same exact issue. I ended up just doing Bjrn's suggestion of direct server return, problem is now gone and didn't have to worry about the session table since that could have posed an issue with several thousand servers querying the VIP.

Thanks,

Dave

> -----Original Message-----
> From: Drew Weaver [mailto:drew.weaver [at] thenap]
> Sent: Friday, June 22, 2012 10:05 AM
> To: David Hubbard; foundry-nsp [at] puck
> Subject: RE: ServerIron dropping empty UDP DNS reply
>
> We actually just fixed this problem in our environment the issue is
> that RHEL 6 is sending multiple requests in the same connection and
> the serveriron isn't expecting this.
>
> We had to do this to get around it.
>
> udp-age 2
> port dns udp-normal-age
>
> it sucks because it keeps the dns connections open for 2 minutes, but
> it sucks less than the 10~ second delay.
>
> Brocade really needs to shorten the minimum udp age.
>
> Thanks,
> -Drew
>
> -----Original Message-----
> From: foundry-nsp-bounces [at] puck
> [mailto:foundry-nsp-bounces [at] puck] On Behalf Of David
> Hubbard
> Sent: Thursday, June 21, 2012 8:38 PM
> To: foundry-nsp [at] puck
> Subject: [f-nsp] ServerIron dropping empty UDP DNS reply
>
> Hi all, we're using ServerIron's to load balance internal DNS queries
> to a series of systems running a caching dns server software called
> dnscache (part of djbdns). The way dnscache works is it sends an
> empty reply if it has no answer to a given query. Apparently the
> ServerIron's drop these valid replies so the querying system has to
> wait for a timeout to the failed lookup instead of knowing it failed
> immediately.
>
> This had not really been a problem in the past however RHEL 6 and most
> of its applications and daemons are now IPv6-aware and will issue
> quad-A queries for a given name even if IPv6 is disabled on the host
> in question. That's causing usability issues at the application level
> because every DNS lookup causes a five second pause while the app
> waits for the quad-A answer that the ServerIron has discarded.
>
> The correct reply looks something like this; just changed the names
> and query, hex is original:
>
> 20:27:41.791038 IP HOST.60850 > CACHE.domain: 54370+ AAAA?
> 012345.012345678901.net. (41)
> 0x0000: 4500 0045 90d7 4000 4011 9c57 d04d 3027
> E..E..@.@..W.M0'
> 0x0010: cc0a 40fa edb2 0035 0031 0dbc d462 0100
> ..@....5.1...b..
> 0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
> .........012345.
> 0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
> 012345678901.net
> 0x0040: 0000 1c00 01 .....
> 20:27:41.810226 IP CACHE.domain > HOST.60850: 54370 0/0/0 (41)
> 0x0000: 4500 0045 0000 4000 3e11 2f2f cc0a 40fa
> E..E..@.>.//..@.
> 0x0010: d04d 3027 0035 edb2 0031 bee4 d462 8180
> .M0'.5...1...b..
> 0x0020: 0001 0000 0000 0000 0677 6562 3030 330c
> .........012345.
> 0x0030: 6d69 7661 6d65 7263 6861 6e74 036e 6574
> 012345678901.net
> 0x0040: 0000 1c00 01 .....
>
> The same request sent to the ServerIron makes it to one of the servers
> behind the ServerIron, the same response is sent back to the
> serverIron (using source-nat on the 'server real'
> definitions), and then it disappears without going back to the
> requesting host.
>
> Any idea of parameters I can set to let those make it back through,
> known bug, etc.?
>
> Thanks,
>
> David
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


bjorn at mork

Jun 23, 2012, 3:30 AM

Post #6 of 7 (996 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

Drew Weaver <drew.weaver [at] thenap> writes:

> Unless you have more than 4,368 requests/sec the session table
> shouldn't ever fill up even on a ServerIron XL which has 524,000
> sessions.

I believe most ISPs of some size will see a *lot* more than that.
No need to consider a load balancer at all if those numbers sound
reasonable to you.


Bjørn

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp


drew.weaver at thenap

Jun 23, 2012, 8:36 AM

Post #7 of 7 (988 views)
Permalink
Re: ServerIron dropping empty UDP DNS reply [In reply to]

Sorry,

I should've said 4,368 sessions from different source IP addresses/sec, not 4,368 queries/sec the number of queries wouldn't be relevant.

Thanks,
-Drew




-----Original Message-----
From: Bjørn Mork [mailto:bjorn [at] mork]
Sent: Saturday, June 23, 2012 6:30 AM
To: Drew Weaver
Cc: 'David Hubbard'; foundry-nsp [at] puck
Subject: Re: [f-nsp] ServerIron dropping empty UDP DNS reply

Drew Weaver <drew.weaver [at] thenap> writes:

> Unless you have more than 4,368 requests/sec the session table
> shouldn't ever fill up even on a ServerIron XL which has 524,000
> sessions.

I believe most ISPs of some size will see a *lot* more than that.
No need to consider a load balancer at all if those numbers sound reasonable to you.


Bjørn

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

nsp foundry RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.