Mailing List Archive: vpnc: devel

Dead Peer Detection fix

Index | Next | Previous | View Threaded

mihai at xcyb

Dec 28, 2010, 2:09 AM

Post #1 of 10 (2974 views)
Permalink

Dead Peer Detection fix

Hello,

The Dead Peer Detection mechanism implemented in vpnc has a small
issue. A lot of people are complaining about connections being dropped
with the following error message logged:

"connection terminated by dead peer detection"

This is caused by vpnc sending incorrect sequence numbers to the VPN
concentrator. The sequence numbers should be sent in big-endian byte
order, however they are sent in little-endian byte-order (on some
architectures).

The initial SeqNo sent by vpnc is randomly chosen and it is
subsequently incremented. However when the least significant byte
rolls over (from vpnc's point of view) the other end notices that the
current SeqNo is less than the previous SeqNo.

This is an error message from a Cisco ASA showing the problem:

Dec 20 02:48:26 10.233.170.24 %ASA-6-713124: Group = somegroup,
Username = user.name, IP = 2.2.1.1, Received DPD sequence number
0x952942 in R_U_THERE, Next expected sequence number should be greater
than 0xff942942

vpnc SeqNo real SeqNo
----------- -----------
42 29 94 fd fd 94 29 42
42 29 94 fe fe 94 29 42
42 29 94 ff ff 94 29 42
42 29 95 00 00 95 29 42 <--- real SeqNo is less than previous one

Depending on the initial [random] sequence number, the bug will
terminate the connection on the first DPD sent or it could survive 255
DPDs (in the most favourable case).

I created a patch [1] that uses htonl/ntohl macros to fix this.

Best regards,
Mihai Maties

[1] http://xcyb.org/vpnc/dpd_big-endian.diff
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

jmvpnc at loplof

Dec 30, 2010, 4:27 AM

Post #2 of 10 (2911 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

On Tue, Dec 28, 2010 at 12:09:26PM +0200, Mihai Maties wrote:
> The Dead Peer Detection mechanism implemented in vpnc has a small
> issue. A lot of people are complaining about connections being dropped
> with the following error message logged:
>
> "connection terminated by dead peer detection"
...
> I created a patch [1] that uses htonl/ntohl macros to fix this.

Great find! I don't know how much time I spent trying to find out the
cause.
I've used your idea and created a different patch. The difference I intend
with my version of the patch is to convert the data to native format
immediately on receiving the data and only converting it back to network
order when sending. The patch is compile tested only.
Can you please test and report back? I'll be on holiday for the next three
week so don't worry if I don't get back immediately - I *will* add this
fix in one form or another.

Thanks!
Joerg
--
Joerg Mayer <jmayer [at] loplof>
We are stuck with technology when what we really want is just stuff that
works. Some say that should read Microsoft instead of technology.

Attachments:

dpd-fix.diff (0.91 KB)

mkienenb at gmail

Dec 30, 2010, 8:43 AM

Post #3 of 10 (2915 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

Mihai,

Thank you!

I merged your patch into my download of the OpenSUSE 11.3
vpnc-0.5.3r449-5.1.src.rpm on Tuesday morning, rebuild the rpm, and
reinstalled, and I haven't had a dropped vpn connection since then.

Joerg,

If I have a chance, I will also try your patch as an alternative.

On Tue, Dec 28, 2010 at 5:09 AM, Mihai Maties <mihai [at] xcyb> wrote:
> Hello,
>
> The Dead Peer Detection mechanism implemented in vpnc has a small
> issue. A lot of people are complaining about connections being dropped
> with the following error message logged:
>
> "connection terminated by dead peer detection"
>
> This is caused by vpnc sending incorrect sequence numbers to the VPN
> concentrator. The sequence numbers should be sent in big-endian byte
> order, however they are sent in little-endian byte-order (on some
> architectures).
>
> The initial SeqNo sent by vpnc is randomly chosen and it is
> subsequently incremented. However when the least significant byte
> rolls over (from vpnc's point of view) the other end notices that the
> current SeqNo is less than the previous SeqNo.
>
> This is an error message from a Cisco ASA showing the problem:
>
> Dec 20 02:48:26 10.233.170.24 %ASA-6-713124: Group = somegroup,
> Username = user.name, IP = 2.2.1.1, Received DPD sequence number
> 0x952942 in R_U_THERE, Next expected sequence number should be greater
> than 0xff942942
>
> vpnc SeqNo real SeqNo
> ----------- -----------
> 42 29 94 fd fd 94 29 42
> 42 29 94 fe fe 94 29 42
> 42 29 94 ff ff 94 29 42
> 42 29 95 00 00 95 29 42 <--- real SeqNo is less than previous one
>
>
> Depending on the initial [random] sequence number, the bug will
> terminate the connection on the first DPD sent or it could survive 255
> DPDs (in the most favourable case).
>
> I created a patch [1] that uses htonl/ntohl macros to fix this.
>
>
> Best regards,
> Mihai Maties
>
>
> [1] http://xcyb.org/vpnc/dpd_big-endian.diff
> _______________________________________________
> vpnc-devel mailing list
> vpnc-devel [at] unix-ag
> https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
> http://www.unix-ag.uni-kl.de/~massar/vpnc/
>
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

mmokrejs at fold

Dec 30, 2010, 11:48 AM

Post #4 of 10 (2959 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

Hi Joerg,
I tried your patch dpd-fix.diff and it breaks my vpnc. So far I
had no problems with
"connection terminated by dead peer detection", now I do have. ;-)

Once it created tunX device but it disappeared in a minute or so,
once although vpnc created the tunnel it died too quickly. I am on
i686-based computer. I havne't seen so far these messages in
/var/log/messages .

Another user whom I have forwarded your patch replied also with
bad results:

<quote>
Thanks for the pointers. I tried the patch, but it is actually worse.
The latest Gentoo version gets me this:

VPNC started in foreground...
lifetime status: 0 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
got r-u-there ack
lifetime status: 300 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 300 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
got r-u-there ack
lifetime status: 600 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 601 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
got r-u-there ack
lifetime status: 900 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 900 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
got r-u-there ack

After applying the patch the r-u-there acks are ignored and I get a
disconnect very soon:

VPNC started in foreground...
lifetime status: 0 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
lifetime status: 5 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 5 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
lifetime status: 10 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 10 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
lifetime status: 15 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 15 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
lifetime status: 20 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 20 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
lifetime status: 25 of 3600 seconds used, 0|0 of 4608000 kbytes used
lifetime status: 25 of 3600 seconds used, 0|0 of 4608000 kbytes used
got late ike packet: 92 bytes
ignoring r-u-there ack 788529152 (expecting 795641883)
dead peer detected, terminating
lifetime status: 30 of 3600 seconds used, 0|0 of 4608000 kbytes used

S7.10 send ipsec termination message
[2010-12-30 18:21:17]

S7.11 send isakmp termination message
[2010-12-30 18:21:17]

S8 close_tunnel
[2010-12-30 18:21:17]

S9 cleanup
[2010-12-30 18:21:18]

</quote>

Joerg Mayer wrote:
> On Tue, Dec 28, 2010 at 12:09:26PM +0200, Mihai Maties wrote:
>> The Dead Peer Detection mechanism implemented in vpnc has a small
>> issue. A lot of people are complaining about connections being dropped
>> with the following error message logged:
>>
>> "connection terminated by dead peer detection"
> ...
>> I created a patch [1] that uses htonl/ntohl macros to fix this.
>
> Great find! I don't know how much time I spent trying to find out the
> cause.
> I've used your idea and created a different patch. The difference I intend
> with my version of the patch is to convert the data to native format
> immediately on receiving the data and only converting it back to network
> order when sending. The patch is compile tested only.
> Can you please test and report back? I'll be on holiday for the next three
> week so don't worry if I don't get back immediately - I *will* add this
> fix in one form or another.
>
> Thanks!
> Joerg
>
>
>
> _______________________________________________
> vpnc-devel mailing list
> vpnc-devel [at] unix-ag
> https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
> http://www.unix-ag.uni-kl.de/~massar/vpnc/
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

mihai at xcyb

Dec 30, 2010, 1:10 PM

Post #5 of 10 (2906 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

On Thu, Dec 30, 2010 at 2:27 PM, Joerg Mayer <jmvpnc [at] loplof> wrote:
> Great find! I don't know how much time I spent trying to find out the
> cause.
> I've used your idea and created a different patch. The difference I intend
> with my version of the patch is to convert the data to native format
> immediately on receiving the data and only converting it back to network
> order when sending. The patch is compile tested only.

Your proposed approach should work, but you have to consider the following:

1) the send_dpd function is used to send both R_U_THERE (local
SeqNo) and R_U_THERE_ACK (remote SeqNo) messages. If you change the
byte order of _all_ sequence numbers you will break the DPD ACKs sent
in reply to the VPN concentrator. The remote SeqNos are already
big-endian, changing the byte order again will force disconnection
(due to wrong SeqNos - you will receive R_U_THERE AABBCCDD and you
will reply R_U_THERE_ACK DDCCBBAA).

2) the SeqNo initialization from your patch doesn't look right to
me. The " *(pl->u.n.data) = htonl(seqno) " assignment does not
accomplish what you want.

Best regards,
Mihai
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

mihai at xcyb

Dec 30, 2010, 1:36 PM

Post #6 of 10 (2918 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

On Thu, Dec 30, 2010 at 9:48 PM, Martin Mokrejs
<mmokrejs [at] fold> wrote:
> Hi Joerg,
> I tried your patch dpd-fix.diff and it breaks my vpnc. So far I
> had no problems with
> "connection terminated by dead peer detection"

Probably the VPN concentrator that you are connecting to is not
bothered by the fact that sequence numbers are not in ascending order.
The RFC does not specify what a peer should do if it does not receive
a SeqNo greater than the previous one. Some implementations choose to
refuse to reply, so the connection will get disconnected.

> , now I do have. ;-)
> ...
> Thanks for the pointers. I tried the patch, but it is actually worse.

Check my previous email, Joerg's current patch has two issues. I
admit, theoretically his approach is better than mine, but the
implementation should be changed a bit.

If you really want to test you can try the
http://xcyb.org/vpnc/dpd_big-endian.diff patch. Since you said the DPD
problem was not an issue in your case I don't expect any changes. From
your point of view it'll work as before.

To test how the sequence numbers are incremented you can run vpnc:

./vpnc --dpd-idle 10 --debug 3 --no-detach ./some-vpnc.conf | tee
-a vpnc.log | egrep "THERE|n.data"

... you will get an output like:

n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 79006e7a
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 79006e7a
n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 79006e7b
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 79006e7b
n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 79006e7c
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 79006e7c

The original vpnc will increment the first byte of n.data while the
patched version will increment the last byte.

Best regards,
Mihai
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

mmokrejs at fold

Dec 30, 2010, 3:00 PM

Post #7 of 10 (2915 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

Hi Mihai,

Mihai Maties wrote:
> On Thu, Dec 30, 2010 at 9:48 PM, Martin Mokrejs
> <mmokrejs [at] fold> wrote:
>> Hi Joerg,
>> I tried your patch dpd-fix.diff and it breaks my vpnc. So far I
>> had no problems with
>> "connection terminated by dead peer detection"
>
> Probably the VPN concentrator that you are connecting to is not
> bothered by the fact that sequence numbers are not in ascending order.
> The RFC does not specify what a peer should do if it does not receive
> a SeqNo greater than the previous one. Some implementations choose to
> refuse to reply, so the connection will get disconnected.
>
>> , now I do have. ;-)
>> ...
>> Thanks for the pointers. I tried the patch, but it is actually worse.
>
> Check my previous email, Joerg's current patch has two issues. I
> admit, theoretically his approach is better than mine, but the
> implementation should be changed a bit.
>
> If you really want to test you can try the
> http://xcyb.org/vpnc/dpd_big-endian.diff patch. Since you said the DPD
> problem was not an issue in your case I don't expect any changes. From
> your point of view it'll work as before.

Yes, your patch does not break anything for me. ;)

n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 3f9ccc21
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 3f9ccc21
n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 3f9ccc22
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 3f9ccc22
n.type: 8d28 (ISAKMP_N_R_U_THERE)
n.data: 3f9ccc23
n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
n.data: 3f9ccc23

Remote Application Version: Cisco Systems, Inc ASA5520 Version 8.3(2) built by builders on Fri 30-Jul-10 17:49

;-)

Thanks,
Martin

>
> To test how the sequence numbers are incremented you can run vpnc:
>
> ./vpnc --dpd-idle 10 --debug 3 --no-detach ./some-vpnc.conf | tee
> -a vpnc.log | egrep "THERE|n.data"
>
> ... you will get an output like:
>
> n.type: 8d28 (ISAKMP_N_R_U_THERE)
> n.data: 79006e7a
> n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
> n.data: 79006e7a
> n.type: 8d28 (ISAKMP_N_R_U_THERE)
> n.data: 79006e7b
> n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
> n.data: 79006e7b
> n.type: 8d28 (ISAKMP_N_R_U_THERE)
> n.data: 79006e7c
> n.type: 8d29 (ISAKMP_N_R_U_THERE_ACK)
> n.data: 79006e7c
>
> The original vpnc will increment the first byte of n.data while the
> patched version will increment the last byte.
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

jmvpnc at loplof

Dec 30, 2010, 3:46 PM

Post #8 of 10 (2905 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

Hello Martin,

On Thu, Dec 30, 2010 at 08:48:40PM +0100, Martin Mokrejs wrote:
> I tried your patch dpd-fix.diff and it breaks my vpnc. So far I
> had no problems with
> "connection terminated by dead peer detection", now I do have. ;-)

OK, thanks for the quick test. That means, I will need to look at the
code in more detail and if I can't come up with the solution as I intend,
your patch will do. As mentioned before: It will take 3 weeks or so.

Ciao
Joerg

--
Joerg Mayer <jmayer [at] loplof>
We are stuck with technology when what we really want is just stuff that
works. Some say that should read Microsoft instead of technology.
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

jmvpnc at loplof

Dec 30, 2010, 6:09 PM

Post #9 of 10 (2907 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

On Fri, Dec 31, 2010 at 01:02:31AM +0100, Martin Mokrejs wrote:
> why don't you commit it meanwhile? You can always revert it if you want
> later. ;-) Anyways, enjoy your holidays!

You are right!
Committed revision 451.

Thanks
Joerg

--
Joerg Mayer <jmayer [at] loplof>
We are stuck with technology when what we really want is just stuff that
works. Some say that should read Microsoft instead of technology.
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

daveyjatin at gmail

Oct 1, 2012, 3:59 AM

Post #10 of 10 (1166 views)
Permalink

Re: Dead Peer Detection fix [In reply to]

Hi Mihai Maties

I am using ubuntu 12.04 with the vpnc client installed using the
software center. The version of the vpnc is : vpnc 0.5.3r512-2ubuntu1

I am still facing the issue of connections getting dropped after 15-20
mins. Could you please let me know if the fix provided by you as per
this email is available in the build mentioned by me. If not then please
let me know what can be done so that this issue gets fixed for me. I
facing tough time attending meetings. Get frequently disconnected while
the meeting is ongoing.

Appreciate your response and help in this regard.

Thanks
Jatin
_______________________________________________
vpnc-devel mailing list
vpnc-devel [at] unix-ag
https://lists.unix-ag.uni-kl.de/mailman/listinfo/vpnc-devel
http://www.unix-ag.uni-kl.de/~massar/vpnc/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads