Mailing List Archive: RANCID: Users

duplicated characters

Index | Next | Previous | View Threaded

tensai at zmonkey

Sep 1, 2011, 1:27 PM

Post #1 of 6 (648 views)
Permalink

duplicated characters

After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
characters in config diffs. Here's an example:

- access-list 102 deny udp any any eq 135
+ access-list 102 denny udp any any eq 135
- access-list 110 deny tcp any any eq 445 log
+ access-list 110 deny tcp any any eq 445 log

The next round of updates will revert the change back. I haven't found
any pattern to which character it will be or even which router. It's
only happening with my Cisco devices, not any of our others (Foundry
switches, Mikrotiks or Lucent TNTs). I'm using 2.3.1, but a fresh
install of 2.3.6 on the same server does the same thing. 2.3.6 on my
Ubuntu 10.04 laptop works just fine.

To verify whether the routers themselves were errant, I did a packet
capture. That confirmed that the data from the router is correct. I
turned on debugging in clogin. WriteTerm shows the duplicate characters.
So then I put in a puts in run_commands in rancid and that also showed
duplicate characters in $expect_out(buffer).

I found this thread from 2004 which hinted at it being an issue with the
telnet client. The OP eventually switched to ssh which in this case
isn't an option.

http://www.gossamer-threads.com/lists/rancid/users/720

The incessant emails are driving me batty and making the cvs history
significantly less useful. I'll debug whatever needs to be debugged but
I'm fresh out of ideas of where to look. I started looking at expect but
was quickly lost in the code.

Any ideas?

Corey
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

heas at shrubbery

Sep 3, 2011, 1:08 PM

Post #2 of 6 (583 views)
Permalink

Re: duplicated characters [In reply to]

Thu, Sep 01, 2011 at 02:27:10PM -0600, Corey Edwards:
> After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
> characters in config diffs. Here's an example:
>
> - access-list 102 deny udp any any eq 135
> + access-list 102 denny udp any any eq 135
> - access-list 110 deny tcp any any eq 445 log
> + access-list 110 deny tcp any any eq 445 log
>
> The next round of updates will revert the change back. I haven't found
> any pattern to which character it will be or even which router. It's
> only happening with my Cisco devices, not any of our others (Foundry
> switches, Mikrotiks or Lucent TNTs). I'm using 2.3.1, but a fresh
> install of 2.3.6 on the same server does the same thing. 2.3.6 on my
> Ubuntu 10.04 laptop works just fine.
>
> To verify whether the routers themselves were errant, I did a packet
> capture. That confirmed that the data from the router is correct. I
> turned on debugging in clogin. WriteTerm shows the duplicate characters.
> So then I put in a puts in run_commands in rancid and that also showed
> duplicate characters in $expect_out(buffer).
>
> I found this thread from 2004 which hinted at it being an issue with the
> telnet client. The OP eventually switched to ssh which in this case
> isn't an option.
>
> http://www.gossamer-threads.com/lists/rancid/users/720

i dont know what the fbsd fix was for this, but they fixed it somehow in
their tcl or expect port.

perhaps its related to a problem that came up in rancid 2.3.5 something
changed in debian's libc that caused problems with string handling
functions, which i believe was related to internationalization.

or could be a bug in buffer handling in expect or tcl.

either way, if you can capture clogin -d -c 'show running' o/p where this
occurs, it should at least reveal where the bug is.

you can also try rebuilding tcl and expect w/o internationalization.

> The incessant emails are driving me batty and making the cvs history
> significantly less useful. I'll debug whatever needs to be debugged but
> I'm fresh out of ideas of where to look. I started looking at expect but
> was quickly lost in the code.
>
> Any ideas?
>
> Corey
> _______________________________________________
> Rancid-discuss mailing list
> Rancid-discuss [at] shrubbery
> http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

tensai at zmonkey

Sep 15, 2011, 8:52 AM

Post #3 of 6 (559 views)
Permalink

Re: duplicated characters [In reply to]

On 09/03/2011 02:08 PM, john heasley wrote:
> Thu, Sep 01, 2011 at 02:27:10PM -0600, Corey Edwards:
>> After a recent upgrade from Debian 5.0 to 6.0 I began seeing duplicate
>> characters in config diffs. Here's an example:
>>
>> - access-list 102 deny udp any any eq 135
>> + access-list 102 denny udp any any eq 135
>> - access-list 110 deny tcp any any eq 445 log
>> + access-list 110 deny tcp any any eq 445 log
>>
>> I found this thread from 2004 which hinted at it being an issue with the
>> telnet client. The OP eventually switched to ssh which in this case
>> isn't an option.
>>
>> http://www.gossamer-threads.com/lists/rancid/users/720
>
> i dont know what the fbsd fix was for this, but they fixed it somehow in
> their tcl or expect port.
>
> perhaps its related to a problem that came up in rancid 2.3.5 something
> changed in debian's libc that caused problems with string handling
> functions, which i believe was related to internationalization.
>
> or could be a bug in buffer handling in expect or tcl.

I moved the RANCID install to a separate server running Debian 6.0 and
it's working just fine, so it appears to be something left over from the
upgrade from 5.0. I just can't imagine what it is.

> either way, if you can capture clogin -d -c 'show running' o/p where this
> occurs, it should at least reveal where the bug is.

I captured this output, but unfortunately it was rife with passwords. I
trimmed it down to what I hope is still a useful summary. In this
example the duplicated character showed up in the word "permitt".

http://pastebin.com/epUCPr2p

> you can also try rebuilding tcl and expect w/o internationalization.

I could not find any way to do this.

Corey

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

heas at shrubbery

Sep 20, 2011, 9:25 AM

Post #4 of 6 (562 views)
Permalink

Re: duplicated characters [In reply to]

Thu, Sep 15, 2011 at 09:52:33AM -0600, Corey Edwards:
> > either way, if you can capture clogin -d -c 'show running' o/p where this
> > occurs, it should at least reveal where the bug is.
>
> I captured this output, but unfortunately it was rife with passwords. I
> trimmed it down to what I hope is still a useful summary. In this
> example the duplicated character showed up in the word "permitt".
>
> http://pastebin.com/epUCPr2p

-expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
+expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no

its occuring inside tcl or expect. i suspect its the same problem that i
had with hpuifilter when some linux users started upgrading. I dont know
what the cause is, but i'd look for old versions of libc and internationization
libraries that tcl or expect are linked with.

> > you can also try rebuilding tcl and expect w/o internationalization.
>
> I could not find any way to do this.

hrm, there doesnt appear to be an easy way to disable it in expect.
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

tensai at zmonkey

Sep 26, 2011, 8:48 AM

Post #5 of 6 (530 views)
Permalink

Re: duplicated characters (SOLVED) [In reply to]

On 09/20/2011 10:25 AM, john heasley wrote:
> Thu, Sep 15, 2011 at 09:52:33AM -0600, Corey Edwards:
>>> either way, if you can capture clogin -d -c 'show running' o/p where this
>>> occurs, it should at least reveal where the bug is.
>>
>> I captured this output, but unfortunately it was rife with passwords. I
>> trimmed it down to what I hope is still a useful summary. In this
>> example the duplicated character showed up in the word "permitt".
>>
>> http://pastebin.com/epUCPr2p
>
> -expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
> +expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
>
> its occuring inside tcl or expect. i suspect its the same problem that i
> had with hpuifilter when some linux users started upgrading. I dont know
> what the cause is, but i'd look for old versions of libc and internationization
> libraries that tcl or expect are linked with.

I ran clogin through strace and looked at all the files that were
touched. A comparison between the working server and the upgraded server
revealed some kerberos packages (libkrb5-dev, krb5-config,
krb5-clients). After removing those, everything is working perfectly. I
guess a kerberized version of telnet must have been triggering this bug.

Corey
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

heas at shrubbery

Sep 26, 2011, 8:58 AM

Post #6 of 6 (532 views)
Permalink

Re: duplicated characters (SOLVED) [In reply to]

Mon, Sep 26, 2011 at 09:48:51AM -0600, Corey Edwards:
> > -expect: does "access-list 1 permit x.x.x.101\r\naccess-li" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
> > +expect: does "access-list 1 permitt x.x.x.101\r\naccess-l" (spawn_id exp6) match regular expression "\u0008+"? (No Gate, RE only) gate=yes re=no
> >
> > its occuring inside tcl or expect. i suspect its the same problem that i
> > had with hpuifilter when some linux users started upgrading. I dont know
> > what the cause is, but i'd look for old versions of libc and internationization
> > libraries that tcl or expect are linked with.
>
> I ran clogin through strace and looked at all the files that were
> touched. A comparison between the working server and the upgraded server
> revealed some kerberos packages (libkrb5-dev, krb5-config,
> krb5-clients). After removing those, everything is working perfectly. I
> guess a kerberized version of telnet must have been triggering this bug.

ah, there is a telnet that is installed with the heimdal package on bsd
that weird stuff too; unsolicited, it prints to the controlling tty in
rather random way, which confuses the login process if the timing is
right.
_______________________________________________
Rancid-discuss mailing list
Rancid-discuss [at] shrubbery
http://www.shrubbery.net/mailman/listinfo.cgi/rancid-discuss

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads