Mailing List Archive: Perl: porters

[perl #70652] Malformed UTF-8 character in bitwise xor when using $1

Index | Next | Previous | View Threaded

perlbug-followup at perl

Nov 19, 2009, 1:14 PM

Post #1 of 4 (173 views)
Permalink

[perl #70652] Malformed UTF-8 character in bitwise xor when using $1

# New Ticket Created by perl [at] marc-s
# Please include the string: [perl #70652]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=70652 >

This is a bug report for perl from perl [at] marc-s,
generated with the help of perlbug 1.35 running under perl v5.8.8.

#!/usr/bin/perl -w

use warnings;
use strict;

#No errors when using bytes!!!
#use bytes;

#Unicode bug

my $unicodestring = "\x{5454}\x{6655}";
my $normalstring="0\36\4\13\200\0\31V\3\0\320\225\342\26\365\4\0\240\r\2\3\0\242_\2\1\0\2\1\0000\0\b\b\b\b\b\b\b\b";
my $iv="\246\205\236\367]\257\304\276";

print "First we need \$1 to be unicode, otherwise the bug won't occour:\n";
$unicodestring=~/(.)/;
print "\$1 is assigned but not yet unicode: UTF8-Flag(\$1): ",utf8::is_utf8($1),"\n";
my $copy=$1;
print "After we copy \$1 the Flag is on: UTF8-Flag(\$1): ",utf8::is_utf8($1),"\n";

print "\nNow we take 8 Bytes of a normal string with m/(.{8})/\n";
print "\$normalstring: UTF8-Flag: ",utf8::is_utf8($normalstring),"\n";

$normalstring=~/(.{8})/;
print "The UTF-8 Flag of \$1 is still on: UTF8-Flag(\$1): ",utf8::is_utf8($1),"\n";
print "We have a second value called (\$iv) without an UTF-8 Flag : UTF8-Flag(\$iv): ",utf8::is_utf8($iv),"\n";
print "Now the UTF-8 Flag of \$1 is off: UTF8-Flag(\$1): ",utf8::is_utf8($1),"\n";
print "Then we do a bitwise ^ opreation of both into \$x (my \$x=\$1 ^ \$iv), which results in a warning.\n\n\n";
# Uncomment this for a fix:
#my $c=$1;

my $x=$1 ^ $iv;
print "\n\n\$1 is now not UTF-8 anymore UTF8-Flag (\$1): ",utf8::is_utf8($1),"\n";
print "\$x is now UTF-8: UTF8-Flag (\$x): ",utf8::is_utf8($x),"\n";
print "\$iv suddenly is also UTF-8: UTF8-Flag (\$iv): ",utf8::is_utf8($iv),"\n";

print "How come 2 non UTF-8 flagged scalars suddenly turn into UTF-8 when a previously UTF-8 flagged \$1 is involved?\n";
print "This will also not happen if \$1 was not UTF-8 before the second m//";

print "This will not happen if we copy \$1 first into another variable or print \$1 before the operation\nThen the UTF-8 Flag will be removed\n";

# The problem occours in Net::SNMP's encrypt_des routine.
---
Flags:
category=core
severity=low
---
Site configuration information for perl v5.8.8:

Configured by Debian Project at Fri Dec 19 00:43:54 EST 2008.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
Platform:
osname=linux, osvers=2.6.18-6-686, archname=i486-linux-gnu-thread-multi
uname='linux etch 2.6.18-6-686 #1 smp fri dec 12 16:48:28 utc 2008 i686 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.8 -Dsitearch=/usr/local/lib/perl/5.8.8 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='4.1.2 20061115 (prerelease) (Debian 4.1.1-21)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.3.6.so, so=so, useshrplib=true, libperl=libperl.so.5.8.8
gnulibc_version='2.3.6'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:

---
@INC for perl v5.8.8:
/etc/perl
/usr/local/lib/perl/5.8.8
/usr/local/share/perl/5.8.8
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8
/usr/share/perl/5.8
/usr/local/lib/site_perl
.

---
Environment for perl v5.8.8:
HOME=/root
LANG=de_DE.UTF-8
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PERL_BADLANG (unset)
SHELL=/bin/bash

zefram at fysh

Nov 20, 2009, 2:19 AM

Post #2 of 4 (160 views)
Permalink

Re: [perl #70652] Malformed UTF-8 character in bitwise xor when using $1 [In reply to]

perl [at] marc-s (via RT) wrote:
># The problem occours in Net::SNMP's encrypt_des routine.

In that case Net::SNMP has a bug. Bitwise ^ on a string containing
non-octets is not a sane operation for cryptographic use. You want to
either limit the input to Latin-1 (or ASCII) or explicitly encode it
into octets using UTF-8/UTF-16/whatever.

-zefram

pagaltzis at gmx

Nov 20, 2009, 7:52 AM

Post #3 of 4 (157 views)
Permalink

Re: [perl #70652] Malformed UTF-8 character in bitwise xor when using $1 [In reply to]

* Zefram <zefram [at] fysh> [2009-11-20 11:20]:
> Bitwise ^ on a string containing non-octets is not a sane
> operation for cryptographic use.

Is it a sane operation at all? Should it be permitted? Silently?

(Wondering aloud.)

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

ikegami at adaelis

Nov 20, 2009, 1:33 PM

Post #4 of 4 (152 views)
Permalink

Re: [perl #70652] Malformed UTF-8 character in bitwise xor when using $1 [In reply to]

On Thu, Nov 19, 2009 at 4:14 PM, perl [at] marc-s (via RT) <
perlbug-followup [at] perl> wrote:

> # New Ticket Created by perl [at] marc-s
> # Please include the string: [perl #70652]
> # in the subject line of all future correspondence about this issue.
> # <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=70652 >
>
>
> This is a bug report for perl from perl [at] marc-s,
> generated with the help of perlbug 1.35 running under perl v5.8.8.
>

I can't replicate the behaviour you describe in 5.8.0 (had to get is_utf8
from Encode), 5.8.8, 5.8.9, 5.10.0 or 5.10.1. Specifically,

>
> #!/usr/bin/perl -w
>
> use warnings;
> use strict;
>
> #No errors when using bytes!!!
> #use bytes;
>
> #Unicode bug
>
> my $unicodestring = "\x{5454}\x{6655}";
> my
> $normalstring="0\36\4\13\200\0\31V\3\0\320\225\342\26\365\4\0\240\r\2\3\0\242_\2\1\0\2\1\0000\0\b\b\b\b\b\b\b\b";
> my $iv="\246\205\236\367]\257\304\276";
>
>
> print "First we need \$1 to be unicode, otherwise the bug won't occour:\n";
> $unicodestring=~/(.)/;
> print "\$1 is assigned but not yet unicode: UTF8-Flag(\$1):
> ",utf8::is_utf8($1),"\n";
> my $copy=$1;
> print "After we copy \$1 the Flag is on: UTF8-Flag(\$1):
> ",utf8::is_utf8($1),"\n";
>
> print "\nNow we take 8 Bytes of a normal string with m/(.{8})/\n";
> print "\$normalstring: UTF8-Flag: ",utf8::is_utf8($normalstring),"\n";
>
> $normalstring=~/(.{8})/;
> print "The UTF-8 Flag of \$1 is still on: UTF8-Flag(\$1):
> ",utf8::is_utf8($1),"\n";
> print "We have a second value called (\$iv) without an UTF-8 Flag :
> UTF8-Flag(\$iv): ",utf8::is_utf8($iv),"\n";
> print "Now the UTF-8 Flag of \$1 is off: UTF8-Flag(\$1):
> ",utf8::is_utf8($1),"\n";
>

No: Now the UTF-8 Flag of $1 is off: UTF8-Flag($1): 1

> print "Then we do a bitwise ^ opreation of both into \$x (my \$x=\$1 ^
> \$iv), which results in a warning.\n\n\n";
>

No warning

> # Uncomment this for a fix:
> #my $c=$1;
>
> my $x=$1 ^ $iv;
> print "\n\n\$1 is now not UTF-8 anymore UTF8-Flag (\$1):
> ",utf8::is_utf8($1),"\n";
> print "\$x is now UTF-8: UTF8-Flag (\$x): ",utf8::is_utf8($x),"\n";
>

No: $x is now UTF-8: UTF8-Flag ($x):

> print "\$iv suddenly is also UTF-8: UTF8-Flag (\$iv):
> ",utf8::is_utf8($iv),"\n";
>

No: $iv suddenly is also UTF-8: UTF8-Flag ($iv):

>
> print "How come 2 non UTF-8 flagged scalars suddenly turn into UTF-8 when a
> previously UTF-8 flagged \$1 is involved?\n";
> print "This will also not happen if \$1 was not UTF-8 before the second
> m//";
>
> print "This will not happen if we copy \$1 first into another variable or
> print \$1 before the operation\nThen the UTF-8 Flag will be removed\n";
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads