m.e at acm
May 19, 2008, 4:34 AM
Post #2 of 9
(287 views)
Permalink
|
|
Re: UTF-16 surrogate message when writing binary data(image)
[In reply to]
|
|
I've finally had the opportunity to test my code a bit more. The $pic_image
is a string of some kind and not a file handle.
Also I tried doing binmode STDOUT but this fails -- the binmode call
returns false.
There could be any binary data in my string, so any UTF-8-bad byte pairs are
definitely possible. I would think the solution is to inform Catalyst that
the data is not UTF-8 but binary. In the normal case of writing to a file,
one can use binmode to tell perl that the data is binary but that does not
work here.
Is this a gap in the Catalyst API?
2008/5/12 Merlyn Kline <merlyn[at]zynet.net>:
> I've just been looking at this same problem as I see it occasionally in
> my logs. In my test case the warning occurs when writing plain HTML which
> I'm certain doesn't contain any UTF-16 surrogates. After some (too much!)
> poking around, I eventually realised that the reported surrogate is actually
> the size of the output (so in your example case, I'm guessing your output
> was DF98 == 57240 bytes). So that suggests to me that it's something to do
> with the FastCGI protocol header. However a cursory examination of the
> contents of the buffer passed to syswrite doesn't reveal this header to be
> present, and a quick look through the calling stack doesn't reveal any
> routine which might add it. The length of the buffer is, however, always the
> number which is later reported as a UTF-16 surrogate. So I conclude that the
> FastCGI protocol header is being put in place on a lower layer, out of my
> reach.
>
> All this neatly explains the previously opaque distribution of the warning
> in my logs (output size isn't often in the UTF-16 reserved range), and why
> it was so difficult for me to find a reliable, simple test case (every time
> I tried to simplify the test case to isolate the cause, the warning went
> away because the simplified case wasn't in the right size range).
> Unfortunately, it also tells me that I'm not going to be able to fix it on
> this particular attack - I've run out of time so I'm going to have to put it
> aside (again) until I can make another chance to investigate further.
>
> Hope all this helps - I look forward to someone else solving it for me ;)
>
> Merlyn Kline
>
>
> -----Original Message-----
> *From:* Martin Ellison [mailto:]
> *Sent:* 12 May 2008 10:05
> *To:* The elegant MVC web framework
> *Subject:* Re: [Catalyst] UTF-16 surrogate message when writing binary
> data(image)
>
> The error is specifically at /usr/lib/perl/5.8/IO/Handle.pm line 199, which
> is the second syswrite call in
>
> sub syswrite {
> @_ >= 2 && @_ <= 4 or croak 'usage: $io->syswrite(BUF [, LEN [,
> OFFSET]])';
> if (defined($_[2])) {
> syswrite($_[0], $_[1], $_[2], $_[3] || 0);
> } else {
> syswrite($_[0], $_[1]);
> }
> }
>
> The error is in production, so I am adding some trace code to investigate
> further, but results will need to wait until I have pushed the code.
>
> 2008/5/10 Matt S Trout <dbix-class[at]trout.me.uk>:
>
>> On Fri, May 09, 2008 at 02:58:41PM +0800, Martin Ellison wrote:
>> > If I write binary data (a JPEG) using $c->res->body then I get all these
>> > errors
>> >
>> > stderr: UTF-16 surrogate 0xdf98 at /usr/lib/perl/5.8/IO/Handle.pm line
>> 199.
>> >
>> > My code looks like
>> >
>> > $c->res->content_type(q{image/jpeg});
>> > $c->res->header( 'Content-Disposition', q{inline} );
>> > $c->res->body($pic_image);
>> >
>> > Presumably, something is assuming that the output is Unicode text and
>> trying
>> > to interpret it accordingly. These error messages are all over the log,
>> > making it difficult to read, besides any impact the situation may be
>> having
>> > on the output.
>> >
>> > Is there any way to fix this issue (eg something like binmode)?
>>
>> My best guess here is that $pic_image is, or looks like, a file handle,
>> and so when Catalyst is sending the response it's doing so by reading
>> from the filehandle and the error's turning up during $fh->read.
>>
>> It'd of course be easier for you to confirm this, since you have the
>> copy of IO/Handle.pm and the line number - maybe you could look?
>>
>> A good way to check would be to loop reading $pic_image yourself and
>> see if you get the same warning ...
>>
>> --
>> Matt S Trout Need help with your Catalyst or DBIx::Class
>> project?
>> Technical Director
>> http://www.shadowcat.co.uk/catalyst/
>> Shadowcat Systems Ltd. Want a managed development or deployment
>> platform?
>> http://chainsawblues.vox.com/
>> http://www.shadowcat.co.uk/servers/
>>
>> _______________________________________________
>> List: Catalyst[at]lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst[at]lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>
>
>
> --
> Regards,
> Martin
> (m.e[at]acm.org)
> IT: http://methodsupport.com Personal: http://thereisnoend.org
>
>
--
Regards,
Martin
(m.e[at]acm.org)
IT: http://methodsupport.com Personal: http://thereisnoend.org
|
