Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Re: GSoC: Replace MS Windows Console with Unicode UI

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


tjreedy at udel

Mar 22, 2009, 8:48 PM

Post #1 of 12 (1605 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI

One of the disappointments of CPython 3.0 on Windows is that the switch
to unicode for text (str), coupled with the continued use of a
unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means
that print can no longer print all str strings, or all legal Python code
(as in a traceback).

Different people have tried and failed fix this bug by fiddling with
'Command Prompt' configeration. This would make a useful summer
project, though I don't know if it would involve enough coding. Call
the project something like 3.0 Unicode UI Improvement.

I see two possibilities.

1) Find an C-coded open-source C-P replacement whose author will license
to PSF and, as needed, modify or integrate it with CPython.

2) IDLE does much better but its support seems to still be imcomplete.
Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's
shell an alternate UI.

If Windows (or other OSes) (to be investigated) does not reliably come
with a full unicode font (at least current BMP), is there a public
domain or open license font that we can include?

Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


v+python at g

Mar 22, 2009, 9:13 PM

Post #2 of 12 (1565 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

On approximately 3/22/2009 8:48 PM, came the following characters from
the keyboard of Terry Reedy:
> One of the disappointments of CPython 3.0 on Windows is that the switch
> to unicode for text (str), coupled with the continued use of a
> unicode-oblivious (obtuse) user interface (MS 'Command Prompt'), means
> that print can no longer print all str strings, or all legal Python code
> (as in a traceback).
>
> Different people have tried and failed fix this bug by fiddling with
> 'Command Prompt' configeration. This would make a useful summer
> project, though I don't know if it would involve enough coding. Call
> the project something like 3.0 Unicode UI Improvement.
>
> I see two possibilities.
>
> 1) Find an C-coded open-source C-P replacement whose author will license
> to PSF and, as needed, modify or integrate it with CPython.
>
> 2) IDLE does much better but its support seems to still be imcomplete.
> Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's
> shell an alternate UI.
>
> If Windows (or other OSes) (to be investigated) does not reliably come
> with a full unicode font (at least current BMP), is there a public
> domain or open license font that we can include?


One can, of course, set CMD into Latin-1 mode (chcp 1252)... not sure
how Python reacts to that, as I've only used it with Perl, until I gave
up on Perl's Unicode support (which someone finally seems to be fixing,
but then there is CPAN to improve). But that doesn't solve the font
problem, nor characters above 255.

One can set CMD into Unicode mode (chcp 65001)... not sure how Python
reacts to that either. But even then...

CMD will only use fixed-width fonts, and none of the standard XP ones
seem to contain all of Unicode. Not sure if that has improved on Vista
or 7, as they don't run here.

It _would_ be nice to get this resolved, somehow.


--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


bignose+hates-spam at benfinney

Mar 22, 2009, 10:25 PM

Post #3 of 12 (1568 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

Terry Reedy <tjreedy [at] udel> writes:

> If Windows (or other OSes) (to be investigated) does not reliably
> come with a full unicode font (at least current BMP), is there a
> public domain or open license font that we can include?

The GNU Unifont at Unifoundry <URL:http://unifoundry.com/> is designed
for this purpose.

--
\ “For my birthday I got a humidifier and a de-humidifier. I put |
`\ them in the same room and let them fight it out.” —Steven Wright |
_o__) |
Ben Finney

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Mar 23, 2009, 12:58 AM

Post #4 of 12 (1563 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

> 2) IDLE does much better but its support seems to still be imcomplete.
> Upgrade tk/tkinter/IDLE (wherever the problems lie) and make IDLE's
> shell an alternate UI.

That is certainly *no* good SoC project. Instead, just report it as
a *specific* bug report (rather than saying "it seems incomplete").

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


tjreedy at udel

Mar 23, 2009, 12:12 PM

Post #5 of 12 (1557 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

Glenn Linderman wrote:

>
> One can set CMD into Unicode mode (chcp 65001)... not sure how Python
> reacts to that either. But even then...

I tried that and others have reported doing so on python-list but no one
has gotten that to work.

> CMD will only use fixed-width fonts, and none of the standard XP ones
> seem to contain all of Unicode. Not sure if that has improved on Vista
> or 7, as they don't run here.
>
> It _would_ be nice to get this resolved, somehow.

Definitely.

tjr



_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


v+python at g

Mar 23, 2009, 2:29 PM

Post #6 of 12 (1553 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

On approximately 3/23/2009 12:12 PM, came the following characters from
the keyboard of Terry Reedy:
> Glenn Linderman wrote:
>
>>
>> One can set CMD into Unicode mode (chcp 65001)... not sure how Python
>> reacts to that either. But even then...
>
> I tried that and others have reported doing so on python-list but no one
> has gotten that to work.


http://support.microsoft.com/kb/247815
http://www.microsoft.com/downloads/details.aspx?familyid=22e69ae4-7e40-4807-8a86-b3d36fab68d3&displaylang=en

(python 3)

import ctypes
k=ctypes.WinDLL('kernel32')
x = k.SetConsoleOutputCP(65001)
if x!= 1:
print("x was ", x )
exit( 1 )
print (''.join(chr(i) for i in range(0x410, 0x430)).encode('utf-8'))

produces a nice b'\xd0\x90\d0\x91....' stream of hex representations of
UTF-8 encoded Unicode characters...

The only thing that seems to be missing is that Python won't emit them
to the screen that way.

So surely some python-dev that is smarter than me, could provide that
magic incantation. Will go search, but that isn't in my current
knowledge banks.

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


songofacandy at gmail

Mar 24, 2009, 10:16 AM

Post #7 of 12 (1542 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

Hi. I'm Japanese and non-ascii charactor user. (cp932)

We have to use "IME" to input non-ascii charactor in Windows.
When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.

So setting codepage to 65001 make output universal but make input ascii-only.
Sit!!!

I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good
IDLE alternative.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Mar 24, 2009, 1:40 PM

Post #8 of 12 (1525 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

> Hi. I'm Japanese and non-ascii charactor user. (cp932)
>
> We have to use "IME" to input non-ascii charactor in Windows.
> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
>
> So setting codepage to 65001 make output universal but make input ascii-only.
> Sit!!!

Is there a code page that still allows IME input, but supports all
of Unicode? I know there is GB18030 - is it any good?

In any case, for *that* problem, I think you need to look for a
different terminal program.

> I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good
> IDLE alternative.

Could well be. To see it integrated into Python (and perhaps replace
IDLE), a lot of steps would need to happen, though. But perhaps you
weren't asking for it to be included in Python - it can also be a good
alternative if it's not included (like several other IDEs)

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


v+python at g

Mar 24, 2009, 5:02 PM

Post #9 of 12 (1531 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

On approximately 3/24/2009 10:16 AM, came the following characters from
the keyboard of INADA Naoki:
> Hi. I'm Japanese and non-ascii charactor user. (cp932)
>
> We have to use "IME" to input non-ascii charactor in Windows.
> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
>
> So setting codepage to 65001 make output universal but make input ascii-only.
> Sit!!!
>
> I hope PyQtShell <http://code.google.com/p/pyqtshell/> become good
> IDLE alternative.


Thanks for the feedback.

So at least one version of the code I posted shows that
programmatically, the code page can be set differently for input and
output, although the last version brought both to 65001. It seems that
the chcp 65001 always does both. If the IME only works for cp932, then
leave input at cp932, and set output to 65001?

I have no idea if that could be a solution for you, but I would be
interested in your results if you find that it is, or isn't, as that
would add to the collective knowledge base about the subject. This is
idea 2, below, where I tried to cover the solution space more broadly.

Looking briefly at the definition of cp932, it seems that it covers most
of the Unicode characters... so perhaps any or several of the following
could happen:

1) the IME could be converted to produce UTF-8 instead of cp932,
allowing use of 65001 for input and output
2) the split code page could be used to avoid the conversion of Unicode
to cp932 for output.
3) Unicode could be converted to cp932 for output, allowing use of cp932
for both input and output.

These are listed in the order of increased overhead for character handling.

Perhaps you could enlighten us all as to the issues with each of these
ideas.

I realize the IME exists today, and is likely coded to use cp932, and
that it would take some work to convert it to produce Unicode. However,
there seems to be a straightforward conversion chart between cp932 and
Unicode at Wikipedia, so perhaps that isn't a huge effort.

It seems that the long term goal of having all software speak Unicode
would increase the efficiency of all software when dealing with
multi-lingual issues, as a common solution can be applied universally,
rather than re-inventing solutions that only work for particular code pages.

But I'm not fully aware of whether or not the design or implementation
of Unicode precludes universal solutions: I have heard rumors that
certain characters must be interpreted differently in different locale
contexts, which seems to be counter to the "one solution fits all"
possibility.

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


songofacandy at gmail

Jul 22, 2009, 6:15 PM

Post #10 of 12 (1151 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

Hi.

>> We have to use "IME" to input non-ascii charactor in Windows.
>> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe.
>>
>> So setting codepage to 65001 make output universal but make input ascii-only.
>> Sit!!!
>
> Is there a code page that still allows IME input, but supports all
> of Unicode? I know there is GB18030 - is it any good?

I found WriteConsoleW() API recently.
This API can write utf16 string to console directly, without change
OutputCodepage.

example:
http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42

I think this API is good for py3k.
When stdout is console and not redirected to [pipe|file],
sys.stdout.write(u"foo")
can avoid encoding and use WriteConsoleW(L"foo")

--
Naoki INADA <songofacandy [at] gmail>
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


aahz at pythoncraft

Jul 22, 2009, 8:03 PM

Post #11 of 12 (1148 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

On Thu, Jul 23, 2009, INADA Naoki wrote:
>
> I found WriteConsoleW() API recently.
> This API can write utf16 string to console directly, without change
> OutputCodepage.
>
> example:
> http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42
>
> I think this API is good for py3k.
> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

Please submit a feature request to bugs.python.org -- with a patch would
be even nicer, of course.
--
Aahz (aahz [at] pythoncraft) <*> http://www.pythoncraft.com/

"The volume of a pizza of thickness 'a' and radius 'z' is
given by pi*z*z*a"
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


martin at v

Jul 22, 2009, 8:36 PM

Post #12 of 12 (1136 views)
Permalink
Re: GSoC: Replace MS Windows Console with Unicode UI [In reply to]

> When stdout is console and not redirected to [pipe|file],
> sys.stdout.write(u"foo")
> can avoid encoding and use WriteConsoleW(L"foo")

I think this would be fairly difficult to implement given
the way the output libraries work. If you think it can be
done, please provide a patch to bugs.python.org.

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.