Mailing List Archive: Python: Python

A thread import problem

Index | Next | Previous | View Threaded

bruce.sherwood at gmail

Jul 18, 2012, 4:03 PM

Post #1 of 20 (633 views)
Permalink

A thread import problem

I'm trying to do something rather tricky, in which a program imports a
module that starts a thread that exec's a (possibly altered) copy of
the source in the original program, and the module doesn't return.
This has to do with an attempt to run VPython in the Mac Cocoa
context, in which Cocoa is required to be the primary thread, making
it necessary to turn the environment inside out, as currently VPython
invokes the Carbon context as a secondary thread.

I've created a simple test case, displayed below, that illustrates
something I don't understand. The module reads the source of the
program that imported it, comments out the import statement in that
source, and performs an exec of the modified source. The module then
enters an infinite loop, so that there is no return to the original
program; only the exec-ed program runs, and it runs in a secondary
thread.

The puzzle is that if there is any later import statement in the exec
source, the exec program halts on that import statement, with no error
message. I saw a discussion that suggested a need for the statement
"global math" to make the math import work, but that doesn't fix the
problem. I've tried with no success various versions of the exec
statement, with respect to its global and local environment.

Can anyone explain why the math import statement causes a problem?
Thanks for any advice you can give.

Bruce Sherwood

---------------------------
The main program:

from import_test import *
print('exec this file')
global math
from math import sin
print(sin(3.14159/6))

-----------------------------
Contents of import_test:

from threading import Thread
from time import sleep
import sys

prog = open(sys.argv[0]).read()
prog = '#'+prog # comment out the import statement
print(prog)

class worker(Thread):
def run(self):
print('start thread')
exec(prog)

w = worker()
w.start()

while True:
sleep(1)
--
http://mail.python.org/mailman/listinfo/python-list

dieter at handshake

Jul 19, 2012, 1:31 AM

Post #2 of 20 (611 views)
Permalink

Re: A thread import problem [In reply to]

Bruce Sherwood <bruce.sherwood [at] gmail> writes:

> I'm trying to do something rather tricky, in which a program imports a
> module that starts a thread that exec's a (possibly altered) copy of
> the source in the original program, and the module doesn't return.
> This has to do with an attempt to run VPython in the Mac Cocoa
> context, in which Cocoa is required to be the primary thread, making
> it necessary to turn the environment inside out, as currently VPython
> invokes the Carbon context as a secondary thread.
>
> I've created a simple test case, displayed below, that illustrates
> something I don't understand. The module reads the source of the
> program that imported it, comments out the import statement in that
> source, and performs an exec of the modified source. The module then
> enters an infinite loop, so that there is no return to the original
> program; only the exec-ed program runs, and it runs in a secondary
> thread.
>
> The puzzle is that if there is any later import statement in the exec
> source, the exec program halts on that import statement, with no error
> message. I saw a discussion that suggested a need for the statement
> "global math" to make the math import work, but that doesn't fix the
> problem. I've tried with no success various versions of the exec
> statement, with respect to its global and local environment.

In a recent discussion in this list someone mentioned that
on module import, you should not start a thread. The reason: apparently,
Python uses some kind of locking during import which can interfere
with "import"s in the started thread.

You can (in principle) easily avoid starting the thread on module import.
Instead of starting the thread as a side effect of the import,
put the start in a function, import the module and then call
the thread starting function.

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 20, 2012, 11:44 AM

Post #3 of 20 (609 views)
Permalink

Re: A thread import problem [In reply to]

Dieter Maurer commented the following on my question about a thread
import problem:

----------------------
In a recent discussion in this list someone mentioned that
on module import, you should not start a thread. The reason: apparently,
Python uses some kind of locking during import which can interfere
with "import"s in the started thread.

You can (in principle) easily avoid starting the thread on module import.
Instead of starting the thread as a side effect of the import,
put the start in a function, import the module and then call
the thread starting function.
-----------------------

Thanks for the helpful information. I tried the experiment of altering
the source code to be exec-ed to import a different module and then
call a function from that module to start a thread, but that didn't
help, perhaps because in this chain of events, which necessarily
involve exec's in a thread, it's the equivalent of the problem you
identify.

I actually do have a solution but it's inelegant. In the module
imported by the user's program, I find all import statements in that
user's program and execute them (then comment them out), thereby
adding their symbols to the module's globals, and I pass the module's
globals to the exec statement.

I'll state the problem in a more general way in the hopes that someone
will see a solution I've missed. The VPython API permits the following
short program, which displays a 3D cube moving to the right, and you
can rotate and zoom the camera with the mouse:

from visual import box, rate
b = box()
while True:
rate(100) # no more than 100 iterations per second
b.pos.x += .01

This works because a GUI environment is invoked by the visual module
in a secondary thread (written mainly in C++, connected to Python by
Boost). The OpenGL rendering of the box in its current position is
driven by a 30-millisecond timer. This works fine with any environment
other than Mac Cocoa.

However, the Mac Cocoa GUI environment and interact loop are required
to be the primary thread, so the challenge is to have the visual
module set up the Cocoa environment, with the user's program running
in a secondary thread. Any ideas?

Bruce Sherwood

On Wed, Jul 18, 2012 at 5:03 PM, Bruce Sherwood
<bruce.sherwood [at] gmail> wrote:
> I'm trying to do something rather tricky, in which a program imports a
> module that starts a thread that exec's a (possibly altered) copy of
> the source in the original program, and the module doesn't return.
> This has to do with an attempt to run VPython in the Mac Cocoa
> context, in which Cocoa is required to be the primary thread, making
> it necessary to turn the environment inside out, as currently VPython
> invokes the Carbon context as a secondary thread.
>
> I've created a simple test case, displayed below, that illustrates
> something I don't understand. The module reads the source of the
> program that imported it, comments out the import statement in that
> source, and performs an exec of the modified source. The module then
> enters an infinite loop, so that there is no return to the original
> program; only the exec-ed program runs, and it runs in a secondary
> thread.
>
> The puzzle is that if there is any later import statement in the exec
> source, the exec program halts on that import statement, with no error
> message. I saw a discussion that suggested a need for the statement
> "global math" to make the math import work, but that doesn't fix the
> problem. I've tried with no success various versions of the exec
> statement, with respect to its global and local environment.
>
> Can anyone explain why the math import statement causes a problem?
> Thanks for any advice you can give.
>
> Bruce Sherwood
>
> ---------------------------
> The main program:
>
> from import_test import *
> print('exec this file')
> global math
> from math import sin
> print(sin(3.14159/6))
>
> -----------------------------
> Contents of import_test:
>
> from threading import Thread
> from time import sleep
> import sys
>
> prog = open(sys.argv[0]).read()
> prog = '#'+prog # comment out the import statement
> print(prog)
>
> class worker(Thread):
> def run(self):
> print('start thread')
> exec(prog)
>
> w = worker()
> w.start()
>
> while True:
> sleep(1)
--
http://mail.python.org/mailman/listinfo/python-list

dieter at handshake

Jul 21, 2012, 1:32 AM

Post #4 of 20 (608 views)
Permalink

Re: A thread import problem [In reply to]

Bruce Sherwood <bruce.sherwood [at] gmail> writes:
> ...
> from visual import box, rate
> b = box()
> while True:
> rate(100) # no more than 100 iterations per second
> b.pos.x += .01
>
> This works because a GUI environment is invoked by the visual module
> in a secondary thread (written mainly in C++, connected to Python by
> Boost). The OpenGL rendering of the box in its current position is
> driven by a 30-millisecond timer. This works fine with any environment
> other than Mac Cocoa.
>
> However, the Mac Cocoa GUI environment and interact loop are required
> to be the primary thread, so the challenge is to have the visual
> module set up the Cocoa environment, with the user's program running
> in a secondary thread. Any ideas?

The usual approach to this situation is to invoke the user code via
a callback from the UI main loop or invoke it explicitely
after the UI system has been set up immediately before its main loop
is called. Might look somehow like this:

main thread:

from thread import start_new_thread
from visual import setup_gui, start_main_loop
setup_gui() # sets up the GUI subsystem
start_new_thread(lambda: __import__(<your module>), ())
start_main_loop()

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 21, 2012, 7:54 AM

Post #5 of 20 (609 views)
Permalink

Re: A thread import problem [In reply to]

Thanks much for this suggestion. I'm not sure I've correctly
understood the operation "start_new_thread(lambda: __import__(<your
imported the module that will execute start_new_thread? It hadn't
occurred to me to have A import B and B import A, though now that you
describe this (if that's indeed what you mean) it makes sense. The
original instance of A won't get past its initial import statement
because the main loop won't return to it.

Bruce Sherwood

On Sat, Jul 21, 2012 at 2:32 AM, Dieter Maurer <dieter [at] handshake> wrote:
> Bruce Sherwood <bruce.sherwood [at] gmail> writes:
>> ...
>> from visual import box, rate
>> b = box()
>> while True:
>> rate(100) # no more than 100 iterations per second
>> b.pos.x += .01
>>
>> This works because a GUI environment is invoked by the visual module
>> in a secondary thread (written mainly in C++, connected to Python by
>> Boost). The OpenGL rendering of the box in its current position is
>> driven by a 30-millisecond timer. This works fine with any environment
>> other than Mac Cocoa.
>>
>> However, the Mac Cocoa GUI environment and interact loop are required
>> to be the primary thread, so the challenge is to have the visual
>> module set up the Cocoa environment, with the user's program running
>> in a secondary thread. Any ideas?
>
> The usual approach to this situation is to invoke the user code via
> a callback from the UI main loop or invoke it explicitely
> after the UI system has been set up immediately before its main loop
> is called. Might look somehow like this:
>
> main thread:
>
> from thread import start_new_thread
> from visual import setup_gui, start_main_loop
> setup_gui() # sets up the GUI subsystem
> start_new_thread(lambda: __import__(<your module>), ())
> start_main_loop()
--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 21, 2012, 9:11 AM

Post #6 of 20 (607 views)
Permalink

Re: A thread import problem [In reply to]

I couldn't get a simple test case to work. I append a listing of the
little test files, all in the same folder. The diagnostic statement
print('after start_new_thread\n') works, but then nothing. Originally
I tried importing testABA.py but was worried that the circular
importing (A imports B which imports A) would be a problem, hence the
test of importing a version of the test program without the import.

The failure of this test case suggests that one cannot do imports
inside secondary threads started in imported modules, something I keep
tripping over. But I hope you'll be able to tell me that I'm doing
something wrong!

Incidentally, a simple test is to execute the file ABA.py, in which
case everything works.

Bruce Sherwood

---------------------------
testABA.py -- execute this file

from ABA import *
print('exec testABA')
from math import sin
print(sin(3.14159/6))

----------------------------
testABA_noimport.py -- a version of testABA.py without the import of ABA

print('exec testABA_noimport')
from math import sin
print(sin(3.14159/6))

-----------------------------
ABA.py

from thread import start_new_thread
from time import sleep
import sys

user = 'testABA_noimport'
start_new_thread(lambda: __import__(user), ())
print('after start_new_thread\n')

while True:
sleep(1)

On Sat, Jul 21, 2012 at 2:32 AM, Dieter Maurer <dieter [at] handshake> wrote:
> The usual approach to this situation is to invoke the user code via
> a callback from the UI main loop or invoke it explicitely
> after the UI system has been set up immediately before its main loop
> is called. Might look somehow like this:
>
> main thread:
>
> from thread import start_new_thread
> from visual import setup_gui, start_main_loop
> setup_gui() # sets up the GUI subsystem
> start_new_thread(lambda: __import__(<your module>), ())
> start_main_loop()
--
http://mail.python.org/mailman/listinfo/python-list

d at davea

Jul 21, 2012, 9:26 AM

Post #7 of 20 (607 views)
Permalink

Re: A thread import problem [In reply to]

On 07/21/2012 10:54 AM, Bruce Sherwood wrote:
> Thanks much for this suggestion. I'm not sure I've correctly
> understood the operation "start_new_thread(lambda: __import__(<your
> module>), ())". By "your module" do you mean the user program which
> imported the module that will execute start_new_thread? It hadn't
> occurred to me to have A import B and B import A, though now that you
> describe this (if that's indeed what you mean) it makes sense. The
> original instance of A won't get past its initial import statement
> because the main loop won't return to it.
>
> Bruce Sherwood
>

Two of the things you mustn't do during an import:

1) start or end any threads
2) import something that's already in the chain of pending imports.
(otherwise known as recursive imports, or import loop). And there's a
special whammy reserved for those who import the script as though it
were a module.

Like any rule, there are possible exceptions. But you're much better
off factoring your code better.

I haven't managed to understand your software description, so i'm not
making a specific suggestion. But I know others have pointed out that
you should do as little as possible in top-level code of an imported
module. Make the work happen in a function, and call that function from
the original script, not from inside some import. An imported module's
top-level code should do nothing more complex than initialize module
constants.

--

DaveA

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 21, 2012, 1:36 PM

Post #8 of 20 (606 views)
Permalink

Re: A thread import problem [In reply to]

Thanks much for this clear statement. I hadn't managed to find any
documentation on this specific issue.

Bruce Sherwood

On Sat, Jul 21, 2012 at 10:26 AM, Dave Angel <d [at] davea> wrote:
> Two of the things you mustn't do during an import:
>
> 1) start or end any threads
> 2) import something that's already in the chain of pending imports.
> (otherwise known as recursive imports, or import loop). And there's a
> special whammy reserved for those who import the script as though it
> were a module.
>
> Like any rule, there are possible exceptions. But you're much better
> off factoring your code better.
>
> I haven't managed to understand your software description, so i'm not
> making a specific suggestion. But I know others have pointed out that
> you should do as little as possible in top-level code of an imported
> module. Make the work happen in a function, and call that function from
> the original script, not from inside some import. An imported module's
> top-level code should do nothing more complex than initialize module
> constants.
>
>
> --
>
> DaveA
>
>
--
http://mail.python.org/mailman/listinfo/python-list

d at davea

Jul 21, 2012, 1:53 PM

Post #9 of 20 (605 views)
Permalink

Re: A thread import problem [In reply to]

On 07/21/2012 04:36 PM, Bruce Sherwood wrote:
> Thanks much for this clear statement. I hadn't managed to find any
> documentation on this specific issue.
>
> Bruce Sherwood
>
> On Sat, Jul 21, 2012 at 10:26 AM, Dave Angel <d [at] davea> wrote:
>> Two of the things you mustn't do during an import:
>>
>> 1) start or end any threads
>> 2) import something that's already in the chain of pending imports.
>> (otherwise known as recursive imports, or import loop). And there's a
>> special whammy reserved for those who import the script as though it
>> were a module.
>>
>> Like any rule, there are possible exceptions. But you're much better
>> off factoring your code better.
>>
>> I haven't managed to understand your software description, so i'm not
>> making a specific suggestion. But I know others have pointed out that
>> you should do as little as possible in top-level code of an imported
>> module. Make the work happen in a function, and call that function from
>> the original script, not from inside some import. An imported module's
>> top-level code should do nothing more complex than initialize module
>> constants.
>>
>>
>> --
>>
>> DaveA
>>
>>

(You top-posted, which makes it harder to figure out who said what.)

For docs on the threading thing, see:

http://docs.python.org/library/threading.html

" ... an import should not have the side effect of spawning a new thread
and then waiting for that thread in any way..."

--

DaveA

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 21, 2012, 2:35 PM

Post #10 of 20 (606 views)
Permalink

Re: A thread import problem [In reply to]

On Sat, Jul 21, 2012 at 2:53 PM, Dave Angel <d [at] davea> wrote:
> On 07/21/2012 04:36 PM, Bruce Sherwood wrote:
>> Thanks much for this clear statement. I hadn't managed to find any
>> documentation on this specific issue.
>>
>> Bruce Sherwood
>>
>> On Sat, Jul 21, 2012 at 10:26 AM, Dave Angel <d [at] davea> wrote:
>>> Two of the things you mustn't do during an import:
>>>
>>> 1) start or end any threads
>>> 2) import something that's already in the chain of pending imports.
>>> (otherwise known as recursive imports, or import loop). And there's a
>>> special whammy reserved for those who import the script as though it
>>> were a module.
>>>
>>> Like any rule, there are possible exceptions. But you're much better
>>> off factoring your code better.
>>>
>>> I haven't managed to understand your software description, so i'm not
>>> making a specific suggestion. But I know others have pointed out that
>>> you should do as little as possible in top-level code of an imported
>>> module. Make the work happen in a function, and call that function from
>>> the original script, not from inside some import. An imported module's
>>> top-level code should do nothing more complex than initialize module
>>> constants.
>>>
>>>
>>> --
>>>
>>> DaveA
>>>
>>>
>
> (You top-posted, which makes it harder to figure out who said what.)
>
> For docs on the threading thing, see:
>
> http://docs.python.org/library/threading.html
>
> " ... an import should not have the side effect of spawning a new thread
> and then waiting for that thread in any way..."
>
>
>
>
> --
>
> DaveA
>

Thanks. I had read that as forbidding "waiting for that thread", not
forbidding spawning a new thread. The following sentence says,
"Failing to abide by this restriction can lead to a deadlock if the
spawned thread directly or indirectly attempts to import a module." I
gather that a clearer, more forceful statement might be, "Failing to
abide by this restriction WILL lead to a deadlock if the spawned
thread directly or indirectly attempts to import a module." All of
which implies the behavior I've seen in various experiments, namely
that as long as the spawned thread doesn't do any imports, I haven't
seen any problems with spawning a thread in an imported module. I take
your word for it that this is a no-no, but I don't know why.

Bruce Sherwood
--
http://mail.python.org/mailman/listinfo/python-list

d at davea

Jul 21, 2012, 3:33 PM

Post #11 of 20 (607 views)
Permalink

Re: A thread import problem [In reply to]

On 07/21/2012 05:35 PM, Bruce Sherwood wrote:
> On Sat, Jul 21, 2012 at 2:53 PM, Dave Angel <d [at] davea> wrote:
>> <SNIP>
>> For docs on the threading thing, see:
>>
>> http://docs.python.org/library/threading.html
>>
>> " ... an import should not have the side effect of spawning a new thread
>> and then waiting for that thread in any way..."
>>
>>
>>
>>
>> --
>>
>> DaveA
>>
> Thanks. I had read that as forbidding "waiting for that thread", not
> forbidding spawning a new thread. The following sentence says,
> "Failing to abide by this restriction can lead to a deadlock if the
> spawned thread directly or indirectly attempts to import a module." I
> gather that a clearer, more forceful statement might be, "Failing to
> abide by this restriction WILL lead to a deadlock if the spawned
> thread directly or indirectly attempts to import a module." All of
> which implies the behavior I've seen in various experiments, namely
> that as long as the spawned thread doesn't do any imports, I haven't
> seen any problems with spawning a thread in an imported module. I take
> your word for it that this is a no-no, but I don't know why.
>
> Bruce Sherwood

I don't know just what will work and what will not; But there are lots
of subtle and indirect ways of "waiting for that thread" and I suspect
that import is one of them.

Since I've never seen a case where we had to break the general rule, it
just seems easier to keep it clean. No threading inside an import.

Same with recursive imports. I could list some of the specific problems
that crop up, but since the only time recursive imports are unavoidable
is when you're constrained by preexisting 3rd party software that does
something strange, it seems easier to make a simple rule that's easy
enough to test for.

if you try to import the script (that imported you) as though it were a
module, you end up with two copies of that module, and if they have any
non-constant globals, you can get very strange symptoms. In other
recursion cases,

--

DaveA

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 21, 2012, 4:10 PM

Post #12 of 20 (605 views)
Permalink

Re: A thread import problem [In reply to]

On Sat, Jul 21, 2012 at 4:16 PM, Dennis Lee Bieber
<wlfraed [at] ix> wrote:
> On Sat, 21 Jul 2012 10:11:30 -0600, Bruce Sherwood
> <bruce.sherwood [at] gmail> declaimed the following in
> gmane.comp.python.general:
>
>
>>
>> ---------------------------
>> testABA.py -- execute this file
>>
>> from ABA import *
>> print('exec testABA')
>> from math import sin
>> print(sin(3.14159/6))
>>
>> ----------------------------
>> testABA_noimport.py -- a version of testABA.py without the import of ABA
>>
>> print('exec testABA_noimport')
>> from math import sin
>> print(sin(3.14159/6))
>>
>> -----------------------------
>> ABA.py
>>
>> from thread import start_new_thread
>> from time import sleep
>> import sys
>>
>> user = 'testABA_noimport'
>> start_new_thread(lambda: __import__(user), ())
>> print('after start_new_thread\n')
>>
>> while True:
>> sleep(1)
>>
>
> And all along you are still coding where imported modules are doing
> work DURING THE IMPORT. Anything beyond initializing module level
> constants or importing modules needed by the module should be placed
> into a function which can be executed by the top-level importer AFTER
> the import has finished.
>
> -=-=-=-=- testABA.py
> print ("testABA: top")
>
> from math import sin, pi
> import time
>
> print ("testABA: defining worker function")
> def runner(): #this is the key -- a function IN the module
> #that does the real work
> print ("testABA.runner: starting work\n")
> time.sleep(2.5)
> print (sin(pi))
> time.sleep(2.5)
> print ("\ntestABA.runner: end of work")
>
> print ("testABA: after set-up")
>
> if __name__ == "__main__":
> print ("testABA: was not an import, running main task")
> runner() #invoke the function if module is not imported
> print ("testABA: end")
> -=-=-=-=-
>
> -=-=-=-=- ABA.py
> import threading
>
> USER = "testABA"
>
> print ("ABA: importing " + USER)
> modl = __import__(USER)
>
> print ("ABA: starting runner")
> th = threading.Thread(target=modl.runner)
> th.start()
>
> print ("ABA: waiting for completion")
> th.join()
>
> print ("ABA: done")
> -=-=-=-=-
>
> And showing the results...
>
> -=-=-=-=-
> E:\UserData\Wulfraed\My Documents\Python Progs>testABA
> testABA: top
> testABA: defining worker function
> testABA: after set-up
> testABA: was not an import, running main task
> testABA.runner: starting work
>
> 1.22460635382e-016
>
> testABA.runner: end of work
> testABA: end
>
> E:\UserData\Wulfraed\My Documents\Python Progs>ABA
> ABA: importing testABA
> testABA: top
> testABA: defining worker function
> testABA: after set-up
> ABA: starting runner
> testABA.runner: starting work
>
> ABA: waiting for completion
> 1.22460635382e-016
>
> testABA.runner: end of work
> ABA: done
> -=-=-=-=-
>
> Note that "testABA.py" is designed to be used as a stand-alone
> program, but is also designed to be imported where all the real work is
> done from a function that is NOT run during the import. The program that
> imports "testABA" is then responsible for actually starting the worker.
>
> I put the sleeps into testABA.runner() so that you can see that the
> main process isn't blocked (note the "ABA: waiting..." output)
>
>
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfraed [at] ix HTTP://wlfraed.home.netcom.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list

Thanks, but the problem I need to solve does not permit putting a
function like runner in the main program. I'm constrained to being
able to handle the API of VPython (vpython.org), which lets you write
programs like the following (call it user.py), which animates a 3D
cube moving to the right, using OpenGL:

from visual import box
b = box()
while True:
b.pos.x += 0.001

In VPython at present, the visual module sets up (using C++ and Boost)
a secondary thread that periodically updates the 3D scene using the
current objects and their attributes, and handles events. On the Mac
this is done with Carbon, which is dying, so I need to base VPython on
the Mac on Cocoa, for which the interact loop must be in the primary
thread. Hence my need to turn the architecture inside out while
nevertheless handling the existing API.

The cleanest scheme is to have the user's program user.py import a
module "visual" that spawns a new process, "python visual2.py
user.py", where visual2.py reads the source from user.py, comments out
the "import visual", and exec's this modified source in a secondary
thread. This works because visual2.py is now the main module.
Unfortunately, if user.py is run from IDLE, print output goes to a
terminal window rather than to the IDLE shell window, and I don't know
how to direct the output to that shell window.

For this reason I've been experimenting with other schemes, and it
took a while to understand that a thread spawned in an imported module
cannot do imports. I've even managed to carry out a real kludge of
executing imports found in user.py at the top level of visual,
creating a dictionary of globals to pass to an exec of the source in
user.py with all imports there commented out. It works, but ugh.

Bruce Sherwood
--
http://mail.python.org/mailman/listinfo/python-list

dieter at handshake

Jul 21, 2012, 11:04 PM

Post #13 of 20 (596 views)
Permalink

Re: A thread import problem [In reply to]

Bruce Sherwood <bruce.sherwood [at] gmail> writes:

> Thanks much for this suggestion. I'm not sure I've correctly
> understood the operation "start_new_thread(lambda: __import__(<your
> module>), ())". By "your module" do you mean the user program which
> imported the module that will execute start_new_thread?

By "your_module", I meant what you have called "user.py" elsewhere
in this thread -- the thing that does the animation.

Of course, my suggestion implies that "visual.py" is somewhat changed.
It is supposed to no longer set up the GUI environment automatically
but do so only when its "setup_gui" function is called, and starting
the GUI main loop, too, is no longer automatic but explicite.

> It hadn't
> occurred to me to have A import B and B import A, though now that you
> describe this (if that's indeed what you mean) it makes sense.

I do not propose to do that -- it can lead to problems.

In my proposal, you have two modules: one the "main" module which
sets up the GUI environment, starts the animation in a separate thread
and then activate the GUI main loop. The second module contains
the code you have shown in a previous message.

Of course, the second module can be eliminated by putting its content
into a function and then calling this function in the "start_new_thread"
(instead of "lambda: __import__(...)").

--
http://mail.python.org/mailman/listinfo/python-list

dieter at handshake

Jul 21, 2012, 11:18 PM

Post #14 of 20 (590 views)
Permalink

Re: A thread import problem [In reply to]

Bruce Sherwood <bruce.sherwood [at] gmail> writes:
> ...
> The failure of this test case suggests that one cannot do imports
> inside secondary threads started in imported modules, something I keep
> tripping over. But I hope you'll be able to tell me that I'm doing
> something wrong!

As you know multiple threads can be dangerous when they concurrently
change common data structures. Locks are used to protect those
data structures. Locking can introduce other problems - like deadlocks
(something you seem to observe).

I have not looked at the concrete implementation. However, the Python
documentation suggests that the import machinery uses its own locks
(beside the "Global Interpreter Lock"). It seems to be a "thread lock",
which would mean that a thread is not blocked when it already
holds the lock - however any other thread would block.
This easily can lead to a deadlock -- when you wait for the other
thread "in any way".

There should be no problem when you complete the whole import chain
without any waiting for the thread. However, should you start
the GUI main loop inside the import chain, you will never complete
this chain.

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 22, 2012, 12:04 PM

Post #15 of 20 (591 views)
Permalink

Re: A thread import problem [In reply to]

On Sat, Jul 21, 2012 at 5:47 PM, Dennis Lee Bieber
<wlfraed [at] ix> wrote:
> On Sat, 21 Jul 2012 17:10:05 -0600, Bruce Sherwood
> <bruce.sherwood [at] gmail> declaimed the following in
> gmane.comp.python.general:
>
>
>> Thanks, but the problem I need to solve does not permit putting a
>> function like runner in the main program. I'm constrained to being
>> able to handle the API of VPython (vpython.org), which lets you write
>> programs like the following (call it user.py), which animates a 3D
>> cube moving to the right, using OpenGL:
>>
>> from visual import box
>> b = box()
>> while True:
>> b.pos.x += 0.001
>
> Well, based on that sample, wrap THAT as "runner"
>
> def runner():
> from visual import box
> b = box()
> while True:
> b.pos.x += 0.0001
>
> and don't /call/ runner() until after all the main system is configured.
> (And runner could, if need be, be "called" as a thread).
>
> Not having a Mac, I can't do tests... but everything I've seen so
> far comes down to NOT IMPORTING anything that tries to spawn threads
> /during the import/.
>
> A properly designed module (as I showed with my testABA.py) only
> "runs" stuff if loaded as the main program; any other use (import) only
> does imports and defines module level entities -- running anything is
> deferred for the program that did the import to invoke AFTER the import
> finished.
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfraed [at] ix HTTP://wlfraed.home.netcom.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list

I don't have the luxury of doing things the "approved" way. I'm
constrained by extensive legacy code (and legacy educational
documentation for student use) to be able to run programs such as this
one:

from visual import box
b = box()
while True:
b.pos.x += 0.001

Another way of saying this is that I'm not building an app, in which
case I would structure things in a simple and straightforward manner.
I am instead trying to maintain and update a library that allows
novice programmers to write programs that generate real-time navigable
3D animations, writing minimalist programs that work cross-platform.

Bruce Sherwood
--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 22, 2012, 4:14 PM

Post #16 of 20 (589 views)
Permalink

Re: A thread import problem [In reply to]

On Sun, Jul 22, 2012 at 3:48 PM, Dennis Lee Bieber
<wlfraed [at] ix> wrote:
> On Sun, 22 Jul 2012 13:04:25 -0600, Bruce Sherwood
> <bruce.sherwood [at] gmail> declaimed the following in
> gmane.comp.python.general:
>
>
>> Another way of saying this is that I'm not building an app, in which
>> case I would structure things in a simple and straightforward manner.
>> I am instead trying to maintain and update a library that allows
>> novice programmers to write programs that generate real-time navigable
>> 3D animations, writing minimalist programs that work cross-platform.
>>
> I suspect you don't need to update the library so much as enforce a
> project style that prevents your import-thread-import cycle. In short,
> something like that hypothetical "runner()" design.
>
> Oh, and document that style in detail: "importable modules shall
> only perform module level definition of entities during import -- no
> code that actually processes data"; "importable modules shall make use
> of the 'if __name__ == "__main__": main()' convention to identify when
> the module is being used stand-alone, and thereby invoke the main
> processing; otherwise the module.main() function is to be invoked only
> be the module doing the imports, after the import has completed"
> etc.
>
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlfraed [at] ix HTTP://wlfraed.home.netcom.com/
>
> --
> http://mail.python.org/mailman/listinfo/python-list

There's nothing wrong with the current VPython architecture, which
does use good style, but there are two absolute, conflicting
requirements that I have to meet.

(1) The simple program API I've shown must be preserved, because there
exist a large number of such programs in existence, used by lots of
people. I can't change the API. Among other uses, every semester there
are about 5000 students in introductory college science courses,
especially physics, who do computational modeling with 3D
visualizations based on instructional materials that teach the
existing API. There is also a large number of instructors who depend
on existing VPython demo programs to continue working even if the
college upgrades Python and VPython. This isn't some little project
where I'm able to teach my small group of collaborators how they
should structure programs.

(2) My hand is forced by Apple no longer supporting Carbon. Among
other aspects of this, Carbon can't be used with 64-bit Python, and
more and more Mac users of VPython want to use 64-bit Python. So there
has to be a version of VPython that is based on Cocoa, but Cocoa is
required to be the primary thread. This requirement, combined with the
absolute requirement that the VPython API cannot be changed, is the
problem I face. I have to turn the architecture inside out,
independent of whether the solution meets all criteria for good Python
programming style.

Bruce Sherwood
--
http://mail.python.org/mailman/listinfo/python-list

jeanpierreda at gmail

Jul 22, 2012, 10:50 PM

Post #17 of 20 (585 views)
Permalink

Re: A thread import problem [In reply to]

On Sun, Jul 22, 2012 at 7:14 PM, Bruce Sherwood
<bruce.sherwood [at] gmail> wrote:
> (2) My hand is forced by Apple no longer supporting Carbon. Among
> other aspects of this, Carbon can't be used with 64-bit Python, and
> more and more Mac users of VPython want to use 64-bit Python. So there
> has to be a version of VPython that is based on Cocoa, but Cocoa is
> required to be the primary thread. This requirement, combined with the
> absolute requirement that the VPython API cannot be changed, is the
> problem I face. I have to turn the architecture inside out,
> independent of whether the solution meets all criteria for good Python
> programming style.

I had exactly this problem with Tkinter on Mac. The API involved
"synchronous" calls to update a GUI written in tkinter, which ran in
another thread (so e.g. students could call the API from the
interactive interpreter). This didn't work on new Mac OS X releases,
because Tkinter had to be run in the main thread after some update --
and also because of this deadlock issue with imports (but I didn't
know that until later).

I ended up solving it by running the Tkinter in the main thread of a
different process, which could handle RPC invocations asynchronously,
and sending remote invocations via a synchronous RPC library in the
parent process.

Maybe you can do something similar in your case?

-- Devin
--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 23, 2012, 7:07 AM

Post #18 of 20 (584 views)
Permalink

Re: A thread import problem [In reply to]

Thanks much for the useful suggestion, and also thanks for your
sympathy and understanding of my plight!

Bruce Sherwood

On Sun, Jul 22, 2012 at 11:50 PM, Devin Jeanpierre
<jeanpierreda [at] gmail> wrote:
> On Sun, Jul 22, 2012 at 7:14 PM, Bruce Sherwood
> <bruce.sherwood [at] gmail> wrote:
>> (2) My hand is forced by Apple no longer supporting Carbon. Among
>> other aspects of this, Carbon can't be used with 64-bit Python, and
>> more and more Mac users of VPython want to use 64-bit Python. So there
>> has to be a version of VPython that is based on Cocoa, but Cocoa is
>> required to be the primary thread. This requirement, combined with the
>> absolute requirement that the VPython API cannot be changed, is the
>> problem I face. I have to turn the architecture inside out,
>> independent of whether the solution meets all criteria for good Python
>> programming style.
>
> I had exactly this problem with Tkinter on Mac. The API involved
> "synchronous" calls to update a GUI written in tkinter, which ran in
> another thread (so e.g. students could call the API from the
> interactive interpreter). This didn't work on new Mac OS X releases,
> because Tkinter had to be run in the main thread after some update --
> and also because of this deadlock issue with imports (but I didn't
> know that until later).
>
> I ended up solving it by running the Tkinter in the main thread of a
> different process, which could handle RPC invocations asynchronously,
> and sending remote invocations via a synchronous RPC library in the
> parent process.
>
> Maybe you can do something similar in your case?
>
> -- Devin
--
http://mail.python.org/mailman/listinfo/python-list

dieter at handshake

Jul 23, 2012, 7:48 AM

Post #19 of 20 (578 views)
Permalink

Re: A thread import problem [In reply to]

Bruce Sherwood <bruce.sherwood [at] gmail> writes:
> ...
> There's nothing wrong with the current VPython architecture, which
> does use good style, but there are two absolute, conflicting
> requirements that I have to meet.
>
> (1) The simple program API I've shown must be preserved, because there
> exist a large number of such programs in existence, used by lots of
> people. I can't change the API. Among other uses, every semester there
> are about 5000 students in introductory college science courses,
> especially physics, who do computational modeling with 3D
> visualizations based on instructional materials that teach the
> existing API. There is also a large number of instructors who depend
> on existing VPython demo programs to continue working even if the
> college upgrades Python and VPython. This isn't some little project
> where I'm able to teach my small group of collaborators how they
> should structure programs.

You might keep the "programs" (one of which you have shown)
but change the way how they are "called" (and change the internal
working of "visual").

In my "proposal", your "program" is not changed in any way -- but
it is not called directly but activated ("imported") from something
like a starting module.

--
http://mail.python.org/mailman/listinfo/python-list

bruce.sherwood at gmail

Jul 23, 2012, 8:33 PM

Post #20 of 20 (569 views)
Permalink

Re: A thread import problem [In reply to]

I'm happy to report that Robin Dunn, the developer of wxPython, showed
me how to solve my VPython architectural problem, using wxPython. I
attach a test program based on wxPython that has all of the properties
I was looking for (though it needs some minor cleanups, including
quitting gracefully, and handling timing better). The attached program
userpoll.py imports the wxpython-based module wxpoll.py which does the
following:

Creates a wxpython environment but does not start the interact loop.
Reads the source of the user program.
Finds the import of the module and adds imported entities to
globals used with exec.
Comments out this import statement in the source.
Does an exec of the modified source, in the primary thread.
Because the exec is not done in a secondary thread, there are no problems
associated with other imports such as math that may be in the user program.

The exec-ed program has a loop in it, containing a rate statement
which clamps to no more than 100 iterations per second (to make the
animation run at a reasonable speed) and which calls a one-shot
interact function in the import module. In other words, polling is
used rather than using an interact loop, thereby avoiding the nearly
intractable threading problems. Happily, Robin found a way to make
this polling work with Cocoa on the Mac. I can now move forward with
implementing this machinery for VPython, to be able to run VPython
programs on 64-bit Pythons on the Mac. Hurray!

You will find massive violations of "good practice" in the attached
module, because in order to pull this off I have to do lots of
calculations at the top level of the module. However, I believe the
structure is in fact entirely reasonable and a good solution to a very
tricky problem.

Thanks to those of you who have made suggestions in this forum that
fed into finding a solution. I'll put in a big plug for wxPython
(wxpython.org), which is a great way to do cross-platform GUI
development with Python.

Bruce Sherwood

Attachments:

userpoll.py (0.20 KB)

wxpoll.py (7.98 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads