Mailing List Archive: Python: Bugs

[issue12655] Expose sched.h functions

Index | Next | Previous | View Threaded

report at bugs

Aug 3, 2012, 5:31 PM

Post #1 of 14 (96 views)
Permalink

[issue12655] Expose sched.h functions

STINNER Victor added the comment:

I'm sorry, I missed this issue. I just saw its API and I have remarks on the cpu_set type:
* Why not reusing the set type for input parameters and the result of sched_getaffinity?
* Method names of cpu_set are very different than names of the set type
* Why do I need to specify the number of CPU?

signal.pthread_sigmask() accepts any iterable object as input, convert it to a sigset_t. For the result, it converts the new sigset_t to a classic Python set object.

Is it possible to get the number of CPU to be able to convert a Python set to a cpu_set?

----------
nosy: +haypo
resolution: fixed ->
status: closed -> open

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 3, 2012, 5:35 PM

Post #2 of 14 (94 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Antoine Pitrou added the comment:

> Is it possible to get the number of CPU to be able to convert a Python
> set to a cpu_set?

sched_getaffinity returns EINVAL if the cpu_set is too small, so it should be easy enough to iterate.
I agree the API should be changed for something saner, and before 3.3. Sorry for not looking at the patch in more detail when it was first submitted. I will try to cook up a new patch this week-end.

----------
nosy: +georg.brandl
priority: normal -> release blocker

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 3, 2012, 9:02 PM

Post #3 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Benjamin Peterson added the comment:

Using a Python set is fine.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 4:08 AM

Post #4 of 14 (93 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Antoine Pitrou added the comment:

Here is a patch removing cpu_set and using sets of integers instead.

----------
Added file: http://bugs.python.org/file26682/cpuset.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 4:13 AM

Post #5 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:

Removed file: http://bugs.python.org/file26682/cpuset.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 4:13 AM

Post #6 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Changes by Antoine Pitrou <pitrou [at] free>:

Added file: http://bugs.python.org/file26683/cpuset.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 6:00 AM

Post #7 of 14 (94 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

STINNER Victor added the comment:

sched_getaffinity() does not fail if the set is smaller than the
number of CPU. Try with an initial value of ncpus=1. So we cannot
start the heuristic with ncpus=16, because it would only return 16
even if the computer has more cpus.

Instead of this heuristic, why not simply alway using ncpus = CPU_SETSIZE?

I don't know if CPU_SETSIZE is part of the standard (POSIX?).

You may also use a constant size (CPU_SETSIZE) of the set used by
sched_setaffinity() to simplify the code.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 6:03 AM

Post #8 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Antoine Pitrou added the comment:

> Try with an initial value of ncpus=1.

Well, I've tried and it works:

>>> os.sched_getaffinity(0)
{0, 1, 2, 3}

> I don't know if CPU_SETSIZE is part of the standard (POSIX?).

These are Linux-specific functions.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 6:10 AM

Post #9 of 14 (87 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

STINNER Victor added the comment:

>> Try with an initial value of ncpus=1.
> Well, I've tried and it works:

Oh, you're right :-) I only checked ncpus (1), not the final result.
It works because CPU_ALLOC_SIZE() rounds the size using
sizeof(unsigned long) (64 bits on my CPU). So CPU_ALLOC_SIZE(1)
returns 8, and sched_getaffinity() takes the size of the set in bytes,
and not the number of CPU.

sched_getaffinity(0) returns {0, 1, 2, 3, 4, 5, 6, 7} with ncpus=1 and
setsize=8.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 6:11 AM

Post #10 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Giampaolo Rodola' added the comment:

+1 for Antoine's patch/approach: it's more usable and pythonic.
I think documentation should mention and link the existence of:
http://docs.python.org/library/multiprocessing.html#multiprocessing.cpu_count

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 6:44 AM

Post #11 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Serhiy Storchaka added the comment:

> You may also use a constant size (CPU_SETSIZE) of the set used by
sched_setaffinity() to simplify the code.

As Antoine pointed out to me (and I was convinced itself, experimented with an example from man:CPU_SET(3)) the cpu_set functions work with a sets of more than CPU_SETSIZE processors.

----------
nosy: +storchaka

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 7:19 AM

Post #12 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Roundup Robot added the comment:

New changeset d6745ddbccbd by Antoine Pitrou in branch 'default':
Issue #12655: Instead of requiring a custom type, os.sched_getaffinity and
http://hg.python.org/cpython/rev/d6745ddbccbd

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 7:20 AM

Post #13 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Antoine Pitrou added the comment:

Ok, I've improved the default ncpus value and committed.

----------
resolution: -> fixed
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 12:01 PM

Post #14 of 14 (89 views)
Permalink

[issue12655] Expose sched.h functions [In reply to]

Roundup Robot added the comment:

New changeset fb975cb8fb45 by Victor Stinner in branch 'default':
Issue #12655: Mention multiprocessing.cpu_count() in os.sched_getaffinity() doc
http://hg.python.org/cpython/rev/fb975cb8fb45

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue12655>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads