Mailing List Archive: Xen: Devel

[PATCH 00 of 10 v3] Automatic NUMA placement for xl

Index | Next | Previous | View Threaded

raistlin at linux

Jul 4, 2012, 9:02 AM

Post #1 of 10 (98 views)
Permalink

[PATCH 00 of 10 v3] Automatic NUMA placement for xl

Hello,

Third version of the NUMA placement series Xen 4.2.

All the comments received during v2's review have been addressed (more details
in single changelogs).

The most notable changes are the following:
- the libxl_cpumap --> libxl_bitmap renaming has been rebased on top of the
recent patches that allows us to allocate bitmaps of different sizes;
- the heuristics for deciding which NUMA placement is the best one has been
redesigned, so that it now provides total ordering.

Here it is what this posting contains (* = acked during previous round):

* [PATCH 01 of 10 v3] libxl: add a new Array type to the IDL
[PATCH 02 of 10 v3] libxl,libxc: introduce libxl_get_numainfo()
* [PATCH 03 of 10 v3] xl: add more NUMA information to `xl info -n'
[PATCH 04 of 10 v3] libxl: rename libxl_cpumap to libxl_bitmap
[PATCH 05 of 10 v3] libxl: expand the libxl_bitmap API a bit
* [PATCH 06 of 10 v3] libxl: introduce some node map helpers
[PATCH 07 of 10 v3] libxl: explicitly check for libmath in autoconf

Is where data structures, utility functions and infrastructure are introduced.

* [PATCH 08 of 10 v3] libxl: enable automatic placement of guests on NUMA nodes
* [PATCH 09 of 10 v3] libxl: have NUMA placement deal with cpupools

Host the core of the mechanism.

* [PATCH 10 of 10 v3] Some automatic NUMA placement documentation

For some more documentation.

Thanks a lot and Regards,
Dario

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

raistlin at linux

Jul 4, 2012, 9:17 AM

Post #2 of 10 (96 views)
Permalink

[PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

raistlin.df at gmail

Jul 4, 2012, 9:21 AM

Post #3 of 10 (95 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Wed, 2012-07-04 at 18:02 +0200, Dario Faggioli wrote:
> Hello,
>
Sorry for this! I don't have the slightest idea of why it `hg email'
hanged after sending only these 3 mails. :-(

Anyway, my second attempt seems to have succeeded, so please, consider
that for the full series (i.e., the thread where the msg-id of the
introductory mail is <patchbomb.1341418679 [at] Solac>), and sorry again for
the noise.

Thanks and Regards,
Dario

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

Ian.Campbell at citrix

Jul 6, 2012, 4:16 AM

Post #4 of 10 (94 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Wed, 2012-07-04 at 17:17 +0100, Dario Faggioli wrote:
> Hello,
>
> Third version of the NUMA placement series Xen 4.2.

I'm afraid I get a segfault with this:

quartz:~# xl dest d32-1 ; gdb --args xl cr /etc/xen/debian-x86_32p-1
[...]
(gdb) r
Starting program: /usr/sbin/xl cr /etc/xen/debian-x86_32p-1
[Thread debugging using libthread_db enabled]
Parsing config from /etc/xen/debian-x86_32p-1

Program received signal SIGSEGV, Segmentation fault.
*__GI___libc_free (mem=0x1) at malloc.c:3710
3710 malloc.c: No such file or directory.
in malloc.c
(gdb) bt
#0 *__GI___libc_free (mem=0x1) at malloc.c:3710
#1 0xb7fa8b78 in libxl_bitmap_dispose (map=0xbffff170) at libxl_utils.c:510
#2 0xb7fadbde in libxl__get_numa_candidates (gc=0x806b7a0, min_free_memkb=141312, min_cpus=4, min_nodes=0, max_nodes=0, suitable_cpumap=0xbffff1f4, cndts=0xbffff208, nr_cndts=0xbffff20c) at libxl_numa.c:397
#3 0xb7fa45ec in numa_place_domain (gc=0x806b7a0, domid=8, info=0xbffff5dc, state=0x806a954) at libxl_dom.c:169
#4 libxl__build_pre (gc=0x806b7a0, domid=8, info=0xbffff5dc, state=0x806a954) at libxl_dom.c:232
#5 0xb7f98234 in libxl__domain_build (gc=0x806b7a0, info=0xbffff5dc, domid=8, state=0x806a954) at libxl_create.c:320
#6 0xb7f9859f in domcreate_bootloader_done (egc=0xbffff43c, bl=0x806a998, rc=0) at libxl_create.c:695
#7 0xb7fb5e50 in bootloader_callback (egc=<value optimized out>, bl=0x806a998, rc=0) at libxl_bootloader.c:256
#8 0xb7fb7482 in libxl__bootloader_run (egc=0xbffff43c, bl=0x806a998) at libxl_bootloader.c:394
#9 0xb7f99535 in initiate_domain_create (ctx=<value optimized out>, d_config=<value optimized out>, domid=0x8068354, restore_fd=-1, ao_how=0x0, aop_console_how=0x0) at libxl_create.c:635
#10 do_domain_create (ctx=<value optimized out>, d_config=<value optimized out>, domid=0x8068354, restore_fd=-1, ao_how=0x0, aop_console_how=0x0) at libxl_create.c:1039
#11 0xb7f9966f in libxl_domain_create_new (ctx=0x8069030, d_config=0xbffff5ac, domid=0x8068354, ao_how=0x0, aop_console_how=0x0) at libxl_create.c:1062
#12 0x0805c479 in create_domain (dom_info=<value optimized out>) at xl_cmdimpl.c:1809
#13 0x0805dd13 in main_create (argc=2, argv=0xbffffd28) at xl_cmdimpl.c:3774
#14 0x0804d1d6 in main (argc=3, argv=0xbffffd24) at xl.c:263
(gdb) frame 1
#1 0xb7fa8b78 in libxl_bitmap_dispose (map=0xbffff170) at libxl_utils.c:510
510 libxl_utils.c: No such file or directory.
in libxl_utils.c
(gdb) print *map
$2 = {size = 3221221764, map = 0x1 <Address 0x1 out of bounds>}
(gdb) frame 2
#2 0xb7fadbde in libxl__get_numa_candidates (gc=0x806b7a0, min_free_memkb=141312, min_cpus=4, min_nodes=0, max_nodes=0, suitable_cpumap=0xbffff1f4, cndts=0xbffff208, nr_cndts=0xbffff20c) at libxl_numa.c:397
397 libxl_numa.c: No such file or directory.
in libxl_numa.c
(gdb) print suitable_nodemap
$3 = {size = 3221221764, map = 0x1 <Address 0x1 out of bounds>}
(gdb) print nodemap
$4 = {size = 0, map = 0x0}

So it looks like suitable_nodemap wasn't initialised?

There are a few "goto out"s before initialising that variable, but none
of them log (really they should) and I didn't investigate which one it
was yet.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

Ian.Campbell at citrix

Jul 6, 2012, 4:20 AM

Post #5 of 10 (91 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Fri, 2012-07-06 at 12:16 +0100, Ian Campbell wrote:
> There are a few "goto out"s before initialising that variable, but none
> of them log (really they should) and I didn't investigate which one it
> was yet.

It seems to be:
/* If we don't have at least 2 nodes, it is useless to proceed */
if (nr_nodes < 2) {
LOG(DEBUG, "only %d node. no placement required", nr_nodes);
rc = 0;
goto out;
}

(LOG is mine...). The other exit paths look like the log further down
the stack but not this one so it is worth adding.

You probably want another libxl_bitmap_init near the top of the
function.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

Ian.Campbell at citrix

Jul 6, 2012, 4:22 AM

Post #6 of 10 (91 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Fri, 2012-07-06 at 12:20 +0100, Ian Campbell wrote:

> You probably want another libxl_bitmap_init near the top of the
> function.

Works for me:

# HG changeset patch
# User Ian Campbell <ian.campbell [at] citrix>
# Date 1341573735 -3600
# Node ID 4f964e2446c935838f54b9cae48a6c62fd8de3d0
# Parent 124ddd91c8de38204e94d3125013a40aaa326774
[mq]: libxl-numa-place-segfault.patch

diff -r 124ddd91c8de -r 4f964e2446c9 tools/libxl/libxl_numa.c
--- a/tools/libxl/libxl_numa.c Wed Jul 04 17:38:44 2012 +0200
+++ b/tools/libxl/libxl_numa.c Fri Jul 06 12:22:15 2012 +0100
@@ -258,6 +258,7 @@ int libxl__get_numa_candidates(libxl__gc
libxl_bitmap suitable_nodemap, nodemap;
int array_size, rc;

+ libxl_bitmap_init(&suitable_nodemap);
libxl_bitmap_init(&nodemap);

/* Get platform info and prepare the map for testing the combinations */
@@ -266,6 +267,7 @@ int libxl__get_numa_candidates(libxl__gc
return ERROR_FAIL;
/* If we don't have at least 2 nodes, it is useless to proceed */
if (nr_nodes < 2) {
+ LOG(DEBUG, "only %d node. no placement required", nr_nodes);
rc = 0;
goto out;
}

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

Ian.Campbell at citrix

Jul 6, 2012, 5:19 AM

Post #7 of 10 (92 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Wed, 2012-07-04 at 17:17 +0100, Dario Faggioli wrote:
> Hello,
>
> Third version of the NUMA placement series Xen 4.2.
>
> All the comments received during v2's review have been addressed (more details
> in single changelogs).
>
> The most notable changes are the following:
> - the libxl_cpumap --> libxl_bitmap renaming has been rebased on top of the
> recent patches that allows us to allocate bitmaps of different sizes;
> - the heuristics for deciding which NUMA placement is the best one has been
> redesigned, so that it now provides total ordering.
>
> Here it is what this posting contains (* = acked during previous round):
>
> * [PATCH 01 of 10 v3] libxl: add a new Array type to the IDL
> [PATCH 02 of 10 v3] libxl,libxc: introduce libxl_get_numainfo()
> * [PATCH 03 of 10 v3] xl: add more NUMA information to `xl info -n'
> [PATCH 04 of 10 v3] libxl: rename libxl_cpumap to libxl_bitmap
> [PATCH 05 of 10 v3] libxl: expand the libxl_bitmap API a bit
> * [PATCH 06 of 10 v3] libxl: introduce some node map helpers
> [PATCH 07 of 10 v3] libxl: explicitly check for libmath in autoconf

These are now sufficiently acked that I have committed them.

> Is where data structures, utility functions and infrastructure are introduced.
>
> * [PATCH 08 of 10 v3] libxl: enable automatic placement of guests on NUMA nodes

As discussed this one has a few issues so I stopped before committing
this one.

Ian.

> * [PATCH 09 of 10 v3] libxl: have NUMA placement deal with cpupools
>
> Host the core of the mechanism.
>
> * [PATCH 10 of 10 v3] Some automatic NUMA placement documentation
>
> For some more documentation.
>
> Thanks a lot and Regards,
> Dario
>

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

raistlin at linux

Jul 6, 2012, 6:05 AM

Post #8 of 10 (92 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Fri, 2012-07-06 at 12:22 +0100, Ian Campbell wrote:
> On Fri, 2012-07-06 at 12:20 +0100, Ian Campbell wrote:
>
> > You probably want another libxl_bitmap_init near the top of the
> > function.
>
Damn. I tested with two, with (fake) 4 and (fake) 8, but I guess I
forgot testing with just one node this time. Sorry for that. :-(

> Works for me:
>
Good. Thank you,
Dario

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachments:

signature.asc (0.19 KB)

ian.campbell at citrix

Jul 8, 2012, 11:32 AM

Post #9 of 10 (92 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

(resend as I don't think I had SMTP setup properly on my laptop, sorry if you get this twice!)

On Wed, 2012-07-04 at 12:17 -0400, Dario Faggioli wrote:
> Hello,
>
> Third version of the NUMA placement series Xen 4.2.

Starting an HVM guest (although I don't suppose it is HVM specific) on a
none NUMA system I get this:

libxl: debug: libxl_numa.c:270:libxl__get_numa_candidates: only 1 node. no placement required
libxl: detail: libxl_dom.c:175:numa_place_domain: 0 NUMA placement candidates found
libxl: notice: libxl_dom.c:180:numa_place_domain: NUMA placement failed, performance might be affected

this being a non-NUMA system I suppose it is not the end of the world.
It'd be nice to avoid the warning though -- perhaps
libxl__get_numa_candidates should either not special case single node
systems or it should manually return the trivial candidate set?

> All the comments received during v2's review have been addressed (more details
> in single changelogs).
>
> The most notable changes are the following:
> - the libxl_cpumap --> libxl_bitmap renaming has been rebased on top of the
> recent patches that allows us to allocate bitmaps of different sizes;
> - the heuristics for deciding which NUMA placement is the best one has been
> redesigned, so that it now provides total ordering.
>
> Here it is what this posting contains (* = acked during previous round):
>
> * [PATCH 01 of 10 v3] libxl: add a new Array type to the IDL
> [PATCH 02 of 10 v3] libxl,libxc: introduce libxl_get_numainfo()
> * [PATCH 03 of 10 v3] xl: add more NUMA information to `xl info -n'
> [PATCH 04 of 10 v3] libxl: rename libxl_cpumap to libxl_bitmap
> [PATCH 05 of 10 v3] libxl: expand the libxl_bitmap API a bit
> * [PATCH 06 of 10 v3] libxl: introduce some node map helpers
> [PATCH 07 of 10 v3] libxl: explicitly check for libmath in autoconf
>
> Is where data structures, utility functions and infrastructure are introduced.
>
> * [PATCH 08 of 10 v3] libxl: enable automatic placement of guests on NUMA nodes
> * [PATCH 09 of 10 v3] libxl: have NUMA placement deal with cpupools
>
> Host the core of the mechanism.
>
> * [PATCH 10 of 10 v3] Some automatic NUMA placement documentation
>
> For some more documentation.
>
> Thanks a lot and Regards,
> Dario
>

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

raistlin at linux

Jul 9, 2012, 7:32 AM

Post #10 of 10 (95 views)
Permalink

Re: [PATCH 00 of 10 v3] Automatic NUMA placement for xl [In reply to]

On Sun, 2012-07-08 at 12:32 -0600, Ian Campbell wrote:
> On Wed, 2012-07-04 at 12:17 -0400, Dario Faggioli wrote:
> > Hello,
> >
> > Third version of the NUMA placement series Xen 4.2.
>
> Starting an HVM guest (although I don't suppose it is HVM specific) on a
> none NUMA system I get this:
>
> libxl: debug: libxl_numa.c:270:libxl__get_numa_candidates: only 1 node. no placement required
> libxl: detail: libxl_dom.c:175:numa_place_domain: 0 NUMA placement candidates found
> libxl: notice: libxl_dom.c:180:numa_place_domain: NUMA placement failed, performance might be affected
>
Mmm... I see. No, it's not HVM specific and you're right, it does not
make much sense.

> this being a non-NUMA system I suppose it is not the end of the world.
> It'd be nice to avoid the warning though -- perhaps
> libxl__get_numa_candidates should either not special case single node
> systems or it should manually return the trivial candidate set?
>
Yep, I can definitely do that, along with the other fix you suggested
and the wording of the doc patch and resend the last three.

Thanks and Regards,
Dario

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachments:

signature.asc (0.19 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads