Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Xen: Devel

xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

 

 

First page Previous page 1 2 Next page Last page  View All Xen devel RSS feed   Index | Next | Previous | View Threaded


konrad.wilk at oracle

Apr 30, 2012, 12:37 PM

Post #1 of 26 (632 views)
Permalink
xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

I somehow thought that this has been fixed but I've been
getting reports that people are running into this.

What kind of fix do I need the in the kernel? I see this:
255 xen_cpuid(&ax, &bx, &cx, &dx);
256
257 xsave_mask =
258 (1 << (X86_FEATURE_XSAVE % 32)) |
259 (1 << (X86_FEATURE_OSXSAVE % 32));
260
261 /* Xen will set CR4.OSXSAVE if supported and not disabled by force */
262 if ((cx & xsave_mask) != xsave_mask)
263 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE & OSXSAVE */
264 }

But do I need some other one?


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


JBeulich at suse

May 2, 2012, 2:00 AM

Post #2 of 26 (591 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk [at] oracle> wrote:
> I somehow thought that this has been fixed but I've been
> getting reports that people are running into this.

"this" being what? I too thought that all xsave related issues were
sorted out by now.

Jan

> What kind of fix do I need the in the kernel? I see this:
> 255 xen_cpuid(&ax, &bx, &cx, &dx);
> 256
> 257 xsave_mask =
> 258 (1 << (X86_FEATURE_XSAVE % 32)) |
> 259 (1 << (X86_FEATURE_OSXSAVE % 32));
> 260
> 261 /* Xen will set CR4.OSXSAVE if supported and not disabled by
> force */
> 262 if ((cx & xsave_mask) != xsave_mask)
> 263 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE &
> OSXSAVE */
> 264 }
>
> But do I need some other one?




_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


apxeng at gmail

May 2, 2012, 11:42 AM

Post #3 of 26 (591 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk [at] oracle> wrote:
>> I somehow thought that this has been fixed but I've been
>> getting reports that people are running into this.
>
> "this" being what? I too thought that all xsave related issues were
> sorted out by now.

I see the following crash if I run without xsave=0 with Ubuntu 11.10
3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.

[ 7.509303] invalid opcode: 0000 [#1] SMP
[ 7.509319] CPU 0
[ 7.509325] Modules linked in:
[ 7.509337]
[ 7.509347] Pid: 0, comm: swapper Not tainted 3.0.0-17-generic
#30-Ubuntu LENOVO 4286CTO/4286CTO
[ 7.509371] RIP: e030:[<ffffffff810140ec>] [<ffffffff810140ec>]
xstate_enable+0x3c/0x50
[ 7.509399] RSP: e02b:ffffffff81c01e58 EFLAGS: 00010046
[ 7.509409] RAX: 0000000000000007 RBX: ffffffff81c01e94 RCX: 0000000000000000
[ 7.509419] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000002660
[ 7.509430] RBP: ffffffff81c01e58 R08: ffffffff81c01e90 R09: ffffffff81c01e94
[ 7.509440] R10: 00000000ffffffff R11: 00000000ffffffff R12: ffffffff81c01e90
[ 7.509501] R13: ffffffff81c01e8c R14: ffffffff81c01e88 R15: ffff8801e97ecd00
[ 7.509519] FS: 0000000000000000(0000) GS:ffff8801e97e0000(0000)
knlGS:0000000000000000
[ 7.509530] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7.509539] CR2: 0000000000000000 CR3: 0000000001c03000 CR4: 0000000000002660
[ 7.509550] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7.509561] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7.509573] Process swapper (pid: 0, threadinfo ffffffff81c00000,
task ffffffff81c0b020)
[ 7.509582] Stack:
[ 7.509589] ffffffff81c01ec8 ffffffff81cd97ac 0000000000000040
0000000000000000
[ 7.509613] ffffffff81007b4f ffffffff81004057 0000024000000007
0000000000000340
[ 7.509637] ffff8801e97eb100 0000000000000008 0000000000000004
0000000000000000
[ 7.509660] Call Trace:
[ 7.509679] [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[ 7.509698] [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[ 7.509712] [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[ 7.509730] [<ffffffff815d06eb>] xsave_init+0x26/0x28
[ 7.509744] [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[ 7.509759] [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[ 7.509774] [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[ 7.509789] [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[ 7.509805] [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
[ 7.509814] Code: 00 04 00 ff 14 25 10 33 c1 81 48 89 c7 48 81 cf
00 00 04 00 ff 14 25 18 33 c1 81 48 8b 05 0d 15 db 00 31 c9 48 89 c2
48 c1 ea 20 <0f> 01 d1 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00
00 55
[ 7.510045] RIP [<ffffffff810140ec>] xstate_enable+0x3c/0x50
[ 7.510062] RSP <ffffffff81c01e58>
[ 7.510086] ---[ end trace a7919e7f17c0a725 ]---
[ 7.510097] Kernel panic - not syncing: Attempted to kill the idle task!
[ 7.510109] Pid: 0, comm: swapper Tainted: G D
3.0.0-17-generic #30-Ubuntu
[ 7.510119] Call Trace:
[ 7.510134] [<ffffffff815dca66>] panic+0x91/0x194
[ 7.510151] [<ffffffff8106344b>] do_exit+0x40b/0x440
[ 7.510168] [<ffffffff815f4350>] oops_end+0xb0/0xf0
[ 7.510181] [<ffffffff8100d938>] die+0x58/0x90
[ 7.510195] [<ffffffff815f3a34>] do_trap+0xc4/0x170
[ 7.510211] [<ffffffff8100af25>] do_invalid_op+0x95/0xb0
[ 7.510226] [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[ 7.510240] [<ffffffff81007b62>] ? check_events+0x12/0x20
[ 7.510255] [<ffffffff81008239>] ? get_phys_to_machine+0x9/0x70
[ 7.510269] [<ffffffff81005c69>] ? pte_mfn_to_pfn+0x89/0xf0
[ 7.510283] [<ffffffff8100743d>] ? xen_force_evtchn_callback+0xd/0x10
[ 7.510299] [<ffffffff815fc2db>] invalid_op+0x1b/0x20
[ 7.510315] [<ffffffff810140ec>] ? xstate_enable+0x3c/0x50
[ 7.510329] [<ffffffff810140dc>] ? xstate_enable+0x2c/0x50
[ 7.510343] [<ffffffff81cd97ac>] xstate_enable_boot_cpu+0xa9/0x174
[ 7.510358] [<ffffffff81007b4f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[ 7.510371] [<ffffffff81004057>] ? xen_write_cr0+0x77/0x90
[ 7.510385] [<ffffffff815d06eb>] xsave_init+0x26/0x28
[ 7.510399] [<ffffffff815d2932>] cpu_init+0x2bb/0x2d8
[ 7.510413] [<ffffffff81cd5ff4>] trap_init+0x169/0x171
[ 7.510427] [<ffffffff81cd0a27>] start_kernel+0x1d0/0x3df
[ 7.510441] [<ffffffff81cd0388>] x86_64_start_reservations+0x132/0x136
[ 7.510457] [<ffffffff81cd3ec6>] xen_start_kernel+0x5ac/0x5b3
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.


>> What kind of fix do I need the in the kernel? I see this:
>>  255         xen_cpuid(&ax, &bx, &cx, &dx);
>>  256
>>  257         xsave_mask =
>>  258                 (1 << (X86_FEATURE_XSAVE % 32)) |
>>  259                 (1 << (X86_FEATURE_OSXSAVE % 32));
>>  260
>>  261         /* Xen will set CR4.OSXSAVE if supported and not disabled by
>> force */
>>  262         if ((cx & xsave_mask) != xsave_mask)
>>  263                 cpuid_leaf1_ecx_mask &= ~xsave_mask; /* disable XSAVE &
>> OSXSAVE */
>>  264 }
>>
>> But do I need some other one?
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel [at] lists
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


JBeulich at suse

May 3, 2012, 2:15 AM

Post #4 of 26 (592 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>>> On 02.05.12 at 20:42, AP <apxeng [at] gmail> wrote:
> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk [at] oracle> wrote:
>>> I somehow thought that this has been fixed but I've been
>>> getting reports that people are running into this.
>>
>> "this" being what? I too thought that all xsave related issues were
>> sorted out by now.
>
> I see the following crash if I run without xsave=0 with Ubuntu 11.10
> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.

And in the thread starting at
http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
I gave debugging instructions that apparently no-one followed so
far. Without someone seeing the problem doing so I don't think we
will ever get anywhere with this (unless, as also indicated there,
someone can spot something wrong with the code that non-obvious
to everyone else).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


apxeng at gmail

May 3, 2012, 11:09 AM

Post #5 of 26 (583 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>> On 02.05.12 at 20:42, AP <apxeng [at] gmail> wrote:
>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk [at] oracle> wrote:
>>>> I somehow thought that this has been fixed but I've been
>>>> getting reports that people are running into this.
>>>
>>> "this" being what? I too thought that all xsave related issues were
>>> sorted out by now.
>>
>> I see the following crash if I run without xsave=0 with Ubuntu 11.10
>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>
> And in the thread starting at
> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
> I gave debugging instructions that apparently no-one followed so
> far. Without someone seeing the problem doing so I don't think we
> will ever get anywhere with this (unless, as also indicated there,
> someone can spot something wrong with the code that non-obvious
> to everyone else).

I missed that thread. I will add some debugging to
pv_guest_cr4_fixup() and the XSETBV handling in
emulate_privileged_op() and post the output.

Thanks,
AP

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


apxeng at gmail

May 4, 2012, 12:30 PM

Post #6 of 26 (587 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Thu, May 3, 2012 at 11:09 AM, AP <apxeng [at] gmail> wrote:
> On Thu, May 3, 2012 at 2:15 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>>> On 02.05.12 at 20:42, AP <apxeng [at] gmail> wrote:
>>> On Wed, May 2, 2012 at 2:00 AM, Jan Beulich <JBeulich [at] suse> wrote:
>>>>>>> On 30.04.12 at 21:37, Konrad Rzeszutek Wilk <konrad.wilk [at] oracle> wrote:
>>>>> I somehow thought that this has been fixed but I've been
>>>>> getting reports that people are running into this.
>>>>
>>>> "this" being what? I too thought that all xsave related issues were
>>>> sorted out by now.
>>>
>>> I see the following crash if I run without xsave=0 with Ubuntu 11.10
>>> 3.0.0-17 kernel (Intel(R) Core(TM) i7-2620M). I don't see this with
>>> Xen 4.1.2. Looks like the OSXSAVE bit is not getting set in CR4.
>>
>> And in the thread starting at
>> http://lists.xen.org/archives/html/xen-devel/2012-04/msg00426.html
>> I gave debugging instructions that apparently no-one followed so
>> far. Without someone seeing the problem doing so I don't think we
>> will ever get anywhere with this (unless, as also indicated there,
>> someone can spot something wrong with the code that non-obvious
>> to everyone else).
>
> I missed that thread. I will add some debugging to
> pv_guest_cr4_fixup() and the XSETBV handling in
> emulate_privileged_op() and post the output.

I ran with the attached xsave_debug patch and saw the following output:
<snip>
(XEN) CPU: After generic identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) CPU: After vendor identify, caps: bfebfbff 28100800 00000000
00000000 17bae3ff 00000000 00000001 00000000
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) CPU: After all inits, caps: bfebfbff 28100800 00000000 00003f40
17bae3ff 00000000 00000001 00000000
<snip>
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[ 6.791945] invalid opcode: 0000 [#1] SMP
<snip>

From the above I realized that X86_CR4_OSXSAVE was never getting set
in v->arch.pv_vcpu.ctrlreg[4]. So I tried the following patch:

diff -r 5a0d60bb536b xen/arch/x86/domain.c
--- a/xen/arch/x86/domain.c Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/arch/x86/domain.c Fri May 04 12:23:57 2012 -0700
@@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
hv_cr4_mask &= ~X86_CR4_DE;
if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
hv_cr4_mask &= ~X86_CR4_FSGSBASE;
- if ( xsave_enabled(v) )
- hv_cr4_mask &= ~X86_CR4_OSXSAVE;

if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
gdprintk(XENLOG_WARNING,
diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
--- a/xen/include/asm-x86/domain.h Fri Apr 27 21:10:59 2012 -0700
+++ b/xen/include/asm-x86/domain.h Fri May 04 12:23:57 2012 -0700
@@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
& ~X86_CR4_DE)
#define real_cr4_to_pv_guest_cr4(c) \
((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD \
- | X86_CR4_OSXSAVE | X86_CR4_SMEP))
+ | X86_CR4_SMEP))

void domain_cpuid(struct domain *d,
unsigned int input,

That allowed the system to boot successfully though I did see the
following message:
(XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660

Not sure if the above patch is right fix but I hope it was at least
helpful in pointing at where the problem might be.

BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.

Thanks,
AP
Attachments: xsave_debug.patch (2.83 KB)


JBeulich at suse

May 7, 2012, 12:15 AM

Post #7 of 26 (578 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>>> On 04.05.12 at 21:30, AP <apxeng [at] gmail> wrote:
> From the above I realized that X86_CR4_OSXSAVE was never getting set
> in v->arch.pv_vcpu.ctrlreg[4].

Yes, that was the observation in the previous thread too, but the
reporter didn't seem interested in continuing on from there.


> So I tried the following patch:
>
> diff -r 5a0d60bb536b xen/arch/x86/domain.c
> --- a/xen/arch/x86/domain.c Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/arch/x86/domain.c Fri May 04 12:23:57 2012 -0700
> @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> hv_cr4_mask &= ~X86_CR4_DE;
> if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> - if ( xsave_enabled(v) )
> - hv_cr4_mask &= ~X86_CR4_OSXSAVE;
>
> if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> gdprintk(XENLOG_WARNING,
> diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> --- a/xen/include/asm-x86/domain.h Fri Apr 27 21:10:59 2012 -0700
> +++ b/xen/include/asm-x86/domain.h Fri May 04 12:23:57 2012 -0700
> @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> & ~X86_CR4_DE)
> #define real_cr4_to_pv_guest_cr4(c) \
> ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD \
> - | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> + | X86_CR4_SMEP))
>
> void domain_cpuid(struct domain *d,
> unsigned int input,

No, this is specifically the wrong thing. From what we know so far
(i.e. the outcome of the above printing you added) the problem in
in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
attempting XSETBV). What your patch efectively does is take away
control from the guest kernels to control the (virtual) CR4 flag...

> That allowed the system to boot successfully though I did see the
> following message:
> (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660

... which is what this message is telling you.

> Not sure if the above patch is right fix but I hope it was at least
> helpful in pointing at where the problem might be.
>
> BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.

Sure, as it's a kernel problem. It's the kernel that needs logging added,
to find out why the CR4 write supposedly happening immediately
prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
happen, or doesn't set the flag. Perhaps something fishy going on
with the paravirt ops patching, since the disassembly of the opcode
bytes shown with the oops message are indicating that the right
thing is being attempted:

ff 14 25 10 33 c1 81 callq *0xffffffff81c13310
48 89 c7 mov %rax,%rdi
48 81 cf 00 00 04 00 or $0x40000,%rdi
^^^^^^^^
ff 14 25 18 33 c1 81 callq *0xffffffff81c13318
48 8b 05 0d 15 db 00 mov 0xdb150d(%rip),%rax
31 c9 xor %ecx,%ecx
48 89 c2 mov %rax,%rdx
48 c1 ea 20 shr $0x20,%rdx
0f 01 d1 xsetbv
5d pop %rbp
c3 retq

The primary thing that strikes me as odd is that both calls are still
indirect ones, even though I thought that they should get replaced
by direct ones (or even the actual instruction, namely in the
read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

And the dumped %rdi value indicates that bit 18 did _not_ get set.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad.wilk at oracle

May 7, 2012, 9:07 AM

Post #8 of 26 (577 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Mon, May 07, 2012 at 08:15:44AM +0100, Jan Beulich wrote:
> >>> On 04.05.12 at 21:30, AP <apxeng [at] gmail> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
>
> Yes, that was the observation in the previous thread too, but the
> reporter didn't seem interested in continuing on from there.
>
>
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> > hv_cr4_mask &= ~X86_CR4_DE;
> > if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> > hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > - if ( xsave_enabled(v) )
> > - hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> >
> > if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> > gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> > & ~X86_CR4_DE)
> > #define real_cr4_to_pv_guest_cr4(c) \
> > ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD \
> > - | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > + | X86_CR4_SMEP))
> >
> > void domain_cpuid(struct domain *d,
> > unsigned int input,
>
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
>
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
>
> ... which is what this message is telling you.
>
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> >
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with xsave=1.
>
> Sure, as it's a kernel problem. It's the kernel that needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> happen, or doesn't set the flag. Perhaps something fishy going on
> with the paravirt ops patching, since the disassembly of the opcode
> bytes shown with the oops message are indicating that the right
> thing is being attempted:
>
> ff 14 25 10 33 c1 81 callq *0xffffffff81c13310 [so xen_read_cr4 which is native_read_cr4]
> 48 89 c7 mov %rax,%rdi
> 48 81 cf 00 00 04 00 or $0x40000,%rdi
> ^^^^^^^^
> ff 14 25 18 33 c1 81 callq *0xffffffff81c13318 [so xen_write_cr4] - which is filtering X86_CR4_PGE and X86_CR4_PSE]
> 48 8b 05 0d 15 db 00 mov 0xdb150d(%rip),%rax
> 31 c9 xor %ecx,%ecx
> 48 89 c2 mov %rax,%rdx
> 48 c1 ea 20 shr $0x20,%rdx
> 0f 01 d1 xsetbv
> 5d pop %rbp
> c3 retq
>
> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

They do get replaced (during runtime - and this is done by the
alternative_instructions). Is this output from objdump or straight
from the memory (so using the Xen debugger?).


>
> And the dumped %rdi value indicates that bit 18 did _not_ get set.

That would imply that xen_write_cr4, which is just mov to cr4
is getting trapped but somehow the hypervisor isn't setting the
rdi value? Or maybe the the native_write_cr4 ends up
filtering in the wrong order?

0xffffffff8102e650 <xen_write_cr4>: push %rbp
0xffffffff8102e651 <xen_write_cr4+1>: and $0x6f,%dil
0xffffffff8102e655 <xen_write_cr4+5>: mov %rsp,%rbp
0xffffffff8102e658 <xen_write_cr4+8>: mov %rdi,%cr4
0xffffffff8102e65b <xen_write_cr4+11>: leaveq
0xffffffff8102e65c <xen_write_cr4+12>: retq


>
> Jan

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jeremy at goop

May 7, 2012, 3:55 PM

Post #9 of 26 (575 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On 05/07/2012 12:15 AM, Jan Beulich wrote:
> The primary thing that strikes me as odd is that both calls are still
> indirect ones, even though I thought that they should get replaced
> by direct ones (or even the actual instruction, namely in the
> read_cr4() case) upon first use. Konrad, Jeremy - am I wrong here?

Patching happens as an explicit step during (moderately late) boot, not
as a side-effect of the first call, so it doesn't surprise me too much
that they haven't been updated at this point.

J

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


apxeng at gmail

May 7, 2012, 4:57 PM

Post #10 of 26 (576 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich [at] suse> wrote:
>
> >>> On 04.05.12 at 21:30, AP <apxeng [at] gmail> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
>
> Yes, that was the observation in the previous thread too, but the
> reporter didn't seem interested in continuing on from there.
>
>
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c   Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c   Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> >
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h    Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h    Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> >
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
>
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
>
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
>
> ... which is what this message is telling you.
>
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> >
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > xsave=1.
>
> Sure, as it's a kernel problem. It's the kernel that needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> happen, or doesn't set the flag. Perhaps something fishy going on

xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

static void xen_write_cr4(unsigned long cr4)
{
    cr4 &= ~X86_CR4_PGE;
    cr4 &= ~X86_CR4_PSE;
    cr4 &= ~X86_CR4_OSXSAVE;

    native_write_cr4(cr4);
}

I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
is what I see:

<snip>
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
[    6.838041] set_in_cr4: cr4: 0x42660
[    6.841743] xen_write_cr4: cr4: 0x2660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
[    6.860546] xstate_enable: Exec xsetbv
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.870509] invalid opcode: 0000 [#1] SMP
<snip>

Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.

(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[ 7.554262] Setting X86_CR4_OSXSAVE
[ 7.557869] set_in_cr4: cr4: 0x42660
[ 7.561551] xen_write_cr4: cr4: 0x42660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x42660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
[ 7.580522] Exec xsetbv
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[ 7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340

Thanks,
AP

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad.wilk at oracle

May 7, 2012, 5:08 PM

Post #11 of 26 (578 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

> > No, this is specifically the wrong thing. From what we know so far
> > (i.e. the outcome of the above printing you added) the problem in
> > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > attempting XSETBV). What your patch efectively does is take away
> > control from the guest kernels to control the (virtual) CR4 flag...
> >
> > > That allowed the system to boot successfully though I did see the
> > > following message:
> > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
> >
> > ... which is what this message is telling you.
> >
> > > Not sure if the above patch is right fix but I hope it was at least
> > > helpful in pointing at where the problem might be.
> > >
> > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > xsave=1.
> >
> > Sure, as it's a kernel problem. It's the kernel that needs logging added,
> > to find out why the CR4 write supposedly happening immediately
> > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > happen, or doesn't set the flag. Perhaps something fishy going on
>
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

Where did you see that code? Looking at the Linus's tree this is what I see
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD

So who added that code? I am not seeing it in v3.0 either?
>
> static void xen_write_cr4(unsigned long cr4)
> {
>     cr4 &= ~X86_CR4_PGE;
>     cr4 &= ~X86_CR4_PSE;
>     cr4 &= ~X86_CR4_OSXSAVE;
>
>     native_write_cr4(cr4);
> }
>
> I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
> is what I see:
>
> <snip>
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
> [    6.838041] set_in_cr4: cr4: 0x42660
> [    6.841743] xen_write_cr4: cr4: 0x2660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x2660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
> [    6.860546] xstate_enable: Exec xsetbv
> (XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
> [    6.870509] invalid opcode: 0000 [#1] SMP
> <snip>
>
> Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.
>
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [ 7.554262] Setting X86_CR4_OSXSAVE
> [ 7.557869] set_in_cr4: cr4: 0x42660
> [ 7.561551] xen_write_cr4: cr4: 0x42660
> (XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
> 0xfffffffffffbfff3 returning cr4: 0x42660
> (XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
> [ 7.580522] Exec xsetbv
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> (XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
> [ 7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
>
> Thanks,
> AP

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


apxeng at gmail

May 7, 2012, 5:41 PM

Post #12 of 26 (577 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
konrad.wilk [at] oracle> wrote:
>
> > > No, this is specifically the wrong thing. From what we know so far
> > > (i.e. the outcome of the above printing you added) the problem in
> > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > attempting XSETBV). What your patch efectively does is take away
> > > control from the guest kernels to control the (virtual) CR4 flag...
> > >
> > > > That allowed the system to boot successfully though I did see the
> > > > following message:
> > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > 00002660
> > >
> > > ... which is what this message is telling you.
> > >
> > > > Not sure if the above patch is right fix but I hope it was at least
> > > > helpful in pointing at where the problem might be.
> > > >
> > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > xsave=1.
> > >
> > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > added,
> > > to find out why the CR4 write supposedly happening immediately
> > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > happen, or doesn't set the flag. Perhaps something fishy going on
> >
> > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
>
> Where did you see that code? Looking at the Linus's tree this is what I
> see
>
>
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
>
> So who added that code? I am not seeing it in v3.0 either?

This is in the Ubuntu 11.10 kernel:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812

After some digging around, it looks like this is an Ubuntu 11.10 only patch:
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Thanks,
AP


JBeulich at suse

May 7, 2012, 11:25 PM

Post #13 of 26 (576 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>>> On 08.05.12 at 01:57, AP <apxeng [at] gmail> wrote:
> On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich [at] suse> wrote:
>> Sure, as it's a kernel problem. It's the kernel that needs logging added,
>> to find out why the CR4 write supposedly happening immediately
>> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
>> happen, or doesn't set the flag. Perhaps something fishy going on
>
> xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
>
> static void xen_write_cr4(unsigned long cr4)
> {
> cr4 &= ~X86_CR4_PGE;
> cr4 &= ~X86_CR4_PSE;
> cr4 &= ~X86_CR4_OSXSAVE;
>
> native_write_cr4(cr4);
> }

That's definitely not the case in _any_ upstream Linux release.
Which means that this must be introduced by a distro-specific
(and broken) patch.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad.wilk at oracle

May 8, 2012, 9:39 AM

Post #14 of 26 (577 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk [at] oracle> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know so far
> > > > (i.e. the outcome of the above printing you added) the problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take away
> > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > >
> > > > > That allowed the system to boot successfully though I did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus's tree this is what I
> > see
> >
> >
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
>
> This is in the Ubuntu 11.10 kernel:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
>
> After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b

Which seems to say that Amazon's HV is advertising the OXSAVE bit?

Lets ping them and see if they have some recomendations.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


msw at amazon

May 8, 2012, 10:02 AM

Post #15 of 26 (582 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Tue, May 08, 2012 at 09:39:57AM -0700, Konrad Rzeszutek Wilk wrote:
> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk [at] oracle> wrote:
> >
> > This is in the Ubuntu 11.10 kernel:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> >
> > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
>
> Which seems to say that Amazon's HV is advertising the OXSAVE bit?
>
> Lets ping them and see if they have some recomendations.

Hi,

The kernel requirements for EC2 are documented in our User Guide here:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/UserProvidedkernels.html#PVGRUB_compatible_kernels

Specifically, there's a callout regarding XSAVE:
"Kernels that disable the pv-ops XSAVE hypercall are known to work
on all instance types, whereas those that enable this hypercall
will fail to launch in some cases. Similarly, non-pv-ops kernels
that do not adhere to the Xen 3.0.2 interface might fail to launch
in some cases."

Apologies for the miswording of the note, it should say something like
"kernels that disable the XSAVE capability" instead.

Cheers,

Matt

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad.wilk at oracle

May 8, 2012, 5:35 PM

Post #16 of 26 (575 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> konrad.wilk [at] oracle> wrote:
> >
> > > > No, this is specifically the wrong thing. From what we know so far
> > > > (i.e. the outcome of the above printing you added) the problem in
> > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > attempting XSETBV). What your patch efectively does is take away
> > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > >
> > > > > That allowed the system to boot successfully though I did see the
> > > > > following message:
> > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > 00002660
> > > >
> > > > ... which is what this message is telling you.
> > > >
> > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > helpful in pointing at where the problem might be.
> > > > >
> > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > xsave=1.
> > > >
> > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > added,
> > > > to find out why the CR4 write supposedly happening immediately
> > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > >
> > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> >
> > Where did you see that code? Looking at the Linus's tree this is what I
> > see
> >
> >
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> >
> > So who added that code? I am not seeing it in v3.0 either?
>
> This is in the Ubuntu 11.10 kernel:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
>
> After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b


Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad at darnok

May 9, 2012, 6:11 AM

Post #17 of 26 (575 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > konrad.wilk [at] oracle> wrote:
> > >
> > > > > No, this is specifically the wrong thing. From what we know so far
> > > > > (i.e. the outcome of the above printing you added) the problem in
> > > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > > attempting XSETBV). What your patch efectively does is take away
> > > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > > >
> > > > > > That allowed the system to boot successfully though I did see the
> > > > > > following message:
> > > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > > 00002660
> > > > >
> > > > > ... which is what this message is telling you.
> > > > >
> > > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > > helpful in pointing at where the problem might be.
> > > > > >
> > > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > > xsave=1.
> > > > >
> > > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > > added,
> > > > > to find out why the CR4 write supposedly happening immediately
> > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > > >
> > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > >
> > > Where did you see that code? Looking at the Linus's tree this is what I
> > > see
> > >
> > >
> > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > >
> > > So who added that code? I am not seeing it in v3.0 either?
> >
> > This is in the Ubuntu 11.10 kernel:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> >
> > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
>
>
> Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
>

CC-ing Intel. It seems that the userspace programs are crashingon
Sandybridge machines even if 'xsave=0' is provided on the command line.
Any advice?
> _______________________________________________
> Xen-devel mailing list
> Xen-devel [at] lists
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


Ian.Campbell at citrix

May 9, 2012, 6:30 AM

Post #18 of 26 (576 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Wed, 2012-05-09 at 14:11 +0100, Konrad Rzeszutek Wilk wrote:
> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> > > On Mon, May 7, 2012 at 5:08 PM, Konrad Rzeszutek Wilk <
> > > konrad.wilk [at] oracle> wrote:
> > > >
> > > > > > No, this is specifically the wrong thing. From what we know so far
> > > > > > (i.e. the outcome of the above printing you added) the problem in
> > > > > > in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> > > > > > attempting XSETBV). What your patch efectively does is take away
> > > > > > control from the guest kernels to control the (virtual) CR4 flag...
> > > > > >
> > > > > > > That allowed the system to boot successfully though I did see the
> > > > > > > following message:
> > > > > > > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 ->
> > > > > > > 00002660
> > > > > >
> > > > > > ... which is what this message is telling you.
> > > > > >
> > > > > > > Not sure if the above patch is right fix but I hope it was at least
> > > > > > > helpful in pointing at where the problem might be.
> > > > > > >
> > > > > > > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > > > > > > xsave=1.
> > > > > >
> > > > > > Sure, as it's a kernel problem. It's the kernel that needs logging
> > > > > > added,
> > > > > > to find out why the CR4 write supposedly happening immediately
> > > > > > prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> > > > > > happen, or doesn't set the flag. Perhaps something fishy going on
> > > > >
> > > > > xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.
> > > >
> > > > Where did you see that code? Looking at the Linus's tree this is what I
> > > > see
> > > >
> > > >
> > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=arch/x86/xen/enlighten.c;h=a8f8844b8d32690b8a189bc37d12cd3f286a81cd;hb=HEAD
> > > >
> > > > So who added that code? I am not seeing it in v3.0 either?
> > >
> > > This is in the Ubuntu 11.10 kernel:
> > > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=blob;f=arch/x86/xen/enlighten.c;h=ce01f6d63507fd44288989ca0ba81a0f5bf04e3f;hb=HEAD#l812
> > >
> > > After some digging around, it looks like this is an Ubuntu 11.10 only patch:
> > > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fba59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b824160b
> >
> >
> > Ugh. There are looks to be a bug in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> >
>
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if 'xsave=0' is provided on the command line.

Is this related to the glibc bug which uses xsave/avx without properly
following the defined procedure to detect if it is available, as
described in http://marc.info/?l=xen-devel&m=133612371602480 ?

That bug has come up a couple of times recently, I'm not sure if this is
the same one though.

Ian.

> Any advice?
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel [at] lists
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel [at] lists
> http://lists.xen.org/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


JBeulich at suse

May 9, 2012, 6:34 AM

Post #19 of 26 (588 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk <konrad [at] darnok> wrote:
> On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
>> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
>> > After some digging around, it looks like this is an Ubuntu 11.10 only
> patch:
>> >
> http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb
> a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> 0b
>>
>>
>> Ugh. There are looks to be a bug in Fedora as well:
> https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
>>
>
> CC-ing Intel. It seems that the userspace programs are crashingon
> Sandybridge machines even if 'xsave=0' is provided on the command line.
> Any advice?

Going through the bug, all I can see are bogus attempts to pass
"xsave=" (with value 0 or 1) to the kernel. That's a hypervisor
option though, and the only kernel option that's relevant here
is "noxsave" afaik.

Further, when the hypervisor has xsave support enabled, there's
(in the pv case) nothing the kernel can do to hide the functionality
from applications, as the hardware's CR4.OSXSAVE will be set, and
hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
command line when "xsave=1" (or xsave enabled by default as in
4.2) just doesn't make any sense.

And finally one always has to keep in mind that there is this nice
glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
when trying to determine whether AVX or FMA is available.

Jan

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad.wilk at oracle

May 9, 2012, 10:38 AM

Post #20 of 26 (581 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On Wed, May 09, 2012 at 02:34:46PM +0100, Jan Beulich wrote:
> >>> On 09.05.12 at 15:11, Konrad Rzeszutek Wilk <konrad [at] darnok> wrote:
> > On Tue, May 08, 2012 at 08:35:11PM -0400, Konrad Rzeszutek Wilk wrote:
> >> On Mon, May 07, 2012 at 05:41:28PM -0700, AP wrote:
> >> > After some digging around, it looks like this is an Ubuntu 11.10 only
> > patch:
> >> >
> > http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-oneiric.git;a=commitdiff;h=3f3fb
> > a59aa5773836d94799d10b692f9b7ea16a0;hp=5e498fdb19f5b27699f063eb10040612b82416
> > 0b
> >>
> >>
> >> Ugh. There are looks to be a bug in Fedora as well:
> > https://bugzilla.redhat.com/show_bug.cgi?id=801650 for this.
> >>
> >
> > CC-ing Intel. It seems that the userspace programs are crashingon
> > Sandybridge machines even if 'xsave=0' is provided on the command line.
> > Any advice?
>
> Going through the bug, all I can see are bogus attempts to pass
> "xsave=" (with value 0 or 1) to the kernel. That's a hypervisor
> option though, and the only kernel option that's relevant here
> is "noxsave" afaik.
>
> Further, when the hypervisor has xsave support enabled, there's
> (in the pv case) nothing the kernel can do to hide the functionality
> from applications, as the hardware's CR4.OSXSAVE will be set, and
> hence CPUID.OSXSAVE will be too. So "noxsave" on the kernel
> command line when "xsave=1" (or xsave enabled by default as in
> 4.2) just doesn't make any sense.
>
> And finally one always has to keep in mind that there is this nice
> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
> when trying to determine whether AVX or FMA is available.

It has this:
146
147 if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
148 {
149 /* Reset the AVX bit in case OSXSAVE is disabled. */
150 if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) != 0
151 && ({ unsigned int xcrlow;
152 unsigned int xcrhigh;
153 asm ("xgetbv"
154 : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
155 (xcrlow & 6) == 6; }))
156 __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
157 }


And when I ran a little silly C program (attached) to probe the CPUID flags, I got:
/root/test-xsave
SSE3 CMOV
AVX XSAVE
Trying XGETBV
Illegal instruction (core dumped)

Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
is _not_ set. But then looking at the sources I see:

# ifdef USE_AS_STRCASECMP_L
102 ENTRY(__strcasecmp)
103 .type __strcasecmp, @gnu_indirect_function
104 cmpl $0, __cpu_features+KIND_OFFSET(%rip)
105 jne 1f
106 call __init_cpu_features
107 1:
108 # ifdef HAVE_AVX_SUPPORT
109 leaq __strcasecmp_avx(%rip), %rax
110 testl $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
111 jnz 2f
112 # endif

which would imply that the AVX bit is sampled here instead of the
YMM one.

Perhaps this is needed:

--- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig 2012-05-09 13:32:10.832000122 -0400
+++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c 2012-05-09 13:33:31.043000138 -0400
@@ -154,6 +154,8 @@ __init_cpu_features (void)
: "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
(xcrlow & 6) == 6; }))
__cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
+ else
+ __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
}

__cpu_features.family = family;

?

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


law at redhat

May 10, 2012, 12:39 PM

Post #21 of 26 (552 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>
>> And finally one always has to keep in mind that there is this nice
>> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
>> when trying to determine whether AVX or FMA is available.
>
> It has this:
> 146
> 147 if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& bit_AVX)
> 148 {
> 149 /* Reset the AVX bit in case OSXSAVE is disabled. */
> 150 if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& bit_OSXSAVE) != 0
> 151&& ({ unsigned int xcrlow;
> 152 unsigned int xcrhigh;
> 153 asm ("xgetbv"
> 154 : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
> 155 (xcrlow& 6) == 6; }))
> 156 __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> 157 }
>
>
> And when I ran a little silly C program (attached) to probe the CPUID flags, I got:
> /root/test-xsave
> SSE3 CMOV
> AVX XSAVE
> Trying XGETBV
> Illegal instruction (core dumped)
>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
> still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:
>
> # ifdef USE_AS_STRCASECMP_L
> 102 ENTRY(__strcasecmp)
> 103 .type __strcasecmp, @gnu_indirect_function
> 104 cmpl $0, __cpu_features+KIND_OFFSET(%rip)
> 105 jne 1f
> 106 call __init_cpu_features
> 107 1:
> 108 # ifdef HAVE_AVX_SUPPORT
> 109 leaq __strcasecmp_avx(%rip), %rax
> 110 testl $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
> 111 jnz 2f
> 112 # endif
>
> which would imply that the AVX bit is sampled here instead of the
> YMM one.
[ ... ]
I think Uli's position is that this code only uses AVX encodings, but
not the YMM registers and thus the right check is for AVX.

That doesn't make sense to me given the text under availability and
support here:

http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/

According to my reading AVX can only be used if the hardware supports
AVX *and* the OS supports XSAVE. The only weasel language is "To use
the Intel AVX extensions reliably in most settings ..." Which Uli might
be relying upon for his position.

Ironically, the code in init-arch used to look like:

if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
{
/* Reset the AVX bit in case OSXSAVE is disabled. */
if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &
bit_OSXSAVE) == 0
|| ({ unsigned int xcrlow;
unsigned int xcrhigh;
asm ("xgetbv"
: "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
(xcrlow & 6) != 6; }))
__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
}


Which I think would have done the right thing. Uli changed it to the
form you quoted just 2 hours after installing the version I quoted.

If i'm going to make the claim Uli is wrong, some clarification from
Intel would be appreciated.


jeff


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


law at redhat

May 10, 2012, 1:15 PM

Post #22 of 26 (554 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:

>
> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit PV guest
> still crashes on that machine) - which means that in multi-lib the bit_YMM_Usable
> is _not_ set. But then looking at the sources I see:
What's even more amusing? Just after installing the incorrect feature
check and introducing bit_YMM_Usable, Uli reverted all the tests which
had been checking for usable YMM regs and made them check AVX again.

one.
>
> Perhaps this is needed:
>
> --- glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c.orig 2012-05-09 13:32:10.832000122 -0400
> +++ glibc-2.15-a316c1f/sysdeps/x86_64/multiarch/init-arch.c 2012-05-09 13:33:31.043000138 -0400
> @@ -154,6 +154,8 @@ __init_cpu_features (void)
> : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
> (xcrlow& 6) == 6; }))
> __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
> + else
> + __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&= ~bit_AVX;
> }
>
> __cpu_features.family = family;
I certainly agree. The big problem here is testing... I still can't
test it :( Against my better judgment I may have to throw a glibc with
that change over the wall and hope. There's been far too much of that
already.

jeff

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


konrad at darnok

May 10, 2012, 1:57 PM

Post #23 of 26 (578 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On May 10, 2012 3:40 PM, "Jeff Law" <law [at] redhat> wrote:
>
> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>>>
>>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at
CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>
>>
>> It has this:
>> 146
>> 147 if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& bit_AVX)
>>
>> 148 {
>> 149 /* Reset the AVX bit in case OSXSAVE is disabled. */
>> 150 if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&
bit_OSXSAVE) != 0
>> 151&& ({ unsigned int xcrlow;
>>
>> 152 unsigned int xcrhigh;
>> 153 asm ("xgetbv"
>> 154 : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>> 155 (xcrlow& 6) == 6; }))
>>
>> 156 __cpu_features.feature[index_YMM_Usable] |= bit_YMM_Usable;
>> 157 }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
flags, I got:
>> /root/test-xsave
>> SSE3 CMOV
>> AVX XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit
PV guest
>> still crashes on that machine) - which means that in multi-lib the
bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103 .type __strcasecmp, @gnu_indirect_function
>> 104 cmpl $0, __cpu_features+KIND_OFFSET(%rip)
>> 105 jne 1f
>> 106 call __init_cpu_features
>> 107 1:
>> 108 # ifdef HAVE_AVX_SUPPORT
>> 109 leaq __strcasecmp_avx(%rip), %rax
>> 110 testl $bit_AVX, __cpu_features+CPUID_OFFSET+index_AVX(%rip)
>> 111 jnz 2f
>> 112 # endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>
> [ ... ]
> I think Uli's position is that this code only uses AVX encodings, but not
the YMM registers and thus the right check is for AVX.
>
> That doesn't make sense to me given the text under availability and
support here:
>
>
http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/
>
> According to my reading AVX can only be used if the hardware supports AVX
*and* the OS supports XSAVE. The only weasel language is "To use the
Intel AVX extensions reliably in most settings ..." Which Uli might be
relying upon for his position.
>
> Ironically, the code in init-arch used to look like:
>
>
> if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
> {
>
> /* Reset the AVX bit in case OSXSAVE is disabled. */
> if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE)
== 0
> || ({ unsigned int xcrlow;
> unsigned int xcrhigh;
> asm ("xgetbv"
>
> : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
> (xcrlow & 6) != 6; }))
>
> __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
> }
>
>
> Which I think would have done the right thing. Uli changed it to the
form you quoted just 2 hours after installing the version I quoted.

Sadly no as it would have executed the xgetv instruction. Since the first
part of the boolean logic returns false.
>
> If i'm going to make the claim Uli is wrong, some clarification from
Intel would be appreciated.
>
>
> jeff
>
On May 10, 2012 3:40 PM, "Jeff Law" <law [at] redhat> wrote:

> On 05/09/2012 11:38 AM, Konrad Rzeszutek Wilk wrote:
>
>>
>>> And finally one always has to keep in mind that there is this nice
>>> glibc bug in that in some versions it is failing to look at CPUID.OSXSAVE
>>> when trying to determine whether AVX or FMA is available.
>>>
>>
>> It has this:
>> 146
>> 147 if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx& bit_AVX)
>> 148 {
>> 149 /* Reset the AVX bit in case OSXSAVE is disabled. */
>> 150 if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx&
>> bit_OSXSAVE) != 0
>> 151&& ({ unsigned int xcrlow;
>> 152 unsigned int xcrhigh;
>> 153 asm ("xgetbv"
>> 154 : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>> 155 (xcrlow& 6) == 6; }))
>> 156 __cpu_features.feature[index_**YMM_Usable] |= bit_YMM_Usable;
>> 157 }
>>
>>
>> And when I ran a little silly C program (attached) to probe the CPUID
>> flags, I got:
>> /root/test-xsave
>> SSE3 CMOV
>> AVX XSAVE
>> Trying XGETBV
>> Illegal instruction (core dumped)
>>
>> Which would imply that the OSXSAVE is not set (but Fedora Core 17 64-bit
>> PV guest
>> still crashes on that machine) - which means that in multi-lib the
>> bit_YMM_Usable
>> is _not_ set. But then looking at the sources I see:
>>
>> # ifdef USE_AS_STRCASECMP_L
>> 102 ENTRY(__strcasecmp)
>> 103 .type __strcasecmp, @gnu_indirect_function
>> 104 cmpl $0, __cpu_features+KIND_OFFSET(%**rip)
>> 105 jne 1f
>> 106 call __init_cpu_features
>> 107 1:
>> 108 # ifdef HAVE_AVX_SUPPORT
>> 109 leaq __strcasecmp_avx(%rip), %rax
>> 110 testl $bit_AVX, __cpu_features+CPUID_OFFSET+**
>> index_AVX(%rip)
>> 111 jnz 2f
>> 112 # endif
>>
>> which would imply that the AVX bit is sampled here instead of the
>> YMM one.
>>
> [ ... ]
> I think Uli's position is that this code only uses AVX encodings, but not
> the YMM registers and thus the right check is for AVX.
>
> That doesn't make sense to me given the text under availability and
> support here:
>
> http://software.intel.com/en-**us/articles/introduction-to-**
> intel-advanced-vector-**extensions/<http://software.intel.com/en-us/articles/introduction-to-intel-advanced-vector-extensions/>
>
> According to my reading AVX can only be used if the hardware supports AVX
> *and* the OS supports XSAVE. The only weasel language is "To use the
> Intel AVX extensions reliably in most settings ..." Which Uli might be
> relying upon for his position.
>
> Ironically, the code in init-arch used to look like:
>
> if (__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx & bit_AVX)
> {
> /* Reset the AVX bit in case OSXSAVE is disabled. */
> if ((__cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx & bit_OSXSAVE)
> == 0
> || ({ unsigned int xcrlow;
> unsigned int xcrhigh;
> asm ("xgetbv"
> : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
> (xcrlow & 6) != 6; }))
> __cpu_features.cpuid[COMMON_**CPUID_INDEX_1].ecx &= ~bit_AVX;
> }
>
>
> Which I think would have done the right thing. Uli changed it to the
> form you quoted just 2 hours after installing the version I quoted.
>
> If i'm going to make the claim Uli is wrong, some clarification from Intel
> would be appreciated.
>
>
> jeff
>
>


konrad at darnok

May 10, 2012, 5:58 PM

Post #24 of 26 (573 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

>> Ironically, the code in init-arch used to look like:
>>
>>
>>  if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_AVX)
>>    {
>>
>>      /* Reset the AVX bit in case OSXSAVE is disabled.  */
>>      if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx & bit_OSXSAVE) ==
>> 0
>>          || ({ unsigned int xcrlow;
>>              unsigned int xcrhigh;
>>              asm ("xgetbv"
>>
>>                   : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>>              (xcrlow & 6) != 6; }))
>>
>>        __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx &= ~bit_AVX;
>>    }
>>
>>
>> Which I think would have done the right thing.   Uli changed it to the
>> form you quoted just 2 hours after installing the version I quoted.
>
> Sadly no as it would have executed the xgetv instruction. Since the first
> part of the boolean logic returns false.

<sigh> And that is what I get from typing this while stopping at
lights and being in a hurry and doing this on a cellphone.
Please ignore what I said above - the earlier version would have worked correct.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


law at redhat

May 10, 2012, 7:27 PM

Post #25 of 26 (555 views)
Permalink
Re: xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable. [In reply to]

On 05/10/2012 06:58 PM, Konrad Rzeszutek Wilk wrote:
>>> Ironically, the code in init-arch used to look like:
>>>
>>>
>>> if (__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& bit_AVX)
>>> {
>>>
>>> /* Reset the AVX bit in case OSXSAVE is disabled. */
>>> if ((__cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx& bit_OSXSAVE) ==
>>> 0
>>> || ({ unsigned int xcrlow;
>>> unsigned int xcrhigh;
>>> asm ("xgetbv"
>>>
>>> : "=a" (xcrlow), "=d" (xcrhigh) : "c" (0));
>>> (xcrlow& 6) != 6; }))
>>>
>>> __cpu_features.cpuid[COMMON_CPUID_INDEX_1].ecx&= ~bit_AVX;
>>> }
>>>
>>>
>>> Which I think would have done the right thing. Uli changed it to the
>>> form you quoted just 2 hours after installing the version I quoted.
>>
>> Sadly no as it would have executed the xgetv instruction. Since the first
>> part of the boolean logic returns false.
>
> <sigh> And that is what I get from typing this while stopping at
> lights and being in a hurry and doing this on a cellphone.
> Please ignore what I said above - the earlier version would have worked correct.
No worries :-)

Had you not gotten involved, I never would have seen the xen-devel
thread which then referred me to the appropriate Intel doc to confirm
that your patch should fix the problem. Your input on this issue has
been greatly appreciated.

jeff


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

First page Previous page 1 2 Next page Last page  View All Xen devel RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.