stefano.stabellini at eu
Aug 6, 2012, 3:23 AM
Post #48 of 48
(168 views)
Permalink
|
On Fri, 3 Aug 2012, David Erickson wrote:
> On Thu, Aug 2, 2012 at 10:38 AM, David Erickson <halcyon1981 [at] gmail> wrote:
> > On Wed, Aug 1, 2012 at 10:52 AM, David Erickson <halcyon1981 [at] gmail> wrote:
> >> On Wed, Aug 1, 2012 at 4:13 AM, Stefano Stabellini
> >> <stefano.stabellini [at] eu> wrote:
> >>> On Tue, 31 Jul 2012, David Erickson wrote:
> >>>> On Tue, Jul 31, 2012 at 4:39 AM, Stefano Stabellini
> >>>> <stefano.stabellini [at] eu> wrote:
> >>>> > On Tue, 31 Jul 2012, David Erickson wrote:
> >>>> >> Just got back in town, following up on the prior discussion. I
> >>>> >> successfully compiled the latest code (25688 and qemu upstream
> >>>> >> 5e3bc7144edd6e4fa2824944e5eb16c28197dd5a), but am still having
> >>>> >> problems during initialization of the card in the guest, in particular
> >>>> >> the unsupported delivery mode 3 which seems to cause interrupt related
> >>>> >> problems during init. I've again attached the qemu-dm-log, and xl
> >>>> >> dmesg log files, and additionally screenshots of the guest dmesg and
> >>>> >> also for comparison starting the same livecd natively on the box.
> >>>> >
> >>>> > "unsupported delivery mode 3" means that the Linux guest is trying to
> >>>> > remap the MSI onto an event channel but Xen is still trying to deliver
> >>>> > the MSI using the emulated code path anyway.
> >>>> >
> >>>> > Adding
> >>>> >
> >>>> > #define XEN_PT_LOGGING_ENABLED 1
> >>>> >
> >>>> > at the top of hw/xen_pt.h and posting the additional QEMU logs could
> >>>> > be helpful.
> >>>> >
> >>>> > The full Xen logs might also be useful. I would add some more tracing to
> >>>> > the hypervisor too:
> >>>> >
> >>>> > diff --git a/xen/drivers/passthrough/io.c b/xen/drivers/passthrough/io.c
> >>>> > index b5975d1..08f4ab7 100644
> >>>> > --- a/xen/drivers/passthrough/io.c
> >>>> > +++ b/xen/drivers/passthrough/io.c
> >>>> > @@ -474,6 +474,11 @@ static void hvm_pci_msi_assert(
> >>>> > {
> >>>> > struct pirq *pirq = dpci_pirq(pirq_dpci);
> >>>> >
> >>>> > + printk("DEBUG %s pirq=%d hvm_domain_use_pirq=%d emuirq=%d\n", __func__,
> >>>> > + pirq->pirq,
> >>>> > + hvm_domain_use_pirq(d, pirq),
> >>>> > + pirq->arch.hvm.emuirq);
> >>>> > +
> >>>> > if ( hvm_domain_use_pirq(d, pirq) )
> >>>> > send_guest_pirq(d, pirq);
> >>>> > else
> >>>>
> >>>> Hi Stefano-
> >>>> I made the modifications (it looks like that DEFINE hasn't been used
> >>>> in awhile, caused a few compilation issues, I had to prefix most of
> >>>> the logged variables with s->hostaddr.), and am attaching the
> >>>> qemu-dm-ubuntu.log and dmesg from xl. You referred to full Xen logs,
> >>>> where do I find those at?
> >>>
> >>> Thanks for the logs!
> >>> You can get the full Xen logs from the serial console but you can also
> >>> grab the last few lines with "xl dmesg", like you did and it seems to be
> >>> enough in this case.
> >>>
> >>>
> >>> The initial MSI remapping has been done:
> >>>
> >>> [00:05.0] msi_msix_setup: requested pirq 4 for MSI-X (vec: 0, entry: 0)
> >>> [00:05.0] msi_msix_update: Updating MSI-X with pirq 4 gvec 0 gflags 0x3037 (entry: 0)
> >>>
> >>> But the guest is not issuing the EVTCHNOP_bind_pirq hypercall that is
> >>> necessary to be able to receive event notifications (emuirq=-1 in the
> >>> Xen logs).
> >>>
> >>> Now we need to figure out why: we still need more logs, this time on the
> >>> guest side.
> >>> What is the kernel version that you are using in the guest?
> >>> Could you please add "debug loglevel=9" to the guest kernel command line
> >>> and then post the guest dmesg again?
> >>> It would be great if you could use the emulated serial to get the logs
> >>> rather than a picture. You can do that by adding serial='pty' to the VM
> >>> config file and console=ttyS0 to the guest command line.
> >>> This additional Xen change could also tell us if the EVTCHNOP_bind_pirq
> >>> has been done:
> >>>
> >>>
> >>> diff --git a/xen/common/event_channel.c b/xen/common/event_channel.c
> >>> index 53777f8..d65a97a 100644
> >>> --- a/xen/common/event_channel.c
> >>> +++ b/xen/common/event_channel.c
> >>> @@ -405,6 +405,8 @@ static long evtchn_bind_pirq(evtchn_bind_pirq_t *bind)
> >>> #ifdef CONFIG_X86
> >>> if ( is_hvm_domain(d) && domain_pirq_to_irq(d, pirq) > 0 )
> >>> map_domain_emuirq_pirq(d, pirq, IRQ_PT);
> >>> + printk("DEBUG %s %d pirq=%d irq=%d emuirq=%d\n", __func__, __LINE__,
> >>> + pirq, domain_pirq_to_irq(d, pirq), domain_pirq_to_emuirq(d, pirq));
> >>> #endif
> >>>
> >>> out:
> >>
> >> The guest is an Ubuntu 11.10 livecd, kernel version 3.0.0-12-generic.
> >> I've also attached all the logs, thanks for the tip on the serial
> >> console, very useful.
> >>
> >> Additionally I've attached logs for booting a solaris livecd (my
> >> ultimate goal is to use this HBA card in Solaris), with the serial
> >> console tip I was able to capture its kernel boot as well.
> >
> > I'm attaching another log from Solaris' kernel debugger, I'm not sure
> > how helpful it is but I found it interesting that it didn't detect an
> > Intel IOMMU/ACPI table and unloaded it, then tried AMD - and I'm new
> > to Solaris but comparing this log to one without PCI Passthrough, the
> > npe module never gets loaded without PCI passthrough, so I assume it
> > failed while setting up the AMD IOMMU module then loaded the npe
> > module to report the error.
unfortunately I don't know much about solaris so it doesn't help me very
much
> Just following up, is there anything I can do to further help debug
> and figure out what is causing the problem here? I'm assuming since
> it isn't working properly in PV or HVM guests there may be multiple
> bugs. Is there an easy way to run Ubuntu as HVM only (more friendly
> than Solaris) to try and isolate if that is a separate bug from what
> is being seen with the PV Ubuntu VM?
I didn't get any specific output on the PV MSI setup, probably because
it is only printed when CONFIG_DEBUG is setup.
Would you be able to rebuild your Ubuntu kernel with the appended patch?
diff --git a/arch/x86/pci/xen.c b/arch/x86/pci/xen.c
index 6e96e65..039f29c 100644
--- a/arch/x86/pci/xen.c
+++ b/arch/x86/pci/xen.c
@@ -90,6 +90,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
struct msi_desc *msidesc;
struct msi_msg msg;
+ printk("DEBUG %s %d %s %s nvec=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),nvec);
list_for_each_entry(msidesc, &dev->msi_list, list) {
__read_msi_msg(msidesc, &msg);
pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
@@ -102,7 +103,9 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
xen_msi_compose_msg(dev, pirq, &msg);
__write_msi_msg(msidesc, &msg);
dev_dbg(&dev->dev, "xen: msi bound to pirq=%d\n", pirq);
+ printk("DEBUG %s %d %s %s pirq=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq);
} else {
+ printk("DEBUG %s %d %s %s pirq=%d already bound\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq);
dev_dbg(&dev->dev,
"xen: msi already bound to pirq=%d\n", pirq);
}
@@ -115,9 +118,11 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
dev_dbg(&dev->dev,
"xen: msi --> pirq=%d --> irq=%d\n", pirq, irq);
}
+ printk("DEBUG %s %d %s %s pirq=%d irq=%d\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev),pirq,irq);
return 0;
error:
+ printk("DEBUG %s %d %s %s error\n",__func__,__LINE__,dev_driver_string(&dev->dev), dev_name(&dev->dev));
dev_err(&dev->dev,
"Xen PCI frontend has not registered MSI/MSI-X support!\n");
return -ENODEV;
_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel
|
xl_dmesg.log
