konrad.wilk at oracle
Aug 12, 2013, 5:49 AM
Post #4 of 6
(1 views)
Permalink
|
|
Re: pvops: Does PVOPS guest os support online "suspend/resume"
[In reply to]
|
|
On Sat, Aug 10, 2013 at 08:29:43AM +0000, Gonglei (Arei) wrote:
>
>
> > -----Original Message-----
> > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk [at] oracle]
> > Sent: Friday, August 09, 2013 3:17 AM
> > To: Gonglei (Arei)
> > Cc: xen-devel [at] lists; Zhangbo (Oscar); Luonengjun; Hanweidong
> > Subject: Re: [Xen-devel] pvops: Does PVOPS guest os support online
> > "suspend/resume"
> >
> > On Thu, Aug 08, 2013 at 02:23:06PM +0000, Gonglei (Arei) wrote:
> > > Hi all,
> > >
> > > While suspend and resume a PVOPS guest os while it's running, we found that
> > it would get its block/net io stucked. However, non-PVOPS guest os has no such
> > problem.
> > >
> >
> > With what version of Linux is this? Have you tried with v3.10?
>
> Thanks for responding. We've tried kernel "3.5.0-17 generic" (ubuntu 12.10), the problem still exists.
So you have not tried v3.10. v3.5 is ancient from the upstream perspective.
> Although we are not sure about the result about kernel 3.10, but suspiciously it would also have the same problem.
Potentially. There were fixes added in 3.5:
commit 569ca5b3f94cd0b3295ec5943aa457cf4a4f6a3a
Author: Jan Beulich <JBeulich [at] suse>
Date: Thu Apr 5 16:10:07 2012 +0100
xen/gnttab: add deferred freeing logic
Rather than just leaking pages that can't be freed at the point where
access permission for the backend domain gets revoked, put them on a
list and run a timer to (infrequently) retry freeing them. (This can
particularly happen when unloading a frontend driver when devices are
still present, and the backend still has them in non-closed state or
hasn't finished closing them yet.)
and that seems to be triggered.
>
> Xen version: 4.3.0
>
> Another method to reproduce:
> 1) xl create dom1.cfg
> 2) xl save -c dom1 /path/to/save/file
> (-c Leave domain running after creating the snapshot.)
>
> As I mentioned before, the problem occurs because PVOPS guest os RESUMEes blkfront when the guest resumes.
> The "blkfront_resume" method seems unnecessary here.
It has to do that otherwise it can't replay the I/Os that might not have
hit the platter when it migrated from the original host.
But you are exercising the case where it does a checkpoint,
not a full save/restore cycle.
In which case you might be indeed hitting a bug.
> non-PVOPS guest os doesn't RESUME blkfront, thus they works fine.
Potentially. The non-PVOPS guests are based on an ancient kernels and
the upstream logic in the generic suspend/resume machinery has also
changed.
>
> So, here comes the 2 questions, is the problem caused because:
> 1) PVOPS kernel doesn't take this situation into accont, and has a bug here?
> or
> 2) PVOPS has other ways to avoid such problem?
Just to make sure I am not confused here. The problem does not
appear if you do NOT use -c, correct?
>
> -Gonglei
> >
> > Thanks.
> > > How reproducible:
> > > -------------------
> > > 1/1
> > >
> > > Steps to reproduce:
> > > ------------------
> > > 1)suspend guest os
> > > Note: do not migrate/shutdown the guest os.
> > > 2)resume guest os
> > >
> > > (Think about rolling-back(resume) during core-dumping(suspend) a guest,
> > such problem would cause the guest os unoprationable.)
> > >
> > >
> > ================================================================
> > ====
> > > we found warning messages in guest os:
> > > --------------------------------------------------------------------
> > > Aug 2 10:17:34 localhost kernel: [38592.985159] platform pcspkr: resume
> > > Aug 2 10:17:34 localhost kernel: [38592.989890] platform vesafb.0: resume
> > > Aug 2 10:17:34 localhost kernel: [38592.996075] input input0: type resume
> > > Aug 2 10:17:34 localhost kernel: [38593.001330] input input1: type resume
> > > Aug 2 10:17:34 localhost kernel: [38593.005496] vbd vbd-51712: legacy
> > resume
> > > Aug 2 10:17:34 localhost kernel: [38593.011506] WARNING: g.e. still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.016909] WARNING: leaking g.e.
> > and page still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.026204] xen vbd-51760: legacy
> > resume
> > > Aug 2 10:17:34 localhost kernel: [38593.033070] vif vif-0: legacy resume
> > > Aug 2 10:17:34 localhost kernel: [38593.039327] WARNING: g.e. still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.045304] WARNING: leaking g.e.
> > and page still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.052101] WARNING: g.e. still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.057965] WARNING: leaking g.e.
> > and page still in use!
> > > Aug 2 10:17:34 localhost kernel: [38593.066795] serial8250 serial8250:
> > resume
> > > Aug 2 10:17:34 localhost kernel: [38593.073556] input input2: type resume
> > > Aug 2 10:17:34 localhost kernel: [38593.079385] platform Fixed MDIO bus.0:
> > resume
> > > Aug 2 10:17:34 localhost kernel: [38593.086285] usb usb1: type resume
> > > ------------------------------------------------------
> > >
> > > which means that we refers to a grant-table while it's in use.
> > >
> > > The reason results in that:
> > > suspend/resume codes:
> > > --------------------------------------------------------
> > > //drivers/xen/manage.c
> > > static void do_suspend(void)
> > > {
> > > int err;
> > > struct suspend_info si;
> > >
> > > shutting_down = SHUTDOWN_SUSPEND;
> > >
> > > ………………
> > > err = dpm_suspend_start(PMSG_FREEZE);
> > > ………………
> > > dpm_resume_start(si.cancelled ? PMSG_THAW : PMSG_RESTORE);
> > >
> > > if (err) {
> > > pr_err("failed to start xen_suspend: %d\n", err);
> > > si.cancelled = 1;
> > > }
> > > //NOTE: si.cancelled = 1
> > >
> > > out_resume:
> > > if (!si.cancelled) {
> > > xen_arch_resume();
> > > xs_resume();
> > > } else
> > > xs_suspend_cancel();
> > >
> > > dpm_resume_end(si.cancelled ? PMSG_THAW : PMSG_RESTORE);
> > //blkfront device got resumed here.
> > >
> > > out_thaw:
> > > #ifdef CONFIG_PREEMPT
> > > thaw_processes();
> > > out:
> > > #endif
> > > shutting_down = SHUTDOWN_INVALID;
> > > }
> > > ------------------------------------
> > >
> > > Func "dpm_suspend_start" suspends devices, and "dpm_resume_end"
> > resumes devices.
> > > However, we found that the device "blkfront" has no SUSPEND method but
> > RESUME method.
> > >
> > > -------------------------------------
> > > //drivers/block/xen-blkfront.c
> > > static DEFINE_XENBUS_DRIVER(blkfront, ,
> > > .probe = blkfront_probe,
> > > .remove = blkfront_remove,
> > > .resume = blkfront_resume, // only RESUME method found here.
> > > .otherend_changed = blkback_changed,
> > > .is_ready = blkfront_is_ready,
> > > );
> > > --------------------------------------
> > >
> > > It resumes blkfront device when it didn't get suspended, which caused the
> > prolem above.
> > >
> > >
> > > =========================================
> > > In order to check whether it's the problem of PVOPS or hypervisor(xen)/dom0,
> > we suspend/resume other non-PVOPS guest oses, no such problem occured.
> > >
> > > Other non-PVOPS are using their own xen drivers, as shown in
> > https://github.com/jpaton/xen-4.1-LJX1/blob/master/unmodified_drivers/linux-
> > 2.6/platform-pci/machine_reboot.c :
> > >
> > > int __xen_suspend(int fast_suspend, void (*resume_notifier)(int))
> > > {
> > > int err, suspend_cancelled, nr_cpus;
> > > struct ap_suspend_info info;
> > >
> > > xenbus_suspend();
> > >
> > > ……………………
> > > preempt_enable();
> > >
> > > if (!suspend_cancelled)
> > > xenbus_resume(); //when the guest os get resumed,
> > suspend_cancelled == 1, thus it wouldn't enter xenbus_resume_uvp here.
> > > else
> > > xenbus_suspend_cancel(); //It gets here. so the blkfront
> > wouldn't resume.
> > >
> > > return 0;
> > > }
> > >
> > >
> > > In non-PVOPS guest os, although they don't have blkfront SUSPEND method
> > either, their xen-driver doesn't resume blkfront device, thus, they would't have
> > any problem after suspend/resume.
> > >
> > >
> > > I'm wondering why the 2 types of driver(PVOPS and non-PVOPS) are different
> > here.
> > > Is that because:
> > > 1) PVOPS kernel doesn't take this situation into accont, and has a bug here?
> > > or
> > > 2) PVOPS has other ways to avoid such problem?
> > >
> > > thank you in advance.
> > >
> > > -Gonglei
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel [at] lists
> > > http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel
|
