Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Xen: Devel

[RFC][PATCH 0/5] Add V4V to Xen

 

 

First page Previous page 1 2 Next page Last page  View All Xen devel RSS feed   Index | Next | Previous | View Threaded


jean.guyader at citrix

May 31, 2012, 8:07 AM

Post #1 of 32 (499 views)
Permalink
[RFC][PATCH 0/5] Add V4V to Xen

V4V is a copy based inter vm communication system.

Please have a look at this thread for more detail
about V4V.
http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html

This patch series is work in progress but I wanted to
post it early enough so I can get feedback from people.

Jean Guyader (5):
xen: add ssize_t to types.h
xen: Add headers to include/Makefile
v4v: Introduce VIRQ_V4V
xen: Enforce casting for guest_handle_cast
xen: Add V4V implementation

xen/arch/x86/hvm/hvm.c | 9 +-
xen/arch/x86/x86_32/entry.S | 2 +
xen/arch/x86/x86_64/compat/entry.S | 2 +
xen/arch/x86/x86_64/entry.S | 2 +
xen/common/Makefile | 1 +
xen/common/domain.c | 11 +-
xen/common/event_channel.c | 1 +
xen/common/v4v.c | 1779 ++++++++++++++++++++++++++++++++++++
xen/include/Makefile | 3 +-
xen/include/asm-x86/guest_access.h | 2 +-
xen/include/public/v4v.h | 544 +++++++++++
xen/include/public/xen.h | 4 +-
xen/include/xen/sched.h | 5 +
xen/include/xen/types.h | 1 +
xen/include/xen/v4v.h | 109 +++
15 files changed, 2467 insertions(+), 8 deletions(-)
create mode 100644 xen/common/v4v.c
create mode 100644 xen/include/public/v4v.h
create mode 100644 xen/include/xen/v4v.h

--
1.7.2.5


jean.guyader at citrix

May 31, 2012, 7:52 AM

Post #2 of 32 (448 views)
Permalink
[RFC][PATCH 0/5] Add V4V to Xen [In reply to]

V4V is a copy based inter vm communication system.

Please have a look at this thread for more detail
about V4V.
http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html

This patch series is work in progress but I wanted to
post it early enough so I can get feedback from people.

Jean Guyader (5):
xen: add ssize_t to types.h
xen: Add headers to include/Makefile
v4v: Introduce VIRQ_V4V
xen: Enforce casting for guest_handle_cast
xen: Add V4V implementation

xen/arch/x86/hvm/hvm.c | 9 +-
xen/arch/x86/x86_32/entry.S | 2 +
xen/arch/x86/x86_64/compat/entry.S | 2 +
xen/arch/x86/x86_64/entry.S | 2 +
xen/common/Makefile | 1 +
xen/common/domain.c | 11 +-
xen/common/event_channel.c | 1 +
xen/common/v4v.c | 1779 ++++++++++++++++++++++++++++++++++++
xen/include/Makefile | 3 +-
xen/include/asm-x86/guest_access.h | 2 +-
xen/include/public/v4v.h | 544 +++++++++++
xen/include/public/xen.h | 4 +-
xen/include/xen/sched.h | 5 +
xen/include/xen/types.h | 1 +
xen/include/xen/v4v.h | 109 +++
15 files changed, 2467 insertions(+), 8 deletions(-)
create mode 100644 xen/common/v4v.c
create mode 100644 xen/include/public/v4v.h
create mode 100644 xen/include/xen/v4v.h

--
1.7.2.5


JBeulich at suse

Jun 1, 2012, 5:41 AM

Post #3 of 32 (445 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

>>> On 31.05.12 at 17:07, Jean Guyader <jean.guyader [at] citrix> wrote:

Is there a particular reason this series got resent with no apparent
change (and also none mentioned in the description here or in the
individual patches)? All of the comments sent yesterday still apply,
and I continue to consider patches 1-4 as unnecessary/broken.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


George.Dunlap at eu

Jun 1, 2012, 6:24 AM

Post #4 of 32 (444 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On Fri, Jun 1, 2012 at 1:41 PM, Jan Beulich <JBeulich [at] suse> wrote:
>>>> On 31.05.12 at 17:07, Jean Guyader <jean.guyader [at] citrix> wrote:
>
> Is there a particular reason this series got resent with no apparent
> change (and also none mentioned in the description here or in the
> individual patches)? All of the comments sent yesterday still apply,
> and I continue to consider patches 1-4 as unnecessary/broken.

The date on the second set of e-mails is 15 minutes before the
original series. I presume this means that Jean sent the series but
it got held up for some reason, so sent it again 15 minutes later; and
the second series made it through but the first was held up until
today.

Never attribute to malice that which can be attributed to computer error. :-)

-George

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


Ian.Campbell at citrix

Jun 1, 2012, 6:47 AM

Post #5 of 32 (444 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On Thu, 2012-05-31 at 16:07 +0100, Jean Guyader wrote:
> V4V is a copy based inter vm communication system.
>
> Please have a look at this thread for more detail
> about V4V.
> http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html
>
> This patch series is work in progress but I wanted to
> post it early enough so I can get feedback from people.

The main thing which is missing here, which makes it rather hard to
review, is any kind of design documentation. It would also be great to
see the rationale for why things have to be done this way.

For example it seems that there is a bunch of stuff being added to the
hypervisor which could live in the context of some guest or service
domain -- putting stuff like that in the hypervisor needs strong
arguments and rationale why it has to be there rather than somewhere
else.

Likewise perhaps the v4v hypercall could effectively be implemented with
a multicall containing some grant copy ops and evtchn manipulations?

But in the absence of any descriptions of the whats, hows and whys of
v4v its rather hard to have these sorts of conversations. The above are
some concrete examples but I'm more interested in seeing the more
general descriptions before we get into those.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at eu

Jun 7, 2012, 1:47 AM

Post #6 of 32 (437 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 01/06 02:47, Ian Campbell wrote:
> On Thu, 2012-05-31 at 16:07 +0100, Jean Guyader wrote:
> > V4V is a copy based inter vm communication system.
> >
> > Please have a look at this thread for more detail
> > about V4V.
> > http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html
> >
> > This patch series is work in progress but I wanted to
> > post it early enough so I can get feedback from people.
>
> The main thing which is missing here, which makes it rather hard to
> review, is any kind of design documentation. It would also be great to
> see the rationale for why things have to be done this way.
>
> For example it seems that there is a bunch of stuff being added to the
> hypervisor which could live in the context of some guest or service
> domain -- putting stuff like that in the hypervisor needs strong
> arguments and rationale why it has to be there rather than somewhere
> else.
>
> Likewise perhaps the v4v hypercall could effectively be implemented with
> a multicall containing some grant copy ops and evtchn manipulations?
>
> But in the absence of any descriptions of the whats, hows and whys of
> v4v its rather hard to have these sorts of conversations. The above are
> some concrete examples but I'm more interested in seeing the more
> general descriptions before we get into those.
>

Here is some documentation of the Xen V4V API and the ring structure.
Let me know if you have other questions.

* Overview
** Architecture

v4v has a very simple architecture. The receiving domain registers a ring with xen.
The sending domain requests that xen inserts data into the receiving domain's ring.
Notification that there is new data to read is delivered by a VIRQ, and a domain can
request an interrupt be generated when sufficient space is available in another domain's
ring. The code that inserts the data into the ring is in xen, and xen writes a header
describing the data and where it came from infront of the data. As both the ring manipulation
and the header are written by the hypervisor, the receiving domain can trust their content
and the format of the ring.

** Addressing
v4v_addr_t identifies an endpoint address.
typedef struct v4v_addr
{
uint32_t port;
domid_t domain;
} v4v_addr_t;
struct v4v_ring_id uniquely identifies a ring on the platform.

struct v4v_ring_id
{
struct v4v_addr addr;
domid_t partner;
};

When a send operation is performed, xen will attempt to find a ring
registered by destination domain, with the required port, and with
a partner of the sending domain. Failing that it will then search
for a ring with the correct port and destination domain, but with
partner set to V4V_DOMID_ANY.

** Setup
A domain first generates a ring, and a structure describing the pages in the ring.
Rings must be page aligned, and contiguous in virtual memory, but need not be
contiguous in physical(pfn) or machine(mfn) memory. A ring is uniquely identified
on the platform by its struct v4v_ring_id.

A ring is contained in a v4v_ring_t
typedef struct v4v_ring
{
uint64_t magic;
/*
* Identifies ring_id - xen only looks at this during register/unregister
* and will fill in id.addr.domain
*/
struct v4v_ring_id id;
/* length of ring[], must be a multiple of 16 */
uint32_t len;
/* rx_ptr - modified by domain */
uint32_t rx_ptr;
/* tx_ptr - modified by xen */
volatile uint32_t tx_ptr;
uint64_t reserved[4];
volatile uint8_t ring[0];
} v4v_ring_t;

To register the ring, xen also needs a list of pfn that the ring occupies.
This is provided in a v4v_pfn_list_t
typedef struct v4v_pfn_list_t
{
uint64_t magic;
uint32_t npage;
uint32_t pad;
uint64_t reserved[3];
v4v_pfn_t pages[0];
} v4v_pfn_list_t;

Registering the ring is done with a hypercall

int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t *pfn_list);

On registering, xen will check that the tx_ptr and rx_ptr values are valid (aligned
and within the size of the ring) and modify them if they are not, it will also update
id.addr.domain with the calling domain's domid and take an internal copy of all the
relevant information. After registration the only field that xen will read is
ring->rx_ptr, xen will write to ring->ring[] and ring->tx_ptr.
Thus pfn_list can be freed after the registration call. Critically, registration is
idempotent, and all a domain need do after hibernation or migration is re-register all
its rings. (NB the id.addr.domain field may change).

** Sending

To send data a guest merely calls the send or sendv hypercall
int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, uint32_t len,
uint32_t protocol);

the caller provides a source address, a destination address, a buffer, a number of bytes to copy
and a 32 bit protocol number. Xen ignores src->domain and instead uses the domain id of the caller.

Xen will attempt to insert into a ring with id.addr equal to dst, and with
id.partner equal to the caller's domid. Otherwise it will insert into a ring
with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the insert is successfull
then Xen will trigger the V4V VIRQ line in the receiving domain.

** Reception

Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is provided
for convience), and inserts a message header infront of all messages it copies onto the ring.

struct v4v_ring_message_header
{
uint32_t len;
struct v4v_addr source;
uint16_t pad;
uint32_t protocol;
uint8_t data[0];
};

len is the number of bytes in the message including the struct v4v_ring_message_header
source specifies the source address from which the message came, source.domain is
set by xen, and source.port is taken from the arguments to the send or sendv call.
protocol is the protocol value given to the send call.

An inline function is provided to make reading from the ring easier:

ssize_t
v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
void *_buf, size_t t, int consume)

v4v_copy_out will copy at most t bytes from the ring r and place them in buf, if it
is non-null. If from and protocol are not NULL pointers then the source address and
protocol number will by copied into them. If consume is non-zero then the ring's
receive pointer will be advanced. The function returns the number of bytes of payload
that the message has (ie. the number of bytes passed to the original send or sendv call).
Thus, v4v_copy_out(r,NULL,NULL,NULL,0,0); will return the number of bytes in the next message,
and v4v_copy_out(r,NULL,NULL,NULL,0,1); will return the number of bytes in the next message
and delete it from the ring.

** Notification
If the receiving ring is full in a send or sendv hypercall, the hypercall will return with
the error -EAGAIN. When sufficient space becomes available in the ring xen will raise the V4V VIRQ
line.


Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 7, 2012, 2:42 AM

Post #7 of 32 (438 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 01/06 02:47, Ian Campbell wrote:
> On Thu, 2012-05-31 at 16:07 +0100, Jean Guyader wrote:
> > V4V is a copy based inter vm communication system.
> >
> > Please have a look at this thread for more detail
> > about V4V.
> > http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html
> >
> > This patch series is work in progress but I wanted to
> > post it early enough so I can get feedback from people.
>
> The main thing which is missing here, which makes it rather hard to
> review, is any kind of design documentation. It would also be great to
> see the rationale for why things have to be done this way.
>
> For example it seems that there is a bunch of stuff being added to the
> hypervisor which could live in the context of some guest or service
> domain -- putting stuff like that in the hypervisor needs strong
> arguments and rationale why it has to be there rather than somewhere
> else.
>
> Likewise perhaps the v4v hypercall could effectively be implemented with
> a multicall containing some grant copy ops and evtchn manipulations?
>
> But in the absence of any descriptions of the whats, hows and whys of
> v4v its rather hard to have these sorts of conversations. The above are
> some concrete examples but I'm more interested in seeing the more
> general descriptions before we get into those.
>
> Ian.
>
>

(resent, sorry if you get this message twice)

* Overview
** Architecture

v4v has a very simple architecture. The receiving domain registers a ring with xen.
The sending domain requests that xen inserts data into the receiving domain's ring.
Notification that there is new data to read is delivered by a VIRQ, and a domain can
request an interrupt be generated when sufficient space is available in another domain's
ring. The code that inserts the data into the ring is in xen, and xen writes a header
describing the data and where it came from infront of the data. As both the ring manipulation
and the header are written by the hypervisor, the receiving domain can trust their content
and the format of the ring.

** Addressing
v4v_addr_t identifies an endpoint address.
typedef struct v4v_addr
{
uint32_t port;
domid_t domain;
} v4v_addr_t;
struct v4v_ring_id uniquely identifies a ring on the platform.

struct v4v_ring_id
{
struct v4v_addr addr;
domid_t partner;
};

When a send operation is performed, xen will attempt to find a ring
registered by destination domain, with the required port, and with
a partner of the sending domain. Failing that it will then search
for a ring with the correct port and destination domain, but with
partner set to V4V_DOMID_ANY.

** Setup
A domain first generates a ring, and a structure describing the pages in the ring.
Rings must be page aligned, and contiguous in virtual memory, but need not be
contiguous in physical(pfn) or machine(mfn) memory. A ring is uniquely identified
on the platform by its struct v4v_ring_id.

A ring is contained in a v4v_ring_t
typedef struct v4v_ring
{
uint64_t magic;
/*
* Identifies ring_id - xen only looks at this during register/unregister
* and will fill in id.addr.domain
*/
struct v4v_ring_id id;
/* length of ring[], must be a multiple of 16 */
uint32_t len;
/* rx_ptr - modified by domain */
uint32_t rx_ptr;
/* tx_ptr - modified by xen */
volatile uint32_t tx_ptr;
uint64_t reserved[4];
volatile uint8_t ring[0];
} v4v_ring_t;

To register the ring, xen also needs a list of pfn that the ring occupies.
This is provided in a v4v_pfn_list_t
typedef struct v4v_pfn_list_t
{
uint64_t magic;
uint32_t npage;
uint32_t pad;
uint64_t reserved[3];
v4v_pfn_t pages[0];
} v4v_pfn_list_t;

Registering the ring is done with a hypercall

int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t *pfn_list);

On registering, xen will check that the tx_ptr and rx_ptr values are valid (aligned
and within the size of the ring) and modify them if they are not, it will also update
id.addr.domain with the calling domain's domid and take an internal copy of all the
relevant information. After registration the only field that xen will read is
ring->rx_ptr, xen will write to ring->ring[] and ring->tx_ptr.
Thus pfn_list can be freed after the registration call. Critically, registration is
idempotent, and all a domain need do after hibernation or migration is re-register all
its rings. (NB the id.addr.domain field may change).

** Sending

To send data a guest merely calls the send or sendv hypercall
int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, uint32_t len,
uint32_t protocol);

the caller provides a source address, a destination address, a buffer, a number of bytes to copy
and a 32 bit protocol number. Xen ignores src->domain and instead uses the domain id of the caller.

Xen will attempt to insert into a ring with id.addr equal to dst, and with
id.partner equal to the caller's domid. Otherwise it will insert into a ring
with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the insert is successfull
then Xen will trigger the V4V VIRQ line in the receiving domain.

** Reception

Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is provided
for convience), and inserts a message header infront of all messages it copies onto the ring.

struct v4v_ring_message_header
{
uint32_t len;
struct v4v_addr source;
uint16_t pad;
uint32_t protocol;
uint8_t data[0];
};

len is the number of bytes in the message including the struct v4v_ring_message_header
source specifies the source address from which the message came, source.domain is
set by xen, and source.port is taken from the arguments to the send or sendv call.
protocol is the protocol value given to the send call.

An inline function is provided to make reading from the ring easier:

ssize_t
v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
void *_buf, size_t t, int consume)

v4v_copy_out will copy at most t bytes from the ring r and place them in buf, if it
is non-null. If from and protocol are not NULL pointers then the source address and
protocol number will by copied into them. If consume is non-zero then the ring's
receive pointer will be advanced. The function returns the number of bytes of payload
that the message has (ie. the number of bytes passed to the original send or sendv call).
Thus, v4v_copy_out(r,NULL,NULL,NULL,0,0); will return the number of bytes in the next message,
and v4v_copy_out(r,NULL,NULL,NULL,0,1); will return the number of bytes in the next message
and delete it from the ring.

** Notification
If the receiving ring is full in a send or sendv hypercall, the hypercall will return with
the error -EAGAIN. When sufficient space becomes available in the ring xen will raise the V4V VIRQ
line.

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 7, 2012, 4:40 AM

Post #8 of 32 (440 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

Hi,

Thanks for this.

Overall, it looks like Xen is doing a few things here:
- nameservice for registering services and finding endpoints;
- ring manipulation arithmetic;
- copying data; and
- notifying endpoints.

The shared-ring logic was able to do all of these, with a few drawbacks:
- The xenstore handshake stuff is really grotty;
- grant maps can cause zombie domains; and
- it doesn't do many-to-one multicast.

Is that a fair summary?

Using copy semantics intead of shared rings fixes the zombie domain
problem, and I think we should definitely take that.

We talked last week about ways we could improve xenstore to provde the
nameservice aspects of v4v; if that can be done I think it should be.

The rest of is within spitting distance of a multicall of grant copies
and event-channel pokes - except for the many-to-one multicast.
That relies on Xen policing the ring headers to make sure the length
fields are correct.

What's the use case for that? Do you use it just to negotiate a
dedicated ring for each sender or is there a need for multiple data
streams to be multiplexed together?

I have a few questions on the design, inline below.

At 10:42 +0100 on 07 Jun (1339065745), Jean Guyader wrote:
> v4v has a very simple architecture. The receiving domain registers a
> ring with xen. The sending domain requests that xen inserts data into
> the receiving domain's ring. Notification that there is new data to
> read is delivered by a VIRQ

Not an evtchn?

> , and a domain can request an interrupt be generated when sufficient
> space is available in another domain's ring.

How is that arranged? It doesn't look like the receiver notifies Xen
when it consumes a ring entry.

> The code that inserts
> the data into the ring is in xen, and xen writes a header describing
> the data and where it came from infront of the data. As both the ring
> manipulation and the header are written by the hypervisor, the
> receiving domain can trust their content and the format of the ring.

Does that save a copy on the receiving side?

> ** Addressing
> v4v_addr_t identifies an endpoint address.
> typedef struct v4v_addr
> {
> uint32_t port;
> domid_t domain;
> } v4v_addr_t;
> struct v4v_ring_id uniquely identifies a ring on the platform.
>
> struct v4v_ring_id
> {
> struct v4v_addr addr;
> domid_t partner;
> };
>
> When a send operation is performed, xen will attempt to find a ring
> registered by destination domain, with the required port, and with
> a partner of the sending domain. Failing that it will then search
> for a ring with the correct port and destination domain, but with
> partner set to V4V_DOMID_ANY.
>
> ** Setup
> A domain first generates a ring, and a structure describing the pages
> in the ring. Rings must be page aligned, and contiguous in virtual
> memory, but need not be contiguous in physical(pfn) or machine(mfn)
> memory. A ring is uniquely identified on the platform by its struct
> v4v_ring_id.
>
> A ring is contained in a v4v_ring_t
> typedef struct v4v_ring
> {
> uint64_t magic;
> /*
> * Identifies ring_id - xen only looks at this during register/unregister
> * and will fill in id.addr.domain
> */
> struct v4v_ring_id id;
> /* length of ring[], must be a multiple of 16 */
> uint32_t len;
> /* rx_ptr - modified by domain */
> uint32_t rx_ptr;
> /* tx_ptr - modified by xen */
> volatile uint32_t tx_ptr;
> uint64_t reserved[4];
> volatile uint8_t ring[0];
> } v4v_ring_t;

(Digressing into code review for a minute, I don't think we should use
'volatile' in the public headers, and possibly it will be more efficient
to use barriers anyway if we're processing the received data in place).

> To register the ring, xen also needs a list of pfn that the ring occupies.
> This is provided in a v4v_pfn_list_t
> typedef struct v4v_pfn_list_t
> {
> uint64_t magic;
> uint32_t npage;
> uint32_t pad;
> uint64_t reserved[3];
> v4v_pfn_t pages[0];
> } v4v_pfn_list_t;
>
> Registering the ring is done with a hypercall
>
> int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t *pfn_list);
>
> On registering, xen will check that the tx_ptr and rx_ptr values are valid (aligned
> and within the size of the ring) and modify them if they are not

Modify them? Not return an error?

>, it will also update
> id.addr.domain with the calling domain's domid and take an internal copy of all the
> relevant information. After registration the only field that xen will read is
> ring->rx_ptr, xen will write to ring->ring[] and ring->tx_ptr.
> Thus pfn_list can be freed after the registration call. Critically, registration is
> idempotent, and all a domain need do after hibernation or migration is re-register all
> its rings. (NB the id.addr.domain field may change).
>
> ** Sending
>
> To send data a guest merely calls the send or sendv hypercall
> int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, uint32_t len,
> uint32_t protocol);

What does the source address do here? Do I need to register a ring in
order to send to another ring?

Why the separate 'protocol' argument when 'port' is bound into the address
struct? A single ring receives messages for a given port but all protocols?

> the caller provides a source address, a destination address, a buffer, a number of bytes to copy
> and a 32 bit protocol number. Xen ignores src->domain and instead uses the domain id of the caller.
>
> Xen will attempt to insert into a ring with id.addr equal to dst, and with
> id.partner equal to the caller's domid. Otherwise it will insert into a ring
> with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the insert is successfull
> then Xen will trigger the V4V VIRQ line in the receiving domain.
>
> ** Reception
>
> Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is provided
> for convience), and inserts a message header infront of all messages it copies onto the ring.
>
> struct v4v_ring_message_header
> {
> uint32_t len;
> struct v4v_addr source;
> uint16_t pad;
> uint32_t protocol;
> uint8_t data[0];
> };
>
> len is the number of bytes in the message including the struct v4v_ring_message_header
> source specifies the source address from which the message came, source.domain is
> set by xen, and source.port is taken from the arguments to the send or sendv call.
> protocol is the protocol value given to the send call.
>
> An inline function is provided to make reading from the ring easier:
>
> ssize_t
> v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
> void *_buf, size_t t, int consume)

I guess that answers my question about avoiding a copy on the receiver. :(

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 7, 2012, 8:36 AM

Post #9 of 32 (440 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 12:40 +0100 on 07 Jun (1339072831), Tim Deegan wrote:
> Hi,
>
> Thanks for this.
>
> Overall, it looks like Xen is doing a few things here:
> - nameservice for registering services and finding endpoints;
> - ring manipulation arithmetic;
> - copying data; and
> - notifying endpoints.
>
> The shared-ring logic was able to do all of these, with a few drawbacks:
> - The xenstore handshake stuff is really grotty;
> - grant maps can cause zombie domains; and
> - it doesn't do many-to-one multicast.

We've just had a discussion of this in person, and from my notes the two
things that stand out are:

- yes, you want to do many-to-one multiplexing, in particular to
avoid the server needing a ring for every client; and
- one reason not to use Xenstore is that there is only one Xenstore
page per VM and it may not be possible to safely share it with other
users (in particular between a BIOS/EFI user and the main OS).

The Xenstore point is understandable, and probably something that ought
to be fixed anyway -- we have seen people run into similar problems with
BIOS drivers for blkfront and netfront.

Using one ring for all clients raises the question of access control and
admission control -- in particular how do you avoid DoS from
badly-behaved clients?

And, given your concerns about sharing an OS with an uncooperative
Xenstore client, how do you handle sharing the OS with a badly behaved
v4v client?

If we _do_ need one ring with multiple writers, and therefore Xen needs
to arbitrate writes, there's still room for the policy-based parts
(controlling connection setup, for example) to live outside the
hypervisor, openvswitch-style.

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 13, 2012, 3:48 AM

Post #10 of 32 (442 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 07/06 04:36, Tim Deegan wrote:
> At 12:40 +0100 on 07 Jun (1339072831), Tim Deegan wrote:
> > Hi,
> >
> > Thanks for this.
> >
> > Overall, it looks like Xen is doing a few things here:
> > - nameservice for registering services and finding endpoints;
> > - ring manipulation arithmetic;
> > - copying data; and
> > - notifying endpoints.
> >
> > The shared-ring logic was able to do all of these, with a few drawbacks:
> > - The xenstore handshake stuff is really grotty;
> > - grant maps can cause zombie domains; and
> > - it doesn't do many-to-one multicast.
>
> We've just had a discussion of this in person, and from my notes the two
> things that stand out are:
>
> - yes, you want to do many-to-one multiplexing, in particular to
> avoid the server needing a ring for every client; and
> - one reason not to use Xenstore is that there is only one Xenstore
> page per VM and it may not be possible to safely share it with other
> users (in particular between a BIOS/EFI user and the main OS).
>
> The Xenstore point is understandable, and probably something that ought
> to be fixed anyway -- we have seen people run into similar problems with
> BIOS drivers for blkfront and netfront.
>
> Using one ring for all clients raises the question of access control and
> admission control -- in particular how do you avoid DoS from
> badly-behaved clients?
>
> And, given your concerns about sharing an OS with an uncooperative
> Xenstore client, how do you handle sharing the OS with a badly behaved
> v4v client?
>
> If we _do_ need one ring with multiple writers, and therefore Xen needs
> to arbitrate writes, there's still room for the policy-based parts
> (controlling connection setup, for example) to live outside the
> hypervisor, openvswitch-style.
>

Thanks for summary Tim.

Today the acl check in V4V (not part of the current patch serie) is done
for every copy by Xen. Moving the policy control outside of Xen would mean that
you still need to have a copy of the acls in Xen and the worst
thing that can happen is for the copy to get out of sync.

What do you think would be the next step going forward?

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 13, 2012, 4:44 AM

Post #11 of 32 (436 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 11:48 +0100 on 13 Jun (1339588138), Jean Guyader wrote:
> On 07/06 04:36, Tim Deegan wrote:
> > Using one ring for all clients raises the question of access control and
> > admission control -- in particular how do you avoid DoS from
> > badly-behaved clients?
> >
> > And, given your concerns about sharing an OS with an uncooperative
> > Xenstore client, how do you handle sharing the OS with a badly behaved
> > v4v client?
> >
> > If we _do_ need one ring with multiple writers, and therefore Xen needs
> > to arbitrate writes, there's still room for the policy-based parts
> > (controlling connection setup, for example) to live outside the
> > hypervisor, openvswitch-style.
>
> Today the acl check in V4V (not part of the current patch serie) is
> done for every copy by Xen.

OK; can you describe roughly how it works? Is it a whitelist of good
domains, or a blacklist of bad? Does it just do access control or is
there rate limiting? Can it detect and block badly-behaved clients,
or is that something you do in the server?

Have you given any thought to my second question -- if you can't rely on
the existing xenstore code, are there problems with sharing a VM with
other v4v drivers? My intuition is that at least it would not be so
bad, since non-malicious drivers ought to be able to coexist, but I'd
like to know your opinion.

> Moving the policy control outside of Xen
> would mean that you still need to have a copy of the acls in Xen and
> the worst thing that can happen is for the copy to get out of sync.

What I meant by openvswitch-style is to have a single low-level ACL in
the hypervisor (maybe as a whitelist of known good connections) and
fault all unusual behaviour (new domain appears, &c) to the more complex
policy engine in user-space.

Really it depends on what kind of policy you want to be able to express.
If a simple yes/no whitelist is good enough (and always will be) then it
may as well live in the hypervisor and we can skip the 'defer to
userspace' part.

> What do you think would be the next step going forward?

Hopefully, to get some feedback from other people (specifically Ian, Ian
and Stefano but anyone else too!) and decide what, if anything, ought to
be changed about the design.

My feedback has mostly been about the hypervisor side of things -- I'm
sure the tools maintainers have some ideas about how this should be
merged with libvchan, to avoid having two separate comms interfaces.

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at gmail

Jun 14, 2012, 3:55 AM

Post #12 of 32 (435 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 13 June 2012 12:44, Tim Deegan <tim [at] xen> wrote:
> At 11:48 +0100 on 13 Jun (1339588138), Jean Guyader wrote:
>> On 07/06 04:36, Tim Deegan wrote:
>> > Using one ring for all clients raises the question of access control and
>> > admission control -- in particular how do you avoid DoS from
>> > badly-behaved clients?
>> >
>> > And, given your concerns about sharing an OS with an uncooperative
>> > Xenstore client, how do you handle sharing the OS with a badly behaved
>> > v4v client?
>> >
>> > If we _do_ need one ring with multiple writers, and therefore Xen needs
>> > to arbitrate writes, there's still room for the policy-based parts
>> > (controlling connection setup, for example) to live outside the
>> > hypervisor, openvswitch-style.
>>
>> Today the acl check in V4V (not part of the current patch serie) is
>> done for every copy by Xen.
>
> OK; can you describe roughly how it works?  Is it a whitelist of good
> domains, or a blacklist of bad?  Does it just do access control or is
> there rate limiting?  Can it detect and block badly-behaved clients,
> or is that something you do in the server?
>

In xen we keep of list of simple acls. Here is the usage of the userspace
tools that controls it. This is a straight forward implementation of the rule
API.

Usage: viptables command [rule]
Commands :
--append -A Append rule
--insert -I <n> Insert rule before rule <n>
--list -L List rules
--delete -D[<n>] Delete rule <n> or the following rule
--flush -F Flush rules
--help -h Help
Rule options:
--dom-in -d <n> Client domid
--dom-out -e <n> Server domid
--port-in -p <n> Client port
--port-out -q <n> Server port
--jump -j {ACCEPT|REJECT} What to do


> Have you given any thought to my second question -- if you can't rely on
> the existing xenstore code, are there problems with sharing a VM with
> other v4v drivers?  My intuition is that at least it would not be so
> bad, since non-malicious drivers ought to be able to coexist, but I'd
> like to know your opinion.
>

Are you talking about having different version of V4V driver running
in the same VM?
I don't think that is a problem they both interact with Xen via
hypercall directly so
if they follow the v4v hypercall interface it's all fine.

>> Moving the policy control outside of Xen
>> would mean that you still need to have a copy of the acls in Xen and
>> the worst thing that can happen is for the copy to get out of sync.
>
> What I meant by openvswitch-style is to have a single low-level ACL in
> the hypervisor (maybe as a whitelist of known good connections) and
> fault all unusual behaviour (new domain appears, &c) to the more complex
> policy engine in user-space.
>

Yes sure.

> Really it depends on what kind of policy you want to be able to express.
> If a simple yes/no whitelist is good enough (and always will be) then it
> may as well live in the hypervisor and we can skip the 'defer to
> userspace' part.
>
>> What do you think would be the next step going forward?
>
> Hopefully, to get some feedback from other people (specifically Ian, Ian
> and Stefano but anyone else too!) and decide what, if anything, ought to
> be changed about the design.
>
> My feedback has mostly been about the hypervisor side of things -- I'm
> sure the tools maintainers have some ideas about how this should be
> merged with libvchan, to avoid having two separate comms interfaces.
>

Ok. Thanks for your feedback.

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 14, 2012, 7:01 AM

Post #13 of 32 (433 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 01/06 02:24, George Dunlap wrote:
> On Fri, Jun 1, 2012 at 1:41 PM, Jan Beulich <JBeulich [at] suse> wrote:
> >>>> On 31.05.12 at 17:07, Jean Guyader <jean.guyader [at] citrix> wrote:
> >
> > Is there a particular reason this series got resent with no apparent
> > change (and also none mentioned in the description here or in the
> > individual patches)? All of the comments sent yesterday still apply,
> > and I continue to consider patches 1-4 as unnecessary/broken.
>
> The date on the second set of e-mails is 15 minutes before the
> original series. I presume this means that Jean sent the series but
> it got held up for some reason, so sent it again 15 minutes later; and
> the second series made it through but the first was held up until
> today.
>
> Never attribute to malice that which can be attributed to computer error. :-)
>

George is absolutely spot on, this is exactly what happen.

Sorry about that :(

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 14, 2012, 7:56 AM

Post #14 of 32 (433 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> Are you talking about having different version of V4V driver running
> in the same VM?

Yes.

> I don't think that is a problem they both interact with Xen via
> hypercall directly so if they follow the v4v hypercall interface it's
> all fine.

AFAICS if they both try to register the same port then one of them will
silently get its ring discarded. And if they both try to communicate
with the same remote port their entries on the pending lists will get
merged (which is probably not too bad). I think the possibility for
confusion depends on how you use the service. Still, it seems better
than the xenstore case, anyway. :)

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 14, 2012, 8:10 AM

Post #15 of 32 (431 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 14/06 03:56, Tim Deegan wrote:
> At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > Are you talking about having different version of V4V driver running
> > in the same VM?
>
> Yes.
>
> > I don't think that is a problem they both interact with Xen via
> > hypercall directly so if they follow the v4v hypercall interface it's
> > all fine.
>
> AFAICS if they both try to register the same port then one of them will
> silently get its ring discarded. And if they both try to communicate
> with the same remote port their entries on the pending lists will get
> merged (which is probably not too bad). I think the possibility for
> confusion depends on how you use the service. Still, it seems better
> than the xenstore case, anyway. :)
>

Not silently, register_ring will return an error.

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 14, 2012, 8:35 AM

Post #16 of 32 (432 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> On 14/06 03:56, Tim Deegan wrote:
> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > > Are you talking about having different version of V4V driver running
> > > in the same VM?
> >
> > Yes.
> >
> > > I don't think that is a problem they both interact with Xen via
> > > hypercall directly so if they follow the v4v hypercall interface it's
> > > all fine.
> >
> > AFAICS if they both try to register the same port then one of them will
> > silently get its ring discarded. And if they both try to communicate
> > with the same remote port their entries on the pending lists will get
> > merged (which is probably not too bad). I think the possibility for
> > confusion depends on how you use the service. Still, it seems better
> > than the xenstore case, anyway. :)
> >
>
> Not silently, register_ring will return an error.

Will it? It looks to me like v4v_ring_add just clobbers the old MFN
list.

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at gmail

Jun 14, 2012, 2:14 PM

Post #17 of 32 (431 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
>> On 14/06 03:56, Tim Deegan wrote:
>> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
>> > > Are you talking about having different version of V4V driver running
>> > > in the same VM?
>> >
>> > Yes.
>> >
>> > > I don't think that is a problem they both interact with Xen via
>> > > hypercall directly so if they follow the v4v hypercall interface it's
>> > > all fine.
>> >
>> > AFAICS if they both try to register the same port then one of them will
>> > silently get its ring discarded.  And if they both try to communicate
>> > with the same remote port their entries on the pending lists will get
>> > merged (which is probably not too bad).  I think the possibility for
>> > confusion depends on how you use the service.  Still, it seems better
>> > than the xenstore case, anyway. :)
>> >
>>
>> Not silently, register_ring will return an error.
>
> Will it?  It looks to me like v4v_ring_add just clobbers the old MFN
> list.
>

Ha yes. It does that now but I think it should return an error
informing up the stack that a ring has already been registered.

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 25, 2012, 2:05 AM

Post #18 of 32 (423 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> >> On 14/06 03:56, Tim Deegan wrote:
> >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> >> > > Are you talking about having different version of V4V driver running
> >> > > in the same VM?
> >> >
> >> > Yes.
> >> >
> >> > > I don't think that is a problem they both interact with Xen via
> >> > > hypercall directly so if they follow the v4v hypercall interface it's
> >> > > all fine.
> >> >
> >> > AFAICS if they both try to register the same port then one of them will
> >> > silently get its ring discarded.  And if they both try to communicate
> >> > with the same remote port their entries on the pending lists will get
> >> > merged (which is probably not too bad).  I think the possibility for
> >> > confusion depends on how you use the service.  Still, it seems better
> >> > than the xenstore case, anyway. :)
> >> >
> >>
> >> Not silently, register_ring will return an error.
> >
> > Will it?  It looks to me like v4v_ring_add just clobbers the old MFN
> > list.
> >
>
> Ha yes. It does that now but I think it should return an error
> informing up the stack that a ring has already been registered.

Actually, I think it's deliberate, to allow a guest to re-register all
its rings after a suspend/resume or migration, without having to worry
about whether it was actually migrated into a new domain or not.

That raises the question of how a v4v client ought to handle migration;
doesn't it have to rely on other OS components to notify it that one has
happened?

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


Ian.Campbell at citrix

Jun 26, 2012, 7:38 AM

Post #19 of 32 (421 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

Hi,

Sorry it's taken me so long to get round to responding to this.

On Mon, 2012-06-25 at 10:05 +0100, Tim Deegan wrote:
> At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> > On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> > >> On 14/06 03:56, Tim Deegan wrote:
> > >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > >> > > Are you talking about having different version of V4V driver running
> > >> > > in the same VM?
> > >> >
> > >> > Yes.
> > >> >
> > >> > > I don't think that is a problem they both interact with Xen via
> > >> > > hypercall directly so if they follow the v4v hypercall interface it's
> > >> > > all fine.
> > >> >
> > >> > AFAICS if they both try to register the same port then one of them will
> > >> > silently get its ring discarded. And if they both try to communicate
> > >> > with the same remote port their entries on the pending lists will get
> > >> > merged (which is probably not too bad). I think the possibility for
> > >> > confusion depends on how you use the service. Still, it seems better
> > >> > than the xenstore case, anyway. :)
> > >> >
> > >>
> > >> Not silently, register_ring will return an error.
> > >
> > > Will it? It looks to me like v4v_ring_add just clobbers the old MFN
> > > list.
> > >
> >
> > Ha yes. It does that now but I think it should return an error
> > informing up the stack that a ring has already been registered.
>
> Actually, I think it's deliberate, to allow a guest to re-register all
> its rings after a suspend/resume or migration, without having to worry
> about whether it was actually migrated into a new domain or not.

Which takes us back to the original issue Tim asked about with
cohabitation of multiple (perhaps just plain buggy or even malicious)
v4v clients in a single domain, doesn't it?

If one of the reasons for not using xenstore is the inability for
multiple clients to co-exist then there is no point moving to a
different scheme with the same (even if lesser) issue. Up until now v4v
has only been used by XenClient so it has avoided this issue but if we
take it upstream then presumably the potential for this sort of problem
to creep in comes back.

Some other things from my notes...

Can you remind me of the reasoning behind using VIRQ_V4V instead of
regular event channels? I can't see any reason why these
shouldn't/couldn't be regular event channels demuxed in the usual way.

Are you going to submit any client code? I think the most important of
these would be code to make libvchan able to use v4v if it is available
(and both ends agree), but any other client code (like the sockets
interface overlay I've heard about) would be interesting to see too.

Have you had any contact from any one else who is interested in building
stuff with these interfaces? (This is just curiosity about potential
wider usage)

I think you and Tim have been discussing access control -- can we get a
summary of what this is going to look like going forward. I suppose
there will be tools for manipulating this stuff -- can you post them?

The patches need plenty of documentation adding to them, both in the
interface docs in xen/include/public (marked up such that it comes out
nicely in docs/html aka
http://xenbits.xen.org/docs/unstable/hypercall/index.html) as well as
design docs and any other useful material you have under docs itself.

Are we generally happy that we could run e.g. block or net traffic over
this channel? As it stands for example the single VIRQ would seem to
provide a scalability limit to how many disks and nics a domain could
have. Are there any other considerations we need to take into account
here?

Lastly -- have you given any thoughts to making the copy operation
asynchronous? This might allow us to take advantage of copy offload
hardware in the future?

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 28, 2012, 3:19 AM

Post #20 of 32 (421 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 25/06 10:05, Tim Deegan wrote:
> At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> > On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> > >> On 14/06 03:56, Tim Deegan wrote:
> > >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > >> > > Are you talking about having different version of V4V driver running
> > >> > > in the same VM?
> > >> >
> > >> > Yes.
> > >> >
> > >> > > I don't think that is a problem they both interact with Xen via
> > >> > > hypercall directly so if they follow the v4v hypercall interface it's
> > >> > > all fine.
> > >> >
> > >> > AFAICS if they both try to register the same port then one of them will
> > >> > silently get its ring discarded.  And if they both try to communicate
> > >> > with the same remote port their entries on the pending lists will get
> > >> > merged (which is probably not too bad).  I think the possibility for
> > >> > confusion depends on how you use the service.  Still, it seems better
> > >> > than the xenstore case, anyway. :)
> > >> >
> > >>
> > >> Not silently, register_ring will return an error.
> > >
> > > Will it?  It looks to me like v4v_ring_add just clobbers the old MFN
> > > list.
> > >
> >
> > Ha yes. It does that now but I think it should return an error
> > informing up the stack that a ring has already been registered.
>
> Actually, I think it's deliberate, to allow a guest to re-register all
> its rings after a suspend/resume or migration, without having to worry
> about whether it was actually migrated into a new domain or not.
>

Yes your are right. The driver will still try to register but a correct error
code could tell it weather or not it has been done.


> That raises the question of how a v4v client ought to handle migration;
> doesn't it have to rely on other OS components to notify it that one has
> happened?
>

Migration will cause the connection to close and the event will propagated
up the stack.

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 28, 2012, 3:38 AM

Post #21 of 32 (422 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 26/06 03:38, Ian Campbell wrote:
> Hi,
>
> Sorry it's taken me so long to get round to responding to this.
>
> On Mon, 2012-06-25 at 10:05 +0100, Tim Deegan wrote:
> > At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> > > On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > > > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> > > >> On 14/06 03:56, Tim Deegan wrote:
> > > >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > > >> > > Are you talking about having different version of V4V driver running
> > > >> > > in the same VM?
> > > >> >
> > > >> > Yes.
> > > >> >
> > > >> > > I don't think that is a problem they both interact with Xen via
> > > >> > > hypercall directly so if they follow the v4v hypercall interface it's
> > > >> > > all fine.
> > > >> >
> > > >> > AFAICS if they both try to register the same port then one of them will
> > > >> > silently get its ring discarded. And if they both try to communicate
> > > >> > with the same remote port their entries on the pending lists will get
> > > >> > merged (which is probably not too bad). I think the possibility for
> > > >> > confusion depends on how you use the service. Still, it seems better
> > > >> > than the xenstore case, anyway. :)
> > > >> >
> > > >>
> > > >> Not silently, register_ring will return an error.
> > > >
> > > > Will it? It looks to me like v4v_ring_add just clobbers the old MFN
> > > > list.
> > > >
> > >
> > > Ha yes. It does that now but I think it should return an error
> > > informing up the stack that a ring has already been registered.
> >
> > Actually, I think it's deliberate, to allow a guest to re-register all
> > its rings after a suspend/resume or migration, without having to worry
> > about whether it was actually migrated into a new domain or not.
>
> Which takes us back to the original issue Tim asked about with
> cohabitation of multiple (perhaps just plain buggy or even malicious)
> v4v clients in a single domain, doesn't it?
>

There is nothing wrong the two v4v driver running in the same guest.
The probably that Tim reported was about trying to create two connections
on the same port. Today with the code that I've submited in the RFC
one will overwrite the other silently which isn't a good thing, that can
easily be changed to notify which one got registered up the stack.

> If one of the reasons for not using xenstore is the inability for
> multiple clients to co-exist then there is no point moving to a
> different scheme with the same (even if lesser) issue. Up until now v4v
> has only been used by XenClient so it has avoided this issue but if we
> take it upstream then presumably the potential for this sort of problem
> to creep in comes back.
>
> Some other things from my notes...
>
> Can you remind me of the reasoning behind using VIRQ_V4V instead of
> regular event channels? I can't see any reason why these
> shouldn't/couldn't be regular event channels demuxed in the usual way.
>

No good reason, the virq can be changed to a event channel. We thought
that event channels required other machinery to function. If we can
deal with the event channel in the v4v driver directly I don't have any
problem changing it to a event channel. What I have in mind is if one
would want to use v4v in a rombios for instance where you can't rely
on any of the xen pv driver code to be present.

> Are you going to submit any client code? I think the most important of
> these would be code to make libvchan able to use v4v if it is available
> (and both ends agree), but any other client code (like the sockets
> interface overlay I've heard about) would be interesting to see too.
>

Yes. I still have to submit the kernel driver code and the v4v library that
talks to it. But now that you've pointed out that we might be able to use
an event channel instead of a virq, I think we might event be able to have
a driver in userspace only.

> Have you had any contact from any one else who is interested in building
> stuff with these interfaces? (This is just curiosity about potential
> wider usage)
>

Today in XenClient we use this interface for:
- USB transport between dom0 and guest
- RPC bus beween dom0 and domU
- Cross domain Citrix ICA connections.

The good thing that this interface can offer is a transport
for all the existing networking programs. I haven't had
any contact from people interested in building stuff with these
interfaces but I think V4V has quite a lot to offer already in
term of thing at the top that can use it.

> I think you and Tim have been discussing access control -- can we get a
> summary of what this is going to look like going forward. I suppose
> there will be tools for manipulating this stuff -- can you post them?
>

I'll post a new series this week that will include some of a feedback
and the access control.

> The patches need plenty of documentation adding to them, both in the
> interface docs in xen/include/public (marked up such that it comes out
> nicely in docs/html aka
> http://xenbits.xen.org/docs/unstable/hypercall/index.html) as well as
> design docs and any other useful material you have under docs itself.
>

I agree such change need a lot of document. Once we agreed on the interfaces
I will start working on a complete documentation for them.

> Are we generally happy that we could run e.g. block or net traffic over
> this channel? As it stands for example the single VIRQ would seem to
> provide a scalability limit to how many disks and nics a domain could
> have. Are there any other considerations we need to take into account
> here?
>
> Lastly -- have you given any thoughts to making the copy operation
> asynchronous? This might allow us to take advantage of copy offload
> hardware in the future?
>

A CPU copy already has to happen once in the guest to put it in the
ring so the second copy that is done in Xen will be really cheap because
it's very linkely going to be in the cache. I don't doing async copy in Xen
will have a huge impact on the performance.

Thanks for taking the time to review.
Jean


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


tim at xen

Jun 28, 2012, 3:50 AM

Post #22 of 32 (422 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

At 11:38 +0100 on 28 Jun (1340883538), Jean Guyader wrote:
> On 26/06 03:38, Ian Campbell wrote:
> > Lastly -- have you given any thoughts to making the copy operation
> > asynchronous? This might allow us to take advantage of copy offload
> > hardware in the future?
>
> A CPU copy already has to happen once in the guest to put it in the
> ring

I don't understand -- isn't the vector-send operation designed to have
Xen do a single copy from scattered sources into the destination ring?
So the sender could be zero-copy?

> so the second copy that is done in Xen will be really cheap because
> it's very linkely going to be in the cache. I don't doing async copy
> in Xen will have a huge impact on the performance.

Using a DMA engine to do the copy could free up CPU cycles that Xen
could use for other vcpus, so even if there isn't a throughput increase
on the v4v channel, there could be a benefit to the system as a whole.
We could do that with synchronous copies, pausing the vcpu while the
copy happens, but I think to get full throughput we'd want to pipeline a
few writes ahead.

Anyway, that's all very forward-looking: AFAICS there's no need to
address it right now except to be careful not to bake things into the
interface that would stop us doing this later on.

Cheers,

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 28, 2012, 4:24 AM

Post #23 of 32 (424 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 28/06 11:50, Tim Deegan wrote:
> At 11:38 +0100 on 28 Jun (1340883538), Jean Guyader wrote:
> > On 26/06 03:38, Ian Campbell wrote:
> > > Lastly -- have you given any thoughts to making the copy operation
> > > asynchronous? This might allow us to take advantage of copy offload
> > > hardware in the future?
> >
> > A CPU copy already has to happen once in the guest to put it in the
> > ring
>
> I don't understand -- isn't the vector-send operation designed to have
> Xen do a single copy from scattered sources into the destination ring?
> So the sender could be zero-copy?
>

Yes sorry that is right. On send a get a pointer from the guest
and then Xen memcpy the data on the receiver's ring.

So the second copy that will probably happened between the ring
and some memory in the receiver will be mostly free.

> > so the second copy that is done in Xen will be really cheap because
> > it's very linkely going to be in the cache. I don't doing async copy
> > in Xen will have a huge impact on the performance.
>
> Using a DMA engine to do the copy could free up CPU cycles that Xen
> could use for other vcpus, so even if there isn't a throughput increase
> on the v4v channel, there could be a benefit to the system as a whole.
> We could do that with synchronous copies, pausing the vcpu while the
> copy happens, but I think to get full throughput we'd want to pipeline a
> few writes ahead.
>

Maybe if you ring is big enough and if the guest sends very big buffer to be
copied that will makes sence to use async copy like i/oat or DMA.

> Anyway, that's all very forward-looking: AFAICS there's no need to
> address it right now except to be careful not to bake things into the
> interface that would stop us doing this later on.
>


Jean


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


Ian.Campbell at citrix

Jun 28, 2012, 4:34 AM

Post #24 of 32 (420 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On Thu, 2012-06-28 at 11:38 +0100, Jean Guyader wrote:
> On 26/06 03:38, Ian Campbell wrote:
> > Hi,
> >
> > Sorry it's taken me so long to get round to responding to this.
> >
> > On Mon, 2012-06-25 at 10:05 +0100, Tim Deegan wrote:
> > > At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> > > > On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > > > > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> > > > >> On 14/06 03:56, Tim Deegan wrote:
> > > > >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > > > >> > > Are you talking about having different version of V4V driver running
> > > > >> > > in the same VM?
> > > > >> >
> > > > >> > Yes.
> > > > >> >
> > > > >> > > I don't think that is a problem they both interact with Xen via
> > > > >> > > hypercall directly so if they follow the v4v hypercall interface it's
> > > > >> > > all fine.
> > > > >> >
> > > > >> > AFAICS if they both try to register the same port then one of them will
> > > > >> > silently get its ring discarded. And if they both try to communicate
> > > > >> > with the same remote port their entries on the pending lists will get
> > > > >> > merged (which is probably not too bad). I think the possibility for
> > > > >> > confusion depends on how you use the service. Still, it seems better
> > > > >> > than the xenstore case, anyway. :)
> > > > >> >
> > > > >>
> > > > >> Not silently, register_ring will return an error.
> > > > >
> > > > > Will it? It looks to me like v4v_ring_add just clobbers the old MFN
> > > > > list.
> > > > >
> > > >
> > > > Ha yes. It does that now but I think it should return an error
> > > > informing up the stack that a ring has already been registered.
> > >
> > > Actually, I think it's deliberate, to allow a guest to re-register all
> > > its rings after a suspend/resume or migration, without having to worry
> > > about whether it was actually migrated into a new domain or not.
> >
> > Which takes us back to the original issue Tim asked about with
> > cohabitation of multiple (perhaps just plain buggy or even malicious)
> > v4v clients in a single domain, doesn't it?
> >
>
> There is nothing wrong the two v4v driver running in the same guest.
> The probably that Tim reported was about trying to create two connections
> on the same port. Today with the code that I've submited in the RFC
> one will overwrite the other silently which isn't a good thing, that can
> easily be changed to notify which one got registered up the stack.

So they'd somehow need to randomise (and retry) their use of source
ports in order to co-exist?

Speaking of ports, is there a registry somewhere of the well known port
numbers and/or any scheme for administering these? (a text file in the
repo would be find by me).

> > If one of the reasons for not using xenstore is the inability for
> > multiple clients to co-exist then there is no point moving to a
> > different scheme with the same (even if lesser) issue. Up until now v4v
> > has only been used by XenClient so it has avoided this issue but if we
> > take it upstream then presumably the potential for this sort of problem
> > to creep in comes back.
> >
> > Some other things from my notes...
> >
> > Can you remind me of the reasoning behind using VIRQ_V4V instead of
> > regular event channels? I can't see any reason why these
> > shouldn't/couldn't be regular event channels demuxed in the usual way.
> >
>
> No good reason, the virq can be changed to a event channel. We thought
> that event channels required other machinery to function. If we can
> deal with the event channel in the v4v driver directly I don't have any
> problem changing it to a event channel. What I have in mind is if one
> would want to use v4v in a rombios for instance where you can't rely
> on any of the xen pv driver code to be present.

I think that will be ok -- under the hood a VIRQ is just an evtchn too
so if you can make the VIRQ work you can a regular evtchn work too.

> > Are you going to submit any client code? I think the most important of
> > these would be code to make libvchan able to use v4v if it is available
> > (and both ends agree), but any other client code (like the sockets
> > interface overlay I've heard about) would be interesting to see too.
> >
>
> Yes. I still have to submit the kernel driver code and the v4v library that
> talks to it. But now that you've pointed out that we might be able to use
> an event channel instead of a virq, I think we might event be able to have
> a driver in userspace only.

That would be a nice property to have!

> > The patches need plenty of documentation adding to them, both in the
> > interface docs in xen/include/public (marked up such that it comes out
> > nicely in docs/html aka
> > http://xenbits.xen.org/docs/unstable/hypercall/index.html) as well as
> > design docs and any other useful material you have under docs itself.
> >
>
> I agree such change need a lot of document. Once we agreed on the interfaces
> I will start working on a complete documentation for them.

Thanks!

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel


jean.guyader at citrix

Jun 28, 2012, 4:43 AM

Post #25 of 32 (420 views)
Permalink
Re: [RFC][PATCH 0/5] Add V4V to Xen [In reply to]

On 28/06 12:34, Ian Campbell wrote:
> On Thu, 2012-06-28 at 11:38 +0100, Jean Guyader wrote:
> > On 26/06 03:38, Ian Campbell wrote:
> > > Hi,
> > >
> > > Sorry it's taken me so long to get round to responding to this.
> > >
> > > On Mon, 2012-06-25 at 10:05 +0100, Tim Deegan wrote:
> > > > At 22:14 +0100 on 14 Jun (1339712061), Jean Guyader wrote:
> > > > > On 14 June 2012 16:35, Tim Deegan <tim [at] xen> wrote:
> > > > > > At 16:10 +0100 on 14 Jun (1339690244), Jean Guyader wrote:
> > > > > >> On 14/06 03:56, Tim Deegan wrote:
> > > > > >> > At 11:55 +0100 on 14 Jun (1339674908), Jean Guyader wrote:
> > > > > >> > > Are you talking about having different version of V4V driver running
> > > > > >> > > in the same VM?
> > > > > >> >
> > > > > >> > Yes.
> > > > > >> >
> > > > > >> > > I don't think that is a problem they both interact with Xen via
> > > > > >> > > hypercall directly so if they follow the v4v hypercall interface it's
> > > > > >> > > all fine.
> > > > > >> >
> > > > > >> > AFAICS if they both try to register the same port then one of them will
> > > > > >> > silently get its ring discarded. And if they both try to communicate
> > > > > >> > with the same remote port their entries on the pending lists will get
> > > > > >> > merged (which is probably not too bad). I think the possibility for
> > > > > >> > confusion depends on how you use the service. Still, it seems better
> > > > > >> > than the xenstore case, anyway. :)
> > > > > >> >
> > > > > >>
> > > > > >> Not silently, register_ring will return an error.
> > > > > >
> > > > > > Will it? It looks to me like v4v_ring_add just clobbers the old MFN
> > > > > > list.
> > > > > >
> > > > >
> > > > > Ha yes. It does that now but I think it should return an error
> > > > > informing up the stack that a ring has already been registered.
> > > >
> > > > Actually, I think it's deliberate, to allow a guest to re-register all
> > > > its rings after a suspend/resume or migration, without having to worry
> > > > about whether it was actually migrated into a new domain or not.
> > >
> > > Which takes us back to the original issue Tim asked about with
> > > cohabitation of multiple (perhaps just plain buggy or even malicious)
> > > v4v clients in a single domain, doesn't it?
> > >
> >
> > There is nothing wrong the two v4v driver running in the same guest.
> > The probably that Tim reported was about trying to create two connections
> > on the same port. Today with the code that I've submited in the RFC
> > one will overwrite the other silently which isn't a good thing, that can
> > easily be changed to notify which one got registered up the stack.
>
> So they'd somehow need to randomise (and retry) their use of source
> ports in order to co-exist?
>

That can be assimilated to two userspace programs trying to bind to the
same TCP port. I think it's not v4v's responsability to solve this problem.

> Speaking of ports, is there a registry somewhere of the well known port
> numbers and/or any scheme for administering these? (a text file in the
> repo would be find by me).
>

Port numbers are 32 bits, by convention the first 65535 will match the TCP onces,
then for the rest we can have a file in the repo to reference them.

> > > If one of the reasons for not using xenstore is the inability for
> > > multiple clients to co-exist then there is no point moving to a
> > > different scheme with the same (even if lesser) issue. Up until now v4v
> > > has only been used by XenClient so it has avoided this issue but if we
> > > take it upstream then presumably the potential for this sort of problem
> > > to creep in comes back.
> > >
> > > Some other things from my notes...
> > >
> > > Can you remind me of the reasoning behind using VIRQ_V4V instead of
> > > regular event channels? I can't see any reason why these
> > > shouldn't/couldn't be regular event channels demuxed in the usual way.
> > >
> >
> > No good reason, the virq can be changed to a event channel. We thought
> > that event channels required other machinery to function. If we can
> > deal with the event channel in the v4v driver directly I don't have any
> > problem changing it to a event channel. What I have in mind is if one
> > would want to use v4v in a rombios for instance where you can't rely
> > on any of the xen pv driver code to be present.
>
> I think that will be ok -- under the hood a VIRQ is just an evtchn too
> so if you can make the VIRQ work you can a regular evtchn work too.
>
> > > Are you going to submit any client code? I think the most important of
> > > these would be code to make libvchan able to use v4v if it is available
> > > (and both ends agree), but any other client code (like the sockets
> > > interface overlay I've heard about) would be interesting to see too.
> > >
> >
> > Yes. I still have to submit the kernel driver code and the v4v library that
> > talks to it. But now that you've pointed out that we might be able to use
> > an event channel instead of a virq, I think we might event be able to have
> > a driver in userspace only.
>
> That would be a nice property to have!
>
> > > The patches need plenty of documentation adding to them, both in the
> > > interface docs in xen/include/public (marked up such that it comes out
> > > nicely in docs/html aka
> > > http://xenbits.xen.org/docs/unstable/hypercall/index.html) as well as
> > > design docs and any other useful material you have under docs itself.
> > >
> >
> > I agree such change need a lot of document. Once we agreed on the interfaces
> > I will start working on a complete documentation for them.
>
> Thanks!
>

Jean

_______________________________________________
Xen-devel mailing list
Xen-devel [at] lists
http://lists.xen.org/xen-devel

First page Previous page 1 2 Next page Last page  View All Xen devel RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.