jean.guyader at citrix
Jun 7, 2012, 2:42 AM
Post #7 of 32
On 01/06 02:47, Ian Campbell wrote:
> On Thu, 2012-05-31 at 16:07 +0100, Jean Guyader wrote:
> > V4V is a copy based inter vm communication system.
> > Please have a look at this thread for more detail
> > about V4V.
> > http://lists.xen.org/archives/html/xen-devel/2012-05/msg01866.html
> > This patch series is work in progress but I wanted to
> > post it early enough so I can get feedback from people.
> The main thing which is missing here, which makes it rather hard to
> review, is any kind of design documentation. It would also be great to
> see the rationale for why things have to be done this way.
> For example it seems that there is a bunch of stuff being added to the
> hypervisor which could live in the context of some guest or service
> domain -- putting stuff like that in the hypervisor needs strong
> arguments and rationale why it has to be there rather than somewhere
> Likewise perhaps the v4v hypercall could effectively be implemented with
> a multicall containing some grant copy ops and evtchn manipulations?
> But in the absence of any descriptions of the whats, hows and whys of
> v4v its rather hard to have these sorts of conversations. The above are
> some concrete examples but I'm more interested in seeing the more
> general descriptions before we get into those.
(resent, sorry if you get this message twice)
v4v has a very simple architecture. The receiving domain registers a ring with xen.
The sending domain requests that xen inserts data into the receiving domain's ring.
Notification that there is new data to read is delivered by a VIRQ, and a domain can
request an interrupt be generated when sufficient space is available in another domain's
ring. The code that inserts the data into the ring is in xen, and xen writes a header
describing the data and where it came from infront of the data. As both the ring manipulation
and the header are written by the hypervisor, the receiving domain can trust their content
and the format of the ring.
v4v_addr_t identifies an endpoint address.
typedef struct v4v_addr
struct v4v_ring_id uniquely identifies a ring on the platform.
struct v4v_addr addr;
When a send operation is performed, xen will attempt to find a ring
registered by destination domain, with the required port, and with
a partner of the sending domain. Failing that it will then search
for a ring with the correct port and destination domain, but with
partner set to V4V_DOMID_ANY.
A domain first generates a ring, and a structure describing the pages in the ring.
Rings must be page aligned, and contiguous in virtual memory, but need not be
contiguous in physical(pfn) or machine(mfn) memory. A ring is uniquely identified
on the platform by its struct v4v_ring_id.
A ring is contained in a v4v_ring_t
typedef struct v4v_ring
* Identifies ring_id - xen only looks at this during register/unregister
* and will fill in id.addr.domain
struct v4v_ring_id id;
/* length of ring, must be a multiple of 16 */
/* rx_ptr - modified by domain */
/* tx_ptr - modified by xen */
volatile uint32_t tx_ptr;
volatile uint8_t ring;
To register the ring, xen also needs a list of pfn that the ring occupies.
This is provided in a v4v_pfn_list_t
typedef struct v4v_pfn_list_t
Registering the ring is done with a hypercall
int hypercall(NR_V4V,V4VOP_register_ring,v4v_ring_t *ring,v4v_pfn_list_t *pfn_list);
On registering, xen will check that the tx_ptr and rx_ptr values are valid (aligned
and within the size of the ring) and modify them if they are not, it will also update
id.addr.domain with the calling domain's domid and take an internal copy of all the
relevant information. After registration the only field that xen will read is
ring->rx_ptr, xen will write to ring->ring and ring->tx_ptr.
Thus pfn_list can be freed after the registration call. Critically, registration is
idempotent, and all a domain need do after hibernation or migration is re-register all
its rings. (NB the id.addr.domain field may change).
To send data a guest merely calls the send or sendv hypercall
int hypercall(NR_V4V,V4VOP_send,v4v_addr_t *src,v4v_addr_t *dst,void *buf, uint32_t len,
the caller provides a source address, a destination address, a buffer, a number of bytes to copy
and a 32 bit protocol number. Xen ignores src->domain and instead uses the domain id of the caller.
Xen will attempt to insert into a ring with id.addr equal to dst, and with
id.partner equal to the caller's domid. Otherwise it will insert into a ring
with id.addr equal to dst and id.partner equal to V4V_DOMID_ANY. If the insert is successfull
then Xen will trigger the V4V VIRQ line in the receiving domain.
Xen pads all messages on the ring to 16 bytes (the macro V4V_ROUNDUP is provided
for convience), and inserts a message header infront of all messages it copies onto the ring.
struct v4v_addr source;
len is the number of bytes in the message including the struct v4v_ring_message_header
source specifies the source address from which the message came, source.domain is
set by xen, and source.port is taken from the arguments to the send or sendv call.
protocol is the protocol value given to the send call.
An inline function is provided to make reading from the ring easier:
v4v_copy_out (struct v4v_ring *r, struct v4v_addr *from, uint32_t * protocol,
void *_buf, size_t t, int consume)
v4v_copy_out will copy at most t bytes from the ring r and place them in buf, if it
is non-null. If from and protocol are not NULL pointers then the source address and
protocol number will by copied into them. If consume is non-zero then the ring's
receive pointer will be advanced. The function returns the number of bytes of payload
that the message has (ie. the number of bytes passed to the original send or sendv call).
Thus, v4v_copy_out(r,NULL,NULL,NULL,0,0); will return the number of bytes in the next message,
and v4v_copy_out(r,NULL,NULL,NULL,0,1); will return the number of bytes in the next message
and delete it from the ring.
If the receiving ring is full in a send or sendv hypercall, the hypercall will return with
the error -EAGAIN. When sufficient space becomes available in the ring xen will raise the V4V VIRQ
Xen-devel mailing list
Xen-devel [at] lists