Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Netstack

Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)

 

 

OpenStack netstack RSS feed   Index | Next | Previous | View Threaded


gkotton at redhat

May 9, 2012, 4:26 AM

Post #1 of 22 (1064 views)
Permalink
Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)

Hi,
I have added a very high level description on how to address the issue.
This can be seen at:
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
Comments will be greatly appreciated.
Questions:
1. Do we want agents to be backward compatible (that is, still maintain
the polling code)
2. The generation of the Agent ID
3. Any other ideas or thoughts about the matter?
I'd like to go ahead with a POC and implement this.
Thanks
Gary

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


irenab at mellanox

May 9, 2012, 5:36 AM

Post #2 of 22 (1060 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Gary,
Great work!
I also express my interest in this blueprint due to the reasons you mentioned.
I would like to address number of points in the proposal.
1. VIF Driver to Quantum Server RPC messages: should it use existing APIs or requires new APIs as code or extension?
2. VIF Un-Plug flow: How it is assured that Quantum Server still has the required information when invoked by Agent? Or what you mean here is the VIF unplug RPC message is a core API 'unpluf_interface' call?
3. Quantum Server to Agent RPC Message content can/should be plug-in specific. Am I right?
4. Agent ID can be generated by using host UUID.
5. Are VIF Plug and Un-Plug the only triggers? Can there be some case when Agent need to get data but it's not due to VIF plug/Un-plug flow?

Thanks a lot,
Irena

-----Original Message-----
From: netstack-bounces+irenab=mellanox.com [at] lists [mailto:netstack-bounces+irenab=mellanox.com [at] lists] On Behalf Of Gary Kotton
Sent: Wednesday, May 09, 2012 2:42 PM
To: <netstack [at] lists>
Subject: [Netstack] Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)

Hi,
I have added a very high level description on how to address the issue.
This can be seen at:
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
Comments will be greatly appreciated.
Questions:
1. Do we want agents to be backward compatible (that is, still maintain the polling code) 2. The generation of the Agent ID 3. Any other ideas or thoughts about the matter?
I'd like to go ahead with a POC and implement this.
Thanks
Gary

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


gkotton at redhat

May 9, 2012, 6:12 AM

Post #3 of 22 (1062 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi,
Please see my inline comments.
Thanks
Gary

On 05/09/2012 03:36 PM, Irena Berezovsky wrote:
> Hi Gary,
> Great work!
> I also express my interest in this blueprint due to the reasons you mentioned.
> I would like to address number of points in the proposal.
> 1. VIF Driver to Quantum Server RPC messages: should it use existing APIs or requires new APIs as code or extension?
Ideally this should make use of the nova common RPC code. This is
currently being addressed by
https://blueprints.launchpad.net/openstack-common/+spec/common-rpc.
> 2. VIF Un-Plug flow: How it is assured that Quantum Server still has the required information when invoked by Agent? Or what you mean here is the VIF unplug RPC message is a core API 'unpluf_interface' call?
Do you mean that the agent indicates that the attachment is unplugged.
When the plugin receives this message the attachment entry has already
been deleted? I think that the plugin should be able to deal with cases
like this.
In the document I missed one item - the agent should continue to check
if the attachment is plugged in - if it is not then it should notify the
Quantum plugin that the attachment is down.
> 3. Quantum Server to Agent RPC Message content can/should be plug-in specific. Am I right?
Yes. This may differ per plugins.
> 4. Agent ID can be generated by using host UUID.
Thanks - this is certainly an option
> 5. Are VIF Plug and Un-Plug the only triggers? Can there be some case when Agent need to get data but it's not due to VIF plug/Un-plug flow?
I think that in case for the open source plugins that this may be the
case. I am still not 100% sure.

A few weeks ago I queried about the actual need of the agents. I am
still not 100% sure that we need the agent support - I think that we
also need to understand how the Melange code is going to be added.
> Thanks a lot,
> Irena
>
> -----Original Message-----
> From: netstack-bounces+irenab=mellanox.com [at] lists [mailto:netstack-bounces+irenab=mellanox.com [at] lists] On Behalf Of Gary Kotton
> Sent: Wednesday, May 09, 2012 2:42 PM
> To:<netstack [at] lists>
> Subject: [Netstack] Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
>
> Hi,
> I have added a very high level description on how to address the issue.
> This can be seen at:
> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
> Comments will be greatly appreciated.
> Questions:
> 1. Do we want agents to be backward compatible (that is, still maintain the polling code) 2. The generation of the Agent ID 3. Any other ideas or thoughts about the matter?
> I'd like to go ahead with a POC and implement this.
> Thanks
> Gary
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


rbryant at redhat

May 9, 2012, 8:22 AM

Post #4 of 22 (1053 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/09/2012 09:12 AM, Gary Kotton wrote:
>
> On 05/09/2012 03:36 PM, Irena Berezovsky wrote:
>> 4. Agent ID can be generated by using host UUID.
> Thanks - this is certainly an option

Is there only one agent on a given host? I assume this is used to
target rpc messages to a specific agent.

In nova, it uses its 'host' configuration option for this, which by
default is just the result of socket.gethostname(). It may make
debugging a bit easier if the queues use that instead of a UUID.

--
Russell Bryant

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


mnewby at internap

May 9, 2012, 12:50 PM

Post #5 of 22 (1046 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Gary,

+1 for adding a framework for plugin to agent communication via rpc.

Regarding your list of shortcomings, the security of the database connection would be easily remedied, but the other arguments are compelling by themselves.

I'm less clear on the need to implement rpc communication between the vif driver and quantum plugin, at least as part of this blueprint. Am I missing a use-case that is not already handled by Quantum's rest interface?

Thanks,


Maru


On 2012-05-09, at 4:26 AM, Gary Kotton wrote:

> Hi,
> I have added a very high level description on how to address the issue. This can be seen at: https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
> Comments will be greatly appreciated.
> Questions:
> 1. Do we want agents to be backward compatible (that is, still maintain the polling code)
> 2. The generation of the Agent ID
> 3. Any other ideas or thoughts about the matter?
> I'd like to go ahead with a POC and implement this.
> Thanks
> Gary
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


snaiksat at cisco

May 9, 2012, 2:55 PM

Post #6 of 22 (1046 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Gary,

Thanks for initiating this. A couple of comments/questions -

1. Do we really need the VIF driver to communicate the agent's identity;
I am referring to the agent ID being sent by the VIF driver in the
message? In general, I am not sure if there is a need to have the VIF
driver send messages/notifications in the first place, but I perhaps
it's being included as a capability in the framework?

2. One model I was thinking of (which is kind of inline with the
existing agent implementations), is where the agents are smart, and they
know what to do in response to changes in the state of the logical
Quantum resources. In such cases, the Quantum plugin need not have to
keep track of sending a message to a particular agent. Instead, can we
have broadcast messages from the plugin to all the agents? If the plugin
has to unicast messages to specific agents, then it needs to maintain a
lot more state/topology information which should not be mandated for
this sole reason.

Thanks,
~Sumit.

> -----Original Message-----
> From: netstack-bounces+snaiksat=cisco.com [at] lists
> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
> Behalf Of Gary Kotton
> Sent: Wednesday, May 09, 2012 4:27 AM
> To: <netstack [at] lists>
> Subject: [Netstack] Scalable
> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
> comms)
>
> Hi,
> I have added a very high level description on how to address the
issue.
> This can be seen at:
>
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
> ZKgOlpvg/edit
> Comments will be greatly appreciated.
> Questions:
> 1. Do we want agents to be backward compatible (that is, still
maintain
> the polling code)
> 2. The generation of the Agent ID
> 3. Any other ideas or thoughts about the matter?
> I'd like to go ahead with a POC and implement this.
> Thanks
> Gary
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


rkukura at redhat

May 9, 2012, 6:26 PM

Post #7 of 22 (1047 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/09/2012 03:50 PM, Maru Newby wrote:
> Hi Gary,
>
> +1 for adding a framework for plugin to agent communication via rpc.
>
> Regarding your list of shortcomings, the security of the database connection would be easily remedied, but the other arguments are compelling by themselves.
>
> I'm less clear on the need to implement rpc communication between the vif driver and quantum plugin, at least as part of this blueprint. Am I missing a use-case that is not already handled by Quantum's rest interface?

At least one of the reasons for this is so that connection of a newly
created tap device to the virtual network by the agent would be
triggered immediately by the RPC, rather than waiting for agent's
periodic poll of the node's local tap devices to discover it.

-Bob

>
> Thanks,
>
>
> Maru
>
>
> On 2012-05-09, at 4:26 AM, Gary Kotton wrote:
>
>> Hi,
>> I have added a very high level description on how to address the issue. This can be seen at: https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
>> Comments will be greatly appreciated.
>> Questions:
>> 1. Do we want agents to be backward compatible (that is, still maintain the polling code)
>> 2. The generation of the Agent ID
>> 3. Any other ideas or thoughts about the matter?
>> I'd like to go ahead with a POC and implement this.
>> Thanks
>> Gary
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp
>
>


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


mnewby at internap

May 9, 2012, 8:23 PM

Post #8 of 22 (1045 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Bob,

On 2012-05-09, at 6:26 PM, Robert Kukura wrote:
>>
>> I'm less clear on the need to implement rpc communication between the vif driver and quantum plugin, at least as part of this blueprint. Am I missing a use-case that is not already handled by Quantum's rest interface?
>
> At least one of the reasons for this is so that connection of a newly
> created tap device to the virtual network by the agent would be
> triggered immediately by the RPC, rather than waiting for agent's
> periodic poll of the node's local tap devices to discover it.
>

I'm afraid I don't see your logic. The path is:

Virt driver -> Quantum -> Plugin -> Agent

The blueprint in question proposes that communication between plugin and agent be conducted via rpc rather than the current shared database. The proposal ensures that there is no polling involved. I see no reason for communication between the virt driver and quantum be conducted over rpc when a rest interface can do the same job with no extra effort or complexity.

Thanks,


Maru


gkotton at redhat

May 10, 2012, 12:05 AM

Post #9 of 22 (1048 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
> Hi Gary,
>
> Thanks for initiating this. A couple of comments/questions -
>
> 1. Do we really need the VIF driver to communicate the agent's identity;
> I am referring to the agent ID being sent by the VIF driver in the
> message? In general, I am not sure if there is a need to have the VIF
> driver send messages/notifications in the first place, but I perhaps
> it's being included as a capability in the framework?
At the moment the open source plugins are not aware of the agents. The
agents poll the data base for updates. The agent ID enables a agent to
regsiter with the plugin, this in trrun enables the plugin to send a
update to the specific agent. The update is initiated by the VIF driver.
In my opinion this does the following:
1. updates the agents as soon as possible regarding a network change
2. limits traffic on the network
3. removes the database interface from the agents
> 2. One model I was thinking of (which is kind of inline with the
> existing agent implementations), is where the agents are smart, and they
> know what to do in response to changes in the state of the logical
> Quantum resources. In such cases, the Quantum plugin need not have to
> keep track of sending a message to a particular agent. Instead, can we
> have broadcast messages from the plugin to all the agents? If the plugin
> has to unicast messages to specific agents, then it needs to maintain a
> lot more state/topology information which should not be mandated for
> this sole reason.
I too thought about this option. In a sense the above proposal is an
optimization of what you mention. This comes at the cost of complexity.
The broadcast option is nice when the number of agents is small. When
this is large, then for each network update there will be
NUMBER_OF_AGENT messages sent for each update. The advantage of what you
mention is that the code is self contained in Quantum.

It may be better to start with the broadcast and then deal with the
optimizations afterwards.

Thanks
Gary

> Thanks,
> ~Sumit.
>
>> -----Original Message-----
>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>> Behalf Of Gary Kotton
>> Sent: Wednesday, May 09, 2012 4:27 AM
>> To:<netstack [at] lists>
>> Subject: [Netstack] Scalable
>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>> comms)
>>
>> Hi,
>> I have added a very high level description on how to address the
> issue.
>> This can be seen at:
>>
> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>> ZKgOlpvg/edit
>> Comments will be greatly appreciated.
>> Questions:
>> 1. Do we want agents to be backward compatible (that is, still
> maintain
>> the polling code)
>> 2. The generation of the Agent ID
>> 3. Any other ideas or thoughts about the matter?
>> I'd like to go ahead with a POC and implement this.
>> Thanks
>> Gary
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


rbryant at redhat

May 10, 2012, 12:16 AM

Post #10 of 22 (1049 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/10/2012 03:05 AM, Gary Kotton wrote:
>> 2. One model I was thinking of (which is kind of inline with the
>> existing agent implementations), is where the agents are smart, and they
>> know what to do in response to changes in the state of the logical
>> Quantum resources. In such cases, the Quantum plugin need not have to
>> keep track of sending a message to a particular agent. Instead, can we
>> have broadcast messages from the plugin to all the agents? If the plugin
>> has to unicast messages to specific agents, then it needs to maintain a
>> lot more state/topology information which should not be mandated for
>> this sole reason.
> I too thought about this option. In a sense the above proposal is an
> optimization of what you mention. This comes at the cost of complexity.
> The broadcast option is nice when the number of agents is small. When
> this is large, then for each network update there will be
> NUMBER_OF_AGENT messages sent for each update. The advantage of what you
> mention is that the code is self contained in Quantum.
>
> It may be better to start with the broadcast and then deal with the
> optimizations afterwards.

Unicasting messages to the agents adds complexity, but is it
prohibitively more complex? By using broadcast, you get rid of the
polling, but sacrifice some of the scalability benefits you would
otherwise get by adopting rpc. There are a whole lot more messages
being sent and agents are having to handle a whole lot more messages
than they really care about.

--
Russell Bryant

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


gkotton at redhat

May 10, 2012, 12:17 AM

Post #11 of 22 (1051 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/09/2012 10:50 PM, Maru Newby wrote:
> Hi Gary,
>
> +1 for adding a framework for plugin to agent communication via rpc.
>
> Regarding your list of shortcomings, the security of the database connection would be easily remedied, but the other arguments are compelling by themselves.
>
> I'm less clear on the need to implement rpc communication between the vif driver and quantum plugin, at least as part of this blueprint. Am I missing a use-case that is not already handled by Quantum's rest interface?
The purpose was for the plugin to push the updates to the quantum agent
instead of the agent polling the database. Sumit raised the idea of a
broadcast. I am happy to start with this and then optimize later.
What are your thoughts?
> Thanks,
>
>
> Maru
>
>
> On 2012-05-09, at 4:26 AM, Gary Kotton wrote:
>
>> Hi,
>> I have added a very high level description on how to address the issue. This can be seen at: https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit
>> Comments will be greatly appreciated.
>> Questions:
>> 1. Do we want agents to be backward compatible (that is, still maintain the polling code)
>> 2. The generation of the Agent ID
>> 3. Any other ideas or thoughts about the matter?
>> I'd like to go ahead with a POC and implement this.
>> Thanks
>> Gary
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


rkukura at redhat

May 10, 2012, 5:23 AM

Post #12 of 22 (1047 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/09/2012 11:23 PM, Maru Newby wrote:
> Hi Bob,
>
> On 2012-05-09, at 6:26 PM, Robert Kukura wrote:
>>>
>>> I'm less clear on the need to implement rpc communication between the
>>> vif driver and quantum plugin, at least as part of this blueprint.
>>> Am I missing a use-case that is not already handled by Quantum's
>>> rest interface?
>>
>> At least one of the reasons for this is so that connection of a newly
>> created tap device to the virtual network by the agent would be
>> triggered immediately by the RPC, rather than waiting for agent's
>> periodic poll of the node's local tap devices to discover it.
>>
>
> I'm afraid I don't see your logic. The path is:
>
> Virt driver -> Quantum -> Plugin -> Agent
>
> The blueprint in question proposes that communication between plugin and
> agent be conducted via rpc rather than the current shared database. The
> proposal ensures that there is no polling involved. I see no reason for
> communication between the virt driver and quantum be conducted over rpc
> when a rest interface can do the same job with no extra effort or
> complexity.
>
> Thanks,
>
>
> Maru
>

Hi Maru,

I agree a REST interface could do the job. I misinterpreted your email
as arguing that the VIF driver should just create the tap device and the
agent should poll to discover the tap device, as is done now.

I'm not sure there would be much difference in code complexity using RPC
vs REST, except maybe that, assuming the VIF driver remains in nova, it
would need to use nova's copy of the openstack-common RPC rather than
quantum's copy, and these would need to remain compatible. RPC might
have the advantage of offering async messaging, but I'm not sure that's
needed here.

But it would seem reasonable for the VIF driver in nova to use the same
mechanism that nova itself will be using to talk with quantum (replacing
QuantumManager), which I'd guess is REST.

I think the general pattern has been that RPC interfaces should be
internal to either the nova or quantum components, and REST interfaces
should be using for communications between different OpenStack
components. In that case, it boils down to whether the VIF drivers are
considered part of nova, where REST should be used, or part of quantum,
where either REST or RPC could be used.

-Bob

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


gkotton at redhat

May 10, 2012, 8:22 AM

Post #13 of 22 (1043 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi,
Below is a table that lists a number of options, a short description,
their advantages and disadvantages. Hopefully this can give an idea of
the scope and complexity.

*Option*
*Description*
*Advantages*
*Disadvantages*
1 .Agent driving data retrieval from plugin
The agent maintains a list of tap devices. If there is a new tap device
then the agent will request the network information for this tap device
from the quantum plugin. In the case of the open source ovs and lb
(linuxbridge) plugins this is tap + 11 letters of the attachment id.
The agent will send an RCP update about the delta to the plugin. The
plugin will answer accordingly. For example if one or more tap devices
are detected then these are sent to the plugin. For each new tap device
the plugin will sent the network information (tags etc) and set the
database attachment as up. For deletion they will be removed (or set as
down).
Simple
Self contained in Quantum
If there is more than 1 attachment ID with the same prefix of 11
characters then this will not work (this currently is a bug)
The agent will still have to poll the network interfaces.
2 .VIF driver driving retrieval from plugin
The VIF driver updates the plugin about a change, which inturn updates
the relevant agent (this was described in the link
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
Event driven.
No polling
VIF driver and agents will need to share communication channels
3. Plugin broadcasting
When the plugin receives a change it broadcasts the change to all of
the registered agents
Relatively simple
Lots of unnecessary messages to agents that do not need to deal with
the traffic


I think that option #1 is a good start. This can later be optimized to
option #2.

Thanks
Gary

On 05/10/2012 10:05 AM, Gary Kotton wrote:
> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>> Hi Gary,
>>
>> Thanks for initiating this. A couple of comments/questions -
>>
>> 1. Do we really need the VIF driver to communicate the agent's identity;
>> I am referring to the agent ID being sent by the VIF driver in the
>> message? In general, I am not sure if there is a need to have the VIF
>> driver send messages/notifications in the first place, but I perhaps
>> it's being included as a capability in the framework?
> At the moment the open source plugins are not aware of the agents. The
> agents poll the data base for updates. The agent ID enables a agent to
> regsiter with the plugin, this in trrun enables the plugin to send a
> update to the specific agent. The update is initiated by the VIF
> driver. In my opinion this does the following:
> 1. updates the agents as soon as possible regarding a network change
> 2. limits traffic on the network
> 3. removes the database interface from the agents
>> 2. One model I was thinking of (which is kind of inline with the
>> existing agent implementations), is where the agents are smart, and they
>> know what to do in response to changes in the state of the logical
>> Quantum resources. In such cases, the Quantum plugin need not have to
>> keep track of sending a message to a particular agent. Instead, can we
>> have broadcast messages from the plugin to all the agents? If the plugin
>> has to unicast messages to specific agents, then it needs to maintain a
>> lot more state/topology information which should not be mandated for
>> this sole reason.
> I too thought about this option. In a sense the above proposal is an
> optimization of what you mention. This comes at the cost of
> complexity. The broadcast option is nice when the number of agents is
> small. When this is large, then for each network update there will be
> NUMBER_OF_AGENT messages sent for each update. The advantage of what
> you mention is that the code is self contained in Quantum.
>
> It may be better to start with the broadcast and then deal with the
> optimizations afterwards.
>
> Thanks
> Gary
>
>> Thanks,
>> ~Sumit.
>>
>>> -----Original Message-----
>>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>>> Behalf Of Gary Kotton
>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>> To:<netstack [at] lists>
>>> Subject: [Netstack] Scalable
>>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>> comms)
>>>
>>> Hi,
>>> I have added a very high level description on how to address the
>> issue.
>>> This can be seen at:
>>>
>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>> ZKgOlpvg/edit
>>> Comments will be greatly appreciated.
>>> Questions:
>>> 1. Do we want agents to be backward compatible (that is, still
>> maintain
>>> the polling code)
>>> 2. The generation of the Agent ID
>>> 3. Any other ideas or thoughts about the matter?
>>> I'd like to go ahead with a POC and implement this.
>>> Thanks
>>> Gary
>>>
>>> --
>>> Mailing list: https://launchpad.net/~netstack
>>> Post to : netstack [at] lists
>>> Unsubscribe : https://launchpad.net/~netstack
>>> More help : https://help.launchpad.net/ListHelp
>
>


dan at nicira

May 10, 2012, 9:42 AM

Post #14 of 22 (1040 views)
Permalink
Re: Scalable Agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On Thu, May 10, 2012 at 5:23 AM, Robert Kukura <rkukura [at] redhat> wrote:
>
>
> Hi Maru,
>
> I agree a REST interface could do the job. I misinterpreted your email
> as arguing that the VIF driver should just create the tap device and the
> agent should poll to discover the tap device, as is done now.


> I'm not sure there would be much difference in code complexity using RPC
> vs REST, except maybe that, assuming the VIF driver remains in nova, it
> would need to use nova's copy of the openstack-common RPC rather than
> quantum's copy, and these would need to remain compatible. RPC might
> have the advantage of offering async messaging, but I'm not sure that's
> needed here.


> But it would seem reasonable for the VIF driver in nova to use the same
> mechanism that nova itself will be using to talk with quantum (replacing
> QuantumManager), which I'd guess is REST.
>
> I think the general pattern has been that RPC interfaces should be
> internal to either the nova or quantum components, and REST interfaces
> should be using for communications between different OpenStack
> components. In that case, it boils down to whether the VIF drivers are
> considered part of nova, where REST should be used, or part of quantum,
> where either REST or RPC could be used.
>

Vif drivers are most certainly part of nova. They are code run by the
nova-compute process.

Definitely agree that communication between nova + quantum should all be
via rest.

Its worth noting that Nova already calls Quantum via REST on instance
creation to create a new port. This call already passes on information
about what host Nova is creating the port on. It might be nice to extend
this to include the exact vswitch port id (if available), further
optimizing the process, but the big win would be notifying the agent about
a new port + attachment, which would eliminate the need for that agent to
poll the central DB to learn about new ports.

In the re-write of the Nova/Quantum integration, this call will be moving
from nova-network (which will no longer exist if Quantum is in use) and
instead be performed directly by nova-compute. With that in mind, it
probably makes sense to have this code be part of the generic communication
between Nova + Quantum, rather than viewing it as part of each vif-driver
(which is specific to the exact type of edge switching technology in use).

Dan




>
> -Bob
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp
>



--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


dan at nicira

May 10, 2012, 9:46 AM

Post #15 of 22 (1035 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On Thu, May 10, 2012 at 12:05 AM, Gary Kotton <gkotton [at] redhat> wrote:

> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>
>> Hi Gary,
>>
>> Thanks for initiating this. A couple of comments/questions -
>>
>> 1. Do we really need the VIF driver to communicate the agent's identity;
>> I am referring to the agent ID being sent by the VIF driver in the
>> message? In general, I am not sure if there is a need to have the VIF
>> driver send messages/notifications in the first place, but I perhaps
>> it's being included as a capability in the framework?
>>
> At the moment the open source plugins are not aware of the agents. The
> agents poll the data base for updates. The agent ID enables a agent to
> regsiter with the plugin, this in trrun enables the plugin to send a update
> to the specific agent. The update is initiated by the VIF driver. In my
> opinion this does the following:
> 1. updates the agents as soon as possible regarding a network change
> 2. limits traffic on the network
> 3. removes the database interface from the agents


One important thing to note is that as we move toward achieving nova parity
(which includes security groups, dhcp, port security, L3, NAT) the amount
and type of data shared between the central node and the agents will take
on many different forms. Many of the existing implementations that we'll
be porting over from Nova rely on a central data store, and my feeling is
that the fastest and least bug prone approach will be to have those agents
continue to fetch data from the database. The biggest win we get from RPC
is eliminating polling by notifying agents about changes via RPC, rather
than having them poll the database to recognize that something has change.
This is how Nova works as well, so I'd focus on using RPC in that fashion
first.

Dan



>
> 2. One model I was thinking of (which is kind of inline with the
>> existing agent implementations), is where the agents are smart, and they
>> know what to do in response to changes in the state of the logical
>> Quantum resources. In such cases, the Quantum plugin need not have to
>> keep track of sending a message to a particular agent. Instead, can we
>> have broadcast messages from the plugin to all the agents? If the plugin
>> has to unicast messages to specific agents, then it needs to maintain a
>> lot more state/topology information which should not be mandated for
>> this sole reason.
>>
> I too thought about this option. In a sense the above proposal is an
> optimization of what you mention. This comes at the cost of complexity. The
> broadcast option is nice when the number of agents is small. When this is
> large, then for each network update there will be NUMBER_OF_AGENT messages
> sent for each update. The advantage of what you mention is that the code is
> self contained in Quantum.
>
> It may be better to start with the broadcast and then deal with the
> optimizations afterwards.
>
> Thanks
> Gary
>
>
> Thanks,
>> ~Sumit.
>>
>> -----Original Message-----
>>> From: netstack-bounces+snaiksat=cisc**o.com [at] lists<cisco.com [at] lists>
>>> [mailto:netstack-bounces+**snaiksat <netstack-bounces%2Bsnaiksat>=
>>> cisco.com [at] lists**launchpad.net <cisco.com [at] lists>] On
>>> Behalf Of Gary Kotton
>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>> To:<netstack [at] lists**net <netstack [at] lists>>
>>> Subject: [Netstack] Scalable
>>> Agents(https://blueprints.**launchpad.net/quantum/+spec/**
>>> scalable-agent-<https://blueprints.launchpad.net/quantum/+spec/scalable-agent->
>>> comms)
>>>
>>> Hi,
>>> I have added a very high level description on how to address the
>>>
>> issue.
>>
>>> This can be seen at:
>>>
>>> https://docs.google.com/**document/d/**1MbcBA2Os4b98ybdgAw2qe_**
>> 68R1NG6KMh8zd<https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd>
>>
>>> ZKgOlpvg/edit
>>> Comments will be greatly appreciated.
>>> Questions:
>>> 1. Do we want agents to be backward compatible (that is, still
>>>
>> maintain
>>
>>> the polling code)
>>> 2. The generation of the Agent ID
>>> 3. Any other ideas or thoughts about the matter?
>>> I'd like to go ahead with a POC and implement this.
>>> Thanks
>>> Gary
>>>
>>> --
>>> Mailing list: https://launchpad.net/~**netstack<https://launchpad.net/~netstack>
>>> Post to : netstack [at] lists
>>> Unsubscribe : https://launchpad.net/~**netstack<https://launchpad.net/~netstack>
>>> More help : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>>>
>>
>
> --
> Mailing list: https://launchpad.net/~**netstack<https://launchpad.net/~netstack>
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~**netstack<https://launchpad.net/~netstack>
> More help : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>



--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


mnewby at internap

May 10, 2012, 10:38 AM

Post #16 of 22 (1039 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Gary,

I appreciate the effort you've put into condensing the options.

I agree with your suggestion that option 1 is a good starting point. How will the agent discover changes to tap devices? Can an agent register for events from linux/kvm or xen, or would the agent just poll? For all I know agents may do this already, so I apologize if this is a silly question.

Regarding option 2, I still see no reason to have the vif driver talk to the agent directly. Ensuring a single point of contact between quantum clients (of which the vif driver is one) and quantum, namely the rest interface, limits complexity and will be easier to maintain and test. If and when performance or other concerns require direct vif driver to agent communication, we can go down that road, but as of now it's answering a question that hasn't been asked. YAGNI.

I would also argue that even RPC communication between the plugin and agent is gold-plating. The problem at hand is that database polling doesn't scale well. The simple answer is for the plugin and agent to communicate directly rather than through a database intermediary. Adding RPC to the mix is an implementation detail, pure and simple, and is not cost-free. RPC introduces queue dependency that can be problematic to debug and as we've seen in nova can cause performance issues all its own.

I'm all for leaving us open going forward to introduce an RPC dependency, but I think the most important thing is to create a clean communication interface between plugin and agent. The initial implementation can be something simple (and relatively dependency free) like secured http. The semantics for implementing and debugging http communication are well-known to all of us. If and when RPC becomes necessary, it will be straightforward to plug in a new transport driver.

Let's keep it simple - distributed computing is complicated enough!

Cheers,


Maru

On 2012-05-10, at 8:22 AM, Gary Kotton wrote:

> Hi,
> Below is a table that lists a number of options, a short description, their advantages and disadvantages. Hopefully this can give an idea of the scope and complexity.
>
> Option
> Description
> Advantages
> Disadvantages
> 1 .Agent driving data retrieval from plugin
> The agent maintains a list of tap devices. If there is a new tap device then the agent will request the network information for this tap device from the quantum plugin. In the case of the open source ovs and lb (linuxbridge) plugins this is tap + 11 letters of the attachment id.
> The agent will send an RCP update about the delta to the plugin. The plugin will answer accordingly. For example if one or more tap devices are detected then these are sent to the plugin. For each new tap device the plugin will sent the network information (tags etc) and set the database attachment as up. For deletion they will be removed (or set as down).
> Simple
> Self contained in Quantum
> If there is more than 1 attachment ID with the same prefix of 11 characters then this will not work (this currently is a bug)
> The agent will still have to poll the network interfaces.
> 2 .VIF driver driving retrieval from plugin
> The VIF driver updates the plugin about a change, which inturn updates the relevant agent (this was described in the link https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
> Event driven.
> No polling
> VIF driver and agents will need to share communication channels
> 3. Plugin broadcasting
> When the plugin receives a change it broadcasts the change to all of the registered agents
> Relatively simple
> Lots of unnecessary messages to agents that do not need to deal with the traffic
>
> I think that option #1 is a good start. This can later be optimized to option #2.
>
> Thanks
> Gary
>
> On 05/10/2012 10:05 AM, Gary Kotton wrote:
>>
>> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>>> Hi Gary,
>>>
>>> Thanks for initiating this. A couple of comments/questions -
>>>
>>> 1. Do we really need the VIF driver to communicate the agent's identity;
>>> I am referring to the agent ID being sent by the VIF driver in the
>>> message? In general, I am not sure if there is a need to have the VIF
>>> driver send messages/notifications in the first place, but I perhaps
>>> it's being included as a capability in the framework?
>> At the moment the open source plugins are not aware of the agents. The agents poll the data base for updates. The agent ID enables a agent to regsiter with the plugin, this in trrun enables the plugin to send a update to the specific agent. The update is initiated by the VIF driver. In my opinion this does the following:
>> 1. updates the agents as soon as possible regarding a network change
>> 2. limits traffic on the network
>> 3. removes the database interface from the agents
>>> 2. One model I was thinking of (which is kind of inline with the
>>> existing agent implementations), is where the agents are smart, and they
>>> know what to do in response to changes in the state of the logical
>>> Quantum resources. In such cases, the Quantum plugin need not have to
>>> keep track of sending a message to a particular agent. Instead, can we
>>> have broadcast messages from the plugin to all the agents? If the plugin
>>> has to unicast messages to specific agents, then it needs to maintain a
>>> lot more state/topology information which should not be mandated for
>>> this sole reason.
>> I too thought about this option. In a sense the above proposal is an optimization of what you mention. This comes at the cost of complexity. The broadcast option is nice when the number of agents is small. When this is large, then for each network update there will be NUMBER_OF_AGENT messages sent for each update. The advantage of what you mention is that the code is self contained in Quantum.
>>
>> It may be better to start with the broadcast and then deal with the optimizations afterwards.
>>
>> Thanks
>> Gary
>>
>>> Thanks,
>>> ~Sumit.
>>>
>>>> -----Original Message-----
>>>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>>>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>>>> Behalf Of Gary Kotton
>>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>>> To:<netstack [at] lists>
>>>> Subject: [Netstack] Scalable
>>>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>>> comms)
>>>>
>>>> Hi,
>>>> I have added a very high level description on how to address the
>>> issue.
>>>> This can be seen at:
>>>>
>>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>>> ZKgOlpvg/edit
>>>> Comments will be greatly appreciated.
>>>> Questions:
>>>> 1. Do we want agents to be backward compatible (that is, still
>>> maintain
>>>> the polling code)
>>>> 2. The generation of the Agent ID
>>>> 3. Any other ideas or thoughts about the matter?
>>>> I'd like to go ahead with a POC and implement this.
>>>> Thanks
>>>> Gary
>>>>
>>>> --
>>>> Mailing list: https://launchpad.net/~netstack
>>>> Post to : netstack [at] lists
>>>> Unsubscribe : https://launchpad.net/~netstack
>>>> More help : https://help.launchpad.net/ListHelp
>>
>>
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp


rbryant at redhat

May 10, 2012, 1:06 PM

Post #17 of 22 (1038 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

On 05/10/2012 12:46 PM, Dan Wendlandt wrote:
>
>
> On Thu, May 10, 2012 at 12:05 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>
> Hi Gary,
>
> Thanks for initiating this. A couple of comments/questions -
>
> 1. Do we really need the VIF driver to communicate the agent's
> identity;
> I am referring to the agent ID being sent by the VIF driver in the
> message? In general, I am not sure if there is a need to have
> the VIF
> driver send messages/notifications in the first place, but I perhaps
> it's being included as a capability in the framework?
>
> At the moment the open source plugins are not aware of the agents.
> The agents poll the data base for updates. The agent ID enables a
> agent to regsiter with the plugin, this in trrun enables the plugin
> to send a update to the specific agent. The update is initiated by
> the VIF driver. In my opinion this does the following:
> 1. updates the agents as soon as possible regarding a network change
> 2. limits traffic on the network
> 3. removes the database interface from the agents
>
>
> One important thing to note is that as we move toward achieving nova
> parity (which includes security groups, dhcp, port security, L3, NAT)
> the amount and type of data shared between the central node and the
> agents will take on many different forms. Many of the
> existing implementations that we'll be porting over from Nova rely on a
> central data store, and my feeling is that the fastest and least bug
> prone approach will be to have those agents continue to fetch data from
> the database. The biggest win we get from RPC is eliminating polling by
> notifying agents about changes via RPC, rather than having them poll the
> database to recognize that something has change. This is how Nova works
> as well, so I'd focus on using RPC in that fashion first.

On a related note, one of the goals we have for nova in Folsom is to cut
down much of this database access in favor of passing objects through rpc.

https://blueprints.launchpad.net/nova/+spec/no-db-messaging

There are a few different motivators for this. One is efficiency in
that it cuts down on how much the database gets hit. Another is
security. By removing direct database access from compute nodes, we can
more easily reduce the potential impact if a compute node were to get
compromised.

--
Russell Bryant

--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


darragh.oreilly at yahoo

May 10, 2012, 2:06 PM

Post #18 of 22 (1037 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

maybe udev events rules/actions could be installed for add/remove tap device events
http://www.reactivated.net/writing_udev_rules.html#external-run




>________________________________
> From: Maru Newby <mnewby [at] internap>
>To: gkotton [at] redhat
>Cc: Christopher Wright <chrisw [at] redhat>; netstack [at] lists
>Sent: Thursday, 10 May 2012, 18:38
>Subject: Re: [Netstack] Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
>
>
>Hi Gary,
>
>
>I appreciate the effort you've put into condensing the options.
>
>
>I agree with your suggestion that option 1 is a good starting point.  How will the agent discover changes to tap devices?  Can an agent register for events from linux/kvm or xen, or would the agent just poll?  For all I know agents may do this already, so I apologize if this is a silly question.
>
>
>Regarding option 2, I still see no reason to have the vif driver talk to the agent directly.  Ensuring a single point of contact between quantum clients (of which the vif driver is one) and quantum, namely the rest interface, limits complexity and will be easier to maintain and test.  If and when performance or other concerns require direct vif driver to agent communication, we can go down that road, but as of now it's answering a question that hasn't been asked.  YAGNI.
>
>
>I would also argue that even RPC communication between the plugin and agent is gold-plating.  The problem at hand is that database polling doesn't scale well.  The simple answer is for the plugin and agent to communicate directly rather than through a database intermediary.  Adding RPC to the mix is an implementation detail, pure and simple, and is not cost-free.  RPC introduces queue dependency that can be problematic to debug and as we've seen in nova can cause performance issues all its own.
>
>
>I'm all for leaving us open going forward to introduce an RPC dependency, but I think the most important thing is to create a clean communication interface between plugin and agent.  The initial implementation can be something simple (and relatively dependency free) like secured http.  The semantics for implementing and debugging http communication are well-known to all of us.  If and when RPC becomes necessary, it will be straightforward to plug in a new transport driver.
>
>
>Let's keep it simple - distributed computing is complicated enough!
>
>
>Cheers,
>
>
>
>
>Maru
>
>
>On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
>
>Hi,
>>Below is a table that lists a number of options, a short
description, their advantages and disadvantages. Hopefully this can
give an idea of the scope and complexity.
>>
>>
>>Option
>> Description
>> Advantages
>> Disadvantages
>>
>>1 .Agent driving data retrieval from plugin
>> The agent maintains a list of tap devices. If there is a new tap device then the agent will request the network information for this tap device from the quantum plugin. In the case of the open source ovs and lb (linuxbridge) plugins this is tap + 11 letters of the attachment id.
>>The agent will send an RCP update about the delta to the
plugin. The plugin will answer accordingly. For example if
one or more tap devices are detected then these are sent to
the plugin. For each new tap device the plugin will sent the
network information (tags etc) and set the database
attachment as up. For deletion they will be removed (or set
as down).
>> Simple
>>Self contained in Quantum
>> If there is more than 1 attachment ID with the same prefix of 11 characters then this will not work (this currently is a bug)
>>The agent will still have to poll the network interfaces.
>>
>>2 .VIF driver driving retrieval from plugin
>> The VIF driver updates the plugin about a change, which inturn updates the relevant agent (this was described in the link https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
>> Event driven.
>>No polling
>> VIF driver and agents will need to share communication channels
>>
>>3. Plugin broadcasting
>> When the plugin receives a change it broadcasts the change to all of the registered agents
>> Relatively simple
>> Lots of unnecessary messages to agents that do not need to deal with the traffic
>>
>>I think that option #1 is a good start. This can later be optimized
to option #2.
>>
>>Thanks
>>Gary
>>
>>On 05/10/2012 10:05 AM, Gary Kotton wrote:
>>On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>>>
>>>Hi Gary,
>>>>
>>>>Thanks for initiating this. A couple of comments/questions -
>>>>
>>>>1. Do we really need the VIF driver to communicate the agent's
identity;
>>>>I am referring to the agent ID being sent by the VIF driver in
the
>>>>message? In general, I am not sure if there is a need to have
the VIF
>>>>driver send messages/notifications in the first place, but I
perhaps
>>>>it's being included as a capability in the framework?
>>>>
At the moment the open source plugins are not aware of the agents. The agents poll the data base for updates. The agent ID enables a agent to regsiter with the plugin, this in trrun enables the plugin to send a update to the specific agent. The update is initiated by the VIF driver. In my opinion this does the following:
>>>1. updates the agents as soon as possible regarding a network
change
>>>2. limits traffic on the network
>>>3. removes the database interface from the agents
>>>
>>>2. One model I was thinking of (which is kind of inline with the
>>>>existing agent implementations), is where the agents are smart,
and they
>>>>know what to do in response to changes in the state of the
logical
>>>>Quantum resources. In such cases, the Quantum plugin need not
have to
>>>>keep track of sending a message to a particular agent. Instead,
can we
>>>>have broadcast messages from the plugin to all the agents? If
the plugin
>>>>has to unicast messages to specific agents, then it needs to
maintain a
>>>>lot more state/topology information which should not be mandated
for
>>>>this sole reason.
>>>>
I too thought about this option. In a sense the above proposal is an optimization of what you mention. This comes at the cost of complexity. The broadcast option is nice when the number of agents is small. When this is large, then for each network update there will be NUMBER_OF_AGENT messages sent for each update. The advantage of what you mention is that the code is self contained in Quantum.
>>>
>>>It may be better to start with the broadcast and then deal with
the optimizations afterwards.
>>>
>>>Thanks
>>>Gary
>>>
>>>
>>>Thanks,
>>>>~Sumit.
>>>>
>>>>
>>>>-----Original Message-----
>>>>>From: netstack-bounces+snaiksat=cisco.com [at] lists
>>>>>[mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>>>>>Behalf Of Gary Kotton
>>>>>Sent: Wednesday, May 09, 2012 4:27 AM
>>>>>To:<netstack [at] lists>
>>>>>Subject: [Netstack] Scalable
>>>>>Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>>>>comms)
>>>>>
>>>>>Hi,
>>>>>I have added a very high level description on how to address
the
>>>>>
issue.
>>>>
>>>>This can be seen at:
>>>>>
>>>>>https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>>>
>>>>ZKgOlpvg/edit
>>>>>Comments will be greatly appreciated.
>>>>>Questions:
>>>>>1. Do we want agents to be backward compatible (that is, still
>>>>>
maintain
>>>>
>>>>the polling code)
>>>>>2. The generation of the Agent ID
>>>>>3. Any other ideas or thoughts about the matter?
>>>>>I'd like to go ahead with a POC and implement this.
>>>>>Thanks
>>>>>Gary
>>>>>
>>>>>--
>>>>>Mailing list: https://launchpad.net/~netstack
>>>>>Post to     : netstack [at] lists
>>>>>Unsubscribe : https://launchpad.net/~netstack
>>>>>More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>
>>>
>>
--
>>Mailing list: https://launchpad.net/~netstack
>>Post to     : netstack [at] lists
>>Unsubscribe : https://launchpad.net/~netstack
>>More help   : https://help.launchpad.net/ListHelp
>>
>
>--
>Mailing list: https://launchpad.net/~netstack
>Post to    : netstack [at] lists
>Unsubscribe : https://launchpad.net/~netstack
>More help  : https://help.launchpad.net/ListHelp
>
>
>


mnewby at internap

May 10, 2012, 2:30 PM

Post #19 of 22 (1038 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Thanks Darragh! That should cover kvm. And apparently it's possible to be notified of vif changes from xen/xcp too, a more xen-savvy co-worker is tracking down details.

Gary, it sounds like it will be possible to have the agent notified directly of device changes. What are you thoughts as to modifying your proposal to take this into account?

Cheers,


Maru


On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:

>
> maybe udev events rules/actions could be installed for add/remove tap device events
> http://www.reactivated.net/writing_udev_rules.html#external-run
>
> From: Maru Newby <mnewby [at] internap>
> To: gkotton [at] redhat
> Cc: Christopher Wright <chrisw [at] redhat>; netstack [at] lists
> Sent: Thursday, 10 May 2012, 18:38
> Subject: Re: [Netstack] Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
>
> Hi Gary,
>
> I appreciate the effort you've put into condensing the options.
>
> I agree with your suggestion that option 1 is a good starting point. How will the agent discover changes to tap devices? Can an agent register for events from linux/kvm or xen, or would the agent just poll? For all I know agents may do this already, so I apologize if this is a silly question.
>
> Regarding option 2, I still see no reason to have the vif driver talk to the agent directly. Ensuring a single point of contact between quantum clients (of which the vif driver is one) and quantum, namely the rest interface, limits complexity and will be easier to maintain and test. If and when performance or other concerns require direct vif driver to agent communication, we can go down that road, but as of now it's answering a question that hasn't been asked. YAGNI.
>
> I would also argue that even RPC communication between the plugin and agent is gold-plating. The problem at hand is that database polling doesn't scale well. The simple answer is for the plugin and agent to communicate directly rather than through a database intermediary. Adding RPC to the mix is an implementation detail, pure and simple, and is not cost-free. RPC introduces queue dependency that can be problematic to debug and as we've seen in nova can cause performance issues all its own.
>
> I'm all for leaving us open going forward to introduce an RPC dependency, but I think the most important thing is to create a clean communication interface between plugin and agent. The initial implementation can be something simple (and relatively dependency free) like secured http. The semantics for implementing and debugging http communication are well-known to all of us. If and when RPC becomes necessary, it will be straightforward to plug in a new transport driver.
>
> Let's keep it simple - distributed computing is complicated enough!
>
> Cheers,
>
>
> Maru
>
> On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
>
>> Hi,
>> Below is a table that lists a number of options, a short description, their advantages and disadvantages. Hopefully this can give an idea of the scope and complexity.
>>
>> Option
>> Description
>> Advantages
>> Disadvantages
>> 1 .Agent driving data retrieval from plugin
>> The agent maintains a list of tap devices. If there is a new tap device then the agent will request the network information for this tap device from the quantum plugin. In the case of the open source ovs and lb (linuxbridge) plugins this is tap + 11 letters of the attachment id.
>> The agent will send an RCP update about the delta to the plugin. The plugin will answer accordingly. For example if one or more tap devices are detected then these are sent to the plugin. For each new tap device the plugin will sent the network information (tags etc) and set the database attachment as up. For deletion they will be removed (or set as down).
>> Simple
>> Self contained in Quantum
>> If there is more than 1 attachment ID with the same prefix of 11 characters then this will not work (this currently is a bug)
>> The agent will still have to poll the network interfaces.
>> 2 .VIF driver driving retrieval from plugin
>> The VIF driver updates the plugin about a change, which inturn updates the relevant agent (this was described in the link https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
>> Event driven.
>> No polling
>> VIF driver and agents will need to share communication channels
>> 3. Plugin broadcasting
>> When the plugin receives a change it broadcasts the change to all of the registered agents
>> Relatively simple
>> Lots of unnecessary messages to agents that do not need to deal with the traffic
>>
>> I think that option #1 is a good start. This can later be optimized to option #2.
>>
>> Thanks
>> Gary
>>
>> On 05/10/2012 10:05 AM, Gary Kotton wrote:
>>>
>>> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>>>> Hi Gary,
>>>>
>>>> Thanks for initiating this. A couple of comments/questions -
>>>>
>>>> 1. Do we really need the VIF driver to communicate the agent's identity;
>>>> I am referring to the agent ID being sent by the VIF driver in the
>>>> message? In general, I am not sure if there is a need to have the VIF
>>>> driver send messages/notifications in the first place, but I perhaps
>>>> it's being included as a capability in the framework?
>>> At the moment the open source plugins are not aware of the agents. The agents poll the data base for updates. The agent ID enables a agent to regsiter with the plugin, this in trrun enables the plugin to send a update to the specific agent. The update is initiated by the VIF driver. In my opinion this does the following:
>>> 1. updates the agents as soon as possible regarding a network change
>>> 2. limits traffic on the network
>>> 3. removes the database interface from the agents
>>>> 2. One model I was thinking of (which is kind of inline with the
>>>> existing agent implementations), is where the agents are smart, and they
>>>> know what to do in response to changes in the state of the logical
>>>> Quantum resources. In such cases, the Quantum plugin need not have to
>>>> keep track of sending a message to a particular agent. Instead, can we
>>>> have broadcast messages from the plugin to all the agents? If the plugin
>>>> has to unicast messages to specific agents, then it needs to maintain a
>>>> lot more state/topology information which should not be mandated for
>>>> this sole reason.
>>> I too thought about this option. In a sense the above proposal is an optimization of what you mention. This comes at the cost of complexity. The broadcast option is nice when the number of agents is small. When this is large, then for each network update there will be NUMBER_OF_AGENT messages sent for each update. The advantage of what you mention is that the code is self contained in Quantum.
>>>
>>> It may be better to start with the broadcast and then deal with the optimizations afterwards.
>>>
>>> Thanks
>>> Gary
>>>
>>>> Thanks,
>>>> ~Sumit.
>>>>
>>>>> -----Original Message-----
>>>>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>>>>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>>>>> Behalf Of Gary Kotton
>>>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>>>> To:<netstack [at] lists>
>>>>> Subject: [Netstack] Scalable
>>>>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>>>> comms)
>>>>>
>>>>> Hi,
>>>>> I have added a very high level description on how to address the
>>>> issue.
>>>>> This can be seen at:
>>>>>
>>>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>>>> ZKgOlpvg/edit
>>>>> Comments will be greatly appreciated.
>>>>> Questions:
>>>>> 1. Do we want agents to be backward compatible (that is, still
>>>> maintain
>>>>> the polling code)
>>>>> 2. The generation of the Agent ID
>>>>> 3. Any other ideas or thoughts about the matter?
>>>>> I'd like to go ahead with a POC and implement this.
>>>>> Thanks
>>>>> Gary
>>>>>
>>>>> --
>>>>> Mailing list: https://launchpad.net/~netstack
>>>>> Post to : netstack [at] lists
>>>>> Unsubscribe : https://launchpad.net/~netstack
>>>>> More help : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp
>
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp
>
>


mnewby at internap

May 10, 2012, 3:43 PM

Post #20 of 22 (1036 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Ah, for some reason I thought xen would need to be supported separately. It looks udev should work for both kvm and xen.

Thanks,


Maru

On 2012-05-10, at 2:30 PM, Maru Newby wrote:

> Thanks Darragh! That should cover kvm. And apparently it's possible to be notified of vif changes from xen/xcp too, a more xen-savvy co-worker is tracking down details.
>
> Gary, it sounds like it will be possible to have the agent notified directly of device changes. What are you thoughts as to modifying your proposal to take this into account?
>
> Cheers,
>
>
> Maru
>
>
> On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:
>
>>
>> maybe udev events rules/actions could be installed for add/remove tap device events
>> http://www.reactivated.net/writing_udev_rules.html#external-run
>>
>> From: Maru Newby <mnewby [at] internap>
>> To: gkotton [at] redhat
>> Cc: Christopher Wright <chrisw [at] redhat>; netstack [at] lists
>> Sent: Thursday, 10 May 2012, 18:38
>> Subject: Re: [Netstack] Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
>>
>> Hi Gary,
>>
>> I appreciate the effort you've put into condensing the options.
>>
>> I agree with your suggestion that option 1 is a good starting point. How will the agent discover changes to tap devices? Can an agent register for events from linux/kvm or xen, or would the agent just poll? For all I know agents may do this already, so I apologize if this is a silly question.
>>
>> Regarding option 2, I still see no reason to have the vif driver talk to the agent directly. Ensuring a single point of contact between quantum clients (of which the vif driver is one) and quantum, namely the rest interface, limits complexity and will be easier to maintain and test. If and when performance or other concerns require direct vif driver to agent communication, we can go down that road, but as of now it's answering a question that hasn't been asked. YAGNI.
>>
>> I would also argue that even RPC communication between the plugin and agent is gold-plating. The problem at hand is that database polling doesn't scale well. The simple answer is for the plugin and agent to communicate directly rather than through a database intermediary. Adding RPC to the mix is an implementation detail, pure and simple, and is not cost-free. RPC introduces queue dependency that can be problematic to debug and as we've seen in nova can cause performance issues all its own.
>>
>> I'm all for leaving us open going forward to introduce an RPC dependency, but I think the most important thing is to create a clean communication interface between plugin and agent. The initial implementation can be something simple (and relatively dependency free) like secured http. The semantics for implementing and debugging http communication are well-known to all of us. If and when RPC becomes necessary, it will be straightforward to plug in a new transport driver.
>>
>> Let's keep it simple - distributed computing is complicated enough!
>>
>> Cheers,
>>
>>
>> Maru
>>
>> On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
>>
>>> Hi,
>>> Below is a table that lists a number of options, a short description, their advantages and disadvantages. Hopefully this can give an idea of the scope and complexity.
>>>
>>> Option
>>> Description
>>> Advantages
>>> Disadvantages
>>> 1 .Agent driving data retrieval from plugin
>>> The agent maintains a list of tap devices. If there is a new tap device then the agent will request the network information for this tap device from the quantum plugin. In the case of the open source ovs and lb (linuxbridge) plugins this is tap + 11 letters of the attachment id.
>>> The agent will send an RCP update about the delta to the plugin. The plugin will answer accordingly. For example if one or more tap devices are detected then these are sent to the plugin. For each new tap device the plugin will sent the network information (tags etc) and set the database attachment as up. For deletion they will be removed (or set as down).
>>> Simple
>>> Self contained in Quantum
>>> If there is more than 1 attachment ID with the same prefix of 11 characters then this will not work (this currently is a bug)
>>> The agent will still have to poll the network interfaces.
>>> 2 .VIF driver driving retrieval from plugin
>>> The VIF driver updates the plugin about a change, which inturn updates the relevant agent (this was described in the link https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd )
>>> Event driven.
>>> No polling
>>> VIF driver and agents will need to share communication channels
>>> 3. Plugin broadcasting
>>> When the plugin receives a change it broadcasts the change to all of the registered agents
>>> Relatively simple
>>> Lots of unnecessary messages to agents that do not need to deal with the traffic
>>>
>>> I think that option #1 is a good start. This can later be optimized to option #2.
>>>
>>> Thanks
>>> Gary
>>>
>>> On 05/10/2012 10:05 AM, Gary Kotton wrote:
>>>>
>>>> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>>>>> Hi Gary,
>>>>>
>>>>> Thanks for initiating this. A couple of comments/questions -
>>>>>
>>>>> 1. Do we really need the VIF driver to communicate the agent's identity;
>>>>> I am referring to the agent ID being sent by the VIF driver in the
>>>>> message? In general, I am not sure if there is a need to have the VIF
>>>>> driver send messages/notifications in the first place, but I perhaps
>>>>> it's being included as a capability in the framework?
>>>> At the moment the open source plugins are not aware of the agents. The agents poll the data base for updates. The agent ID enables a agent to regsiter with the plugin, this in trrun enables the plugin to send a update to the specific agent. The update is initiated by the VIF driver. In my opinion this does the following:
>>>> 1. updates the agents as soon as possible regarding a network change
>>>> 2. limits traffic on the network
>>>> 3. removes the database interface from the agents
>>>>> 2. One model I was thinking of (which is kind of inline with the
>>>>> existing agent implementations), is where the agents are smart, and they
>>>>> know what to do in response to changes in the state of the logical
>>>>> Quantum resources. In such cases, the Quantum plugin need not have to
>>>>> keep track of sending a message to a particular agent. Instead, can we
>>>>> have broadcast messages from the plugin to all the agents? If the plugin
>>>>> has to unicast messages to specific agents, then it needs to maintain a
>>>>> lot more state/topology information which should not be mandated for
>>>>> this sole reason.
>>>> I too thought about this option. In a sense the above proposal is an optimization of what you mention. This comes at the cost of complexity. The broadcast option is nice when the number of agents is small. When this is large, then for each network update there will be NUMBER_OF_AGENT messages sent for each update. The advantage of what you mention is that the code is self contained in Quantum.
>>>>
>>>> It may be better to start with the broadcast and then deal with the optimizations afterwards.
>>>>
>>>> Thanks
>>>> Gary
>>>>
>>>>> Thanks,
>>>>> ~Sumit.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>>>>>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
>>>>>> Behalf Of Gary Kotton
>>>>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>>>>> To:<netstack [at] lists>
>>>>>> Subject: [Netstack] Scalable
>>>>>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>>>>> comms)
>>>>>>
>>>>>> Hi,
>>>>>> I have added a very high level description on how to address the
>>>>> issue.
>>>>>> This can be seen at:
>>>>>>
>>>>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>>>>> ZKgOlpvg/edit
>>>>>> Comments will be greatly appreciated.
>>>>>> Questions:
>>>>>> 1. Do we want agents to be backward compatible (that is, still
>>>>> maintain
>>>>>> the polling code)
>>>>>> 2. The generation of the Agent ID
>>>>>> 3. Any other ideas or thoughts about the matter?
>>>>>> I'd like to go ahead with a POC and implement this.
>>>>>> Thanks
>>>>>> Gary
>>>>>>
>>>>>> --
>>>>>> Mailing list: https://launchpad.net/~netstack
>>>>>> Post to : netstack [at] lists
>>>>>> Unsubscribe : https://launchpad.net/~netstack
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~netstack
>>> Post to : netstack [at] lists
>>> Unsubscribe : https://launchpad.net/~netstack
>>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>


gkotton at redhat

May 11, 2012, 1:28 AM

Post #21 of 22 (1039 views)
Permalink
Re: Scalable Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi,
Thanks for the input and comments. This is really great.

I would like to propose the following staged development (enable to
stabilize, test, and then optimize):
1. Stage 1 - have the agent detect a change, initially by polling. When
the agent detects and update then it will contact the plugin for a
detailed update about the network.
2. Stage 2 - Event driven support. One option is to have the operating
system notify the agent (as suggested by Darragh) another is to have the
VIF driver notify the agent. I am in favour of the latter. The VIF
driver is essentially creating the new tap device or deleting the
existing tap device. It seems logical that this would drive the update
on the agent.

What I would like to do is a quick POC of the above and then write a
detailed design of the flow so that we can all review. If it "compiles
and runs" on paper then it will speed up the development, testing and
deployment. It will also enable us to document for future reference.
This will also save time with review and the ping/pong with the -1's.

Have a good weekend and thanks for the inputs and comments. Hopefully
next week I'll have an update on the progress.
Thanks
Gary


On 05/11/2012 12:30 AM, Maru Newby wrote:
> Thanks Darragh! That should cover kvm. And apparently it's possible
> to be notified of vif changes from xen/xcp too, a more xen-savvy
> co-worker is tracking down details.
>
> Gary, it sounds like it will be possible to have the agent notified
> directly of device changes. What are you thoughts as to modifying
> your proposal to take this into account?
>
> Cheers,
>
>
> Maru
>
>
> On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:
>
>>
>> maybe udev events rules/actions could be installed for add/remove tap
>> device events
>> http://www.reactivated.net/writing_udev_rules.html#external-run
>>
>> ------------------------------------------------------------------------
>> *From:* Maru Newby <mnewby [at] internap <mailto:mnewby [at] internap>>
>> *To:* gkotton [at] redhat <mailto:gkotton [at] redhat>
>> *Cc:* Christopher Wright <chrisw [at] redhat
>> <mailto:chrisw [at] redhat>>; netstack [at] lists
>> <mailto:netstack [at] lists>
>> *Sent:* Thursday, 10 May 2012, 18:38
>> *Subject:* Re: [Netstack] Scalable
>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms)
>>
>> Hi Gary,
>>
>> I appreciate the effort you've put into condensing the options.
>>
>> I agree with your suggestion that option 1 is a good starting
>> point. How will the agent discover changes to tap devices? Can
>> an agent register for events from linux/kvm or xen, or would the
>> agent just poll? For all I know agents may do this already, so I
>> apologize if this is a silly question.
>>
>> Regarding option 2, I still see no reason to have the vif driver
>> talk to the agent directly. Ensuring a single point of contact
>> between quantum clients (of which the vif driver is one) and
>> quantum, namely the rest interface, limits complexity and will be
>> easier to maintain and test. If and when performance or other
>> concerns require direct vif driver to agent communication, we can
>> go down that road, but as of now it's answering a question that
>> hasn't been asked. YAGNI.
>>
>> I would also argue that even RPC communication between the plugin
>> and agent is gold-plating. The problem at hand is that database
>> polling doesn't scale well. The simple answer is for the plugin
>> and agent to communicate directly rather than through a database
>> intermediary. Adding RPC to the mix is an implementation detail,
>> pure and simple, and is not cost-free. RPC introduces queue
>> dependency that can be problematic to debug and as we've seen in
>> nova can cause performance issues all its own.
>>
>> I'm all for leaving us open going forward to introduce an RPC
>> dependency, but I think the most important thing is to create a
>> clean communication interface between plugin and agent. The
>> initial implementation can be something simple (and relatively
>> dependency free) like secured http. The semantics for
>> implementing and debugging http communication are well-known to
>> all of us. If and when RPC becomes necessary, it will be
>> straightforward to plug in a new transport driver.
>>
>> Let's keep it simple - distributed computing is complicated enough!
>>
>> Cheers,
>>
>>
>> Maru
>>
>> On 2012-05-10, at 8:22 AM, Gary Kotton wrote:
>>
>>> Hi,
>>> Below is a table that lists a number of options, a short
>>> description, their advantages and disadvantages. Hopefully this
>>> can give an idea of the scope and complexity.
>>>
>>> *Option*
>>> *Description*
>>> *Advantages*
>>> *Disadvantages*
>>> 1 .Agent driving data retrieval from plugin
>>> The agent maintains a list of tap devices. If there is a new
>>> tap device then the agent will request the network information
>>> for this tap device from the quantum plugin. In the case of the
>>> open source ovs and lb (linuxbridge) plugins this is tap + 11
>>> letters of the attachment id.
>>> The agent will send an RCP update about the delta to the plugin.
>>> The plugin will answer accordingly. For example if one or more
>>> tap devices are detected then these are sent to the plugin. For
>>> each new tap device the plugin will sent the network information
>>> (tags etc) and set the database attachment as up. For deletion
>>> they will be removed (or set as down).
>>> Simple
>>> Self contained in Quantum
>>> If there is more than 1 attachment ID with the same prefix of
>>> 11 characters then this will not work (this currently is a bug)
>>> The agent will still have to poll the network interfaces.
>>> 2 .VIF driver driving retrieval from plugin
>>> The VIF driver updates the plugin about a change, which inturn
>>> updates the relevant agent (this was described in the link
>>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>> )
>>> Event driven.
>>> No polling
>>> VIF driver and agents will need to share communication channels
>>> 3. Plugin broadcasting
>>> When the plugin receives a change it broadcasts the change to
>>> all of the registered agents
>>> Relatively simple
>>> Lots of unnecessary messages to agents that do not need to deal
>>> with the traffic
>>>
>>>
>>> I think that option #1 is a good start. This can later be
>>> optimized to option #2.
>>>
>>> Thanks
>>> Gary
>>>
>>> On 05/10/2012 10:05 AM, Gary Kotton wrote:
>>>> On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:
>>>>> Hi Gary,
>>>>>
>>>>> Thanks for initiating this. A couple of comments/questions -
>>>>>
>>>>> 1. Do we really need the VIF driver to communicate the agent's
>>>>> identity;
>>>>> I am referring to the agent ID being sent by the VIF driver in
>>>>> the
>>>>> message? In general, I am not sure if there is a need to have
>>>>> the VIF
>>>>> driver send messages/notifications in the first place, but I
>>>>> perhaps
>>>>> it's being included as a capability in the framework?
>>>> At the moment the open source plugins are not aware of the
>>>> agents. The agents poll the data base for updates. The agent ID
>>>> enables a agent to regsiter with the plugin, this in trrun
>>>> enables the plugin to send a update to the specific agent. The
>>>> update is initiated by the VIF driver. In my opinion this does
>>>> the following:
>>>> 1. updates the agents as soon as possible regarding a network
>>>> change
>>>> 2. limits traffic on the network
>>>> 3. removes the database interface from the agents
>>>>> 2. One model I was thinking of (which is kind of inline with the
>>>>> existing agent implementations), is where the agents are
>>>>> smart, and they
>>>>> know what to do in response to changes in the state of the
>>>>> logical
>>>>> Quantum resources. In such cases, the Quantum plugin need not
>>>>> have to
>>>>> keep track of sending a message to a particular agent.
>>>>> Instead, can we
>>>>> have broadcast messages from the plugin to all the agents? If
>>>>> the plugin
>>>>> has to unicast messages to specific agents, then it needs to
>>>>> maintain a
>>>>> lot more state/topology information which should not be
>>>>> mandated for
>>>>> this sole reason.
>>>> I too thought about this option. In a sense the above proposal
>>>> is an optimization of what you mention. This comes at the cost
>>>> of complexity. The broadcast option is nice when the number of
>>>> agents is small. When this is large, then for each network
>>>> update there will be NUMBER_OF_AGENT messages sent for each
>>>> update. The advantage of what you mention is that the code is
>>>> self contained in Quantum.
>>>>
>>>> It may be better to start with the broadcast and then deal with
>>>> the optimizations afterwards.
>>>>
>>>> Thanks
>>>> Gary
>>>>
>>>>> Thanks,
>>>>> ~Sumit.
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: netstack-bounces+snaiksat=cisco.com [at] lists
>>>>>> <mailto:netstack-bounces+snaiksat=cisco.com [at] lists>
>>>>>> [mailto:netstack-bounces+snaiksat=cisco.com [at] lists]
>>>>>> On
>>>>>> Behalf Of Gary Kotton
>>>>>> Sent: Wednesday, May 09, 2012 4:27 AM
>>>>>> To:<netstack [at] lists>
>>>>>> <mailto:netstack [at] lists>
>>>>>> Subject: [Netstack] Scalable
>>>>>> Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
>>>>>>
>>>>>> comms)
>>>>>>
>>>>>> Hi,
>>>>>> I have added a very high level description on how to address the
>>>>> issue.
>>>>>> This can be seen at:
>>>>>>
>>>>> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
>>>>>
>>>>>> ZKgOlpvg/edit
>>>>>> Comments will be greatly appreciated.
>>>>>> Questions:
>>>>>> 1. Do we want agents to be backward compatible (that is, still
>>>>> maintain
>>>>>> the polling code)
>>>>>> 2. The generation of the Agent ID
>>>>>> 3. Any other ideas or thoughts about the matter?
>>>>>> I'd like to go ahead with a POC and implement this.
>>>>>> Thanks
>>>>>> Gary
>>>>>>
>>>>>> --
>>>>>> Mailing list: https://launchpad.net/~netstack
>>>>>> <https://launchpad.net/%7Enetstack>
>>>>>> Post to : netstack [at] lists
>>>>>> <mailto:netstack [at] lists>
>>>>>> Unsubscribe : https://launchpad.net/~netstack
>>>>>> <https://launchpad.net/%7Enetstack>
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~netstack
>>> <https://launchpad.net/%7Enetstack>
>>> Post to : netstack [at] lists
>>> <mailto:netstack [at] lists>
>>> Unsubscribe : https://launchpad.net/~netstack
>>> <https://launchpad.net/%7Enetstack>
>>> More help : https://help.launchpad.net/ListHelp
>>
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> Post to : netstack [at] lists
>> <mailto:netstack [at] lists>
>> Unsubscribe : https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> More help : https://help.launchpad.net/ListHelp
>>
>>
>


snaiksat at cisco

May 12, 2012, 2:36 PM

Post #22 of 22 (987 views)
Permalink
Re: ScalableAgents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms) [In reply to]

Hi Gary, All,



Some comments inline.



Thanks,

~Sumit.



From: netstack-bounces+snaiksat=cisco.com [at] lists
[mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
Behalf Of Gary Kotton
Sent: Friday, May 11, 2012 1:29 AM
To: Maru Newby
Cc: Christopher Wright; netstack [at] lists
Subject: Re: [Netstack]
ScalableAgents(https://blueprints.launchpad.net/quantum/+spec/scalable-a
gent-comms)



Hi,
Thanks for the input and comments. This is really great.

I would like to propose the following staged development (enable to
stabilize, test, and then optimize):
1. Stage 1 - have the agent detect a change, initially by polling. When
the agent detects and update then it will contact the plugin for a
detailed update about the network.



<Sumit> Currently the agents work by first polling the Quantum DB to
detect changes in the network/port state, and then accordingly react
locally (to check for the presence of a tap device, etc.). As I
understand, you are proposing to switch the sequence of this logic,
i.e., the agent first detects a change in the local state (e.g., a new
tap device has been created) and then communicates with the Quantum
plugin to obtain more context for this change. If this is the thought, I
believe it is reasonable and will eliminate the overhead from polling
the DB. The premise here of course is that the agent is able to locally
detect all the changes that it needs to react to, and in the basic case
of the Linux Bridge plugin, I don't think there is anything beyond
creation of tap devices, so this should work. If there are other state
changes introduced by the Quantum plugin that the agent needs to react
to, then the agent would not know about these in the absence of a
notification mechanism.</Sumit>


2. Stage 2 - Event driven support. One option is to have the operating
system notify the agent (as suggested by Darragh) another is to have the
VIF driver notify the agent. I am in favour of the latter. The VIF
driver is essentially creating the new tap device or deleting the
existing tap device. It seems logical that this would drive the update
on the agent.



<Sumit> I would actually prefer the former approach. It's better to
decouple the components to the extent possible so as to be able to
update/reuse them independently. I doubt that we will get any
significant performance advantage out of making the VIF driver aware of
the agents and having them communicate with those explicitly. However,
it does introduce a stronger coupling and we should probably avoid
it.</Sumit>



What I would like to do is a quick POC of the above and then write a
detailed design of the flow so that we can all review. If it "compiles
and runs" on paper then it will speed up the development, testing and
deployment. It will also enable us to document for future reference.
This will also save time with review and the ping/pong with the -1's.



<Sumit> Great, thanks for doing this! </Sumit>



Have a good weekend and thanks for the inputs and comments. Hopefully
next week I'll have an update on the progress.
Thanks
Gary


On 05/11/2012 12:30 AM, Maru Newby wrote:

Thanks Darragh! That should cover kvm. And apparently it's possible to
be notified of vif changes from xen/xcp too, a more xen-savvy co-worker
is tracking down details.



Gary, it sounds like it will be possible to have the agent notified
directly of device changes. What are you thoughts as to modifying your
proposal to take this into account?



Cheers,





Maru





On 2012-05-10, at 2:06 PM, Darragh OReilly wrote:







maybe udev events rules/actions could be installed for add/remove tap
device events

http://www.reactivated.net/writing_udev_rules.html#external-run





________________________________

From: Maru Newby <mnewby [at] internap>
To: gkotton [at] redhat
Cc: Christopher Wright <chrisw [at] redhat>; netstack [at] lists

Sent: Thursday, 10 May 2012, 18:38
Subject: Re: [Netstack] Scalable
Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-com
ms)



Hi Gary,



I appreciate the effort you've put into condensing the options.



I agree with your suggestion that option 1 is a good starting point.
How will the agent discover changes to tap devices? Can an agent
register for events from linux/kvm or xen, or would the agent just poll?
For all I know agents may do this already, so I apologize if this is a
silly question.



Regarding option 2, I still see no reason to have the vif driver talk to
the agent directly. Ensuring a single point of contact between quantum
clients (of which the vif driver is one) and quantum, namely the rest
interface, limits complexity and will be easier to maintain and test.
If and when performance or other concerns require direct vif driver to
agent communication, we can go down that road, but as of now it's
answering a question that hasn't been asked. YAGNI.



I would also argue that even RPC communication between the plugin and
agent is gold-plating. The problem at hand is that database polling
doesn't scale well. The simple answer is for the plugin and agent to
communicate directly rather than through a database intermediary.
Adding RPC to the mix is an implementation detail, pure and simple, and
is not cost-free. RPC introduces queue dependency that can be
problematic to debug and as we've seen in nova can cause performance
issues all its own.



I'm all for leaving us open going forward to introduce an RPC
dependency, but I think the most important thing is to create a clean
communication interface between plugin and agent. The initial
implementation can be something simple (and relatively dependency free)
like secured http. The semantics for implementing and debugging http
communication are well-known to all of us. If and when RPC becomes
necessary, it will be straightforward to plug in a new transport driver.



Let's keep it simple - distributed computing is complicated enough!



Cheers,





Maru



On 2012-05-10, at 8:22 AM, Gary Kotton wrote:





Hi,
Below is a table that lists a number of options, a short description,
their advantages and disadvantages. Hopefully this can give an idea of
the scope and complexity.

Option

Description

Advantages

Disadvantages

1 .Agent driving data retrieval from plugin

The agent maintains a list of tap devices. If there is a new tap device
then the agent will request the network information for this tap device
from the quantum plugin. In the case of the open source ovs and lb
(linuxbridge) plugins this is tap + 11 letters of the attachment id.
The agent will send an RCP update about the delta to the plugin. The
plugin will answer accordingly. For example if one or more tap devices
are detected then these are sent to the plugin. For each new tap device
the plugin will sent the network information (tags etc) and set the
database attachment as up. For deletion they will be removed (or set as
down).

Simple
Self contained in Quantum

If there is more than 1 attachment ID with the same prefix of 11
characters then this will not work (this currently is a bug)
The agent will still have to poll the network interfaces.

2 .VIF driver driving retrieval from plugin

The VIF driver updates the plugin about a change, which inturn updates
the relevant agent (this was described in the link
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd
)

Event driven.
No polling

VIF driver and agents will need to share communication channels

3. Plugin broadcasting

When the plugin receives a change it broadcasts the change to all of the
registered agents

Relatively simple

Lots of unnecessary messages to agents that do not need to deal with the
traffic


I think that option #1 is a good start. This can later be optimized to
option #2.

Thanks
Gary

On 05/10/2012 10:05 AM, Gary Kotton wrote:

On 05/10/2012 12:55 AM, Sumit Naiksatam (snaiksat) wrote:



Hi Gary,

Thanks for initiating this. A couple of comments/questions -

1. Do we really need the VIF driver to communicate the agent's identity;

I am referring to the agent ID being sent by the VIF driver in the
message? In general, I am not sure if there is a need to have the VIF
driver send messages/notifications in the first place, but I perhaps
it's being included as a capability in the framework?

At the moment the open source plugins are not aware of the agents. The
agents poll the data base for updates. The agent ID enables a agent to
regsiter with the plugin, this in trrun enables the plugin to send a
update to the specific agent. The update is initiated by the VIF driver.
In my opinion this does the following:
1. updates the agents as soon as possible regarding a network change
2. limits traffic on the network
3. removes the database interface from the agents



2. One model I was thinking of (which is kind of inline with the
existing agent implementations), is where the agents are smart, and they

know what to do in response to changes in the state of the logical
Quantum resources. In such cases, the Quantum plugin need not have to
keep track of sending a message to a particular agent. Instead, can we
have broadcast messages from the plugin to all the agents? If the plugin

has to unicast messages to specific agents, then it needs to maintain a
lot more state/topology information which should not be mandated for
this sole reason.

I too thought about this option. In a sense the above proposal is an
optimization of what you mention. This comes at the cost of complexity.
The broadcast option is nice when the number of agents is small. When
this is large, then for each network update there will be
NUMBER_OF_AGENT messages sent for each update. The advantage of what you
mention is that the code is self contained in Quantum.

It may be better to start with the broadcast and then deal with the
optimizations afterwards.

Thanks
Gary




Thanks,
~Sumit.




-----Original Message-----
From: netstack-bounces+snaiksat=cisco.com [at] lists
[mailto:netstack-bounces+snaiksat=cisco.com [at] lists] On
Behalf Of Gary Kotton
Sent: Wednesday, May 09, 2012 4:27 AM
To:<netstack [at] lists> <mailto:netstack [at] lists>

Subject: [Netstack] Scalable
Agents(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-
comms)

Hi,
I have added a very high level description on how to address the

issue.



This can be seen at:

https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zd



ZKgOlpvg/edit
Comments will be greatly appreciated.
Questions:
1. Do we want agents to be backward compatible (that is, still

maintain



the polling code)
2. The generation of the Agent ID
3. Any other ideas or thoughts about the matter?
I'd like to go ahead with a POC and implement this.
Thanks
Gary

--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp





--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp




--
Mailing list: https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
<https://launchpad.net/%7Enetstack>
More help : https://help.launchpad.net/ListHelp

OpenStack netstack RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.