Mailing List Archive: OpenStack: Dev

[Quantum] Scalable agents

Index | Next | Previous | View Threaded

gkotton at redhat

Jul 16, 2012, 3:30 AM

Post #1 of 7 (227 views)
Permalink

[Quantum] Scalable agents

Hi,
The patch https://review.openstack.org/#/c/9591/ contains the initial
support for the scalable agents (this is currently implemented on the
linux bridge). At the moment this does not support a network or port
update, that is, the user can set 'admin_status_up' to 0. This means
that either the network or the port should stop handling traffic.
The network/port update is challenging in a number of respects. First
and foremost the quantum plugin is not aware of the agent on which the
port may have been allocated (this is where the VM has been deployed).
In addition to this there may be a number of agents running.
There are a number of options to perform the port update. They are
listed below:
1. Make use of the openstack-common notifier support. This would have
the plugin notify "all" of the agents. I have yet to look at the code
but guess that it is similar to the next item.
2. Make use of the RPC mechanism to have the plugin notify the agents.
At the moment the plugin has the topic of all of the agents (this is
used for a health check to ensure that the configuration on the agent is
in sync with that of the plugin). It is described in detail in
https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit?pli=1

If I understand correctly then both of the above would require that the
agents are also RPC consumers. In both of the above the when there is a
update to either a network or port then there will be a lot of traffic
broadcast on the network.

Another alternative is to piggy back onto the health check message. This
will contain the ID's of the networks/ports that were updated prior to
the last check. When an agent receives these, if they are using the the
network or port then they will request the details from the plugin. This
will certainly have less traffic on the network.

If anyone has any ideas then it would be great to hear them.
Hopefully we can discuss this in tonight's meeting.
Thanks
Gary

_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack [at] lists
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

dan at nicira

Jul 17, 2012, 6:23 PM

Post #2 of 7 (200 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On Mon, Jul 16, 2012 at 3:30 AM, Gary Kotton <gkotton [at] redhat> wrote:

> Hi,
> The patch https://review.openstack.org/#**/c/9591/<https://review.openstack.org/#/c/9591/>contains the initial support for the scalable agents (this is currently
> implemented on the linux bridge). At the moment this does not support a
> network or port update, that is, the user can set 'admin_status_up' to 0.
> This means that either the network or the port should stop handling traffic.
> The network/port update is challenging in a number of respects. First and
> foremost the quantum plugin is not aware of the agent on which the port may
> have been allocated (this is where the VM has been deployed). In addition
> to this there may be a number of agents running.
> There are a number of options to perform the port update. They are listed
> below:
> 1. Make use of the openstack-common notifier support. This would have the
> plugin notify "all" of the agents. I have yet to look at the code but guess
> that it is similar to the next item.
> 2. Make use of the RPC mechanism to have the plugin notify the agents. At
> the moment the plugin has the topic of all of the agents (this is used for
> a health check to ensure that the configuration on the agent is in sync
> with that of the plugin). It is described in detail in
> https://docs.google.com/**document/d/**1MbcBA2Os4b98ybdgAw2qe_**
> 68R1NG6KMh8zdZKgOlpvg/edit?**pli=1<https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit?pli=1>
>
> If I understand correctly then both of the above would require that the
> agents are also RPC consumers. In both of the above the when there is a
> update to either a network or port then there will be a lot of traffic
> broadcast on the network.
>

Hi Gary,

Yes, I think either way, to eliminate the polling, we need to have some
mechanism to inform the agents that they need to update state. My goal
would be to build a standard mechanism for this that to the degree possible
leverages existing APIs and data formats, so that we can avoid having
multiple formats for the same data and avoid any RPC-call sprawl.

I agree that we don't want to broadcast all data everyone. At the same
time, I'd like to avoid having to make the the core plugin code running
within quantum-server be aware of all of the different agents. What I
think would be idea is that we have a fine-grained notification mechanism
for when objects (networks, subnets, ports) are updated, and that agents
could choose to register for updates on particular objects. For example, a
DHCP agent handling all DHCP for a deployment might register for
create/update/delete operations on subnets + ports, whereas a plugin agent
might only register for updates from the ports that it sees locally on the
hypervisor. Conceptually, you could think of there being a 'topic' per
port in this case, though we may need to implement it differently in
practice.

In general, I think it is ideal if these external agents can use standard
mechanisms and formats as much as possible. For example, after learning
that port X was created, the DHCP agent can actually use a standard
webservice GET to learn about the configuration of the port (or if people
feel that such information should be included in the notification itself,
this notification data uses the same format as the webservice API).

So in sum, I'm hoping that we can take an approach to this problem that
build a base framework that will continue to work as we add more rich
functionality to quantum networks, recognizing that in most cases, agents
will need to follow the pattern of triggering off of changes to API
objects. I'm not sure whether this is inline with your thinking or not, so
I'd be curious to hear your thoughts. Thanks,

Dan

>
> Another alternative is to piggy back onto the health check message. This
> will contain the ID's of the networks/ports that were updated prior to the
> last check. When an agent receives these, if they are using the the network
> or port then they will request the details from the plugin. This will
> certainly have less traffic on the network.
>
> If anyone has any ideas then it would be great to hear them.
> Hopefully we can discuss this in tonight's meeting.
> Thanks
> Gary
>
>
> ______________________________**_________________
> Mailing list: https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
> Post to : openstack [at] lists
> Unsubscribe : https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
> More help : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

gkotton at redhat

Jul 18, 2012, 5:17 AM

Post #3 of 7 (191 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On 07/18/2012 04:23 AM, Dan Wendlandt wrote:
>
>
> On Mon, Jul 16, 2012 at 3:30 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
> Hi,
> The patch https://review.openstack.org/#/c/9591/ contains the
> initial support for the scalable agents (this is currently
> implemented on the linux bridge). At the moment this does not
> support a network or port update, that is, the user can set
> 'admin_status_up' to 0. This means that either the network or the
> port should stop handling traffic.
> The network/port update is challenging in a number of respects.
> First and foremost the quantum plugin is not aware of the agent on
> which the port may have been allocated (this is where the VM has
> been deployed). In addition to this there may be a number of
> agents running.
> There are a number of options to perform the port update. They are
> listed below:
> 1. Make use of the openstack-common notifier support. This would
> have the plugin notify "all" of the agents. I have yet to look at
> the code but guess that it is similar to the next item.
> 2. Make use of the RPC mechanism to have the plugin notify the
> agents. At the moment the plugin has the topic of all of the
> agents (this is used for a health check to ensure that the
> configuration on the agent is in sync with that of the plugin). It
> is described in detail in
> https://docs.google.com/document/d/1MbcBA2Os4b98ybdgAw2qe_68R1NG6KMh8zdZKgOlpvg/edit?pli=1
>
> If I understand correctly then both of the above would require
> that the agents are also RPC consumers. In both of the above the
> when there is a update to either a network or port then there will
> be a lot of traffic broadcast on the network.
>
>
> Hi Gary,
>
> Yes, I think either way, to eliminate the polling, we need to have
> some mechanism to inform the agents that they need to update state.
> My goal would be to build a standard mechanism for this that to the
> degree possible leverages existing APIs and data formats, so that we
> can avoid having multiple formats for the same data and avoid any
> RPC-call sprawl.

I agree with you wholeheartedly here. In my opinion this is what I have
started with the RPC inclusion and initial support. At the moment this
lacks the "update from the service" side (which essentially is what this
mail is about :))

>
> I agree that we don't want to broadcast all data everyone. At the
> same time, I'd like to avoid having to make the the core plugin code
> running within quantum-server be aware of all of the different agents.
> What I think would be idea is that we have a fine-grained
> notification mechanism for when objects (networks, subnets, ports) are
> updated, and that agents could choose to register for updates on
> particular objects.

This is along the sames lines that I was thinking. In the current
implementation the agent connects to the service to sync with the
configuration. I was thinking of having the agent publicize it what
information it would like to receive, for example:
- quantum agent - needs port and network updates (port creation and
deletion are treated in the current implementation)
- dhcp-agent - port creation, deletion and updates
- firewall agent - ...

When the service performs an operation and one of the agents supports
the operation type then that agent should be updated. These are not
"real time" opertaions and for the first phase we can use the broadcast
mechnism. I do think that we should optimize for very large scale
environments.

> For example, a DHCP agent handling all DHCP for a deployment might
> register for create/update/delete operations on subnets + ports,
> whereas a plugin agent might only register for updates from the ports
> that it sees locally on the hypervisor. Conceptually, you could think
> of there being a 'topic' per port in this case, though we may need to
> implement it differently in practice.

The agent ID is currently stored in the database (this is for the
configuration sync mechanism). I think that adding an extra column
indicating the capabilities enables the service to notify the agents.
The issue is how refined can the updates be - we want to ensure that we
have a scalable architecture.

>
> In general, I think it is ideal if these external agents can use
> standard mechanisms and formats as much as possible. For example,
> after learning that port X was created, the DHCP agent can actually
> use a standard webservice GET to learn about the configuration of the
> port (or if people feel that such information should be included in
> the notification itself, this notification data uses the same format
> as the webservice API).

I am not sure that I agree here. If the service is notifying the agent
then why not have the information being passed in the message (IP + mac
etc.) There is no need for the GET operation.

>
> So in sum, I'm hoping that we can take an approach to this problem
> that build a base framework that will continue to work as we add more
> rich functionality to quantum networks, recognizing that in most
> cases, agents will need to follow the pattern of triggering off of
> changes to API objects. I'm not sure whether this is inline with your
> thinking or not, so I'd be curious to hear your thoughts. Thanks,
>
> Dan
>
>
> Another alternative is to piggy back onto the health check
> message. This will contain the ID's of the networks/ports that
> were updated prior to the last check. When an agent receives
> these, if they are using the the network or port then they will
> request the details from the plugin. This will certainly have less
> traffic on the network.
>
> If anyone has any ideas then it would be great to hear them.
> Hopefully we can discuss this in tonight's meeting.
> Thanks
> Gary
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> Post to : openstack [at] lists
> <mailto:openstack [at] lists>
> Unsubscribe : https://launchpad.net/~openstack
> <https://launchpad.net/%7Eopenstack>
> More help : https://help.launchpad.net/ListHelp
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

dan at nicira

Jul 19, 2012, 9:11 AM

Post #4 of 7 (189 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On Wed, Jul 18, 2012 at 5:17 AM, Gary Kotton <gkotton [at] redhat> wrote:

> **
> On 07/18/2012 04:23 AM, Dan Wendlandt wrote:
>
>
>
Hi Gary,

Removing much of the thread history, as I think we agree on the high-level
goals. Now just focusing on the differences.

>
>
> For example, a DHCP agent handling all DHCP for a deployment might
> register for create/update/delete operations on subnets + ports, whereas a
> plugin agent might only register for updates from the ports that it sees
> locally on the hypervisor. Conceptually, you could think of there being a
> 'topic' per port in this case, though we may need to implement it
> differently in practice.
>
>
> The agent ID is currently stored in the database (this is for the
> configuration sync mechanism). I think that adding an extra column
> indicating the capabilities enables the service to notify the agents. The
> issue is how refined can the updates be - we want to ensure that we have a
> scalable architecture.
>

I think either we can implement the filtering ourselves using a mechanism
like this, or we can rely on the message bus to do it for us. I'm not
really familiar with the scalability of various message bus
implementations, but a simple model would be that there's a topic for:
- port creation
- net creation
- subnet creation

and a specific topic for each entity after its created to learn about
updates and deletes.

as I said, we may need to implement this logic ourselves is using many such
topics would not be scalable, but this seems like the kind of think a
message bus should be good at..

> In general, I think it is ideal if these external agents can use
> standard mechanisms and formats as much as possible. For example, after
> learning that port X was created, the DHCP agent can actually use a
> standard webservice GET to learn about the configuration of the port (or if
> people feel that such information should be included in the notification
> itself, this notification data uses the same format as the webservice
> API).
>
>
> I am not sure that I agree here. If the service is notifying the agent
> then why not have the information being passed in the message (IP + mac
> etc.) There is no need for the GET operation.
>

My general bias here is that if there are now two ways to fetch every type
of information (one via the standard "public" interface and another via the
"internal" interface with a different implementation) that is twice the
testing, updating, documenting that we have to do. Perhaps the two
problems we're trying to solve are sufficiently different that they require
two different mechanisms, but in my use cases I haven't seen that yet.

Dan

~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

gkotton at redhat

Jul 22, 2012, 5:51 AM

Post #5 of 7 (175 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On 07/19/2012 07:11 PM, Dan Wendlandt wrote:
>
>
> On Wed, Jul 18, 2012 at 5:17 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
> On 07/18/2012 04:23 AM, Dan Wendlandt wrote:
>>
>
> Hi Gary,
>
> Removing much of the thread history, as I think we agree on the
> high-level goals. Now just focusing on the differences.
>
>
>
>> For example, a DHCP agent handling all DHCP for a deployment
>> might register for create/update/delete operations on subnets +
>> ports, whereas a plugin agent might only register for updates
>> from the ports that it sees locally on the hypervisor.
>> Conceptually, you could think of there being a 'topic' per port
>> in this case, though we may need to implement it differently in
>> practice.
>
> The agent ID is currently stored in the database (this is for the
> configuration sync mechanism). I think that adding an extra column
> indicating the capabilities enables the service to notify the
> agents. The issue is how refined can the updates be - we want to
> ensure that we have a scalable architecture.
>
>
> I think either we can implement the filtering ourselves using a
> mechanism like this, or we can rely on the message bus to do it for
> us. I'm not really familiar with the scalability of various message
> bus implementations, but a simple model would be that there's a topic
> for:
> - port creation
> - net creation
> - subnet creation

This is an interesting idea. In addition to the creation we will also
need the update. I would prefer that the agents would have one topic -
that is for all updates. When an agent connects to the plugin it will
register the type of operations that are supported on the specific
agent. The agent operations can be specific as bit masks.

I have implemented something similar in
https://review.openstack.org/#/c/9591

This can certainly be improved and optimized. What are your thoughts?

In addition to this we have a number of issues where the plugin does not
expose the information via the standard API's - for example the VLAN tag
(this is being addressed via extensions in the provider networks feature)

There are a number of things that we need to address:
1. Support for different plugins - if acceptable then the model above
needs to be more generic and a common interface should be defined.
2. Support for different agents. This is pretty simple - for example the
DHCP agent. It has to do the following:
i. use the health check mechanism (this registers the mask for the
notification updates)
ii. add in support for port creation (I guess that I can add this
as part of this patch)
3. Logging. At the moment the agents do not have a decent logging
mechanism. This makes debugging the RPC code terribly difficult. This
was scheduled for F-3. I'll be happy to add this if there are no objections.
4. We need to discuss the notifications that Yong added and how these
two methods can interact together. More specifically I think that we
need to address the configuration files.

The RPC code requires that the eventlet monkey patch be set. This cause
havoc when I was using the events from pyudev for new device creation.
At the moment I have moved the event driven support to polling (if
anyone who reads this is familiar with the issue or has an idea on how
to address it any help will be great)

> and a specific topic for each entity after its created to learn about
> updates and deletes.

I prefer having a cast to a specific topic than a broadcast all. (please
look at
https://review.openstack.org/#/c/9591/3/quantum/plugins/linuxbridge/lb_quantum_plugin.py
- method update_port - line 174).

>
> as I said, we may need to implement this logic ourselves is using many
> such topics would not be scalable, but this seems like the kind of
> think a message bus should be good at..
>
>> In general, I think it is ideal if these external agents can use
>> standard mechanisms and formats as much as possible. For
>> example, after learning that port X was created, the DHCP agent
>> can actually use a standard webservice GET to learn about the
>> configuration of the port (or if people feel that such
>> information should be included in the notification itself, this
>> notification data uses the same format as the webservice API).
>
> I am not sure that I agree here. If the service is notifying the
> agent then why not have the information being passed in the
> message (IP + mac etc.) There is no need for the GET operation.
>
>
> My general bias here is that if there are now two ways to fetch every
> type of information (one via the standard "public" interface and
> another via the "internal" interface with a different implementation)
> that is twice the testing, updating, documenting that we have to do.
> Perhaps the two problems we're trying to solve are sufficiently
> different that they require two different mechanisms, but in my use
> cases I haven't seen that yet.

This is a tough one. On one hand I agree with you. On the other I think
that we should have a better tuned and optimized system. Yes, this may
require a bit more effort but I think that it is more robust. Another
issue is that each plugin has its own traits and characteristics.
Private additional data may have to be transferred.

Thanks
Gary
> Dan
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

dan at nicira

Jul 23, 2012, 1:02 AM

Post #6 of 7 (173 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On Sun, Jul 22, 2012 at 5:51 AM, Gary Kotton <gkotton [at] redhat> wrote:

> **
>
>
> This is an interesting idea. In addition to the creation we will also need
> the update. I would prefer that the agents would have one topic - that is
> for all updates. When an agent connects to the plugin it will register the
> type of operations that are supported on the specific agent. The agent
> operations can be specific as bit masks.
>
> I have implemented something similar in
> https://review.openstack.org/#/c/9591
>
> This can certainly be improved and optimized. What are your thoughts?
>

Based on your follow-up emails, I think we're now thinking similarly about
this. Just to be clear though, for updates I was talking about a different
topic for each entity that has its own UUID (e.g., topic
port-update-f01c8dcb-d9c1-4bd6-9101-1924790b4b45)

>
> In addition to this we have a number of issues where the plugin does not
> expose the information via the standard API's - for example the VLAN tag
> (this is being addressed via extensions in the provider networks feature)
>

Agreed. There are a couple options here: direct DB access (no polling,
just direct fetching), admin API extensions, or custom RPC calls. Each has
pluses and minuses. Perhaps my real goal here would be better described as
"if there's an existing plugin agnostic way to doing X, our strong bias
should be to use it until presented with concrete evidence to the
contrary". For example, should a DHCP client create a port for the DHCP
server via the standard API, or via a custom API or direct DB access? My
strong bias would be toward using the standard API.

> 3. Logging. At the moment the agents do not have a decent logging
> mechanism. This makes debugging the RPC code terribly difficult. This was
> scheduled for F-3. I'll be happy to add this if there are no objections.
>

That sounds valuable.

> 4. We need to discuss the notifications that Yong added and how these two
> methods can interact together. More specifically I think that we need to
> address the configuration files.
>

Agreed. I think we need to decide on this at monday's IRC meeting, so we
can move forward. Given F-3 deadlines, I'm well aware that I'll have to be
pragmatic here :)

>
> The RPC code requires that the eventlet monkey patch be set. This cause
> havoc when I was using the events from pyudev for new device creation. At
> the moment I have moved the event driven support to polling (if anyone who
> reads this is familiar with the issue or has an idea on how to address it
> any help will be great)
>

Sorry, wish I could help, but I'm probably in the same boat as you on this
one.

I'm going to make sure we have a good chunk of time to discuss this during
the IRC meeting on monday (sorry, I know that's late night for you...).

Dan

>
> Thanks
> Gary
>
> Dan
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

gkotton at redhat

Jul 23, 2012, 3:00 AM

Post #7 of 7 (175 views)
Permalink

Re: [Quantum] Scalable agents [In reply to]

On 07/23/2012 11:02 AM, Dan Wendlandt wrote:
>
>
> On Sun, Jul 22, 2012 at 5:51 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
>
>
> This is an interesting idea. In addition to the creation we will
> also need the update. I would prefer that the agents would have
> one topic - that is for all updates. When an agent connects to the
> plugin it will register the type of operations that are supported
> on the specific agent. The agent operations can be specific as bit
> masks.
>
> I have implemented something similar in
> https://review.openstack.org/#/c/9591
>
> This can certainly be improved and optimized. What are your thoughts?
>
>
> Based on your follow-up emails, I think we're now thinking similarly
> about this. Just to be clear though, for updates I was talking about
> a different topic for each entity that has its own UUID (e.g., topic
> port-update-f01c8dcb-d9c1-4bd6-9101-1924790b4b45)

Printout from the rabbit queues (this is for linux bridge where at the
moment the port update and network deletion are of interest (unless we
decide to change the way in which the agent is implemented))

openstack [at] openstac:~/devstack$ sudo rabbitmqctl list_queues
Listing queues ...
q-agent-network-update 0
q-agent-network-update.10351797001a4a231279 0
q-plugin 0
q-agent-port-update.10351797001a4a231279 0

10351797001a4a231279 == IP and MAC of the host

>
> In addition to this we have a number of issues where the plugin
> does not expose the information via the standard API's - for
> example the VLAN tag (this is being addressed via extensions in
> the provider networks feature)
>
>
> Agreed. There are a couple options here: direct DB access (no
> polling, just direct fetching), admin API extensions, or custom RPC
> calls. Each has pluses and minuses. Perhaps my real goal here would
> be better described as "if there's an existing plugin agnostic way to
> doing X, our strong bias should be to use it until presented with
> concrete evidence to the contrary". For example, should a DHCP
> client create a port for the DHCP server via the standard API, or via
> a custom API or direct DB access? My strong bias would be toward
> using the standard API.

Good question. I think that if the standard API's can be used then we
should go for it. Problem is that these require additional configurations.

> 3. Logging. At the moment the agents do not have a decent logging
> mechanism. This makes debugging the RPC code terribly difficult.
> This was scheduled for F-3. I'll be happy to add this if there are
> no objections.
>
>
> That sounds valuable.

Hopefully I'll be able to find some time for this.
>
> 4. We need to discuss the notifications that Yong added and how
> these two methods can interact together. More specifically I think
> that we need to address the configuration files.
>
>
> Agreed. I think we need to decide on this at monday's IRC meeting, so
> we can move forward. Given F-3 deadlines, I'm well aware that I'll
> have to be pragmatic here :)
>
>
> The RPC code requires that the eventlet monkey patch be set. This
> cause havoc when I was using the events from pyudev for new device
> creation. At the moment I have moved the event driven support to
> polling (if anyone who reads this is familiar with the issue or
> has an idea on how to address it any help will be great)
>
>
> Sorry, wish I could help, but I'm probably in the same boat as you on
> this one.

I have a solution that works. In the long term it would be better if
this was event driven. This all depends on how the discussions above
play out.
> I'm going to make sure we have a good chunk of time to discuss this
> during the IRC meeting on monday (sorry, I know that's late night for
> you...).

:). Tomorrow is jet lag day!
>
> Dan
>
>
>
> Thanks
> Gary
>
>> Dan
>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Dan Wendlandt
>> Nicira, Inc: www.nicira.com <http://www.nicira.com>
>> twitter: danwendlandt
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads