Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenStack: Netstack

Database common blueprint

 

 

OpenStack netstack RSS feed   Index | Next | Previous | View Threaded


gkotton at redhat

May 3, 2012, 7:36 AM

Post #1 of 8 (605 views)
Permalink
Database common blueprint

Hi,
I have taken a more thorough look at the database implementations. On
the plugin side all of the plugins make use of the common database code
(../quantum/quantum/db/*). Today I tested a number of scenarios with
connectivity to the server. This works well with the linuxbridge and the
openvswitch plugins. Taking into account that we working on the
scalability of the agents
(https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I
am pretty convinced that a large part of the blueprint is redundant.
Nonetheless I still think we need to address
1. Common configuration for the database strings (similarly to those in
OpenStack)
2. If the Quantum service starts and the database is down. Quantum
service terminates. Output:
[f17-dev [at] localhos ~]$ quantum list_nets default
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237,
in list_nets_v11
res = client.list_networks(**filters)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145,
in with_params
ret = self.function(instance, *args)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522,
in list_networks
return self.get(self.networks_path, params=filters)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 379,
in get
headers=headers, params=params)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 371,
in retry_request
raise exceptions.ConnectionFailed(reason=_("Maximum attempts
reached"))
ConnectionFailed: Connection to quantum failed: Maximum attempts
reached
3. Error handling - for example when using the CLI and the DB is down we
get the attached output. It would be better if the trace was looged and
the user would receive the specific error message.
[f17-dev [at] localhos ~]$ quantum list_nets default
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237,
in list_nets_v11
res = client.list_networks(**filters)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145,
in with_params
ret = self.function(instance, *args)
File
"/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522,
in list_networks
...
raise exceptions.QuantumClientException(message=msg)
QuantumClientException: 500-{'message': 'Traceback (most recent
call last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py",
line 336, in handle_one_response\n result =
self.application(self.environ, start_response)\n File
"/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in
__call__\n return app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
__call__\n return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
__call__\n return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
__call__\n return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
__call__\n return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in
__call__\n resp = self.call_func(req, *args, **self.kwargs)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in
call_func\n return self.func(req, *args, **kwargs)\n File
"/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in
__call__\n action_result = self.dispatch(request, action, args)\n
File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in
dispatch\n return controller_method(request=request,
**action_args)\n File
"/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108,
in the_func\n return func(*args, **kwargs)\n File
"/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in
index\n return self._items(request, tenant_id)\n File
"/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in
_items\n filter_opts=filter_opts)\n File
"/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
line 71, in get_all_networks\n networks_list =
db.network_list(tenant_id)\n File
"/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in
network_list\n filter_by(tenant_id=tenant_id).\\\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066,
in all\n return list(self)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176,
in __iter__\n return self._execute_and_instances(context)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189,
in _execute_and_instances\n close_with_result=True)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180,
in _connection_from_session\n **kw)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
729, in connection\n close_with_result=close_with_result)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
735, in _connection_for_bind\n return
engine.contextual_connect(**kwargs)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
2476, in contextual_connect\n self.pool.connect(),\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in
connect\n return _ConnectionFairy(self).checkout()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in
checkout\n self.connection =
self._connection_record.get_connection()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in
get_connection\n self.connection = self.__connect()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in
__connect\n connection = self.__pool._creator()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
line 80, in connect\n return dialect.connect(*cargs, **cparams)\n
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py",
line 285, in connect\n return self.dbapi.connect(*cargs,
**cparams)\n File
"/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
Connect\n return Connection(*args, **kwargs)\n File
"/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187,
in __init__\n super(Connection, self).__init__(*args,
**kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect
to MySQL server on \'localhost.localdomain\' (111)") None None\n'}

Any thought and or comments. Any objections if I update the blueprint to
deal with the the above items.
Thanks
Gary


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


dan at nicira

May 3, 2012, 9:15 AM

Post #2 of 8 (571 views)
Permalink
Re: Database common blueprint [In reply to]

Hi Gary,

Great to see all the effort on improving usability + consistency.

On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkotton [at] redhat> wrote:
>
>
> Hi,
> I have taken a more thorough look at the database implementations. On the
plugin side all of the plugins make use of the common database code
(../quantum/quantum/db/*).


Yes, or at least, any plugin that uses a database uses the common database
code. Some plugins don't have a local database at all, as they rely on an
external system (NVP is the only example in the repo, but I've heard of
other people developing plugins that don't use a local DB).

>
> Today I tested a number of scenarios with connectivity to the server.
This works well with the linuxbridge and the openvswitch plugins. Taking
into account that we working on the scalability of the agents (
https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I am
pretty convinced that a large part of the blueprint is redundant.
Nonetheless I still think we need to address
> 1. Common configuration for the database strings (similarly to those in
OpenStack)


Agreed on this one.

>
> 2. If the Quantum service starts and the database is down. Quantum
service terminates. Output:
> [f17-dev [at] localhos ~]$ quantum list_nets default
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 145, in with_params
> ret = self.function(instance, *args)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 522, in list_networks
> return self.get(self.networks_path, params=filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 379, in get
> headers=headers, params=params)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 371, in retry_request
> raise exceptions.ConnectionFailed(reason=_("Maximum attempts
reached"))
> ConnectionFailed: Connection to quantum failed: Maximum attempts
reached


Does the service terminate? Or is it just that each request fails? The
former would definitely be a problem, but the stack trace at least suggests
that the service actually catches the error.

One improvement we definitely need is to hide such errors from the tenant
(who should just get generic errors). We had a bug filed on that already,
and Guilherme proposed a patch for it yesterday:
https://review.openstack.org/#/c/7019/ . Perhaps that is what you're
suggesting below?

Thanks!

dan

>
> 3. Error handling - for example when using the CLI and the DB is down we
get the attached output. It would be better if the trace was looged and the
user would receive the specific error message.
> [f17-dev [at] localhos ~]$ quantum list_nets default
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 145, in with_params
> ret = self.function(instance, *args)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
line 522, in list_networks
> ...
> raise exceptions.QuantumClientException(message=msg)
> QuantumClientException: 500-{'message': 'Traceback (most recent call
last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line
336, in handle_one_response\n result =
self.application(self.environ, start_response)\n File
"/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n
return app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n
resp = self.call_func(req, *args, **self.kwargs)\n File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n
return self.func(req, *args, **kwargs)\n File
"/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n
action_result = self.dispatch(request, action, args)\n File
"/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n
return controller_method(request=request, **action_args)\n File
"/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in
the_func\n return func(*args, **kwargs)\n File
"/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in
index\n return self._items(request, tenant_id)\n File
"/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in
_items\n filter_opts=filter_opts)\n File
"/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
line 71, in get_all_networks\n networks_list =
db.network_list(tenant_id)\n File
"/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in
network_list\n filter_by(tenant_id=tenant_id).\\\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in
all\n return list(self)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in
__iter__\n return self._execute_and_instances(context)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in
_execute_and_instances\n close_with_result=True)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in
_connection_from_session\n **kw)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729,
in connection\n close_with_result=close_with_result)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735,
in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
2476, in contextual_connect\n self.pool.connect(),\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in
connect\n return _ConnectionFairy(self).checkout()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in
checkout\n self.connection = self._connection_record.get_connection()\n
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in
get_connection\n self.connection = self.__connect()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in
__connect\n connection = self.__pool._creator()\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line
80, in connect\n return dialect.connect(*cargs, **cparams)\n File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File
"/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
Connect\n return Connection(*args, **kwargs)\n File
"/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in
__init__\n super(Connection, self).__init__(*args,
**kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to
MySQL server on \'localhost.localdomain\' (111)") None None\n'}
>
> Any thought and or comments. Any objections if I update the blueprint to
deal with the the above items.
> Thanks
> Gary
>
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp




--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


gkotton at redhat

May 3, 2012, 10:01 AM

Post #3 of 8 (581 views)
Permalink
Re: Database common blueprint [In reply to]

On 05/03/2012 07:15 PM, Dan Wendlandt wrote:
> Hi Gary,
>
> Great to see all the effort on improving usability + consistency.
>
> On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
> >
> >
> > Hi,
> > I have taken a more thorough look at the database implementations.
> On the plugin side all of the plugins make use of the common database
> code (../quantum/quantum/db/*).
>
>
> Yes, or at least, any plugin that uses a database uses the common
> database code. Some plugins don't have a local database at all, as
> they rely on an external system (NVP is the only example in the repo,
> but I've heard of other people developing plugins that don't use a
> local DB).
>
> >
> > Today I tested a number of scenarios with connectivity to the
> server. This works well with the linuxbridge and the openvswitch
> plugins. Taking into account that we working on the scalability of the
> agents
> (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms),
> I am pretty convinced that a large part of the blueprint is redundant.
> Nonetheless I still think we need to address
> > 1. Common configuration for the database strings (similarly to those
> in OpenStack)
>
>
> Agreed on this one.
>
> >
> > 2. If the Quantum service starts and the database is down. Quantum
> service terminates. Output:
> > [f17-dev [at] localhos ~]$ quantum list_nets default
> > Traceback (most recent call last):
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line
> 237, in list_nets_v11
> > res = client.list_networks(**filters)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 145, in with_params
> > ret = self.function(instance, *args)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 522, in list_networks
> > return self.get(self.networks_path, params=filters)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 379, in get
> > headers=headers, params=params)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 371, in retry_request
> > raise exceptions.ConnectionFailed(reason=_("Maximum attempts
> reached"))
> > ConnectionFailed: Connection to quantum failed: Maximum attempts
> reached
>
>
> Does the service terminate? Or is it just that each request fails?
> The former would definitely be a problem, but the stack trace at
> least suggests that the service actually catches the error.
The service terminates. I'll use the fixes below and get back to you if
it resolved.
>
> One improvement we definitely need is to hide such errors from the
> tenant (who should just get generic errors). We had a bug filed on
> that already, and Guilherme proposed a patch for it yesterday:
> https://review.openstack.org/#/c/7019/ . Perhaps that is what you're
> suggesting below?
Brilliant - this addresses part of it (saved a few -1's for me :))
>
> Thanks!
>
> dan
>
> >
> > 3. Error handling - for example when using the CLI and the DB is
> down we get the attached output. It would be better if the trace was
> looged and the user would receive the specific error message.
> > [f17-dev [at] localhos ~]$ quantum list_nets default
> > Traceback (most recent call last):
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line
> 237, in list_nets_v11
> > res = client.list_networks(**filters)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 145, in with_params
> > ret = self.function(instance, *args)
> > File
> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line
> 522, in list_networks
> > ...
> > raise exceptions.QuantumClientException(message=msg)
> > QuantumClientException: 500-{'message': 'Traceback (most recent
> call last):\n File
> "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in
> handle_one_response\n result = self.application(self.environ,
> start_response)\n File
> "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in
> __call__\n return app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in
> __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in
> call_func\n return self.func(req, *args, **kwargs)\n File
> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in
> __call__\n action_result = self.dispatch(request, action, args)\n
> File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in
> dispatch\n return controller_method(request=request,
> **action_args)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line
> 108, in the_func\n return func(*args, **kwargs)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101,
> in index\n return self._items(request, tenant_id)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84,
> in _items\n filter_opts=filter_opts)\n File
> "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
> line 71, in get_all_networks\n networks_list =
> db.network_list(tenant_id)\n File
> "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in
> network_list\n filter_by(tenant_id=tenant_id).\\\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
> 2066, in all\n return list(self)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line
> 2176, in __iter__\n return self._execute_and_instances(context)\n
> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2189, in _execute_and_instances\n close_with_result=True)\n
> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
> line 2180, in _connection_from_session\n **kw)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 729, in connection\n close_with_result=close_with_result)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 735, in _connection_for_bind\n return
> engine.contextual_connect(**kwargs)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
> 2476, in contextual_connect\n self.pool.connect(),\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in
> connect\n return _ConnectionFairy(self).checkout()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in
> checkout\n self.connection =
> self._connection_record.get_connection()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in
> get_connection\n self.connection = self.__connect()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in
> __connect\n connection = self.__pool._creator()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
> line 80, in connect\n return dialect.connect(*cargs, **cparams)\n
> File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py",
> line 285, in connect\n return self.dbapi.connect(*cargs,
> **cparams)\n File
> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
> Connect\n return Connection(*args, **kwargs)\n File
> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187,
> in __init__\n super(Connection, self).__init__(*args,
> **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t
> connect to MySQL server on \'localhost.localdomain\' (111)") None None\n'}
> >
> > Any thought and or comments. Any objections if I update the
> blueprint to deal with the the above items.
> > Thanks
> > Gary
> >
> >
> > --
> > Mailing list: https://launchpad.net/~netstack
> <https://launchpad.net/%7Enetstack>
> > Post to : netstack [at] lists
> <mailto:netstack [at] lists>
> > Unsubscribe : https://launchpad.net/~netstack
> <https://launchpad.net/%7Enetstack>
> > More help : https://help.launchpad.net/ListHelp
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~


dan at nicira

May 3, 2012, 10:13 AM

Post #4 of 8 (583 views)
Permalink
Re: Database common blueprint [In reply to]

On Thu, May 3, 2012 at 10:01 AM, Gary Kotton <gkotton [at] redhat> wrote:

> **
> On 05/03/2012 07:15 PM, Dan Wendlandt wrote:
>
> Hi Gary,
>
> Great to see all the effort on improving usability + consistency.
>
> On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkotton [at] redhat> wrote:
> >
> >
> > Hi,
> > I have taken a more thorough look at the database implementations. On
> the plugin side all of the plugins make use of the common database code
> (../quantum/quantum/db/*).
>
>
> Yes, or at least, any plugin that uses a database uses the common database
> code. Some plugins don't have a local database at all, as they rely on an
> external system (NVP is the only example in the repo, but I've heard of
> other people developing plugins that don't use a local DB).
>
> >
> > Today I tested a number of scenarios with connectivity to the server.
> This works well with the linuxbridge and the openvswitch plugins. Taking
> into account that we working on the scalability of the agents (
> https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I
> am pretty convinced that a large part of the blueprint is redundant.
> Nonetheless I still think we need to address
> > 1. Common configuration for the database strings (similarly to those in
> OpenStack)
>
>
> Agreed on this one.
>
> >
> > 2. If the Quantum service starts and the database is down. Quantum
> service terminates. Output:
> > [f17-dev [at] localhos ~]$ quantum list_nets default
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
> line 237, in list_nets_v11
> > res = client.list_networks(**filters)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 145, in with_params
> > ret = self.function(instance, *args)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 522, in list_networks
> > return self.get(self.networks_path, params=filters)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 379, in get
> > headers=headers, params=params)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 371, in retry_request
> > raise exceptions.ConnectionFailed(reason=_("Maximum attempts
> reached"))
> > ConnectionFailed: Connection to quantum failed: Maximum attempts
> reached
>
>
> Does the service terminate? Or is it just that each request fails? The
> former would definitely be a problem, but the stack trace at least suggests
> that the service actually catches the error.
>
> The service terminates. I'll use the fixes below and get back to you if it
> resolved.
>

Perhaps our setup is different, as I am not seeing the service (i.e.,
quantum-server) terminate if the database is down. Below I list networks,
stop mysql, confirm that mysql is down, list networks with quantum client
(get client error), use screen to confirm that service continues to run
despite error, restart mysql, list networks with quantum client again
demonstrating that quantum-server is again accessing the restarted database.

Are you seeing something else for the same pattern of events?

Dan


root [at] devstack-multi:~# quantum list_nets default
Virtual Networks for Tenant default
Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
root [at] devstack-multi:~# /etc/init.d/mysql stop
root [at] devstack-multi:~# mysql -uroot -pnova
ERROR 2002 (HY000): Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)
root [at] devstack-multi:~# su stack
stack [at] devstack-multi:/root$ screen -x stack

[1]+ Stopped screen -x stack
stack [at] devstack-multi:/root$ quantum list_nets default
Traceback (most recent call last):
File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py", line
237, in list_nets_v11
res = client.list_networks(**filters)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
145, in with_params
ret = self.function(instance, *args)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
522, in list_networks
return self.get(self.networks_path, params=filters)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
379, in get
headers=headers, params=params)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
364, in retry_request
headers=headers, params=params)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
305, in do_request
self._handle_fault_response(status_code, data)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
228, in _handle_fault_response
EXCEPTION_HANDLERS[self.version](status_code, des_error_body)
File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
119, in exception_handler_v11
raise exceptions.QuantumClientException(message=msg)
QuantumClientException: 500-{'message': 'Traceback (most recent call
last):\n File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line
336, in handle_one_response\n result = self.application(self.environ,
start_response)\n File "/usr/lib/python2.7/dist-packages/paste/urlmap.py",
line 203, in __call__\n return app(environ, start_response)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
__call__\n response = self.app(environ, start_response)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
return resp(environ, start_response)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__\n
resp = self.call_func(req, *args, **self.kwargs)\n File
"/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func\n
return self.func(req, *args, **kwargs)\n File
"/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n
action_result = self.dispatch(request, action, args)\n File
"/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n return
controller_method(request=request, **action_args)\n File
"/opt/stack/quantum/quantum/api/api_common.py", line 108, in the_func\n
return func(*args, **kwargs)\n File
"/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n
return self._items(request, tenant_id)\n File
"/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n
filter_opts=filter_opts)\n File
"/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py",
line 122, in get_all_networks\n for x in db.network_list(tenant_id):\n
File "/opt/stack/quantum/quantum/db/api.py", line 131, in network_list\n
filter_by(tenant_id=tenant_id).\\\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2066, in all\n return list(self)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2176, in __iter__\n return self._execute_and_instances(context)\n
File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2189, in _execute_and_instances\n close_with_result=True)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
line 2180, in _connection_from_session\n **kw)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
line 729, in connection\n close_with_result=close_with_result)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
line 735, in _connection_for_bind\n return
engine.contextual_connect(**kwargs)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py",
line 2476, in contextual_connect\n self.pool.connect(),\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
line 224, in connect\n return _ConnectionFairy(self).checkout()\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
line 466, in checkout\n self.connection =
self._connection_record.get_connection()\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
line 300, in get_connection\n self.connection = self.__connect()\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
line 330, in __connect\n connection = self.__pool._creator()\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py",
line 80, in connect\n return dialect.connect(*cargs, **cparams)\n File
"/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py",
line 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n
File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in
Connect\n return Connection(*args, **kwargs)\n File
"/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in
__init__\n super(Connection, self).__init__(*args,
**kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t connect to
local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2)")
None None\n'}
stack [at] devstack-multi:/root$ screen -x stack

[2]+ Stopped screen -x stack
stack [at] devstack-multi:/root$ sudo service mysql start
mysql start/running, process 27677
stack [at] devstack-multi:/root$ quantum list_nets default
Virtual Networks for Tenant default
Network ID: f6513015-2ba5-443b-8055-a2809cd34c20










>
> One improvement we definitely need is to hide such errors from the tenant
> (who should just get generic errors). We had a bug filed on that already,
> and Guilherme proposed a patch for it yesterday:
> https://review.openstack.org/#/c/7019/ . Perhaps that is what you're
> suggesting below?
>
> Brilliant - this addresses part of it (saved a few -1's for me :))
>
>
> Thanks!
>
> dan
>
> >
> > 3. Error handling - for example when using the CLI and the DB is down we
> get the attached output. It would be better if the trace was looged and the
> user would receive the specific error message.
> > [f17-dev [at] localhos ~]$ quantum list_nets default
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
> line 237, in list_nets_v11
> > res = client.list_networks(**filters)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 145, in with_params
> > ret = self.function(instance, *args)
> > File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
> line 522, in list_networks
> > ...
> > raise exceptions.QuantumClientException(message=msg)
> > QuantumClientException: 500-{'message': 'Traceback (most recent call
> last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line
> 336, in handle_one_response\n result =
> self.application(self.environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n
> return app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n
> resp = self.call_func(req, *args, **self.kwargs)\n File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n
> return self.func(req, *args, **kwargs)\n File
> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n
> action_result = self.dispatch(request, action, args)\n File
> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n
> return controller_method(request=request, **action_args)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in
> the_func\n return func(*args, **kwargs)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in
> index\n return self._items(request, tenant_id)\n File
> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in
> _items\n filter_opts=filter_opts)\n File
> "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
> line 71, in get_all_networks\n networks_list =
> db.network_list(tenant_id)\n File
> "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in
> network_list\n filter_by(tenant_id=tenant_id).\\\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in
> all\n return list(self)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in
> __iter__\n return self._execute_and_instances(context)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in
> _execute_and_instances\n close_with_result=True)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in
> _connection_from_session\n **kw)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729,
> in connection\n close_with_result=close_with_result)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735,
> in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n
> File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
> 2476, in contextual_connect\n self.pool.connect(),\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in
> connect\n return _ConnectionFairy(self).checkout()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in
> checkout\n self.connection = self._connection_record.get_connection()\n
> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in
> get_connection\n self.connection = self.__connect()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in
> __connect\n connection = self.__pool._creator()\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line
> 80, in connect\n return dialect.connect(*cargs, **cparams)\n File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
> 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File
> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
> Connect\n return Connection(*args, **kwargs)\n File
> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in
> __init__\n super(Connection, self).__init__(*args,
> **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to
> MySQL server on \'localhost.localdomain\' (111)") None None\n'}
> >
> > Any thought and or comments. Any objections if I update the blueprint to
> deal with the the above items.
> > Thanks
> > Gary
> >
> >
> > --
> > Mailing list: https://launchpad.net/~netstack
> > Post to : netstack [at] lists
> > Unsubscribe : https://launchpad.net/~netstack
> > More help : https://help.launchpad.net/ListHelp
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>


--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


gkotton at redhat

May 3, 2012, 10:31 AM

Post #5 of 8 (580 views)
Permalink
Re: Database common blueprint [In reply to]

On 05/03/2012 08:13 PM, Dan Wendlandt wrote:
>
>
> On Thu, May 3, 2012 at 10:01 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
> On 05/03/2012 07:15 PM, Dan Wendlandt wrote:
>> Hi Gary,
>>
>> Great to see all the effort on improving usability + consistency.
>>
>> On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkotton [at] redhat
>> <mailto:gkotton [at] redhat>> wrote:
>> >
>> >
>> > Hi,
>> > I have taken a more thorough look at the database
>> implementations. On the plugin side all of the plugins make use
>> of the common database code (../quantum/quantum/db/*).
>>
>>
>> Yes, or at least, any plugin that uses a database uses the common
>> database code. Some plugins don't have a local database at all,
>> as they rely on an external system (NVP is the only example in
>> the repo, but I've heard of other people developing plugins that
>> don't use a local DB).
>>
>> >
>> > Today I tested a number of scenarios with connectivity to the
>> server. This works well with the linuxbridge and the openvswitch
>> plugins. Taking into account that we working on the scalability
>> of the agents
>> (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms),
>> I am pretty convinced that a large part of the blueprint is
>> redundant. Nonetheless I still think we need to address
>> > 1. Common configuration for the database strings (similarly to
>> those in OpenStack)
>>
>>
>> Agreed on this one.
>>
>> >
>> > 2. If the Quantum service starts and the database is down.
>> Quantum service terminates. Output:
>> > [f17-dev [at] localhos ~]$ quantum list_nets default
>> > Traceback (most recent call last):
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
>> line 237, in list_nets_v11
>> > res = client.list_networks(**filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 145, in with_params
>> > ret = self.function(instance, *args)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 522, in list_networks
>> > return self.get(self.networks_path, params=filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 379, in get
>> > headers=headers, params=params)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 371, in retry_request
>> > raise exceptions.ConnectionFailed(reason=_("Maximum
>> attempts reached"))
>> > ConnectionFailed: Connection to quantum failed: Maximum
>> attempts reached
>>
>>
>> Does the service terminate? Or is it just that each request
>> fails? The former would definitely be a problem, but the stack
>> trace at least suggests that the service actually catches the error.
> The service terminates. I'll use the fixes below and get back to
> you if it resolved.
>
>
> Perhaps our setup is different, as I am not seeing the service (i.e.,
> quantum-server) terminate if the database is down. Below I list
> networks, stop mysql, confirm that mysql is down, list networks with
> quantum client (get client error), use screen to confirm that service
> continues to run despite error, restart mysql, list networks with
> quantum client again demonstrating that quantum-server is again
> accessing the restarted database.
>
> Are you seeing something else for the same pattern of events?

I did the following:
1. rebooted the applaince
2. killed the mysql service
3. killed the quantum service
4. started the quantime service
5. when the service starts it exists with an exception at db.create_engine
I should have been more specific. I provided the message from the
quantum CLI and not the quantum log. My bad.
Yes, this is an edge case. When we first encountered it we resolved it
by starting the mysql service prior to the quantum. It would be best if
the service tried to reconnect.

Thanks
Gary

> Dan
>
>
> root [at] devstack-multi:~# quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
> root [at] devstack-multi:~# /etc/init.d/mysql stop
> root [at] devstack-multi:~# mysql -uroot -pnova
> ERROR 2002 (HY000): Can't connect to local MySQL server through socket
> '/var/run/mysqld/mysqld.sock' (2)
> root [at] devstack-multi:~# su stack
> stack [at] devstack-multi:/root$ screen -x stack
>
> [1]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ quantum list_nets default
> Traceback (most recent call last):
> File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py",
> line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 145, in with_params
> ret = self.function(instance, *args)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 522, in list_networks
> return self.get(self.networks_path, params=filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 379, in get
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 364, in retry_request
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 305, in do_request
> self._handle_fault_response(status_code, data)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 228, in _handle_fault_response
> EXCEPTION_HANDLERS[self.version](status_code, des_error_body)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 119, in exception_handler_v11
> raise exceptions.QuantumClientException(message=msg)
> QuantumClientException: 500-{'message': 'Traceback (most recent call
> last):\n File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py",
> line 336, in handle_one_response\n result =
> self.application(self.environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in
> __call__\n return app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in
> __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in
> call_func\n return self.func(req, *args, **kwargs)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n
> action_result = self.dispatch(request, action, args)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n
> return controller_method(request=request, **action_args)\n File
> "/opt/stack/quantum/quantum/api/api_common.py", line 108, in
> the_func\n return func(*args, **kwargs)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n
> return self._items(request, tenant_id)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n
> filter_opts=filter_opts)\n File
> "/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py", line
> 122, in get_all_networks\n for x in db.network_list(tenant_id):\n
> File "/opt/stack/quantum/quantum/db/api.py", line 131, in
> network_list\n filter_by(tenant_id=tenant_id).\\\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2066, in all\n return list(self)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2176, in __iter__\n return
> self._execute_and_instances(context)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2189, in _execute_and_instances\n close_with_result=True)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2180, in _connection_from_session\n **kw)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 729, in connection\n close_with_result=close_with_result)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 735, in _connection_for_bind\n return
> engine.contextual_connect(**kwargs)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py",
> line 2476, in contextual_connect\n self.pool.connect(),\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 224, in connect\n return _ConnectionFairy(self).checkout()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 466, in checkout\n self.connection =
> self._connection_record.get_connection()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 300, in get_connection\n self.connection = self.__connect()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 330, in __connect\n connection = self.__pool._creator()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py",
> line 80, in connect\n return dialect.connect(*cargs, **cparams)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py",
> line 285, in connect\n return self.dbapi.connect(*cargs,
> **cparams)\n File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py",
> line 81, in Connect\n return Connection(*args, **kwargs)\n File
> "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in
> __init__\n super(Connection, self).__init__(*args,
> **kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t
> connect to local MySQL server through socket
> \'/var/run/mysqld/mysqld.sock\' (2)") None None\n'}
> stack [at] devstack-multi:/root$ screen -x stack
>
> [2]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ sudo service mysql start
> mysql start/running, process 27677
> stack [at] devstack-multi:/root$ quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
>
>
>
>
>
>
>
>
>>
>> One improvement we definitely need is to hide such errors from
>> the tenant (who should just get generic errors). We had a bug
>> filed on that already, and Guilherme proposed a patch for it
>> yesterday: https://review.openstack.org/#/c/7019/ . Perhaps that
>> is what you're suggesting below?
> Brilliant - this addresses part of it (saved a few -1's for me :))
>
>>
>> Thanks!
>>
>> dan
>>
>> >
>> > 3. Error handling - for example when using the CLI and the DB
>> is down we get the attached output. It would be better if the
>> trace was looged and the user would receive the specific error
>> message.
>> > [f17-dev [at] localhos ~]$ quantum list_nets default
>> > Traceback (most recent call last):
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
>> line 237, in list_nets_v11
>> > res = client.list_networks(**filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 145, in with_params
>> > ret = self.function(instance, *args)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 522, in list_networks
>> > ...
>> > raise exceptions.QuantumClientException(message=msg)
>> > QuantumClientException: 500-{'message': 'Traceback (most
>> recent call last):\n File
>> "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in
>> handle_one_response\n result =
>> self.application(self.environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in
>> __call__\n return app(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line
>> 131, in __call__\n response = self.app(environ,
>> start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line
>> 131, in __call__\n response = self.app(environ,
>> start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in
>> __call__\n resp = self.call_func(req, *args, **self.kwargs)\n
>> File "/usr/lib/python2.7/site-packages/webob/dec.py", line 210,
>> in call_func\n return self.func(req, *args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in
>> __call__\n action_result = self.dispatch(request, action,
>> args)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py",
>> line 780, in dispatch\n return
>> controller_method(request=request, **action_args)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/api_common.py",
>> line 108, in the_func\n return func(*args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
>> 101, in index\n return self._items(request, tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
>> 84, in _items\n filter_opts=filter_opts)\n File
>> "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
>> line 71, in get_all_networks\n networks_list =
>> db.network_list(tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131,
>> in network_list\n filter_by(tenant_id=tenant_id).\\\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2066, in all\n return list(self)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2176, in __iter__\n return
>> self._execute_and_instances(context)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2189, in _execute_and_instances\n
>> close_with_result=True)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2180, in _connection_from_session\n **kw)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",
>> line 729, in connection\n
>> close_with_result=close_with_result)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",
>> line 735, in _connection_for_bind\n return
>> engine.contextual_connect(**kwargs)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py",
>> line 2476, in contextual_connect\n self.pool.connect(),\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
>> line 224, in connect\n return
>> _ConnectionFairy(self).checkout()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
>> 466, in checkout\n self.connection =
>> self._connection_record.get_connection()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
>> 300, in get_connection\n self.connection = self.__connect()\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
>> line 330, in __connect\n connection = self.__pool._creator()\n
>> File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
>> line 80, in connect\n return dialect.connect(*cargs,
>> **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
>> 285, in connect\n return self.dbapi.connect(*cargs,
>> **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line
>> 81, in Connect\n return Connection(*args, **kwargs)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line
>> 187, in __init__\n super(Connection, self).__init__(*args,
>> **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t
>> connect to MySQL server on \'localhost.localdomain\' (111)") None
>> None\n'}
>> >
>> > Any thought and or comments. Any objections if I update the
>> blueprint to deal with the the above items.
>> > Thanks
>> > Gary
>> >
>> >
>> > --
>> > Mailing list: https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> > Post to : netstack [at] lists
>> <mailto:netstack [at] lists>
>> > Unsubscribe : https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Dan Wendlandt
>> Nicira, Inc: www.nicira.com <http://www.nicira.com>
>> twitter: danwendlandt
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>


dan at nicira

May 3, 2012, 11:05 AM

Post #6 of 8 (589 views)
Permalink
Re: Database common blueprint [In reply to]

On Thu, May 3, 2012 at 10:31 AM, Gary Kotton <gkotton [at] redhat> wrote:

> **
> On 05/03/2012 08:13 PM, Dan Wendlandt wrote:
>
>
> I did the following:
> 1. rebooted the applaince
> 2. killed the mysql service
> 3. killed the quantum service
> 4. started the quantime service
> 5. when the service starts it exists with an exception at db.create_engine
> I should have been more specific. I provided the message from the quantum
> CLI and not the quantum log. My bad.
> Yes, this is an edge case. When we first encountered it we resolved it by
> starting the mysql service prior to the quantum. It would be best if the
> service tried to reconnect.
>

I see, we were talking about two different scenarios. Your point is that
the service fails to start if the DB is not running. Seems similar to the
issue you are fixing with the agents. Agree that this makes sense to fix.

Dan


>
> Thanks
> Gary
>
>
> Dan
>
>
> root [at] devstack-multi:~# quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
> root [at] devstack-multi:~# /etc/init.d/mysql stop
> root [at] devstack-multi:~# mysql -uroot -pnova
> ERROR 2002 (HY000): Can't connect to local MySQL server through socket
> '/var/run/mysqld/mysqld.sock' (2)
> root [at] devstack-multi:~# su stack
> stack [at] devstack-multi:/root$ screen -x stack
>
> [1]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ quantum list_nets default
> Traceback (most recent call last):
> File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py", line
> 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 145, in with_params
> ret = self.function(instance, *args)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 522, in list_networks
> return self.get(self.networks_path, params=filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 379, in get
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 364, in retry_request
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 305, in do_request
> self._handle_fault_response(status_code, data)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 228, in _handle_fault_response
> EXCEPTION_HANDLERS[self.version](status_code, des_error_body)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py", line
> 119, in exception_handler_v11
> raise exceptions.QuantumClientException(message=msg)
> QuantumClientException: 500-{'message': 'Traceback (most recent call
> last):\n File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line
> 336, in handle_one_response\n result = self.application(self.environ,
> start_response)\n File "/usr/lib/python2.7/dist-packages/paste/urlmap.py",
> line 203, in __call__\n return app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__\n
> return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__\n
> resp = self.call_func(req, *args, **self.kwargs)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func\n
> return self.func(req, *args, **kwargs)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n
> action_result = self.dispatch(request, action, args)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n return
> controller_method(request=request, **action_args)\n File
> "/opt/stack/quantum/quantum/api/api_common.py", line 108, in the_func\n
> return func(*args, **kwargs)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n
> return self._items(request, tenant_id)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n
> filter_opts=filter_opts)\n File
> "/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py",
> line 122, in get_all_networks\n for x in db.network_list(tenant_id):\n
> File "/opt/stack/quantum/quantum/db/api.py", line 131, in network_list\n
> filter_by(tenant_id=tenant_id).\\\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2066, in all\n return list(self)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2176, in __iter__\n return self._execute_and_instances(context)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2189, in _execute_and_instances\n close_with_result=True)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2180, in _connection_from_session\n **kw)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 729, in connection\n close_with_result=close_with_result)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 735, in _connection_for_bind\n return
> engine.contextual_connect(**kwargs)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py",
> line 2476, in contextual_connect\n self.pool.connect(),\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 224, in connect\n return _ConnectionFairy(self).checkout()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 466, in checkout\n self.connection =
> self._connection_record.get_connection()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 300, in get_connection\n self.connection = self.__connect()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 330, in __connect\n connection = self.__pool._creator()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py",
> line 80, in connect\n return dialect.connect(*cargs, **cparams)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py",
> line 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n
> File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in
> Connect\n return Connection(*args, **kwargs)\n File
> "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in
> __init__\n super(Connection, self).__init__(*args,
> **kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t connect to
> local MySQL server through socket \'/var/run/mysqld/mysqld.sock\' (2)")
> None None\n'}
> stack [at] devstack-multi:/root$ screen -x stack
>
> [2]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ sudo service mysql start
> mysql start/running, process 27677
> stack [at] devstack-multi:/root$ quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
>
>
>
>
>
>
>
>
>
>
>>
>> One improvement we definitely need is to hide such errors from the tenant
>> (who should just get generic errors). We had a bug filed on that already,
>> and Guilherme proposed a patch for it yesterday:
>> https://review.openstack.org/#/c/7019/ . Perhaps that is what you're
>> suggesting below?
>>
>> Brilliant - this addresses part of it (saved a few -1's for me :))
>>
>>
>> Thanks!
>>
>> dan
>>
>> >
>> > 3. Error handling - for example when using the CLI and the DB is down
>> we get the attached output. It would be better if the trace was looged and
>> the user would receive the specific error message.
>> > [f17-dev [at] localhos ~]$ quantum list_nets default
>> > Traceback (most recent call last):
>> > File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
>> line 237, in list_nets_v11
>> > res = client.list_networks(**filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145, in
>> with_params
>> > ret = self.function(instance, *args)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522, in
>> list_networks
>> > ...
>> > raise exceptions.QuantumClientException(message=msg)
>> > QuantumClientException: 500-{'message': 'Traceback (most recent call
>> last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line
>> 336, in handle_one_response\n result =
>> self.application(self.environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n
>> return app(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
>> return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
>> __call__\n response = self.app(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
>> return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
>> return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
>> __call__\n response = self.app(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n
>> return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n
>> resp = self.call_func(req, *args, **self.kwargs)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n
>> return self.func(req, *args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n
>> action_result = self.dispatch(request, action, args)\n File
>> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n
>> return controller_method(request=request, **action_args)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in
>> the_func\n return func(*args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in
>> index\n return self._items(request, tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in
>> _items\n filter_opts=filter_opts)\n File
>> "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
>> line 71, in get_all_networks\n networks_list =
>> db.network_list(tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in
>> network_list\n filter_by(tenant_id=tenant_id).\\\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in
>> all\n return list(self)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in
>> __iter__\n return self._execute_and_instances(context)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in
>> _execute_and_instances\n close_with_result=True)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in
>> _connection_from_session\n **kw)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729,
>> in connection\n close_with_result=close_with_result)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735,
>> in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line
>> 2476, in contextual_connect\n self.pool.connect(),\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in
>> connect\n return _ConnectionFairy(self).checkout()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in
>> checkout\n self.connection = self._connection_record.get_connection()\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in
>> get_connection\n self.connection = self.__connect()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in
>> __connect\n connection = self.__pool._creator()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line
>> 80, in connect\n return dialect.connect(*cargs, **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
>> 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
>> Connect\n return Connection(*args, **kwargs)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in
>> __init__\n super(Connection, self).__init__(*args,
>> **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to
>> MySQL server on \'localhost.localdomain\' (111)") None None\n'}
>> >
>> > Any thought and or comments. Any objections if I update the blueprint
>> to deal with the the above items.
>> > Thanks
>> > Gary
>> >
>> >
>> > --
>> > Mailing list: https://launchpad.net/~netstack
>> > Post to : netstack [at] lists
>> > Unsubscribe : https://launchpad.net/~netstack
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Dan Wendlandt
>> Nicira, Inc: www.nicira.com
>> twitter: danwendlandt
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>>
>>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>


--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


mnewby at internap

May 3, 2012, 11:19 AM

Post #7 of 8 (573 views)
Permalink
Re: Database common blueprint [In reply to]

Hi Gary,

Agreed that the 3 issues you raise need to be fixed. I think they are outside the scope of the blueprint and could be reasonably filed as bugs.

I have a modest proposal:

The agents should not be connect to a database. Agents should communicate with a service that allows them to receive and transmit state. If we're going to rewrite this, let's do it right.

Thoughts?


Maru

On 2012-05-03, at 7:36 AM, Gary Kotton wrote:

>
> Hi,
> I have taken a more thorough look at the database implementations. On the plugin side all of the plugins make use of the common database code (../quantum/quantum/db/*). Today I tested a number of scenarios with connectivity to the server. This works well with the linuxbridge and the openvswitch plugins. Taking into account that we working on the scalability of the agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I am pretty convinced that a large part of the blueprint is redundant. Nonetheless I still think we need to address
> 1. Common configuration for the database strings (similarly to those in OpenStack)
> 2. If the Quantum service starts and the database is down. Quantum service terminates. Output:
> [f17-dev [at] localhos ~]$ quantum list_nets default
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145, in with_params
> ret = self.function(instance, *args)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522, in list_networks
> return self.get(self.networks_path, params=filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 379, in get
> headers=headers, params=params)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 371, in retry_request
> raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
> ConnectionFailed: Connection to quantum failed: Maximum attempts reached
> 3. Error handling - for example when using the CLI and the DB is down we get the attached output. It would be better if the trace was looged and the user would receive the specific error message.
> [f17-dev [at] localhos ~]$ quantum list_nets default
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145, in with_params
> ret = self.function(instance, *args)
> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522, in list_networks
> ...
> raise exceptions.QuantumClientException(message=msg)
> QuantumClientException: 500-{'message': 'Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in handle_one_response\n result = self.application(self.environ, start_response)\n File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n return app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n return self.func(req, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n action_result = self.dispatch(request, action, args)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n return controller_method(request=request, **action_args)\n File "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in the_func\n return func(*args, **kwargs)\n File "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in index\n return self._items(request, tenant_id)\n File "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in _items\n filter_opts=filter_opts)\n File "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py", line 71, in get_all_networks\n networks_list = db.network_list(tenant_id)\n File "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in network_list\n filter_by(tenant_id=tenant_id).\\\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in all\n return list(self)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in __iter__\n return self._execute_and_instances(context)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in _execute_and_instances\n close_with_result=True)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in _connection_from_session\n **kw)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729, in connection\n close_with_result=close_with_result)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735, in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2476, in contextual_connect\n self.pool.connect(),\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in connect\n return _ConnectionFairy(self).checkout()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in checkout\n self.connection = self._connection_record.get_connection()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in get_connection\n self.connection = self.__connect()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in __connect\n connection = self.__pool._creator()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__\n super(Connection, self).__init__(*args, **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to MySQL server on \'localhost.localdomain\' (111)") None None\n'}
>
> Any thought and or comments. Any objections if I update the blueprint to deal with the the above items.
> Thanks
> Gary
>
>
> --
> Mailing list: https://launchpad.net/~netstack
> Post to : netstack [at] lists
> Unsubscribe : https://launchpad.net/~netstack
> More help : https://help.launchpad.net/ListHelp


--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp


gkotton at redhat

May 3, 2012, 11:41 AM

Post #8 of 8 (570 views)
Permalink
Re: Database common blueprint [In reply to]

On 05/03/2012 09:19 PM, Maru Newby wrote:
> Hi Gary,
>
> Agreed that the 3 issues you raise need to be fixed. I think they are outside the scope of the blueprint and could be reasonably filed as bugs.
Great. Does this involve deleting the blueprint?
> I have a modest proposal:
>
> The agents should not be connect to a database. Agents should communicate with a service that allows them to receive and transmit state. If we're going to rewrite this, let's do it right.
I agree. As far as I understand this is going to be addressed with
https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms
> Thoughts?
>
>
> Maru
>
> On 2012-05-03, at 7:36 AM, Gary Kotton wrote:
>
>> Hi,
>> I have taken a more thorough look at the database implementations. On the plugin side all of the plugins make use of the common database code (../quantum/quantum/db/*). Today I tested a number of scenarios with connectivity to the server. This works well with the linuxbridge and the openvswitch plugins. Taking into account that we working on the scalability of the agents (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms), I am pretty convinced that a large part of the blueprint is redundant. Nonetheless I still think we need to address
>> 1. Common configuration for the database strings (similarly to those in OpenStack)
>> 2. If the Quantum service starts and the database is down. Quantum service terminates. Output:
>> [f17-dev [at] localhos ~]$ quantum list_nets default
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237, in list_nets_v11
>> res = client.list_networks(**filters)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145, in with_params
>> ret = self.function(instance, *args)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522, in list_networks
>> return self.get(self.networks_path, params=filters)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 379, in get
>> headers=headers, params=params)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 371, in retry_request
>> raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
>> ConnectionFailed: Connection to quantum failed: Maximum attempts reached
>> 3. Error handling - for example when using the CLI and the DB is down we get the attached output. It would be better if the trace was looged and the user would receive the specific error message.
>> [f17-dev [at] localhos ~]$ quantum list_nets default
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py", line 237, in list_nets_v11
>> res = client.list_networks(**filters)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 145, in with_params
>> ret = self.function(instance, *args)
>> File "/usr/lib/python2.7/site-packages/quantum/client/__init__.py", line 522, in list_networks
>> ...
>> raise exceptions.QuantumClientException(message=msg)
>> QuantumClientException: 500-{'message': 'Traceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in handle_one_response\n result = self.application(self.environ, start_response)\n File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__\n return app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__\n response = self.app(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in __call__\n return resp(environ, start_response)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File "/usr/lib/python2.7/site-packages/webob/dec.py", line 210, in call_func\n return self.func(req, *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in __call__\n action_result = self.dispatch(request, action, args)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 780, in dispatch\n return controller_method(request=request, **action_args)\n File "/usr/lib/python2.7/site-packages/quantum/api/api_common.py", line 108, in the_func\n return func(*args, **kwargs)\n File "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 101, in index\n return self._items(request, tenant_id)\n File "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line 84, in _items\n filter_opts=filter_opts)\n File "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py", line 71, in get_all_networks\n networks_list = db.network_list(tenant_id)\n File "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131, in network_list\n filter_by(tenant_id=tenant_id).\\\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2066, in all\n return list(self)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2176, in __iter__\n return self._execute_and_instances(context)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2189, in _execute_and_instances\n close_with_result=True)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2180, in _connection_from_session\n **kw)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 729, in connection\n close_with_result=close_with_result)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 735, in _connection_for_bind\n return engine.contextual_connect(**kwargs)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2476, in contextual_connect\n self.pool.connect(),\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 224, in connect\n return _ConnectionFairy(self).checkout()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 466, in checkout\n self.connection = self._connection_record.get_connection()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 300, in get_connection\n self.connection = self.__connect()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 330, in __connect\n connection = self.__pool._creator()\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 285, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__\n super(Connection, self).__init__(*args, **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t connect to MySQL server on \'localhost.localdomain\' (111)") None None\n'}
>>
>> Any thought and or comments. Any objections if I update the blueprint to deal with the the above items.
>> Thanks
>> Gary
>>
>>
>> --
>> Mailing list: https://launchpad.net/~netstack
>> Post to : netstack [at] lists
>> Unsubscribe : https://launchpad.net/~netstack
>> More help : https://help.launchpad.net/ListHelp



--
Mailing list: https://launchpad.net/~netstack
Post to : netstack [at] lists
Unsubscribe : https://launchpad.net/~netstack
More help : https://help.launchpad.net/ListHelp

OpenStack netstack RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.