gkotton at redhat
May 3, 2012, 10:31 AM
Post #5 of 8
(256 views)
Permalink
|
On 05/03/2012 08:13 PM, Dan Wendlandt wrote:
>
>
> On Thu, May 3, 2012 at 10:01 AM, Gary Kotton <gkotton [at] redhat
> <mailto:gkotton [at] redhat>> wrote:
>
> On 05/03/2012 07:15 PM, Dan Wendlandt wrote:
>> Hi Gary,
>>
>> Great to see all the effort on improving usability + consistency.
>>
>> On Thu, May 3, 2012 at 7:36 AM, Gary Kotton <gkotton [at] redhat
>> <mailto:gkotton [at] redhat>> wrote:
>> >
>> >
>> > Hi,
>> > I have taken a more thorough look at the database
>> implementations. On the plugin side all of the plugins make use
>> of the common database code (../quantum/quantum/db/*).
>>
>>
>> Yes, or at least, any plugin that uses a database uses the common
>> database code. Some plugins don't have a local database at all,
>> as they rely on an external system (NVP is the only example in
>> the repo, but I've heard of other people developing plugins that
>> don't use a local DB).
>>
>> >
>> > Today I tested a number of scenarios with connectivity to the
>> server. This works well with the linuxbridge and the openvswitch
>> plugins. Taking into account that we working on the scalability
>> of the agents
>> (https://blueprints.launchpad.net/quantum/+spec/scalable-agent-comms),
>> I am pretty convinced that a large part of the blueprint is
>> redundant. Nonetheless I still think we need to address
>> > 1. Common configuration for the database strings (similarly to
>> those in OpenStack)
>>
>>
>> Agreed on this one.
>>
>> >
>> > 2. If the Quantum service starts and the database is down.
>> Quantum service terminates. Output:
>> > [f17-dev [at] localhos ~]$ quantum list_nets default
>> > Traceback (most recent call last):
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
>> line 237, in list_nets_v11
>> > res = client.list_networks(**filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 145, in with_params
>> > ret = self.function(instance, *args)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 522, in list_networks
>> > return self.get(self.networks_path, params=filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 379, in get
>> > headers=headers, params=params)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 371, in retry_request
>> > raise exceptions.ConnectionFailed(reason=_("Maximum
>> attempts reached"))
>> > ConnectionFailed: Connection to quantum failed: Maximum
>> attempts reached
>>
>>
>> Does the service terminate? Or is it just that each request
>> fails? The former would definitely be a problem, but the stack
>> trace at least suggests that the service actually catches the error.
> The service terminates. I'll use the fixes below and get back to
> you if it resolved.
>
>
> Perhaps our setup is different, as I am not seeing the service (i.e.,
> quantum-server) terminate if the database is down. Below I list
> networks, stop mysql, confirm that mysql is down, list networks with
> quantum client (get client error), use screen to confirm that service
> continues to run despite error, restart mysql, list networks with
> quantum client again demonstrating that quantum-server is again
> accessing the restarted database.
>
> Are you seeing something else for the same pattern of events?
I did the following:
1. rebooted the applaince
2. killed the mysql service
3. killed the quantum service
4. started the quantime service
5. when the service starts it exists with an exception at db.create_engine
I should have been more specific. I provided the message from the
quantum CLI and not the quantum log. My bad.
Yes, this is an edge case. When we first encountered it we resolved it
by starting the mysql service prior to the quantum. It would be best if
the service tried to reconnect.
Thanks
Gary
> Dan
>
>
> root [at] devstack-multi:~# quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
> root [at] devstack-multi:~# /etc/init.d/mysql stop
> root [at] devstack-multi:~# mysql -uroot -pnova
> ERROR 2002 (HY000): Can't connect to local MySQL server through socket
> '/var/run/mysqld/mysqld.sock' (2)
> root [at] devstack-multi:~# su stack
> stack [at] devstack-multi:/root$ screen -x stack
>
> [1]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ quantum list_nets default
> Traceback (most recent call last):
> File "/opt/stack/python-quantumclient/quantum/client/cli_lib.py",
> line 237, in list_nets_v11
> res = client.list_networks(**filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 145, in with_params
> ret = self.function(instance, *args)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 522, in list_networks
> return self.get(self.networks_path, params=filters)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 379, in get
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 364, in retry_request
> headers=headers, params=params)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 305, in do_request
> self._handle_fault_response(status_code, data)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 228, in _handle_fault_response
> EXCEPTION_HANDLERS[self.version](status_code, des_error_body)
> File "/opt/stack/python-quantumclient/quantum/client/__init__.py",
> line 119, in exception_handler_v11
> raise exceptions.QuantumClientException(message=msg)
> QuantumClientException: 500-{'message': 'Traceback (most recent call
> last):\n File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py",
> line 336, in handle_one_response\n result =
> self.application(self.environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 203, in
> __call__\n return app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in
> __call__\n response = self.app(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in
> __call__\n return resp(environ, start_response)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in
> __call__\n resp = self.call_func(req, *args, **self.kwargs)\n File
> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in
> call_func\n return self.func(req, *args, **kwargs)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 748, in __call__\n
> action_result = self.dispatch(request, action, args)\n File
> "/opt/stack/quantum/quantum/wsgi.py", line 780, in dispatch\n
> return controller_method(request=request, **action_args)\n File
> "/opt/stack/quantum/quantum/api/api_common.py", line 108, in
> the_func\n return func(*args, **kwargs)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 101, in index\n
> return self._items(request, tenant_id)\n File
> "/opt/stack/quantum/quantum/api/networks.py", line 84, in _items\n
> filter_opts=filter_opts)\n File
> "/opt/stack/quantum/quantum/plugins/openvswitch/ovs_quantum_plugin.py", line
> 122, in get_all_networks\n for x in db.network_list(tenant_id):\n
> File "/opt/stack/quantum/quantum/db/api.py", line 131, in
> network_list\n filter_by(tenant_id=tenant_id).\\\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2066, in all\n return list(self)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2176, in __iter__\n return
> self._execute_and_instances(context)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2189, in _execute_and_instances\n close_with_result=True)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2180, in _connection_from_session\n **kw)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 729, in connection\n close_with_result=close_with_result)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 735, in _connection_for_bind\n return
> engine.contextual_connect(**kwargs)\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py",
> line 2476, in contextual_connect\n self.pool.connect(),\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 224, in connect\n return _ConnectionFairy(self).checkout()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 466, in checkout\n self.connection =
> self._connection_record.get_connection()\n File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 300, in get_connection\n self.connection = self.__connect()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 330, in __connect\n connection = self.__pool._creator()\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py",
> line 80, in connect\n return dialect.connect(*cargs, **cparams)\n
> File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.6-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py",
> line 285, in connect\n return self.dbapi.connect(*cargs,
> **cparams)\n File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py",
> line 81, in Connect\n return Connection(*args, **kwargs)\n File
> "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in
> __init__\n super(Connection, self).__init__(*args,
> **kwargs2)\nOperationalError: (OperationalError) (2002, "Can\'t
> connect to local MySQL server through socket
> \'/var/run/mysqld/mysqld.sock\' (2)") None None\n'}
> stack [at] devstack-multi:/root$ screen -x stack
>
> [2]+ Stopped screen -x stack
> stack [at] devstack-multi:/root$ sudo service mysql start
> mysql start/running, process 27677
> stack [at] devstack-multi:/root$ quantum list_nets default
> Virtual Networks for Tenant default
> Network ID: f6513015-2ba5-443b-8055-a2809cd34c20
>
>
>
>
>
>
>
>
>>
>> One improvement we definitely need is to hide such errors from
>> the tenant (who should just get generic errors). We had a bug
>> filed on that already, and Guilherme proposed a patch for it
>> yesterday: https://review.openstack.org/#/c/7019/ . Perhaps that
>> is what you're suggesting below?
> Brilliant - this addresses part of it (saved a few -1's for me :))
>
>>
>> Thanks!
>>
>> dan
>>
>> >
>> > 3. Error handling - for example when using the CLI and the DB
>> is down we get the attached output. It would be better if the
>> trace was looged and the user would receive the specific error
>> message.
>> > [f17-dev [at] localhos ~]$ quantum list_nets default
>> > Traceback (most recent call last):
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/cli_lib.py",
>> line 237, in list_nets_v11
>> > res = client.list_networks(**filters)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 145, in with_params
>> > ret = self.function(instance, *args)
>> > File
>> "/usr/lib/python2.7/site-packages/quantum/client/__init__.py",
>> line 522, in list_networks
>> > ...
>> > raise exceptions.QuantumClientException(message=msg)
>> > QuantumClientException: 500-{'message': 'Traceback (most
>> recent call last):\n File
>> "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 336, in
>> handle_one_response\n result =
>> self.application(self.environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in
>> __call__\n return app(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line
>> 131, in __call__\n response = self.app(environ,
>> start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line
>> 131, in __call__\n response = self.app(environ,
>> start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 159, in
>> __call__\n return resp(environ, start_response)\n File
>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 147, in
>> __call__\n resp = self.call_func(req, *args, **self.kwargs)\n
>> File "/usr/lib/python2.7/site-packages/webob/dec.py", line 210,
>> in call_func\n return self.func(req, *args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/wsgi.py", line 748, in
>> __call__\n action_result = self.dispatch(request, action,
>> args)\n File "/usr/lib/python2.7/site-packages/quantum/wsgi.py",
>> line 780, in dispatch\n return
>> controller_method(request=request, **action_args)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/api_common.py",
>> line 108, in the_func\n return func(*args, **kwargs)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
>> 101, in index\n return self._items(request, tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/api/networks.py", line
>> 84, in _items\n filter_opts=filter_opts)\n File
>> "/usr/lib/python2.7/site-packages/quantum/plugins/linuxbridge/LinuxBridgePlugin.py",
>> line 71, in get_all_networks\n networks_list =
>> db.network_list(tenant_id)\n File
>> "/usr/lib/python2.7/site-packages/quantum/db/api.py", line 131,
>> in network_list\n filter_by(tenant_id=tenant_id).\\\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2066, in all\n return list(self)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2176, in __iter__\n return
>> self._execute_and_instances(context)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2189, in _execute_and_instances\n
>> close_with_result=True)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py",
>> line 2180, in _connection_from_session\n **kw)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",
>> line 729, in connection\n
>> close_with_result=close_with_result)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py",
>> line 735, in _connection_for_bind\n return
>> engine.contextual_connect(**kwargs)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py",
>> line 2476, in contextual_connect\n self.pool.connect(),\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
>> line 224, in connect\n return
>> _ConnectionFairy(self).checkout()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
>> 466, in checkout\n self.connection =
>> self._connection_record.get_connection()\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line
>> 300, in get_connection\n self.connection = self.__connect()\n
>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py",
>> line 330, in __connect\n connection = self.__pool._creator()\n
>> File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py",
>> line 80, in connect\n return dialect.connect(*cargs,
>> **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
>> 285, in connect\n return self.dbapi.connect(*cargs,
>> **cparams)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line
>> 81, in Connect\n return Connection(*args, **kwargs)\n File
>> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line
>> 187, in __init__\n super(Connection, self).__init__(*args,
>> **kwargs2)\nOperationalError: (OperationalError) (2003, "Can\'t
>> connect to MySQL server on \'localhost.localdomain\' (111)") None
>> None\n'}
>> >
>> > Any thought and or comments. Any objections if I update the
>> blueprint to deal with the the above items.
>> > Thanks
>> > Gary
>> >
>> >
>> > --
>> > Mailing list: https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> > Post to : netstack [at] lists
>> <mailto:netstack [at] lists>
>> > Unsubscribe : https://launchpad.net/~netstack
>> <https://launchpad.net/%7Enetstack>
>> > More help : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Dan Wendlandt
>> Nicira, Inc: www.nicira.com <http://www.nicira.com>
>> twitter: danwendlandt
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com <http://www.nicira.com>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
|
