Debugging nova — a small illustration with pdb

The other day, I was testing non-ascii characters with nova commands. I started with adding an ssh key to nova with a non-ascii name, not realizing key-pair names won’t support them. This is how we ended up figuring it after a small debugging session with Flavio.

First, let’s try to add an ssh keypair & attempt to provide a non-ascii name:

$ nova keypair-add --pub_key ~/.ssh/id_rsa.pub äçë
ERROR: Keypair data is invalid (HTTP 400) (Request-ID: req-5a1a2683-d658-417b-a805-91346d5202dc)

Note – supplying --debug provides more info on the stdout. However, we’re interested to find out the root cause.

Let’s find out why the key-pair data is invalid by stepping through code using pdb. To start, stop the nova-api service, and ensure none of its processes are running :

$ service openstack-nova-api stop
$ ps aux | grep -i nova-api

Ensure to have the config attribute osapi_compute_workers set to 1 — I had it set to 5 while testing rhbz-81349 — so they won’t run in parallel, in /etc/nova/nova.conf :

$ grep osapi_compute_workers \
  /etc/nova/nova.conf | grep -v ^$ | grep -v ^# \
  osapi_compute_workers=1

Let’s set the pdb trace — import pdb; pdb.set_trace() — in the source code to break into the debugger — /usr/lib/python2.6/site-packages/nova/compute/api.py

Code context — I’m setting the breakpoint just after the self-explanatory function _validate_keypair_name :

.
.
class KeypairAPI(base.Base):
    """Sub-set of the Compute Manager API for managing key pairs."""
    def __init__(self, **kwargs):
        super(KeypairAPI, self).__init__(**kwargs)

    def _validate_keypair_name(self, context, user_id, key_name):

        import pdb; pdb.set_trace()       

        safechars = "_- " + string.digits + string.ascii_letters
        clean_value = "".join(x for x in key_name if x in safechars)
.
.

Now, invoke the nova-api service manually:

$ /usr/bin/nova-api --config-file \
  /usr/share/nova/nova-dist.conf --config-file \
  /etc/nova/nova.conf --logfile /var/log/nova/api.log

From a different terminal, again, try to create a keypair with non-ascii characters — this time, it’ll let us break into the debugger

$ nova keypair-add --pub-key ~/.ssh/id_rsa.pub äçë

Now, from shell from where we invoked the nova-api server daemon, a debugger Pdb would be waiting, to step through the code, do:

• type n — which continues execution until the next line in the current function is reached
• print the value safechars — indicates what values are considered safe — which is what we’re trying to find out
• quit the debugger

.
.
.
2013-03-26 19:56:26 71243 DEBUG nova.utils [-] Reloading cached file /etc/nova/policy.json read_cached_file /usr/lib/python2.6/site-packages/nova/uti1
2013-03-26 19:56:27 INFO nova.api.openstack.wsgi [req-9ef1fd31-0cec-4465-894a-f1ae0fa9a77e 320ce46de7e24a75a7ff8906d7355ff7 57ff99aae24b4035b52177a72s
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2198)_validate_keypair_name()
-> safechars = "_- " + string.digits + string.ascii_letters
(Pdb) 
(Pdb)  n
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2199)_validate_keypair_name()
-> clean_value = "".join(x for x in key_name if x in safechars)
(Pdb)  safechars
 '_- 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' 
(Pdb) 
(Pdb)  quit

So, the above indicates non-ascii characters are not allowed for key-pair names.

Clean up

Remove the debugging fragment from the source file. Start the nova-api server daemon gracefully —

 $ service openstack-nova-api start 

Advertisement