Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Qmail: users

Help on CDB file format

 

 

Qmail users RSS feed   Index | Next | Previous | View Threaded


hugo.monteiro at fct

May 8, 2008, 2:33 PM

Post #1 of 3 (215 views)
Permalink
Help on CDB file format

Hello all,

I'm sorry for posting to all three lists with what might be a subject a
bit OT. Happens that Qmail, derivated Qmail-LDAP and DJBDNS are the "the
facto" pieces of software that use and take advantage of CDBs.

Now that you all have excused me for the intrusion, on to the facts.

I'm in the process of writing a small program that, hopefully, will
allow to do recipient checking and lookup email addresses on multiple
distinct backends. Happens that for legacy and performance reasons, i
would like to include CDB in the supported backends. The program, or the
part of the relative code, is already able to read and search the
matching values for the queried key, but i've come across a situation
that i was hoping someone could clarify.

From the documentation i've seen, the representation of the data isn't
more than a record like "+keylen,datalen:key->value", where there's the
possibility of the same key to point to different values. So i started
testing with a self made cdb file like the described above and the code
behaved properly. Hurray!

Then i tried with an aliases.cdb file and a cdb file generated with
qmail-newu and i didn't get anything. Taking a closer look to the dumps
of those cdbs, i saw that the data was represented differently from the
cdb i made, and even from each other.

aliases.cdb keeps data in the format "+keylen,datalen::key[at]->&data",
where the ":" and "@" are part of the key, and the "&" part of the data.

then, cdb generated with qmail-newu has the format
"+keylen,datalen:!key->data", where the "!" is part of the key.

Of course i could write code to handle those "artifacts", but i'm not
confortable doing so without understanding them. I'd like to know if
there is any reason for them, what that reason is and if in terms of
recipient checking/lookup it would be a bad thing if the code would just
circumvent them. I would like to know also if there are more "artifacts"
that i should be aware of.


Thank you all for your time and input.

Hugo Monteiro.

--
ci.fct.unl.pt:~# cat .signature

Hugo Monteiro
Email : hugo.monteiro[at]fct.unl.pt
Telefone : +351 212948300 Ext.15307

Centro de Informática
Faculdade de Ciências e Tecnologia da
Universidade Nova de Lisboa
Quinta da Torre 2829-516 Caparica Portugal
Telefone: +351 212948596 Fax: +351 212948548
www.ci.fct.unl.pt apoio[at]fct.unl.pt

ci.fct.unl.pt:~# _


kchr-list-qmail at rac

May 8, 2008, 11:53 PM

Post #2 of 3 (196 views)
Permalink
Re: Help on CDB file format [In reply to]

Hugo Monteiro wrote:
> Hello all,
>
> I'm sorry for posting to all three lists with what might be a subject a
> bit OT. Happens that Qmail, derivated Qmail-LDAP and DJBDNS are the "the
> facto" pieces of software that use and take advantage of CDBs.

Hugo,

There's a list dedicated to cdb for discussing this.

http://cr.yp.to/lists.html#cdb


Regards,
Kim Christensen


jms1 at jms1

May 9, 2008, 10:19 PM

Post #3 of 3 (156 views)
Permalink
Re: Help on CDB file format [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2008-05-08, at 1733, Hugo Monteiro wrote:
>
> From the documentation i've seen, the representation of the data
> isn't more than a record like "+keylen,datalen:key->value", where
> there's the possibility of the same key to point to different
> values. So i started testing with a self made cdb file like the
> described above and the code behaved properly. Hurray!

note that the "+keylen,datalen:" and "->" portions are added by
cdbdump to make the output more easily processed by humans. the same
tags are read by cdbmake when building a cdb file.

for a line which looks like this:

+13,34:jms1[at]jms1.net->$1$xxxxxxxx$xxxxxxxxxxxxxxxxxxxxxx

the key is actually the 13 bytes "jms1[at]jms1.net", and the value is
actually the 34 bytes beginning with "$1$". by explicitly specifying
the lengths ahead of time, it becomes possible for both keys and data
to include any bytes, including newlines and nulls.


> Then i tried with an aliases.cdb file and a cdb file generated with
> qmail-newu and i didn't get anything. Taking a closer look to the
> dumps of those cdbs, i saw that the data was represented differently
> from the cdb i made, and even from each other.

the qmail-newu program reads users/assign and writes users/cdb. i'm
not sure where "aliases.cdb" comes into play, unless this is a
mangling of the filename.

the actual formats of the keys and data within users/cdb are changed
somewhat from how they appear in users/assign; this is done to allow
qmail-lspawn to more easily work with the data.

the keys are stored as "!" followed by the "local" part (the virtual
userid), possibly followed by a "wildcard" character. the list of
wildcard characters is also found in the cdb file, as the value
assigned to the key which is the empty string (i.e. the line starting
with "+0," in the output of "cdbdump".)

the data are stored pretty much as-is from the users/assign file,
although the ":" field separators are stored as null bytes (hex 00) so
that when they're read into memory, each string will be self-
terminating (in C, a null byte marks the end of a string.)


> aliases.cdb keeps data in the format "+keylen,datalen::key[at]->&data",
> where the ":" and "@" are part of the key, and the "&" part of the
> data.

i'm not sure where this file comes from (unless you're using that add-
on program which lets you use a sendmail-style "aliases" file instead
of maintaining .qmail-* files in /var/qmail/alias... is that it?) so i
don't know how those are built, stored, or used.


> then, cdb generated with qmail-newu has the format "+keylen,datalen:!
> key->data", where the "!" is part of the key.

the "!" will always be there in the key. qmail-newu adds it, and qmail-
lspawn searches for it.


> Of course i could write code to handle those "artifacts", but i'm
> not confortable doing so without understanding them. I'd like to
> know if there is any reason for them, what that reason is and if in
> terms of recipient checking/lookup it would be a bad thing if the
> code would just circumvent them. I would like to know also if there
> are more "artifacts" that i should be aware of.

the best documentation is qmail-newu.c and qmail-lspawn.c in the qmail
source code- that's what i just read in order to figure out the
explanation above.

as for why they're there... djb's documentation is a bit sparse. i
would imagine he had good reasons at the time- if you want to email
and ask him, and he actually answers you, let us know. the point is
that they ARE there.

as for there being any more "artifacts", as you call them... examine
the source code for the programs which read and write the cdb files
you're interested in.


- --------------------------------------------------------
| John M. Simpson -- KG4ZOW -- Programmer At Large |
| http://www.jms1.net/ <jms1[at]jms1.net> |
- --------------------------------------------------------
| Hope for America -- http://www.ronpaul2008.com/ |
- --------------------------------------------------------





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkglMGIACgkQEB9RczMG/PuJgACgjwFThF2QW6yGuRb5dH1o1pDo
HE4AnAqJM7Riz+EuvzRrbNFPj46HFkiX
=v21a
-----END PGP SIGNATURE-----

Qmail users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.