dbmauth

Andrey Gerzhov (kittle@Deimos.Ldc.Net)
Sun, 15 Feb 1998 02:49:40 +0200 (EET)

Secure. Really Secure. Stronghold: Secure Web Server

Apache Week Home | Other Features | Issues | Apache 1.2

DBM User Authentication

With more than a few users, keeping user passwords in a .htpasswd
file can get inefficient and slow down page accesses considerable.
DBM user files let sites efficient store many tens or thousands of
users (or more) with very quick access. This feature explains what
DBM is, and how to use it with Apache.

First published in Apache Week issue 42 (22nd November 1996).

DBM User Authentication
[INLINE]

This week, we explain how to store user authentication information in
DBM files for faster access when you have thousands of users.

Recently (see Apache Week 37) we explained how to use user
authentication to restrict pages to selected people. We showed how to
use the htpasswd program to create the necessary .htpasswd files, and
how to create group files to provide more control over the users. We
also said that .htpasswd files and group files like this are not very
efficient when a large number of users are involved. This is because
these are plain text files and for every request in the authenticated
area Apache has to read through the file looking for the user. A much
faster way to store the user information is to use files in DBM
format. This article explains how to create and manage DBM format user
authentication files.
[INLINE]

What is DBM?

DBM files are a simple and relatively standard method of storing
information for quick retrieval. Each item of information stored in a
DBM file consists of two parts: a key and a value. If you know the key
you can access the value very quickly. The DBM file maintains an
'index' of the keys, each of which points to where the value is stored
within the file, and the index is usually arranged such that values
can be accessed with the minimum number of file system accesses even
for very large numbers of keys.

In practice, on many systems a DBM 'file' is actually stored in two
files on the disk. If, for example, a DBM file called 'users' is
created, it will actually be stored in files called users.pag and
users.dir. If you ever need to rename or delete a DBM from the command
line, remember to change both the files, keeping the extensions (.pag
and .dir) the same. Some newer versions of DBM only create one file.

Provided the key is known in advance DBM format files are a very
efficient way of accessing information associated with that key. For
web user authentication, the key will be the username, and the value
will store their (encrypted) password. Looking up usernames and their
passwords in a DBM file will be more efficient than using a plain text
file when more than a few users are involved. This will be
particularly important for sites with lots of users (say, over 10,000)
or where there are lots of accesses to authenticated pages.
[INLINE]

Preparing Apache for DBM Files

If you want to use DBM format files with Apache, you will need to make
sure it is compiled with DBM support. By default, Apache cannot use
DBM files for user authentication, so the optional DBM authentication
module needs to be included. Note that this is included in addition to
the normal user authentication module (which uses plain text files, as
explained in the previous article). It is possible to have support for
multiple file formats compiled into Apache at the same time.

To add the DBM authentication module, edit your Configuration file in
the Apache src directory. Remove the comment from the line which
currently says

# Module dbm_auth_module mod_auth_dbm.o

To remove the comment, delete the # and space character at the
right-hand end of the line. Now update the Apache configuration by
running ./Configure, then re-make the executable with make.

However, before compiling you might also need to tell Apache where to
find the DBM functions. On some systems this is automatic. On others
you will need to add the text -lndbm or -ldbm to the EXTRA_LIBS line
in the Configuration file. (Apache 1.2 will attempt to do this
automatically if needed, but you might still need to configure it
manually in some cases). If you are not sure what your system
requires, try leaving it blank and compiling. If at the end of the
compilation you see errors about functions such as _dbm_fetch() not
being found, try each of these choices in turn. (Remember to re-run
./Configure after changing Configuration). If you still cannot get it
to compile, you might have a system where the DBM library is installed
in a non-standard directory, or where the there is no DBM library
available. You could either contact you system administrator, or
download and compile your own copy of the DBM libraries (a good choice
might be GDBM: read about it or download it).
[INLINE]

Creating A DBM Users File

For standard (htpasswd) user authentication password files, the
program htpasswd is used to add new users and set their passwords. To
create and manage DBM format user files another program from the
Apache support directory is used. The program is called dbmmanage and
is written in perl (so you will need perl on your system, and it will
need to have been compiled with support for the same DBM library you
compiled into Apache. If you have only just installed DBM on your
system you will might need to re-compile perl to build in DBM
support).

This program can be used to create a new DBM file, add users and
passwords to it, change passwords, or delete users. To start by
creating a new DBM file and adding a user to it, run the command:

dbmmanage /usr/local/etc/httpd/users adduser martin hamster

The creates the DBM file /usr/local/etc/httpd/usersdbm (which might
actually consist of /usr/local/etc/httpd/usersdbm.dir and
/usr/local/etc/httpd/usersdbm.pag), if it does not already exist. It
then adds the user 'martin' with password 'hamster'. This command can
be used with other usernames and passwords to add more users, or with
an existing username to change that user's password. A user can be
deleted from the password file with

dbmmanage /usr/local/etc/httpd/usersdbm delete martin

You can get a list of all the users in the DBM file with
dbmmanage /usr/local/etc/httpd/usersdbm view

[INLINE]

Restricting a Directory

Now you have a DBM user authentication file with some users in it, you
are ready to create an authenticated area. You can restrict a
directory either using a <Directory> section in access.conf or by
using a .htaccess file. The article in Apache Week 37 explained how
you can setup a basic .htaccess file, using this example:

AuthName "restricted stuff"
AuthType Basic
AuthUserFile /usr/local/etc/httpd/users

require valid-user

To use DBM files, the only change is to replace the directive
AuthUserFile line with

AuthDBMUserFile

/usr/local/etc/httpd/usersdbm This single change tells Apache that the
user file is now in a DBM format, rather than plain text. All the rest
of the user authentication setup remains the same (so the
authentication type is still Basic, and the syntax of require is the
same as before).
[INLINE]

Using Groups

Each user can be in one or more "groups", and you can restrict access
to people just in a specified group. This makes it possible to manage
all your users on your site in a single database, and customise the
areas that each can access. The use of DBM files for storing group
information is particularly efficient because you can use the same
file to store both password and group information.

The dbmmanage command can be used to set group information for users.
For example, to add the user "martin" to the group "staff", you would
use

dbmmanage /usr/local/etc/httpd/users adduser martin hamster staff

You put a user into multiple groups but listing them, separated by
commas. For example,

dbmmanage /usr/local/etc/httpd/users adduser martin hamster staff,admin

Note that dbmmanage has to be told the password as well, and there is
no way to set or change group information for a user without knowing
their password. This means in practice that dbmmanage is not suitable
for managing users in groups, and you will have to write your own
management scripts. Some help writing perl to manage DBM files is
given later in this article.

After creating a user and group file containing details of which users
are in which groups, you can restrict access by these groups. For
example, to restrict access to an area to only people in the group
staff, you could use:

AuthName "restricted stuff"
AuthType Basic
AuthDBMUserFile /usr/local/etc/httpd/users
AuthDBMGroupFile /usr/local/etc/httpd/users

require group staff

[INLINE]

Custom Management of DBM Files

The supplied dbmmanage script to manage DBM files is adequate for
basic editing, but cannot handle advanced use, such as managing group
information. It is also command line driven, while a Web interface
might be a better choice in many situations. To do either of these
things you will have to write programs to manage DBM files yourself.
Using perl this is not too difficult.

As a simple example, say you have an existing .htpasswd file and you
want to convert it to a DBM file, putting all the users in a specific
group. We will introduce the concepts here, and there is a link below
to the completed program for you to download. It will be written in
Perl which is quick to write and easy to customise, although the
principles of DBM use are the same whatever language is used.

The basic way to look in a DBM file is given here. DBM files are
opened in Perl as 'hashed arrays'. The "key" is the user name, and the
value is the encrypted password and optionally group information. A
simple script to lookup all the keys and values in a DBM is:

dbmopen(%DBM, "/usr/local/etc/httpd", 0644) ||
die "Cannot open file: $!\n";
while (($key, $value) = each %DBM) {
print "key=$key, value=$value\n";
}
dbmclose(%DBM);

Note that if the given DBM file does not exist, it will be created.
This script will work with both perl 4 and perl 5 (although Perl 5
users might prefer to use the new tie facility instead of dbmopen). To
lookup a known key you would use:

$key = "martin";

dbmopen(%DBM, "/usr/local/etc/httpd", 0644) ||
die "Cannot open file: $!\n";
$value = $DBM{$key};
if (!defined($value)) {
print "$key not stored\n";
} else {
print "key=$key, value=$value\n";
}
dbmclose(%DBM);

Now we can write a script to convert a htpasswd file into a DBM
database, optionally putting each user into one or more groups. The
script is htpasswd2dbm.pl, and is used like this:

cd /usr/local/etc/httpd
htpasswd2dbm.pl -htpasswd users usersdbm

The -htpasswd option specifies the htpasswd file to be read, the the
final argument is the DBM file to create (or add to). To set a group,
use the -group argument. For example, to put all the users from this
file into the groups admin and staff, use

htpasswd2dbm.pl -htpasswd users -group admin,staff usersdbm

The program will add users to an existing DBM database, so it can be
used to merge multiple htpasswd files. If you give users from
different files different groups, you will be able to setup access
restrictions on a group-by-group basis, and manage all your users in
one database. Note that if there is already a user with the same
username in the DBM file it will be overwritten by the new
information.

Group information stored in a DBM file as part of the value. If no
group information is stored, the key associated with a username just
consists of the encrypted password. To store group information, the
encrypted password is followed by a colon, then a list of groups that
the user is in, each separated by a comma. So a typical key might look
like this:

E7yT67YGht65:admin,staff

A program written in perl can easily extract the group information,
for example:

$value = $DBM{$key};
($enc, $groupfield) = split(/:/, $value);
@groups = split(/,/, $groupfield);

It is also possible to store additional information in the DBM file,
by following the groups list with a colon. Apache will ignore any data
after a colon following the groups list, so it could be used, for
example, to store the real name and contact details for the user, and
an expiry date. This could be stored in the DBM like this:

$DBM{$key} = join(":", $enc, join(",", @groups),
$realname, $company, $emailaddr,
$expdate);

Keeping all the user information together in a database like this,
which Apache can also use for user authentication, can make
administering a site with many users simpler.
_________________________________________________________________

Comments or criticisms? Please email us at editors@apacheweek.com.

Apache Week is copyright 1996-1998 by C2Net Europe Limited. C2Net
sells network security and encryption products.