Readable/writable database in Perl

[Eric]

Hello,

Currently, our code is utilizing a database that is neither humanly
readable or writable. My task is to modify the existing code so that
the database is both. A sample of the code is below - in this
particular example, to request a mount point. (It does some other
stuff, such as incrementing/decrementing counts, etc., but that can be
ignored in relation to this request for assistance.)

I'm not looking for the solution in the posting (that wouldn't be any
fun ; what I am looking for is some initial direction on how I might
go about accomplishing this. There are no shortage of database related
core and CPAN modules. In particular, I was told to use the 'mySQL'
approach (if that means anything to anybody).

Anyone have an idea on how I might go about this task?

Thanks to advance to all that respond.

Eric

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
sub RequestMountPoint {
my $self = shift;
my $protocol = shift;
my $bldNum = shift;
my $mntPnt = undef;
my $mntDBfile = $self->Config->HomeDir()."/mntDBfile";

my $mntsDB = {};
unless (open SEMAPHORE, "> /tmp/mnt.lock") {
$self->Config->Env->ReleaseMach();
die "problem allocating semaphore";
}
flock SEMAPHORE, Fcntl::LOCK_EX;

tie( %$mntsDB, "MLDBM", $mountsDBFile, O_CREAT|O_RDWR, 0666,
$DB_File:B_BTREE );

my $mountPoint = $self->Config->MountPointCount();
for (my $label = 0; $label < $mountPoint; $label++){
$mntPnt = "mntpnt".$label;
unless (defined($mntsDB->{$mntPnt})){
$mntsDB->{$mntPnt} = {
BldNum => $bldNum,
Protocol => $protocol,
RefCnt => 1,
};
last;
}

if (($mntsDB->{$mntPnt}->{BldNum} == $bldNum) and
($mntsDB->{$mntPnt}->{Protocol} eq $protocol)) {
$mntsDB->{$mntPnt}->{RefCnt}++;
last;
}

$mntPnt = undef;
}
my $ref = $mntsDB->{$mntPnt}->{RefCnt};

untie(%$mntsDB);

close(SEMAPHORE);

return $mntPnt, $ref;
}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 

[xhoster]

Hello,

Currently, our code is utilizing a database that is neither humanly
readable or writable. My task is to modify the existing code so that
the database is both.

There is are good reasons that no "real" databases use human readable
and even more so, human writable) data files.

What features of real databases are you willing to surrender to accomplish
this aim? Concurrency? Isolation? Atomicity? Durability? Performance
scalability with increasing database size? All of the above?

A sample of the code is below - in this
particular example, to request a mount point. (It does some other
stuff, such as incrementing/decrementing counts, etc., but that can be
ignored in relation to this request for assistance.)

If you want us to ignore code, then you should remove before you post it,
rather than just posting it and asking us to ignore it.

I'm not looking for the solution in the posting (that wouldn't be any
fun ; what I am looking for is some initial direction on how I might
go about accomplishing this. There are no shortage of database related
core and CPAN modules. In particular, I was told to use the 'mySQL'
approach (if that means anything to anybody).

MySQL does not use human readable or writable data files. Well, unless you
are an extraordinarily gifted human.

Anyone have an idea on how I might go about this task?

Sure. Step one, specify exactly what the task is, including the answers to
the questions I asked above about you are willing to sacrifice.

Step two, wait for step one.

Xho

 

[Brian McCauley]

Currently, our code is utilizing a database that is neither humanly
readable or writable. My task is to modify the existing code so that
the database is both.

I'm not looking for the solution in the posting (that wouldn't be any
fun ; what I am looking for is some initial direction on how I might
go about accomplishing this. There are no shortage of database related
core and CPAN modules. In particular, I was told to use the 'mySQL'
approach (if that means anything to anybody).

Well, if you want to use a human readable database you'd do better to
use a database module that uses flat files, like say DBD::CSV or
DBD::AnyData.

 

[Eric]

Xho,

Thanks for your reply to my posting. My response inline:

There is are good reasons that no "real" databases use human readable
and even more so, human writable) data files.

I'm no expert in databases, period. But I did point out to my group
that allowing someone to edit a database might corrupt it. I am only
following instructions on this one.

What features of real databases are you willing to surrender to accomplish
this aim? Concurrency? Isolation? Atomicity? Durability? Performance
scalability with increasing database size? All of the above?

As stated, I am not an expert in databases. So I don't know what we
are willing to give up. Having said that however, the database is will
never grow to great bounds. So I don't think we'd be sacrificing a
whole lot.

If you want us to ignore code, then you should remove before you post it,
rather than just posting it and asking us to ignore it.

In the past I have been criticized for posting to this group without
including all of the code. Seems like when I leave something out or
try to simplify things, people seem to focus on that rather than the
question I am asking. So I can't seem to win with that one.

MySQL does not use human readable or writable data files. Well, unless you
are an extraordinarily gifted human.

That's good information. Perhaps the person giving me this direction
is unaware of this and we need to go a different route.

Sure. Step one, specify exactly what the task is, including the answers to
the questions I asked above about you are willing to sacrifice.

The task is to have a humanly readable/writable database. I don't know
what we'd be willing to sacrifice, but as mentioned already, the
database will be small by database standards.

If we can't have what we want in a database, then I was told that we
could create simple text files instead.

Eric

 

[Eric]

Thanks for this direction, Brian. I will certainly look into this.

Eric

 

[xhoster]

Xho,

Thanks for your reply to my posting. My response inline:


I'm no expert in databases, period. But I did point out to my group
that allowing someone to edit a database might corrupt it. I am only
following instructions on this one.


As stated, I am not an expert in databases. So I don't know what we
are willing to give up. Having said that however, the database is will
never grow to great bounds. So I don't think we'd be sacrificing a
whole lot.

Then something that reads the entire file in and writes the entire file
back out when done, like Config::Simple, might be a good option. Or just
using Data:umper as a serializer, if the humans who will be reading it
are Perl people and like that Perlish format for data.

In the past I have been criticized for posting to this group without
including all of the code. Seems like when I leave something out or
try to simplify things, people seem to focus on that rather than the
question I am asking. So I can't seem to win with that one.

The keys is that while the code you post should be "real" in the sense
that it can be run and in doing so it demonstrates the issues at hand, it
doesn't need to be identical to the end-goal code you are working on. Make
a reduced copy of the code that gets rid of all the stuff that you are
confident is irrelevant to the topic at hand. I do that not just for
posting, but also for my own testing purposes.

That's good information. Perhaps the person giving me this direction
is unaware of this and we need to go a different route.


The task is to have a humanly readable/writable database. I don't know
what we'd be willing to sacrifice, but as mentioned already, the
database will be small by database standards.

You need to figure this out. The fact that it will be small mitigates
indexing/performance problems--you can just read and parse the whole thing
and then write out the whole thing each time. But it does little to
address the concurrency, atomicity, durability issues. You (or your boss,
since it seems you are mostly just following orders) absolutely must get a
handle on these things. Assuming the entire file will be re-written upon
any changes, then you need to figure out how to lock the file against one
Perl program accessing it while other Perl programs also accessing it,
between Perl and people accessing, and between people and other people
accessing. Will people change the file only while the program is quescient
or not running at all, or will they change it at any time? How will the
program know a change has been made, and how will it respond? If a human
is holding a lock for a long time, what should the program do? (presumably
the programs won't need to hold locks for extended period, so that
shouldn't be as much as an issue the other way around.)

If we can't have what we want in a database, then I was told that we
could create simple text files instead.

That is almost by definition what you will need to do. While it might be
possible to build some fancy structures into a file and still have it
human readable (through creative use of white-space for example), I don't
think it would be possible to do so and still have it human writable. If
the human doesn't understand the significance of the white space, then it
would not be possible for them to edit the file without corrupting it.

So it is just a matter of what modules (if any) to use to read and write
the file. I'd turn that around. Instead of looking at every possible
module and asking if the format it uses is acceptable to the humans, ask
format do the human readers/writers want to use.

Xho

 

[Joe Smith]

Currently, our code is utilizing a database that is neither humanly
readable or writable. My task is to modify the existing code so that
the database is both.

If it is to be readable and writable by humans but read-only to machines,
then you could do what sendmail does.

The file /etc/mail/aliases has a very simple format - it is designed to
be read and written by the system administrator. But performing a
sequential search on the plain-text file is very inefficient. So the
sendmail package comes with a 'newaliases' command. That command reads
/etc/mail/aliases and creates /etc/mail/aliases.db, which is a database
optimized for doing fast lookups. The master file is the plain-text
one and the database is derived from it.

An alternative is to have the database be the definitive version.
Whenever the data needs to be edited by a human, the entire database
could be converted to a temporary plain-text file. Then when the
human is finished editing, the database can be re-created from the
modified plain-text file. (You'd have to have some sort of locking
mechanism to ensure that no machine-generated updates are allowed
while the human is editing.)

Back in the days of Perl 4, I had a program that kept track of files
and their checksums so that I could avoid duplicates and avoid doing
unnecessary I/O to magneto-optical disks. During it development, I
made sure it had an 'export' command to dump the data into a flat file,
and an 'import' command to rebuild the dbm file from it. The flat file
would exist just long enough to do a global search-and-replace, and would
be deleted as soon as the database of binary data was rebuilt.

-Joe

 

[Dr.Ruud]

Eric schreef:

In the past I have been criticized for posting to this group without
including all of the code.

Impossible. You should only share a complete (meaning compilable AND
runnable, so with test data, and using warnings AND strictures) script,
never "all of the code".

 

[Eric]

Thanks to everyone who replied to this posting. I think I have enough
to get me going on this adventure.

I apologize for any breech of etiquette I may have committed. I'm not
what you call an ongoing, experienced usenet participant. I'll make
note of the suggestions people offered in regards to posting and
replying. This group has always been helpful in giving me good
direction.

Eric