Readable/writable database in Perl

Discussion in 'Perl Misc' started by Eric, Apr 26, 2007.

  1. Eric

    Eric Guest

    Hello,

    Currently, our code is utilizing a database that is neither humanly
    readable or writable. My task is to modify the existing code so that
    the database is both. A sample of the code is below - in this
    particular example, to request a mount point. (It does some other
    stuff, such as incrementing/decrementing counts, etc., but that can be
    ignored in relation to this request for assistance.)

    I'm not looking for the solution in the posting (that wouldn't be any
    fun :); what I am looking for is some initial direction on how I might
    go about accomplishing this. There are no shortage of database related
    core and CPAN modules. In particular, I was told to use the 'mySQL'
    approach (if that means anything to anybody).

    Anyone have an idea on how I might go about this task?

    Thanks to advance to all that respond.

    Eric

    vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
    sub RequestMountPoint {
    my $self = shift;
    my $protocol = shift;
    my $bldNum = shift;
    my $mntPnt = undef;
    my $mntDBfile = $self->Config->HomeDir()."/mntDBfile";

    my $mntsDB = {};
    unless (open SEMAPHORE, "> /tmp/mnt.lock") {
    $self->Config->Env->ReleaseMach();
    die "problem allocating semaphore";
    }
    flock SEMAPHORE, Fcntl::LOCK_EX;

    tie( %$mntsDB, "MLDBM", $mountsDBFile, O_CREAT|O_RDWR, 0666,
    $DB_File::DB_BTREE );

    my $mountPoint = $self->Config->MountPointCount();
    for (my $label = 0; $label < $mountPoint; $label++){
    $mntPnt = "mntpnt".$label;
    unless (defined($mntsDB->{$mntPnt})){
    $mntsDB->{$mntPnt} = {
    BldNum => $bldNum,
    Protocol => $protocol,
    RefCnt => 1,
    };
    last;
    }

    if (($mntsDB->{$mntPnt}->{BldNum} == $bldNum) and
    ($mntsDB->{$mntPnt}->{Protocol} eq $protocol)) {
    $mntsDB->{$mntPnt}->{RefCnt}++;
    last;
    }

    $mntPnt = undef;
    }
    my $ref = $mntsDB->{$mntPnt}->{RefCnt};

    untie(%$mntsDB);

    close(SEMAPHORE);

    return $mntPnt, $ref;
    }
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     
    Eric, Apr 26, 2007
    #1
    1. Advertising

  2. Eric

    Guest

    Eric <> wrote:
    > Hello,
    >
    > Currently, our code is utilizing a database that is neither humanly
    > readable or writable. My task is to modify the existing code so that
    > the database is both.


    There is are good reasons that no "real" databases use human readable
    and even more so, human writable) data files.

    What features of real databases are you willing to surrender to accomplish
    this aim? Concurrency? Isolation? Atomicity? Durability? Performance
    scalability with increasing database size? All of the above?


    > A sample of the code is below - in this
    > particular example, to request a mount point. (It does some other
    > stuff, such as incrementing/decrementing counts, etc., but that can be
    > ignored in relation to this request for assistance.)


    If you want us to ignore code, then you should remove before you post it,
    rather than just posting it and asking us to ignore it.

    > I'm not looking for the solution in the posting (that wouldn't be any
    > fun :); what I am looking for is some initial direction on how I might
    > go about accomplishing this. There are no shortage of database related
    > core and CPAN modules. In particular, I was told to use the 'mySQL'
    > approach (if that means anything to anybody).


    MySQL does not use human readable or writable data files. Well, unless you
    are an extraordinarily gifted human.

    > Anyone have an idea on how I might go about this task?


    Sure. Step one, specify exactly what the task is, including the answers to
    the questions I asked above about you are willing to sacrifice.

    Step two, wait for step one.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 26, 2007
    #2
    1. Advertising

  3. On Apr 26, 6:08 pm, Eric <> wrote:
    > Currently, our code is utilizing a database that is neither humanly
    > readable or writable. My task is to modify the existing code so that
    > the database is both.


    > I'm not looking for the solution in the posting (that wouldn't be any
    > fun :); what I am looking for is some initial direction on how I might
    > go about accomplishing this. There are no shortage of database related
    > core and CPAN modules. In particular, I was told to use the 'mySQL'
    > approach (if that means anything to anybody).


    Well, if you want to use a human readable database you'd do better to
    use a database module that uses flat files, like say DBD::CSV or
    DBD::AnyData.
     
    Brian McCauley, Apr 26, 2007
    #3
  4. Eric

    Eric Guest

    Xho,

    Thanks for your reply to my posting. My response inline:

    > There is are good reasons that no "real" databases use human readable
    > and even more so, human writable) data files.


    I'm no expert in databases, period. But I did point out to my group
    that allowing someone to edit a database might corrupt it. I am only
    following instructions on this one.

    > What features of real databases are you willing to surrender to accomplish
    > this aim? Concurrency? Isolation? Atomicity? Durability? Performance
    > scalability with increasing database size? All of the above?


    As stated, I am not an expert in databases. So I don't know what we
    are willing to give up. Having said that however, the database is will
    never grow to great bounds. So I don't think we'd be sacrificing a
    whole lot.

    > If you want us to ignore code, then you should remove before you post it,
    > rather than just posting it and asking us to ignore it.


    In the past I have been criticized for posting to this group without
    including all of the code. Seems like when I leave something out or
    try to simplify things, people seem to focus on that rather than the
    question I am asking. So I can't seem to win with that one.

    > MySQL does not use human readable or writable data files. Well, unless you
    > are an extraordinarily gifted human.


    That's good information. Perhaps the person giving me this direction
    is unaware of this and we need to go a different route.

    > Sure. Step one, specify exactly what the task is, including the answers to
    > the questions I asked above about you are willing to sacrifice.


    The task is to have a humanly readable/writable database. I don't know
    what we'd be willing to sacrifice, but as mentioned already, the
    database will be small by database standards.

    If we can't have what we want in a database, then I was told that we
    could create simple text files instead.

    Eric


    wrote:
    > Eric <> wrote:
    > > Hello,
    > >
    > > Currently, our code is utilizing a database that is neither humanly
    > > readable or writable. My task is to modify the existing code so that
    > > the database is both.

    >
    > There is are good reasons that no "real" databases use human readable
    > and even more so, human writable) data files.
    >
    > What features of real databases are you willing to surrender to accomplish
    > this aim? Concurrency? Isolation? Atomicity? Durability? Performance
    > scalability with increasing database size? All of the above?
    >
    >
    > > A sample of the code is below - in this
    > > particular example, to request a mount point. (It does some other
    > > stuff, such as incrementing/decrementing counts, etc., but that can be
    > > ignored in relation to this request for assistance.)

    >
    > If you want us to ignore code, then you should remove before you post it,
    > rather than just posting it and asking us to ignore it.
    >
    > > I'm not looking for the solution in the posting (that wouldn't be any
    > > fun :); what I am looking for is some initial direction on how I might
    > > go about accomplishing this. There are no shortage of database related
    > > core and CPAN modules. In particular, I was told to use the 'mySQL'
    > > approach (if that means anything to anybody).

    >
    > MySQL does not use human readable or writable data files. Well, unless you
    > are an extraordinarily gifted human.
    >
    > > Anyone have an idea on how I might go about this task?

    >
    > Sure. Step one, specify exactly what the task is, including the answers to
    > the questions I asked above about you are willing to sacrifice.
    >
    > Step two, wait for step one.
    >
    > Xho
    >
    > --
    > -------------------- http://NewsReader.Com/ --------------------
    > Usenet Newsgroup Service $9.95/Month 30GB
     
    Eric, Apr 26, 2007
    #4
  5. Eric

    Eric Guest

    Thanks for this direction, Brian. I will certainly look into this.

    Eric


    Brian McCauley wrote:
    > On Apr 26, 6:08 pm, Eric <> wrote:
    > > Currently, our code is utilizing a database that is neither humanly
    > > readable or writable. My task is to modify the existing code so that
    > > the database is both.

    >
    > > I'm not looking for the solution in the posting (that wouldn't be any
    > > fun :); what I am looking for is some initial direction on how I might
    > > go about accomplishing this. There are no shortage of database related
    > > core and CPAN modules. In particular, I was told to use the 'mySQL'
    > > approach (if that means anything to anybody).

    >
    > Well, if you want to use a human readable database you'd do better to
    > use a database module that uses flat files, like say DBD::CSV or
    > DBD::AnyData.
     
    Eric, Apr 26, 2007
    #5
  6. Eric

    Guest

    Eric <> wrote:
    > Xho,
    >
    > Thanks for your reply to my posting. My response inline:
    >
    > > There is are good reasons that no "real" databases use human readable
    > > and even more so, human writable) data files.

    >
    > I'm no expert in databases, period. But I did point out to my group
    > that allowing someone to edit a database might corrupt it. I am only
    > following instructions on this one.
    >
    > > What features of real databases are you willing to surrender to
    > > accomplish this aim? Concurrency? Isolation? Atomicity? Durability?
    > > Performance scalability with increasing database size? All of the
    > > above?

    >
    > As stated, I am not an expert in databases. So I don't know what we
    > are willing to give up. Having said that however, the database is will
    > never grow to great bounds. So I don't think we'd be sacrificing a
    > whole lot.


    Then something that reads the entire file in and writes the entire file
    back out when done, like Config::Simple, might be a good option. Or just
    using Data::Dumper as a serializer, if the humans who will be reading it
    are Perl people and like that Perlish format for data.

    >
    > > If you want us to ignore code, then you should remove before you post
    > > it, rather than just posting it and asking us to ignore it.

    >
    > In the past I have been criticized for posting to this group without
    > including all of the code. Seems like when I leave something out or
    > try to simplify things, people seem to focus on that rather than the
    > question I am asking. So I can't seem to win with that one.


    The keys is that while the code you post should be "real" in the sense
    that it can be run and in doing so it demonstrates the issues at hand, it
    doesn't need to be identical to the end-goal code you are working on. Make
    a reduced copy of the code that gets rid of all the stuff that you are
    confident is irrelevant to the topic at hand. I do that not just for
    posting, but also for my own testing purposes.

    >
    > > MySQL does not use human readable or writable data files. Well, unless
    > > you are an extraordinarily gifted human.

    >
    > That's good information. Perhaps the person giving me this direction
    > is unaware of this and we need to go a different route.
    >
    > > Sure. Step one, specify exactly what the task is, including the
    > > answers to the questions I asked above about you are willing to
    > > sacrifice.

    >
    > The task is to have a humanly readable/writable database. I don't know
    > what we'd be willing to sacrifice, but as mentioned already, the
    > database will be small by database standards.


    You need to figure this out. The fact that it will be small mitigates
    indexing/performance problems--you can just read and parse the whole thing
    and then write out the whole thing each time. But it does little to
    address the concurrency, atomicity, durability issues. You (or your boss,
    since it seems you are mostly just following orders) absolutely must get a
    handle on these things. Assuming the entire file will be re-written upon
    any changes, then you need to figure out how to lock the file against one
    Perl program accessing it while other Perl programs also accessing it,
    between Perl and people accessing, and between people and other people
    accessing. Will people change the file only while the program is quescient
    or not running at all, or will they change it at any time? How will the
    program know a change has been made, and how will it respond? If a human
    is holding a lock for a long time, what should the program do? (presumably
    the programs won't need to hold locks for extended period, so that
    shouldn't be as much as an issue the other way around.)

    >
    > If we can't have what we want in a database, then I was told that we
    > could create simple text files instead.


    That is almost by definition what you will need to do. While it might be
    possible to build some fancy structures into a file and still have it
    human readable (through creative use of white-space for example), I don't
    think it would be possible to do so and still have it human writable. If
    the human doesn't understand the significance of the white space, then it
    would not be possible for them to edit the file without corrupting it.

    So it is just a matter of what modules (if any) to use to read and write
    the file. I'd turn that around. Instead of looking at every possible
    module and asking if the format it uses is acceptable to the humans, ask
    format do the human readers/writers want to use.

    Xho

    --
    -------------------- http://NewsReader.Com/ --------------------
    Usenet Newsgroup Service $9.95/Month 30GB
     
    , Apr 27, 2007
    #6
  7. Eric

    Joe Smith Guest

    Eric wrote:

    > Currently, our code is utilizing a database that is neither humanly
    > readable or writable. My task is to modify the existing code so that
    > the database is both.


    If it is to be readable and writable by humans but read-only to machines,
    then you could do what sendmail does.

    The file /etc/mail/aliases has a very simple format - it is designed to
    be read and written by the system administrator. But performing a
    sequential search on the plain-text file is very inefficient. So the
    sendmail package comes with a 'newaliases' command. That command reads
    /etc/mail/aliases and creates /etc/mail/aliases.db, which is a database
    optimized for doing fast lookups. The master file is the plain-text
    one and the database is derived from it.

    An alternative is to have the database be the definitive version.
    Whenever the data needs to be edited by a human, the entire database
    could be converted to a temporary plain-text file. Then when the
    human is finished editing, the database can be re-created from the
    modified plain-text file. (You'd have to have some sort of locking
    mechanism to ensure that no machine-generated updates are allowed
    while the human is editing.)

    Back in the days of Perl 4, I had a program that kept track of files
    and their checksums so that I could avoid duplicates and avoid doing
    unnecessary I/O to magneto-optical disks. During it development, I
    made sure it had an 'export' command to dump the data into a flat file,
    and an 'import' command to rebuild the dbm file from it. The flat file
    would exist just long enough to do a global search-and-replace, and would
    be deleted as soon as the database of binary data was rebuilt.

    -Joe
     
    Joe Smith, Apr 27, 2007
    #7
  8. Eric

    Dr.Ruud Guest

    Eric schreef:

    > In the past I have been criticized for posting to this group without
    > including all of the code.


    Impossible. You should only share a complete (meaning compilable AND
    runnable, so with test data, and using warnings AND strictures) script,
    never "all of the code".

    --
    Affijn, Ruud

    "Gewoon is een tijger."
     
    Dr.Ruud, Apr 27, 2007
    #8
  9. Eric

    Eric Guest

    Thanks to everyone who replied to this posting. I think I have enough
    to get me going on this adventure.

    I apologize for any breech of etiquette I may have committed. I'm not
    what you call an ongoing, experienced usenet participant. I'll make
    note of the suggestions people offered in regards to posting and
    replying. This group has always been helpful in giving me good
    direction.

    Eric
     
    Eric, Apr 27, 2007
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Craig Deelsnyder
    Replies:
    1
    Views:
    4,668
    =?Utf-8?B?U2hhd24gS2VubmV5?=
    Jul 19, 2004
  2. roger
    Replies:
    2
    Views:
    1,026
    roger
    Oct 23, 2004
  3. hazz
    Replies:
    0
    Views:
    667
  4. Replies:
    2
    Views:
    494
    lordy
    Jun 30, 2006
  5. PGPS
    Replies:
    10
    Views:
    644
Loading...

Share This Page