Python database of plain text editable by notepad or vi

Discussion in 'Python' started by James Harris, Mar 25, 2010.

  1. James Harris

    James Harris Guest

    I am looking to store named pieces of text in a form that can be
    edited by a standard editor such as notepad (under Windows) or vi
    (under Unix) and then pulled into Python as needed. The usual record
    locking and transactions of databases are not required.

    Another way to look at it is to treat the separate files as entries in
    a dictionary. The file name would be the key and the lines of the file
    the value.

    Anyone know of a database (with a Python interface) which will allow
    text files to be treated as database fields? If not I can just write
    it but I thought it best to ask if there was an existing solution
    first.

    James
    James Harris, Mar 25, 2010
    #1
    1. Advertising

  2. James Harris

    jkn Guest

    jkn, Mar 25, 2010
    #2
    1. Advertising

  3. James Harris

    Jon Clements Guest

    On 25 Mar, 22:40, James Harris <> wrote:
    > I am looking to store named pieces of text in a form that can be
    > edited by a standard editor such as notepad (under Windows) or vi
    > (under Unix) and then pulled into Python as needed. The usual record
    > locking and transactions of databases are not required.
    >
    > Another way to look at it is to treat the separate files as entries in
    > a dictionary. The file name would be the key and the lines of the file
    > the value.
    >
    > Anyone know of a database (with a Python interface) which will allow
    > text files to be treated as database fields? If not I can just write
    > it but I thought it best to ask if there was an existing solution
    > first.
    >
    > James


    I could be missing something here, but aren't you basically just
    talking about an OS's filesystem?

    glob or listdir somewhere, then create a dict using the file contents
    would meet your criteria, with very little lines of code -- but I
    would be interested to know what the use-case was for this... Is it
    read completely at start up time, or if each file contains a large
    amount of lines and aren't fixed width (or has no other indexing
    support without maintenance), then is a complete sequential-scan
    required each time, or do you just tell the user to not update when
    running (unless I s'pose something along the lines of a SIGHUP for
    config files is applicable).

    Sorry, just don't understand why you'd want this.

    Jon.
    Jon Clements, Mar 25, 2010
    #3
  4. James Harris

    James Harris Guest

    On 25 Mar, 22:56, Jon Clements <> wrote:
    > On 25 Mar, 22:40, James Harris <> wrote:
    >
    > > I am looking to store named pieces of text in a form that can be
    > > edited by a standard editor such as notepad (under Windows) or vi
    > > (under Unix) and then pulled into Python as needed. The usual record
    > > locking and transactions of databases are not required.

    >
    > > Another way to look at it is to treat the separate files as entries in
    > > a dictionary. The file name would be the key and the lines of the file
    > > the value.

    >
    > > Anyone know of a database (with a Python interface) which will allow
    > > text files to be treated as database fields? If not I can just write
    > > it but I thought it best to ask if there was an existing solution
    > > first.

    ....
    > I could be missing something here, but aren't you basically just
    > talking about an OS's filesystem?


    For storage, yes. The files would be marked-up text stored in the
    filesystem. The "dbms" (I use the term loosely!) would provide access
    to them by some full or partial key mechanism yet to be determined.
    Read-only access would do - at least for now.

    > glob or listdir somewhere, then create a dict using the file contents
    > would meet your criteria, with very little lines of code -- but I
    > would be interested to know what the use-case was for this... Is it
    > read completely at start up time, or if each file contains a large
    > amount of lines and aren't fixed width (or has no other indexing
    > support without maintenance), then is a complete sequential-scan
    > required each time, or do you just tell the user to not update when
    > running (unless I s'pose something along the lines of a SIGHUP for
    > config files is applicable).


    All good questions. For now, at least, the files can be read-only and
    I'd want those on disk to be the master copies at all times. If I was
    writing it myself I'd probably 'cache' some files in memory and stat
    them before use. If newer I would reread the file.

    >
    > Sorry, just don't understand why you'd want this.


    I tried to avoid boring folks with the details. I'm toying with some
    ideas for a way to help generate source code (in various languages,
    not just Python). If it goes ahead the text files would be mainly
    marked-up code snippets - with or without symbols that need to be
    replaced.

    Rather than write one single monolithic app I thought to split it into
    reusable components. One part being data access could perhaps be an
    existing database (and I'll take a look at jkn's suggestion).

    Think of the database as similar to an associative array stored on
    disk. The only difference is I may want to play fast and loose with
    the keys in some ways - e.g. check for partial key matches or return a
    list of part-matched keys. The language name could be part of the key
    but I'd also need to store variants for specific language versions.
    I'm not sure yet how it will all pan out. As I say, just throwing
    around some ideas.

    James
    James Harris, Mar 26, 2010
    #4
  5. James Harris

    Harishankar Guest

    On Fri, 26 Mar 2010 02:49:53 -0700 (PDT)
    James Harris <> wrote:

    > On 25 Mar, 22:56, Jon Clements <> wrote:
    > > On 25 Mar, 22:40, James Harris <>
    > > wrote:
    > >
    > > > I am looking to store named pieces of text in a form that can be
    > > > edited by a standard editor such as notepad (under Windows) or vi
    > > > (under Unix) and then pulled into Python as needed. The usual
    > > > record locking and transactions of databases are not required.

    > >
    > > > Another way to look at it is to treat the separate files as
    > > > entries in a dictionary. The file name would be the key and the
    > > > lines of the file the value.

    > >
    > > > Anyone know of a database (with a Python interface) which will
    > > > allow text files to be treated as database fields? If not I can
    > > > just write it but I thought it best to ask if there was an
    > > > existing solution first.

    > ...
    > > I could be missing something here, but aren't you basically just
    > > talking about an OS's filesystem?

    >
    > For storage, yes. The files would be marked-up text stored in the
    > filesystem. The "dbms" (I use the term loosely!) would provide access
    > to them by some full or partial key mechanism yet to be determined.
    > Read-only access would do - at least for now.
    >
    > > glob or listdir somewhere, then create a dict using the file
    > > contents would meet your criteria, with very little lines of code
    > > -- but I would be interested to know what the use-case was for
    > > this... Is it read completely at start up time, or if each file
    > > contains a large amount of lines and aren't fixed width (or has no
    > > other indexing support without maintenance), then is a complete
    > > sequential-scan required each time, or do you just tell the user to
    > > not update when running (unless I s'pose something along the lines
    > > of a SIGHUP for config files is applicable).

    >
    > All good questions. For now, at least, the files can be read-only and
    > I'd want those on disk to be the master copies at all times. If I was
    > writing it myself I'd probably 'cache' some files in memory and stat
    > them before use. If newer I would reread the file.
    >
    > >
    > > Sorry, just don't understand why you'd want this.

    >
    > I tried to avoid boring folks with the details. I'm toying with some
    > ideas for a way to help generate source code (in various languages,
    > not just Python). If it goes ahead the text files would be mainly
    > marked-up code snippets - with or without symbols that need to be
    > replaced.
    >
    > Rather than write one single monolithic app I thought to split it into
    > reusable components. One part being data access could perhaps be an
    > existing database (and I'll take a look at jkn's suggestion).
    >
    > Think of the database as similar to an associative array stored on
    > disk. The only difference is I may want to play fast and loose with
    > the keys in some ways - e.g. check for partial key matches or return a
    > list of part-matched keys. The language name could be part of the key
    > but I'd also need to store variants for specific language versions.
    > I'm not sure yet how it will all pan out. As I say, just throwing
    > around some ideas.
    >
    > James


    I am not sure exactly what you need, but would you consider using
    something like ConfigParser module provided by Python? It appears to be
    something similar to what you need.


    --
    V.Harishankar

    http://literaryforums.org
    http://harishankar.org
    Harishankar, Mar 26, 2010
    #5
  6. James Harris

    Jon Clements Guest

    On 26 Mar, 09:49, James Harris <> wrote:
    > On 25 Mar, 22:56, Jon Clements <> wrote:
    >
    >
    >
    > > On 25 Mar, 22:40, James Harris <> wrote:

    >
    > > > I am looking to store named pieces of text in a form that can be
    > > > edited by a standard editor such as notepad (under Windows) or vi
    > > > (under Unix) and then pulled into Python as needed. The usual record
    > > > locking and transactions of databases are not required.

    >
    > > > Another way to look at it is to treat the separate files as entries in
    > > > a dictionary. The file name would be the key and the lines of the file
    > > > the value.

    >
    > > > Anyone know of a database (with a Python interface) which will allow
    > > > text files to be treated as database fields? If not I can just write
    > > > it but I thought it best to ask if there was an existing solution
    > > > first.

    > ...
    > > I could be missing something here, but aren't you basically just
    > > talking about an OS's filesystem?

    >
    > For storage, yes. The files would be marked-up text stored in the
    > filesystem. The "dbms" (I use the term loosely!) would provide access
    > to them by some full or partial key mechanism yet to be determined.
    > Read-only access would do - at least for now.
    >
    > > glob or listdir somewhere, then create a dict using the file contents
    > > would meet your criteria, with very little lines of code -- but I
    > > would be interested to know what the use-case was for this... Is it
    > > read completely at start up time, or if each file contains a large
    > > amount of lines and aren't fixed width (or has no other indexing
    > > support without maintenance), then is a complete sequential-scan
    > > required each time, or do you just tell the user to not update when
    > > running (unless I s'pose something along the lines of a SIGHUP for
    > > config files is applicable).

    >
    > All good questions. For now, at least, the files can be read-only and
    > I'd want those on disk to be the master copies at all times. If I was
    > writing it myself I'd probably 'cache' some files in memory and stat
    > them before use. If newer I would reread the file.
    >


    It's hard to bore this group :)

    >
    >
    > > Sorry, just don't understand why you'd want this.

    >
    > I tried to avoid boring folks with the details. I'm toying with some
    > ideas for a way to help generate source code (in various languages,
    > not just Python). If it goes ahead the text files would be mainly
    > marked-up code snippets - with or without symbols that need to be
    > replaced.
    >
    > Rather than write one single monolithic app I thought to split it into
    > reusable components. One part being data access could perhaps be an
    > existing database (and I'll take a look at jkn's suggestion).
    >
    > Think of the database as similar to an associative array stored on
    > disk. The only difference is I may want to play fast and loose with
    > the keys in some ways - e.g. check for partial key matches or return a
    > list of part-matched keys. The language name could be part of the key
    > but I'd also need to store variants for specific language versions.
    > I'm not sure yet how it will all pan out. As I say, just throwing
    > around some ideas.
    >
    > James


    Thanks for the explanation.

    Who admins and, who's editing this data?

    I couldn't 100% guarantee that I could modify a text file and always
    put the right
    delimiter in the right place and remember to escape the relevant chars
    (and I'm
    probably not the 'average' user).

    Any opposition to just putting it in a 'proper' DB, then 'blobbing'
    the values?
    (or just integrate a procedure/script/function whatever your chosen
    RDBMS calls to choose it).
    Or in some systems, 'externally referencing'... loads of DB's have
    free front-ends,
    and there are lots of Python libraries.

    I think perhaps, all I'm saying is, I'd choose a different approach.
    I'd provide a front-end, rather than choose to re-write the wheel over
    DB's.

    Be nice to know how you get on, if you'd be so kind?

    Cheers,

    Jon.
    Jon Clements, Mar 26, 2010
    #6
  7. James Harris

    Ethan Furman Guest

    Jon Clements wrote:
    > On 26 Mar, 09:49, James Harris <> wrote:
    >
    >>On 25 Mar, 22:56, Jon Clements <> wrote:
    >>
    >>
    >>
    >>
    >>>On 25 Mar, 22:40, James Harris <> wrote:

    >>
    >>>>I am looking to store named pieces of text in a form that can be
    >>>>edited by a standard editor such as notepad (under Windows) or vi
    >>>>(under Unix) and then pulled into Python as needed. The usual record
    >>>>locking and transactions of databases are not required.

    >>
    >>>>Another way to look at it is to treat the separate files as entries in
    >>>>a dictionary. The file name would be the key and the lines of the file
    >>>>the value.

    >>
    >>>>Anyone know of a database (with a Python interface) which will allow
    >>>>text files to be treated as database fields? If not I can just write
    >>>>it but I thought it best to ask if there was an existing solution
    >>>>first.

    >>
    >>...
    >>
    >>>I could be missing something here, but aren't you basically just
    >>>talking about an OS's filesystem?

    >>
    >>For storage, yes. The files would be marked-up text stored in the
    >>filesystem. The "dbms" (I use the term loosely!) would provide access
    >>to them by some full or partial key mechanism yet to be determined.
    >>Read-only access would do - at least for now.
    >>
    >>
    >>>glob or listdir somewhere, then create a dict using the file contents
    >>>would meet your criteria, with very little lines of code -- but I
    >>>would be interested to know what the use-case was for this... Is it
    >>>read completely at start up time, or if each file contains a large
    >>>amount of lines and aren't fixed width (or has no other indexing
    >>>support without maintenance), then is a complete sequential-scan
    >>>required each time, or do you just tell the user to not update when
    >>>running (unless I s'pose something along the lines of a SIGHUP for
    >>>config files is applicable).

    >>
    >>All good questions. For now, at least, the files can be read-only and
    >>I'd want those on disk to be the master copies at all times. If I was
    >>writing it myself I'd probably 'cache' some files in memory and stat
    >>them before use. If newer I would reread the file.
    >>

    >
    >
    > It's hard to bore this group :)
    >
    >
    >>
    >>>Sorry, just don't understand why you'd want this.

    >>
    >>I tried to avoid boring folks with the details. I'm toying with some
    >>ideas for a way to help generate source code (in various languages,
    >>not just Python). If it goes ahead the text files would be mainly
    >>marked-up code snippets - with or without symbols that need to be
    >>replaced.
    >>
    >>Rather than write one single monolithic app I thought to split it into
    >>reusable components. One part being data access could perhaps be an
    >>existing database (and I'll take a look at jkn's suggestion).
    >>
    >>Think of the database as similar to an associative array stored on
    >>disk. The only difference is I may want to play fast and loose with
    >>the keys in some ways - e.g. check for partial key matches or return a
    >>list of part-matched keys. The language name could be part of the key
    >>but I'd also need to store variants for specific language versions.
    >>I'm not sure yet how it will all pan out. As I say, just throwing
    >>around some ideas.
    >>
    >>James

    >
    >
    > Thanks for the explanation.
    >
    > Who admins and, who's editing this data?
    >
    > I couldn't 100% guarantee that I could modify a text file and always
    > put the right
    > delimiter in the right place and remember to escape the relevant chars
    > (and I'm
    > probably not the 'average' user).
    >
    > Any opposition to just putting it in a 'proper' DB, then 'blobbing'
    > the values?
    > (or just integrate a procedure/script/function whatever your chosen
    > RDBMS calls to choose it).
    > Or in some systems, 'externally referencing'... loads of DB's have
    > free front-ends,
    > and there are lots of Python libraries.
    >
    > I think perhaps, all I'm saying is, I'd choose a different approach.
    > I'd provide a front-end, rather than choose to re-write the wheel over
    > DB's.


    Just to provide a counter-viewpoint:

    The (one) system I have worked with that was like that (program source
    files kept in blobs in a database) I absolutely hated. I was stuck with
    using the tools provided by the app (which, amazingly enough, I also
    hated ;), and unable to use my own tools because because *my* source
    files were _not_ saved as *files*. Okay, venting over.

    My point is, if what you are storing is plain ol' source files,
    providing a way to directly access them is a good thing. If what you
    are storing is a mangled version, the ability to let the user choose any
    editor to use is a good thing. :)

    My $0.02.

    ~Ethan~
    Ethan Furman, Mar 27, 2010
    #7
  8. James Harris

    James Harris Guest

    On 26 Mar, 14:58, Jon Clements <> wrote:
    > On 26 Mar, 09:49, James Harris <> wrote:

    ....
    > > I'm toying with some
    > > ideas for a way to help generate source code (in various languages,
    > > not just Python). If it goes ahead the text files would be mainly
    > > marked-up code snippets - with or without symbols that need to be
    > > replaced.

    >
    > > Rather than write one single monolithic app I thought to split it into
    > > reusable components. One part being data access could perhaps be an
    > > existing database (and I'll take a look at jkn's suggestion).

    >
    > > Think of the database as similar to an associative array stored on
    > > disk. The only difference is I may want to play fast and loose with
    > > the keys in some ways - e.g. check for partial key matches or return a
    > > list of part-matched keys. The language name could be part of the key
    > > but I'd also need to store variants for specific language versions.
    > > I'm not sure yet how it will all pan out. As I say, just throwing
    > > around some ideas.

    ....
    > Thanks for the explanation.


    No problem. Thanks for taking an interest!

    > Who admins and, who's editing this data?


    For the app I have in mind a programmer would admin and edit the
    files.

    > I couldn't 100% guarantee that I could modify a text file and always
    > put the right
    > delimiter in the right place and remember to escape the relevant chars
    > (and I'm
    > probably not the 'average' user).


    Apart from any markup each file would be just source code or, maybe,
    individual fragments of source code so there would be few delimiters
    to get right. And any that were wrong could be detected either by the
    code writer or the compiler when used.

    > Any opposition to just putting it in a 'proper' DB, then 'blobbing'
    > the values?
    > (or just integrate a procedure/script/function whatever your chosen
    > RDBMS calls to choose it).
    > Or in some systems, 'externally referencing'... loads of DB's have
    > free front-ends,
    > and there are lots of Python libraries.


    I've thought of that. Like Ethan, though, I'd prefer simple text for
    this. Databases, for good reasons, tend to obscure text. For one thing
    it makes them more secure but it also makes them less transparent and
    harder to examine and edit. For this app simple text files seem to be
    the best option at the moment.

    >
    > I think perhaps, all I'm saying is, I'd choose a different approach.
    > I'd provide a front-end, rather than choose to re-write the wheel over
    > DB's.


    Agreed. That was my reason for asking the question initially.

    > Be nice to know how you get on, if you'd be so kind?


    I don't know yet if it will be feasible but if I do eventually write
    something I'll report back.

    James
    James Harris, Mar 27, 2010
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?Uk4gRGFz?=
    Replies:
    5
    Views:
    8,250
    srajput
    May 23, 2008
  2. RNDAS
    Replies:
    1
    Views:
    12,565
    Nate Hekman
    Apr 26, 2005
  3. PhuWong
    Replies:
    0
    Views:
    421
    PhuWong
    Feb 25, 2006
  4. felciano
    Replies:
    5
    Views:
    934
    Joshua J. Kugler
    Jun 26, 2007
  5. geoffbache
    Replies:
    8
    Views:
    591
    Stefan Behnel
    Feb 11, 2008
Loading...

Share This Page