Database vs Text for text search

[http://links.i6networks.com]

At what level a database will be better for a read only text search?

I have some records, each has 6 fields. Each record has a pararent record.

Each parent has less than 100 childs.

Total records is about 1000-1500.

I want to put everthing in one single file, or put each parent and their
children in a file so I can have 20-30 files.

I am afraid to go to the database route, because the limitation of concurent
connections and the overhead of database search.

Is there a cut-off number of records, below that line, text is better, above
it, database is better?

 

[ctcgag]

At what level a database will be better for a read only text search?

What kind of searching? What kind of joins?

I have some records, each has 6 fields. Each record has a pararent
record.

Each parent has less than 100 childs.

Total records is about 1000-1500.

I want to put everthing in one single file, or put each parent and their
children in a file so I can have 20-30 files.

I am afraid to go to the database route, because the limitation of
concurent connections and the overhead of database search.

Databases are designed to handle concurrent connections, and are designed
to do searches efficiently. That is what they are for.

Is there a cut-off number of records, below that line, text is better,
above it, database is better?

No. I have database tables with as few as 5 row, and as many 100,000,000.
I have text file schemas with as few as few as 1 records, and as many as
50,000,000,000.

Which one you choose depends on the concurrency, the data model, the
algorithmic and indexing schemes necessary to achieve acceptable
performance under the access conditions, etc.


Xho

 

[Bob Walton]

At what level a database will be better for a read only text search?

I have some records, each has 6 fields. Each record has a pararent record.

Each parent has less than 100 childs.

Total records is about 1000-1500.

I want to put everthing in one single file, or put each parent and their
children in a file so I can have 20-30 files.

I am afraid to go to the database route, because the limitation of concurent
connections and the overhead of database search.

Is there a cut-off number of records, below that line, text is better, above
it, database is better?

Well, you don't provide nearly enough information to make such a
decision. Is this a CGI app? Batch lookup? Something else? What
response time is required? You mention concurrent connections. How
many? You also mention the "overhead" of a "database search". That
would be a lot less than the overhead of a brute-force search of a
flat-file text file (assuming a decent database). The overhead of
establishing a database connection, however, may well exceed that of a
brute force text file search. Your file seems quite small -- will it
ever be bigger, or is that it? How complicated are the lookups?

Assuming your app is CGI with less than a large number of hits, it seems
like the brute force text file search would be reasonable, given the
indicated size of the file. Since it is read-only, you don't have to
worry about file locking, or adding/deleting/changing data, all of which
dramatically simplifies things.

 

[Sherm Pendley]

At what level a database will be better for a read only text search?

Whenever profiling your app shows that searching the text has become a
bottleneck, and when profiling shows that replacing the offending code
with a database-backed alternative removes that bottleneck.

Why guess when you can measure?

sherm--

 

[http://links.i6networks.com]

At what level a database will be better for a read only text search?

Well, you don't provide nearly enough information to make such a
decision. Is this a CGI app? Batch lookup? Something else? What
response time is required? You mention concurrent connections. How
many? You also mention the "overhead" of a "database search". That
would be a lot less than the overhead of a brute-force search of a
flat-file text file (assuming a decent database). The overhead of
establishing a database connection, however, may well exceed that of a
brute force text file search. Your file seems quite small -- will it
ever be bigger, or is that it? How complicated are the lookups?

Assuming your app is CGI with less than a large number of hits, it seems
like the brute force text file search would be reasonable, given the
indicated size of the file. Since it is read-only, you don't have to
worry about file locking, or adding/deleting/changing data, all of which
dramatically simplifies things.

each record has those fields:
description
product id
current price
old price

i only need to "grep case insentively keys in the description" either OR or
AND operations between searched keys.
It is a CGI/Perl

I have trouble to build the data structure, as I don't know how to store
them in a text file. I am think doing =~ m//i search in perl, but also
consider to run a system call of grep, which seems easier and simpler.

 

[Bob Walton]

http://links.i6networks.com wrote:

....

I have trouble to build the data structure, as I don't know how to store
them in a text file. I am think doing =~ m//i search in perl, but also
consider to run a system call of grep, which seems easier and simpler.

The "trouble to build the data structure, as I don't know how to store
them in a text file" would point to using a real database, even if the
quantity of data doesn't.