Random Access File as Set

C

carlbernardi

Hi,

I need to be able to use the Random Access File class as a Set class so
there will be no duplicate entries. I thought about building a Random
Access File that scans itself and won't add a similar entry but I have
a few million unique entries of which many could be similar.

Thanks,

Carl
 
J

John W. Kennedy

Hi,

I need to be able to use the Random Access File class as a Set class so
there will be no duplicate entries. I thought about building a Random
Access File that scans itself and won't add a similar entry but I have
a few million unique entries of which many could be similar.

Before there was SQL, and before there were B-trees, there were hash
files. You're basically asking the first question asked by the first
programmer who, nearly half a century ago. designed the first IBM RAMAC
305 disk file.
 
M

Matt Humphrey

Hi,

I need to be able to use the Random Access File class as a Set class so
there will be no duplicate entries. I thought about building a Random
Access File that scans itself and won't add a similar entry but I have
a few million unique entries of which many could be similar.

With a million entries this really sounds like it should be a database.
I've built file-based object sets by using an index that contains the hash
code and points to the address in the Random Access File where the object is
found, but I was dealing with only tens of thousands. What are you storing
in the file--arbitrary objects or strings or numbers? Are you expecting to
delete items from the file? How big are the objects and are they easy to
test for equality? Can you make a truly unique hash code?

Matt Humphrey (e-mail address removed) http://www.iviz.com/
 
M

Mark Rafn

I need to be able to use the Random Access File class as a Set class so
there will be no duplicate entries. I thought about building a Random
Access File that scans itself and won't add a similar entry but I have
a few million unique entries of which many could be similar.

How big is your dataset? Don't overlook the idea of just dumping it all into
a HashSet or TreeSet in memory (and deciding which to use will be useful even
if you decide to go disk-based). Modern systems with multiple GiB of RAM can
handle things that would have been lunacy a few years ago.

If you intend to scale to many gigs, then on-disk solutions are needed. Look
into sleepycat or some other disk hashing or tree storage sytem. Don't try to
write it yourself unless you're doing it as a learning project and don't mind
making bunches of mistakes.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top