[ANN] Localmemcache-0.3.0: A persistent key-value database based onmmap()'ed shared memory

[Sven C. Koehler]

Hi, Localmemcache-0.3.0 is ready! Persistence is officially supported
now so it's now a real key-value database like GDBM and the like.

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data. It's based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

FEATURES
========

- blazingly fast: On my machine it's about 10% slower to store strings in
LocalMemCache than storing them in a Ruby Hash of strings
- persistent
- parallel writes are supported by default
- uses transactions internally to avoid data corruption
- lightweight: the core library is just about 1400 lines of C code

* http://localmemcache.rubyforge.org/

EXAMPLE
=======

require 'localmemcache'
# 1. the memcached way
# $lm = LocalMemCache.new :namespace => :viewcounters
# 2. the GDBM way
$lm = LocalMemCache.new :filename => "./viewcounters.lmc"
$lm[:foo] = 1
$lm[:foo]
$lm.deletefoo)

INSTALL
=======

# gem install localmemcache

(In case rubyforge has not yet updated the mirrors, fetch the 0.3.0 gem
from here: http://github.com/sck/localmemcache/downloads and then do
# gem install localmemcache-0.3.0.gem )

CONTACT
=======

Please contact me with bugs, suggestions and patches at: schween + snafu # de

LINKS
=====

Localmemcache: http://localmemcache.rubyforge.org/
Rubyforge project: http://localmemcache.rubyforge.org/

Source code is hosted on github: http://github.com/sck/localmemcache/

Best,

Sven C. Koehler

3100e63c98860d2393bb79e6493a415b localmemcache-0.3.0.gem
c817f4eadc86aae7435bb91b52039197 localmemcache-0.3.0.tar.gz

 

[Roger Pack]

Localmemcache is a library for C and ruby that aims to provide an
interface similar to memcached but for accessing local data instead
of remote data. It's based on mmap()'ed shared memory for maximum
speed. Since version 0.3.0 it supports persistence, also making it a
fastalternative to GDBM and Berkeley DB.

Wow that's an interesting idea. I assume it's a ruby wrapper for a C
library that does the storage?
-=r

 

[Sven C. Koehler]

Wow that's an interesting idea.

Thank you!

I assume it's a ruby wrapper for a C
library that does the storage?

Yeah, that's the case. It also has a C API. My original motivation for
writing localmemcache was that I did not want to serve requests for
images in Ruby just to update a viewcounter in the database. I wanted to
do this right in the webserver. So with localmemcache I just can do it
very easily in a webserver module in C, but I still can access the
counters when I do more complex operations in Ruby for other requests.

Best,

Sven

 

[Rados³aw Bu³at]

From README

"
SUPPORTED SYSTEMS
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

- a >=3D64bit Unix
"

Does it mean that it can't run on 32bit?

--=20
Pozdrawiam

Rados=B3aw Bu=B3at
http://radarek.jogger.pl - m=F3j blog

 

[Rados³aw Bu³at]

Hi, Localmemcache-0.3.0 is ready! =A0Persistence is officially supported
now so it's now a real key-value database like GDBM and the like.

It means that all data written to localmemcache store are later
available and never disappear (like in memcached)? I greped for
'persistence' and didn't find too much. Also one line from C source is
strange for me:
/src/lmc_shm.c: mc->use_persistence =3D 0;

I looked at the source and don't see where it can assigned with 1
(true) value etc.

--=20
Pozdrawiam

Rados=B3aw Bu=B3at
http://radarek.jogger.pl - m=F3j blog

 

[Sven C. Koehler]

"
SUPPORTED SYSTEMS
=================

- a >=64bit Unix
"

Does it mean that it can't run on 32bit?

I think it can run with 32bit, but this means your maximum memory pool
size will be about 4G, depending on how much memory you already use for
other stuff...
64Bit is more than enough for setting the memory pool size to the maximum
file size of your file system... So yeah, you might be able to use
LocalMemCache, but at one point you just could run out of virtual address
space in 32bit, and then other software which doesn't fully rely on
mmap() would probably a better solution for you.

-S.

 

[Sven C. Koehler]

It means that all data written to localmemcache store are later
available and never disappear (like in memcached)? I greped for
'persistence' and didn't find too much. Also one line from C source is
strange for me:
./src/lmc_shm.c: mc->use_persistence = 0;

I looked at the source and don't see where it can assigned with 1
(true) value etc.

It's persistent by default, this done by mmap(). Yes, it means your data
doesn't go away. That's also the reason why LocalMemCache doesn't need a
server process like memcached.

[lmc_shm.c: lmc_shm_create]
| lmc_file_path_for_namespace((char *)&fn, mc->namespace);
| if (!lmc_handle_error((mc->fd = open(fn, O_RDWR, (mode_t)0777)) == -1,
| "open", "ShmError", e)) goto open_failed;
| if (!lmc_handle_error(lseek(mc->fd, mc->size - 1, SEEK_SET) == -1,
| "lseek", "ShmError", e)) goto failed;
| if (!lmc_handle_error(write(mc->fd, "", 1) != 1, "write",
| "ShmError", e)) goto failed;
| mc->base = mmap(0, mc->size, PROT_READ | PROT_WRITE, MAP_SHARED, mc->fd,
| (off_t)0);


-S.

 

[Roger Pack]

It's persistent by default, this done by mmap(). Yes, it means your
data
doesn't go away. That's also the reason why LocalMemCache doesn't need

As long as you don't reboot you're persistent, I suppose
-=r

 

[Sven C. Koehler]

As long as you don't reboot you're persistent, I suppose

Nope, the file will be still there if you do a normal reboot and the
buffer cache will be purged before that happens. ;-)

The only thing that could happen is that your computer crashes and the
buffer cache is not purged. If that happens you might lose some data,
but the file should not be corrupted (If we suppose that the buffer cache
is purged in an LRU fashion, which should be the default on Unix iirc).
But these problems you will also have with Tokyo Tyrant and GDBM, so
there's no difference.

-S.