Memcached: Dealing with unknown keys

[Tony Garcia]

I was wondering if there is a way to handle reading keys when you don't
know what the keys are, using memcache-client...

 

[Ben Bleything]

I was wondering if there is a way to handle reading keys when you don't
know what the keys are, using memcache-client...

Do you mean discovering what keys are stored in the cache or reading the
value of MYSTERY KEYS(tm)?

Ben

 

[Marcin Raczkowski]

Do you mean discovering what keys are stored in the cache or reading the
value of MYSTERY KEYS(tm)?

Ben

turn on debug in memcached pipe it to your ruby process, parse output,
get keys, make money

 

[Tony Garcia]

Do you mean discovering what keys are stored in the cache or reading the
value of MYSTERY KEYS(tm)?

Ben

I suppose a little more clarification would help:

I need to pass information (for example, an email address) based on user
name from a rails application to a disconnected ruby process through the
cache. The external process has no way to know what users are being
passed to it. I had wanted to use the user name as the key, but it seems
that I can't find the values of current keys in the cache - at least not
with memcache-client.

So far, about the best I've found is to set the cache as a hash:
Cache["users"] = { "user1"=>"(e-mail address removed)",
"user2"=>"(e-mail address removed)" }

Is there any better way?

 

[Ben Bleything]

I need to pass information (for example, an email address) based on user
name from a rails application to a disconnected ruby process through the
cache. The external process has no way to know what users are being
passed to it. I had wanted to use the user name as the key, but it seems
that I can't find the values of current keys in the cache - at least not
with memcache-client.

It sounds like you're trying to use the cache as a message queue, when
you should probably be using both in tandem... that is, store user data
in the cache, but send messages along the queue with the user name, so
the disconnected process can look it up in the cache.

Caching in general (and memcache in particular) are intended to speed up
slow data access, not to act as data repositories.

So far, about the best I've found is to set the cache as a hash:
Cache["users"] = { "user1"=>"(e-mail address removed)",
"user2"=>"(e-mail address removed)" }

If you need to iterate over the usernames and the iterator has no way of
knowing what they are, that's your best bet. Not a great idea, though.

Ben

 

[Tony Garcia]

It sounds like you're trying to use the cache as a message queue, when
you should probably be using both in tandem... that is, store user data
in the cache, but send messages along the queue with the user name, so
the disconnected process can look it up in the cache.

Only the username, email address, and a code are being stored - there
are no messages or other items that need to be passed along.

Caching in general (and memcache in particular) are intended to speed up
slow data access, not to act as data repositories.

Granted - and I am using memcache as a queue in this case. I've done a
lot of looking and can't find much else that will do what I need - and
Joyent has memcache already installed.

If anyone has a better suggestion, I'd love to hear it. This is the best
I've been able to figure out on my own.

So far, about the best I've found is to set the cache as a hash:
Cache["users"] = { "user1"=>"(e-mail address removed)",
"user2"=>"(e-mail address removed)" }

If you need to iterate over the usernames and the iterator has no way of
knowing what they are, that's your best bet. Not a great idea, though.

I'm not too thrilled with using that method either, but as I said, it's
the best I can come up with on my own.

Any suggestions?

 

[Marcin Raczkowski]

WEll.

There are libraries ready to do exactly what you want.
I dunno why you try to reinvite the well that so many people did before you.

anyway check out ruby-talk archive some days ago someone created library
to crate message queue using memcache - not that i think it's good idea
- there are much better solutions - or even tuplespace which is in
stdlib doing just that.

anyway have phun

 

[Ben Bleything]

Only the username, email address, and a code are being stored - there
are no messages or other items that need to be passed along.

Ah, okay. It sounds to me like the username, email, and the code *are*
a message, but it's a matter of semantics for sure

Granted - and I am using memcache as a queue in this case. I've done a
lot of looking and can't find much else that will do what I need - and
Joyent has memcache already installed.

Twitter uses a queue that talks the memcache protocol, but I can't
remember what it's called. Geoffrey Grosenbach recently posted a cool
blog post about message queues, too.

http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue

If anyone has a better suggestion, I'd love to hear it. This is the best
I've been able to figure out on my own.

I think the main problem is that memcache just isn't intended for what
you're trying to do. The hash solution you came up with will get the
job done, but it's a bit of a hack (as we both seem to agree). If it
works though, there's not really any reason to change.

I think if you want to change, you'll need to find some way to pass
messages, but hopefully someone else will have a more creative solution.

Cheers,
Ben

 

[Tony Garcia]

Twitter uses a queue that talks the memcache protocol, but I can't
remember what it's called. Geoffrey Grosenbach recently posted a cool
blog post about message queues, too.

http://nubyonrails.com/articles/about-this-blog-beanstalk-messaging-queue

I read that. It's what pointed me toward memcache, actually. Again, part
of the problem is that I'm on a shared Joyent server, and I was trying
to work with what they had. Installing a beanstalk server isn't a very
good option for me at the moment.

I think the main problem is that memcache just isn't intended for what
you're trying to do. The hash solution you came up with will get the
job done, but it's a bit of a hack (as we both seem to agree). If it
works though, there's not really any reason to change.

My biggest concern is scalability. I'm pretty sure I'm going to have to
change things before too long.

Thanks

 

[Blaine Cook]

I read that. It's what pointed me toward memcache, actually. Again, part
of the problem is that I'm on a shared Joyent server, and I was trying
to work with what they had. Installing a beanstalk server isn't a very
good option for me at the moment.



My biggest concern is scalability. I'm pretty sure I'm going to have to
change things before too long.

FWIW, we (Twitter) were on Joyent and used Starling (our message queue
that speaks the memcache protocol) there with no problems. All you
need to get started is to do a "gem install starling" and run it with
write permissions to /var/spool/starling (you can specify a different
path since it's unlikely you can write to /var on joyent). Once it's
running, you can use the normal memcache library to connect to <server
ip>:22122 and use set() to put things in the queue and get() to take
them out again.

Hope that helps!

b.

 

[Tony Garcia]

FWIW, we (Twitter) were on Joyent and used Starling (our message queue
that speaks the memcache protocol) there with no problems. All you
need to get started is to do a "gem install starling" and run it with
write permissions to /var/spool/starling (you can specify a different
path since it's unlikely you can write to /var on joyent). Once it's
running, you can use the normal memcache library to connect to <server
ip>:22122 and use set() to put things in the queue and get() to take
them out again.

Hope that helps!

b.

I'll check into that - thanks!

 

[Ben Bleything]

I read that. It's what pointed me toward memcache, actually. Again, part
of the problem is that I'm on a shared Joyent server, and I was trying
to work with what they had. Installing a beanstalk server isn't a very
good option for me at the moment.

Well, blaine made my point. You can easily install stuff on joyent
Good luck, and report back with what you end up doing.

Ben

 

[Eric Hodel]

Only the username, email address, and a code are being stored - there
are no messages or other items that need to be passed along.


Granted - and I am using memcache as a queue in this case. I've done a
lot of looking and can't find much else that will do what I need - and
Joyent has memcache already installed.

If anyone has a better suggestion, I'd love to hear it. This is the
best
I've been able to figure out on my own.

memcached is the wrong tool for this purpose. It is intended to be a
cache for some other persistent storage service.

 

[Tony Garcia]

Well, blaine made my point. You can easily install stuff on joyent
Good luck, and report back with what you end up doing.

Ben

Turns out that is not the case. When I tried installing stuff like that,
I was prevented from doing it. A request to tech support resulted in me
being told that I can't do all that on a *shared* server. Of course, it
turns out that memcached isn't available, either.

Final solution - I'm scrapping the Joyent account and getting a
dedicated server where I can do what I need.

Oh, and Eric:

memcached is the wrong tool for this purpose. It is intended to be a
cache for some other persistent storage service.

Got that already. Did you miss my request for better suggestions? Do you
*have* any better suggestions?