Key-Value Stores

Notes from my learning about simple NoSQL storage solutions.

14

SEP
2009

Setting up the Redis Server

Before we can play with Redis, you will need to get the server running locally. Luckily, that's very easy.

Installing Redis

Building Redis is a simple matter of grabbing the code and compiling it. Once built, you can place the executables in a convenient location in your PATH. On my box, I can do all of that with these commands:

curl -O http://redis.googlecode.com/files/redis-1.0.tar.gz
tar xzvf redis-1.0.tar.gz 
cd redis-1.0
make
sudo cp redis-server redis-cli redis-benchmark /usr/local/bin

Those commands build version 1.0 of the server, which is the current stable release as of this writing. You may need to adjust the version numbers down the road to get the latest releases though.

I also copied the executables to where I prefer to have them: /usr/local/bin. Feel free to change that directory in the last command to whatever you prefer.

If you will be talking to Redis from Ruby, as I will show in all of my examples, you are going to need a client library. I recommend Ezra Zygmuntowicz's redis-rb. You can install that gem with:

gem install redis

Running and Configuring Redis

That's it for the install. Launching the server is even easier. The pattern is just:

redis-server path/to/redis.conf

The argument is the path to the configuration file that tells Redis how you want it to behave. There's a sample configuration file in the Redis source code that shows the options.

I'm not going to discuss all of the configuration options. They are already well commented in the sample file. However, I do want to mention a few things.

First, if you will only be connecting to a local Redis instance, uncomment the bind configuration in the sample file:

bind 127.0.0.1

That tells Redis not to listen for external connections.

If you do need to accept external connections, you may want to set a limit for the number of simultaneous connections to avoid exhausting the file descriptors on your server. You can also adjust the timeout for inactive connections to reclaim those resources:

maxclients 128
timeout 60

There are some other non-network limits you may wish to fiddle with as well.

By default, Redis supports multiple databases. It even has a move() command that allows you to transfer keys between databases. I find I usually only want one though. I'm more likely to launch multiple servers if I want more. This would allow me to control resources, like memory consumption, on a per database basis, at the cost of losing the atomic move(). Even if I didn't just want one database though, I think it would be rare to need the 16 that are configured by default. You can easily turn that down:

databases 1

Another limit you may want to consider setting is the maximum memory limit. If you plan to use Redis as a memcached replacement, you will likely wish to control how much memory it can consume. You can set the maximum number of bytes Redis can allocate, after which it will start purging volatile keys. If it cannot reclaim any more memory it will start refusing write commands. Here's a sample setting for a 100MB limit:

maxmemory 104857600

Note that the above setting is really only a good idea when using Redis as a cache. If you are using it as a general database, you will need to monitor its memory consumption and take action before too many resources are consumed.

The last setting I want to talk about is probably the most important for using Redis.

The server will periodically fork and asynchronously dump the current contents of the database to disk. The dump is actually made to a temporary file and then moved to replace any older dump, so the operation is atomic and won't leave you with a partially dumped database. If Redis is eventually shutdown and reloaded, it will restore from this dump file.

How often it dumps the keys is configureable by the amount of time that passes and the number of changes that have been made to the data. For example, the following settings tell Redis to dump the database after 60 seconds if 100 changes have been made or after five minutes if there has been at least 1 change:

save 300 1
save 60 100

As you can see, you can set several different conditions. As soon as any one line of conditions matches, meaning both the time and the changes much match, the database is dumped and both counts restart.

Note that the time condition can be met before the changes. This means that, using the settings above, I can launch a Redis server, let it sit for five or more minutes, and then change a single key to trigger an immediate dump. The time will have already passed and as soon as I make the changes condition true as well, that is enough. In other words, I don't have to wait five minutes after I make the change.

That covers plenty about installing and running the Redis server. You are now all set to play with it.

Comments (11)
  1. Marcelo Barbudas
    Marcelo Barbudas September 14th, 2009 Reply Link

    Why Redis? Why not Cassandra for instance?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II September 14th, 2009 Reply Link

      There are a lot of key-value stores out there. I'm just showing the tools I have played with and found good uses for so far.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  2. Kuroki Kaze
    Kuroki Kaze January 20th, 2010 Reply Link

    I want to mention that by default Redis starts in foreground mode and constantly spamming the console with usage statistics. To fix that, change demonize option in redis.conf file to yes.

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  3. sar
    sar August 10th, 2010 Reply Link

    How do I uninstall the redis server?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II August 10th, 2010 Reply Link

      If you installed it using the instructions above, it's:

      sudo rm /usr/local/bin/redis-*
      sudo gem uninstall redis
      
      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  4. Sal Uryasev
    Sal Uryasev December 2nd, 2010 Reply Link

    This is a good recipe. Thanks for your help!

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  5. sarthak
    sarthak March 2nd, 2011 Reply Link

    thanks for posting this...!!

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
  6. expert
    expert March 12th, 2011 Reply Link

    Is there a way to use Redis as a pure cache server?
    or simply change the settings for save?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II March 12th, 2011 Reply Link

      Sure, if you don't add any save lines to your redis.conf file, the server will not perform background saves.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
  7. Hengjie
    Hengjie April 17th, 2012 Reply Link

    You mentioned:

    Note that the above setting is really only a good idea when using Redis as a cache. If you are using it as a general database, you will need to monitor its memory consumption and take action before too many resources are consumed.

    Is there anyway you can monitor the memory limit? If so, how can I?

    1. Reply (using GitHub Flavored Markdown)

      Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

      Ajax loader
    2. James Edward Gray II
      James Edward Gray II April 17th, 2012 Reply Link

      Sure. A tool like Monit could handle this for you.

      1. Reply (using GitHub Flavored Markdown)

        Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

        Ajax loader
Leave a Comment (using GitHub Flavored Markdown)

Comments on this blog are moderated. Spam is removed, formatting is fixed, and there's a zero tolerance policy on intolerance.

Ajax loader