Rectangle 27 18

AWS has integrated Elastic Beanstalk (EB) with the EC2 Container Service (ECS) to support multi-container Docker environments. An EB environment composed of multiple instances in an autoscaling group can run multiple containers per instance, managed through the ECS agent and its API. Use .ebextensions to map multiple ELB listeners to the containers running on EC2 instances.

See this tutorial for running an Nginx + PHP-FPM in separate containers.

Thanks Ben! Your blog post is a very interesting read (actually I got stuck on your website reading all the other stuff as well ;) I didn't expect such severe limitations of Beanstalk, for now I am going to go ahead and just run Docker containers inside a small EC2 instance. I'm curious how well that will run.

What if you were just running ONE high-load application that just ran on a single container and connected to an RDS database. In that scenario, would you just run a single docker instance, or a multiple-container docker instance to somehow maximize resources?

Amazon Elastic Beanstalk vs EC2 instance with docker containers - Stac...

amazon-web-services amazon-ec2 docker elastic-beanstalk
Rectangle 27 44

EB vs ECS really comes down to control. Do you want to control your scaling and capacity or do you want to have that more abstracted and instead focus primarily on your app. ECS will give you control, as you have to specify the size and number of nodes in the cluster and whether or not auto-scaling should be used. With EB, you simply provide a Dockerfile and EB takes care of scaling your provisioning of number and size of nodes, you basically can forget about the infrastructure with the EB route.

With ECS you'll have to build the infrastructure first before you can start deploying the the Dockerfile so it really comes down to 1) your familiarity with infrastructure and 2) level of effort that you want to spend on the infrastructure vs the app.

Yes, but how does the auto-scaling mechanism of both services work, does Elastic Beanstalk scale the containers, per Container, so if only one is under load, it scales only this one, or does it always scale up the instances and start all containers, no matter under what load they are ?

Now that ECS is GA, EB leverages ECS to provide it's multi-container infrastructure. The auto-scaling is done using the typical EC2 auto scaling group primitive. The triggering factor to scale up or down is not the container however but the instance node. I.e. if network interface traffic, or CPU load or disk load reach a certain threshold, then the cluster can scale out or in. So in your example, if the 5th container's node was under heavy CPU load you could set an auto scaling group trigger based on that.

jboss - Should I use AWS Elastic Beanstalk or the Amazon EC2 Container...

amazon-web-services jboss docker amazon-elastic-beanstalk amazon-ecs
Rectangle 27 2

Yes your assumptions are correct. The difference is that for doing it on EC2 you will have to manage and monitor your containers all by yourself. So I guess for production environments you'll have to have your own team for that, or else go with BeanStalk.

Amazon Elastic Beanstalk vs EC2 instance with docker containers - Stac...

amazon-web-services amazon-ec2 docker elastic-beanstalk
Rectangle 27 258

The bottom line is you should almost always use EBS backed instances.

  • EBS backed instances can be set so that they cannot be (accidentally) terminated through the API.
  • EBS backed instances can be stopped when you're not using them and resumed when you need them again (like pausing a Virtual PC), at least with my usage patterns saving much more money than I spend on a few dozen GB of EBS storage.
  • EBS backed instances don't lose their instance storage when they crash (not a requirement for all users, but makes recovery much faster)
  • You can dynamically resize EBS instance storage.
  • You can transfer the EBS instance storage to a brand new instance (useful if the hardware at Amazon you were running on gets flaky or dies, which does happen from time to time)
  • It is faster to launch an EBS backed instance because the image does not have to be fetched from S3.
  • If the hardware your EBS-backed instance is scheduled for maintenance, stopping and starting the instance automatically migrates to new hardware. I was also able to move an EBS-backed instance on failed hardware by force-stopping the instance and launching it again (your mileage may vary on failed hardware).

I'm a heavy user of Amazon and switched all of my instances to EBS backed storage as soon as the technology came out of beta. I've been very happy with the result.

EBS can still fail - not a silver bullet

Keep in mind that any piece of cloud-based infrastructure can fail at any time. Plan your infrastructure accordingly. While EBS-backed instances provide certain level of durability compared to ephemeral storage instances, they can and do fail. Have an AMI from which you can launch new instances as needed in any availability zone, back up your important data (e.g. databases), and if your budget allows it, run multiple instances of servers for load balancing and redundancy (ideally in multiple availability zones).

At some points in time, it may be cheaper to achieve faster IO on Instance Store instances. There was a time when it was certainly true. Now there are many options for EBS storage, catering to many needs. The options and their pricing evolve constantly as technology changes. If you have a significant amount of instances that are truly disposable (they don't affect your business much if they just go away), do the math on cost vs. performance. EBS-backed instances can also die at any point in time, but my practical experience is that EBS is more durable.

Yes, the above were my thoughts as well... Hopefully somehow here writes about their preferences for instance-store as a comparison...

Instance store backed EC2 can also be set to not accidentally terminate.

I'm actually switching most of my EBS backed EC2 instances to using instance stores. It really depends on what you want to achieve. I'm switching because of better IO and because I view each EC2 instance as disposable at all moments, or: it will break down any minute and I will lose everything that's on such an instance. Architecting that way helps to get a real HA system. See also stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html

@Jim: At least when I wrote the answer a year ago, you got much better IO by striping a number of EBS instances into a software RAID configuration than using instance storage. It's also much faster to launch a replacement instance from EBS backing than from S3 backing (instance storage is loaded from S3, which can be slow). I have not done much on AWS the last 6 months or so; things may have changed.

Seems a little lop-sided -- though it's possible to run EBS-Backed instances and maintain a heavy emphasis on recyclability, I think that having newcomers looking at this post and subsequently creating EBS-Backed instances is dangerous because they likely will not maintain that same emphasis on recyclability, which is perhaps the most crucial component of any cloud infrastructure. And the good majority of people looking at this are sure to be new to this stuff

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 258

The bottom line is you should almost always use EBS backed instances.

  • EBS backed instances can be set so that they cannot be (accidentally) terminated through the API.
  • EBS backed instances can be stopped when you're not using them and resumed when you need them again (like pausing a Virtual PC), at least with my usage patterns saving much more money than I spend on a few dozen GB of EBS storage.
  • EBS backed instances don't lose their instance storage when they crash (not a requirement for all users, but makes recovery much faster)
  • You can dynamically resize EBS instance storage.
  • You can transfer the EBS instance storage to a brand new instance (useful if the hardware at Amazon you were running on gets flaky or dies, which does happen from time to time)
  • It is faster to launch an EBS backed instance because the image does not have to be fetched from S3.
  • If the hardware your EBS-backed instance is scheduled for maintenance, stopping and starting the instance automatically migrates to new hardware. I was also able to move an EBS-backed instance on failed hardware by force-stopping the instance and launching it again (your mileage may vary on failed hardware).

I'm a heavy user of Amazon and switched all of my instances to EBS backed storage as soon as the technology came out of beta. I've been very happy with the result.

EBS can still fail - not a silver bullet

Keep in mind that any piece of cloud-based infrastructure can fail at any time. Plan your infrastructure accordingly. While EBS-backed instances provide certain level of durability compared to ephemeral storage instances, they can and do fail. Have an AMI from which you can launch new instances as needed in any availability zone, back up your important data (e.g. databases), and if your budget allows it, run multiple instances of servers for load balancing and redundancy (ideally in multiple availability zones).

At some points in time, it may be cheaper to achieve faster IO on Instance Store instances. There was a time when it was certainly true. Now there are many options for EBS storage, catering to many needs. The options and their pricing evolve constantly as technology changes. If you have a significant amount of instances that are truly disposable (they don't affect your business much if they just go away), do the math on cost vs. performance. EBS-backed instances can also die at any point in time, but my practical experience is that EBS is more durable.

Yes, the above were my thoughts as well... Hopefully somehow here writes about their preferences for instance-store as a comparison...

Instance store backed EC2 can also be set to not accidentally terminate.

I'm actually switching most of my EBS backed EC2 instances to using instance stores. It really depends on what you want to achieve. I'm switching because of better IO and because I view each EC2 instance as disposable at all moments, or: it will break down any minute and I will lose everything that's on such an instance. Architecting that way helps to get a real HA system. See also stu.mp/2011/04/the-cloud-is-not-a-silver-bullet.html

@Jim: At least when I wrote the answer a year ago, you got much better IO by striping a number of EBS instances into a software RAID configuration than using instance storage. It's also much faster to launch a replacement instance from EBS backing than from S3 backing (instance storage is loaded from S3, which can be slow). I have not done much on AWS the last 6 months or so; things may have changed.

Seems a little lop-sided -- though it's possible to run EBS-Backed instances and maintain a heavy emphasis on recyclability, I think that having newcomers looking at this post and subsequently creating EBS-Backed instances is dangerous because they likely will not maintain that same emphasis on recyclability, which is perhaps the most crucial component of any cloud infrastructure. And the good majority of people looking at this are sure to be new to this stuff

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 1

This is the main discussion of comparing Memcached and Redis Memcached vs. Redis? Both AWS and Azure for sure will upgrade in the future to the newer versions of Redis, but when and how they will roll out it will depend only on them. Meanwhile you could install Redis 3.0.2 yourself, but you need to see if you really need Redis 3 which actually gives you the cluster support. And if you don't need the cluster then you can go with 2.8 from Elasticache.

amazon web services - Using memcached or Redis on aws-elasticache - St...

amazon-web-services redis memcached
Rectangle 27 1

This is the main discussion of comparing Memcached and Redis Memcached vs. Redis? Both AWS and Azure for sure will upgrade in the future to the newer versions of Redis, but when and how they will roll out it will depend only on them. Meanwhile you could install Redis 3.0.2 yourself, but you need to see if you really need Redis 3 which actually gives you the cluster support. And if you don't need the cluster then you can go with 2.8 from Elasticache.

amazon web services - Using memcached or Redis on aws-elasticache - St...

amazon-web-services redis memcached
Rectangle 27 62

99% of our AWS setup is recyclable. So for me it doesn't really matter if I terminate an instance -- nothing is lost ever. E.g. my application is automatically deployed on an instance from SVN, our logs are written to a central syslog server.

The only benefit of instance storage that I see are cost-savings. Otherwise EBS-backed instances win. Eric mentioned all the advantages.

I haven't had any good experience with EBS-backed instances in the past year or so. The last downtimes on AWS pretty much wrecked EBS as well.

I am guessing that a service like RDS uses some kind of EBS as well and that seems to work for the most part. On the instances we manage ourselves, we have got rid off EBS where possible.

Getting rid to an extend where we moved a database cluster back to iron (= real hardware). The only remaining piece in our infrastructure is a DB server where we stripe multiple EBS volumes into a software RAID and backup twice a day. Whatever would be lost in between backups, we can live with.

EBS is a somewhat flakey technology since it's essentially a network volume: a volume attached to your server from remote. I am not negating the work done with it it is an amazing product since essentially unlimited persistent storage is just an API call away. But it's hardly fit for scenarios where I/O performance is key.

And in addition to how network storage behaves, all network is shared on EC2 instances. The smaller an instance (e.g. t1.micro, m1.small) the worse it gets because your network interfaces on the actual host system are shared among multiple VMs (= your EC2 instance) which run on top of it.

The larger instance you get, the better it gets of course. Better here means within reason.

When persistence is required, I would always advice people to use something like S3 to centralize between instances. S3 is a very stable service. Then automate your instance setup to a point where you can boot a new server and it gets ready by itself. Then there is no need to have network storage which lives longer than the instance.

So all in all, I see no benefit to EBS-backed instances what so ever. I rather add a minute to bootstrap, then run with a potential SPOF.

Is there any significant improvement of IO performance with EBS IOPS-kind of volumes compared to standard? Supposing, the above said holds for EBS IOPS volumes, as well.

Both technologies evolve. I'm wirting this comment in 2014, when I have "Provisioned IOPS" EBS, but - the "instance store" is now SSD, which is even faster than before!! Ephemeral storage will always win in terms of speed. So I use both - keep the "persistent" stuff on EBS, having all the temp files, logs, "TempDB" database, swap-file and other stuff on Instance-store. BENEFIT FROM BOTH!

What if you needed a distributed database which needs to store its data in a distributed and persistent manner. Wouldn't you need EBS because instance storage is not persistent?

@CMCDragonkai Of course you do. There are a lot of options these days, e.g. AWS started offering SSD-based storage. I would look into those and re-do the analysis (single vs. RAID, etc.). I would also look into getting the biggest instances possible because of network throughput. EBS is still an issue on instances like t1.micro.

The part of this answer about network performance is fairly obsolete - for quite a while now, there have existed a variety of instances that can be "EBS-optimized" at a small extra cost, and some that are such by default (with no surcharge), which have dedicated network interfaces towards EBS, cf. docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 62

99% of our AWS setup is recyclable. So for me it doesn't really matter if I terminate an instance -- nothing is lost ever. E.g. my application is automatically deployed on an instance from SVN, our logs are written to a central syslog server.

The only benefit of instance storage that I see are cost-savings. Otherwise EBS-backed instances win. Eric mentioned all the advantages.

I haven't had any good experience with EBS-backed instances in the past year or so. The last downtimes on AWS pretty much wrecked EBS as well.

I am guessing that a service like RDS uses some kind of EBS as well and that seems to work for the most part. On the instances we manage ourselves, we have got rid off EBS where possible.

Getting rid to an extend where we moved a database cluster back to iron (= real hardware). The only remaining piece in our infrastructure is a DB server where we stripe multiple EBS volumes into a software RAID and backup twice a day. Whatever would be lost in between backups, we can live with.

EBS is a somewhat flakey technology since it's essentially a network volume: a volume attached to your server from remote. I am not negating the work done with it it is an amazing product since essentially unlimited persistent storage is just an API call away. But it's hardly fit for scenarios where I/O performance is key.

And in addition to how network storage behaves, all network is shared on EC2 instances. The smaller an instance (e.g. t1.micro, m1.small) the worse it gets because your network interfaces on the actual host system are shared among multiple VMs (= your EC2 instance) which run on top of it.

The larger instance you get, the better it gets of course. Better here means within reason.

When persistence is required, I would always advice people to use something like S3 to centralize between instances. S3 is a very stable service. Then automate your instance setup to a point where you can boot a new server and it gets ready by itself. Then there is no need to have network storage which lives longer than the instance.

So all in all, I see no benefit to EBS-backed instances what so ever. I rather add a minute to bootstrap, then run with a potential SPOF.

Is there any significant improvement of IO performance with EBS IOPS-kind of volumes compared to standard? Supposing, the above said holds for EBS IOPS volumes, as well.

Both technologies evolve. I'm wirting this comment in 2014, when I have "Provisioned IOPS" EBS, but - the "instance store" is now SSD, which is even faster than before!! Ephemeral storage will always win in terms of speed. So I use both - keep the "persistent" stuff on EBS, having all the temp files, logs, "TempDB" database, swap-file and other stuff on Instance-store. BENEFIT FROM BOTH!

What if you needed a distributed database which needs to store its data in a distributed and persistent manner. Wouldn't you need EBS because instance storage is not persistent?

@CMCDragonkai Of course you do. There are a lot of options these days, e.g. AWS started offering SSD-based storage. I would look into those and re-do the analysis (single vs. RAID, etc.). I would also look into getting the biggest instances possible because of network throughput. EBS is still an issue on instances like t1.micro.

The part of this answer about network performance is fairly obsolete - for quite a while now, there have existed a variety of instances that can be "EBS-optimized" at a small extra cost, and some that are such by default (with no surcharge), which have dedicated network interfaces towards EBS, cf. docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 33

We like instance-store. It forces us to make our instances completely recyclable, and we can easily automate the process of building a server from scratch on a given AMI. This also means we can easily swap out AMIs. Also, EBS still has performance problems from time to time.

Netflix makes the same recommendations as well.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 33

We like instance-store. It forces us to make our instances completely recyclable, and we can easily automate the process of building a server from scratch on a given AMI. This also means we can easily swap out AMIs. Also, EBS still has performance problems from time to time.

Netflix makes the same recommendations as well.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 1

For keeping your webserver sessions in sync you can easily switch to Redis or Memcached as your session handler. This is a simple setting in the PHP.ini and they can all access the same Redis or Memcached server to do sessions. You can use Amazon's Elasticache which will manage the Redis or Memcache instance for you.

You could use Git. When you deploy you can setup multiple servers and it will push your branch (master) to the multiple servers. So every new build goes out to all webserver.

What about new machines that launch?

I would setup new machines to run a rsync script from a trusted source, your master web server. That way they sync their web folders with the master when they boot and would be identical even if the AMI had old web files in it.

What about files that change and need to be live updated?

Store any user uploaded files in S3. So if user uploads a document on Server 1 then the file is stored in s3 and location is stored in a database. Then if a different user is on server 2 he can see the same file and access it as if it was on server 2. The file would be retrieved from s3 and served to the client.

amazon web services - How to setup shared persistent storage for multi...

amazon-web-services amazon-ec2
Rectangle 27 16

Eric pretty much nailed it. We (Bitnami) are a popular provider of free AMIs for popular applications and development frameworks (PHP, Joomla, Drupal, you get the idea). I can tell you that EBS-backed AMIs are significantly more popular than S3-backed. In general I think s3-backed instances are used for distributed, time-limited jobs (for example, large scale processing of data) where if one machine fails, another one is simply spinned up. EBS-backed AMIS tend to be used for 'traditional' server tasks, such as web or database servers that keep state locally and thus require the data to be available in the case of crashing.

One aspect I did not see mentioned is the fact that you can take snapshots of an EBS-backed instance while running, effectively allowing you to have very cost-effective backups of your infrastructure (the snapshots are block-based and incremental)

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 16

Eric pretty much nailed it. We (Bitnami) are a popular provider of free AMIs for popular applications and development frameworks (PHP, Joomla, Drupal, you get the idea). I can tell you that EBS-backed AMIs are significantly more popular than S3-backed. In general I think s3-backed instances are used for distributed, time-limited jobs (for example, large scale processing of data) where if one machine fails, another one is simply spinned up. EBS-backed AMIS tend to be used for 'traditional' server tasks, such as web or database servers that keep state locally and thus require the data to be available in the case of crashing.

One aspect I did not see mentioned is the fact that you can take snapshots of an EBS-backed instance while running, effectively allowing you to have very cost-effective backups of your infrastructure (the snapshots are block-based and incremental)

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 14

I've had the exact same experience as Eric at my last position. Now in my new job, I'm going through the same process I performed at my last job... rebuilding all their AMIs for EBS backed instances - and possibly as 32bit machines (cheaper - but can't use same AMI on 32 and 64 machines).

EBS backed instances launch quickly enough that you can begin to make use of the Amazon AutoScaling API which lets you use CloudWatch metrics to trigger the launch of additional instances and register them to the ELB (Elastic Load Balancer), and also to shut them down when no longer required.

This kind of dynamic autoscaling is what AWS is all about - where the real savings in IT infrastructure can come into play. It's pretty much impossible to do autoscaling right with the old s3 "InstanceStore"-backed instances.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 14

I've had the exact same experience as Eric at my last position. Now in my new job, I'm going through the same process I performed at my last job... rebuilding all their AMIs for EBS backed instances - and possibly as 32bit machines (cheaper - but can't use same AMI on 32 and 64 machines).

EBS backed instances launch quickly enough that you can begin to make use of the Amazon AutoScaling API which lets you use CloudWatch metrics to trigger the launch of additional instances and register them to the ELB (Elastic Load Balancer), and also to shut them down when no longer required.

This kind of dynamic autoscaling is what AWS is all about - where the real savings in IT infrastructure can come into play. It's pretty much impossible to do autoscaling right with the old s3 "InstanceStore"-backed instances.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 1378

Bitmap and HyperLogLog

Redis is more powerful, more popular, and better supported than memcached. Memcached can only do a small fraction of the things Redis can do. Redis is better even where their features overlap.

Both tools are powerful, fast, in-memory data stores that are useful as a cache. Both can help speed up your application by caching database results, HTML fragments, or anything else that might be expensive to generate.

When used for the same thing, here is how they compare using the original question's "Points to Consider":

  • Read/write speed: Both are extremely fast. Benchmarks vary by workload, versions, and many other factors but generally show redis to be as fast or almost as fast as memcached. I recommend redis, but not because memcached is slow. It's not.
  • memcached: You specify the cache size and as you insert items the daemon quickly grows to a little more than this size. There is never really a way to reclaim any of that space, short of restarting memcached. All your keys could be expired, you could flush the database, and it would still use the full chunk of RAM you configured it with.
  • I stored 100,000 ~2KB strings (~200MB) of random sentences into both. Memcached RAM usage grew to ~225MB. Redis RAM usage grew to ~228MB. After flushing both, redis dropped to ~29MB and memcached stayed at ~225MB. They are similarly efficient in how they store data, but only one is capable of reclaiming it.
  • Disk I/O dumping: A clear win for redis since it does this by default and has very configurable persistence. Memcached has no mechanisms for dumping to disk without 3rd party tools.
  • Scaling: Both give you tons of headroom before you need more than a single instance as a cache. Redis includes tools to help you go beyond that while memcached does not.

Memcached is a simple volatile cache server. It allows you to store key/value pairs where the value is limited to being a string up to 1MB.

If you need high performance or high availability there are 3rd party tools, products, and services available.

Redis can do the same jobs as memcached can, and can do them better.

Redis can act as a cache as well. It can store key/value pairs too. In redis they can even be up to 512MB.

You can turn off persistence and it will happily lose your data on restart too. If you want your cache to survive restarts it lets you do that as well. In fact, that's the default.

If one instance of redis/memcached isn't enough performance for your workload, redis is the clear choice. Redis includes cluster support and comes with high availability tools (redis-sentinel) right "in the box". Over the past few years redis has also emerged as the clear leader in 3rd party tooling. Companies like Redis Labs, Amazon, and others offer many useful redis tools and services. The ecosystem around redis is much larger. The number of large scale deployments is now likely greater than for memcached.

Redis is more than a cache. It is an in-memory data structure server. Below you will find a quick overview of things Redis can do beyond being a simple key/value cache like memcached. Most of redis' features are things memcached cannot do.

Redis is better documented than memcached. While this can be subjective, it seems to be more and more true all the time.

redis.io is a fantastic easily navigated resource. It lets you try redis in the browser and even gives you live interactive examples with each command in the docs.

There are now 2x as many stackoverflow results for redis as memcached. 2x as many Google results. More readily accessible examples in more languages. More active development. More active client development. These measurements might not mean much individually, but in combination they paint a clear picture that support and documentation for redis is greater and much more up-to-date.

By default redis persists your data to disk using a mechanism called snapshotting. If you have enough RAM available it's able to write all of your data to disk with almost no performance degradation. It's almost free!

In snapshot mode there is a chance that a sudden crash could result in a small amount of lost data. If you absolutely need to make sure no data is ever lost, don't worry, redis has your back there too with AOF (Append Only File) mode. In this persistence mode data can be synced to disk as it is written. This can reduce maximum write throughput to however fast your disk can write, but should still be quite fast.

There are many configuration options to fine tune persistence if you need, but the defaults are very sensible. These options make it easy to setup redis as a safe, redundant place to store data. It is a real database.

Memcached is limited to strings, but Redis is a data structure server that can serve up many different data types. It also provides the commands you need to make the most of those data types.

Simple text or binary values that can be up to 512MB in size. This is the only data type redis and memcached share, though memcached strings are limited to 1MB.

Redis gives you more tools for leveraging this datatype by offering commands for bitwise operations, bit-level manipulation, floating point increment/decrement support, range queries, and multi-key operations. Memcached doesn't support any of that.

Strings are useful for all sorts of use cases, which is why memcached is fairly useful with this data type alone.

Hashes are sort of like a key value store within a key value store. They map between string fields and string values. Field->value maps using a hash are slightly more space efficient than key->value maps using regular strings.

Hashes are useful as a namespace, or when you want to logically group many keys. With a hash you can grab all the members efficiently, expire all the members together, delete all the members together, etc. Great for any use case where you have several key/value pairs that need to grouped.

One example use of a hash is for storing user profiles between applications. A redis hash stored with the user ID as the key will allow you to store as many bits of data about a user as needed while keeping them stored under a single key. The advantage of using a hash instead of serializing the profile into a string is that you can have different applications read/write different fields within the user profile without having to worry about one app overriding changes made by others (which can happen if you serialize stale data).

Redis lists are ordered collections of strings. They are optimized for inserting, reading, or removing values from the top or bottom (aka: left or right) of the list.

Lists make great durable, atomic, queues. These work great for job queues, logs, buffers, and many other use cases.

Sets are unordered collections of unique values. They are optimized to let you quickly check if a value is in the set, quickly add/remove values, and to measure overlap with other sets.

These are great for things like access control lists, unique visitor trackers, and many other things. Most programming languages have something similar (usually called a Set). This is like that, only distributed.

Redis provides several commands to manage sets. Obvious ones like adding, removing, and checking the set are present. So are less obvious commands like popping/reading a random item and commands for performing unions and intersections with other sets.

Sorted Sets are also collections of unique values. These ones, as the name implies, are ordered. They are ordered by a score, then lexicographically.

This data type is optimized for quick lookups by score. Getting the highest, lowest, or any range of values in between is extremely fast.

If you add users to a sorted set along with their high score, you have yourself a perfect leader-board. As new high scores come in, just add them to the set again with their high score and it will re-order your leader-board. Also great for keeping track of the last time users visited and who is active in your application.

Storing values with the same score causes them to be ordered lexicographically (think alphabetically). This can be useful for things like auto-complete features.

Many of the sorted set commands are similar to commands for sets, sometimes with an additional score parameter. Also included are commands for managing scores and querying by score.

Redis has several commands for storing, retrieving, and measuring geographic data. This includes radius queries and measuring distances between points.

Technically geographic data in redis is stored within sorted sets, so this isn't a truly separate data type. It is more of an extension on top of sorted sets.

Like geo, these aren't completely separate data types. These are commands that allow you to treat string data as if it's either a bitmap or a hyperloglog.

Bitmaps are what the bit-level operators I referenced under Strings are for. This data type was the basic building block for reddit's recent collaborative art project: r/Place.

HyperLogLog allows you to use a constant extremely small amount of space to count almost unlimited unique values with shocking accuracy. Using only ~16KB you could efficiently count the number of unique visitors to your site, even if that number is in the millions.

Commands in redis are atomic, meaning you can be sure that as soon as you write a value to redis that value is visible to all clients connected to redis. There is no wait for that value to propagate. Technically memcached is atomic as well, but with redis adding all this functionality beyond memcached it is worth noting and somewhat impressive that all these additional data types and features are also atomic.

While not quite the same as transactions in relational databases, redis also has transactions that use "optimistic locking" (WATCH/MULTI/EXEC).

Redis provides a feature called 'pipelining'. If you have many redis commands you want to execute you can use pipelining to send them to redis all-at-once instead of one-at-a-time.

Normally when you execute a command to either redis or memcached, each command is a separate request/response cycle. With pipelining, redis can buffer several commands and execute them all at once, responding with all of the responses to all of your commands in a single reply.

This can allow you to achieve even greater throughput on bulk importing or other actions that involve lots of commands.

Redis has commands dedicated to pub/sub functionality, allowing redis to act as a high speed message broadcaster. This allows a single client to publish messages to many other clients connected to a channel.

Redis does pub/sub as well as almost any tool. Dedicated message brokers like RabbitMQ may have advantages in certain areas, but the fact that the same server can also give you persistent durable queues and other data structures your pub/sub workloads likely need, Redis will often prove to be the best and most simple tool for the job.

You can kind of think of lua scripts like redis's own SQL or stored procedures. It's both more and less than that, but the analogy mostly works.

Maybe you have complex calculations you want redis to perform. Maybe you can't afford to have your transactions roll back and need guarantees every step of a complex process will happen atomically. These problems and many more can be solved with lua scripting.

The entire script is executed atomically, so if you can fit your logic into a lua script you can often avoid messing with optimistic locking transactions.

As mentioned above, redis includes built in support for clustering and is bundled with its own high availability tool called redis-sentinel.

Without hesitation I would recommend redis over memcached for any new projects, or existing projects that don't already use memcached.

The above may sound like I don't like memcached. On the contrary: it is a powerful, simple, stable, mature, and hardened tool. There are even some use cases where it's a little faster than redis. I love memcached. I just don't think it makes much sense for future development.

Redis does everything memcached does, often better. Any performance advantage for memcached is minor and workload specific. There are also workloads for which redis will be faster, and many more workloads that redis can do which memcached simply can't. The tiny performance differences seem minor in the face of the giant gulf in functionality and the fact that both tools are so fast and efficient they may very well be the last piece of your infrastructure you'll ever have to worry about scaling.

There is only one scenario where memcached makes more sense: where memcached is already in use as a cache. If you are already caching with memcached then keep using it, if it meets your needs. It is likely not worth the effort to move to redis and if you are going to use redis just for caching it may not offer enough benefit to be worth your time. If memcached isn't meeting your needs, then you should probably move to redis. This is true whether you need to scale beyond memcached or you need additional functionality.

How does Memcached offer clustering in a way that exists in the server themselves? I've always used libraries that distributed to a pool of memcached servers using hashing algorithms or a modulus. The same is said for Redis. I mostly use Python and there seem to be quite a few modules that don't rely on the memcached library to handle connection pools.

"Transactions with optimistic locking (WATCH/MULTI/EXEC)" - Redis has no right transactions. I.e. if [multi, cmd1, cmd2, cmd3 (exception) , exec] then cmd1 and cmd2 will be executed.

@Oleg that is not actually true. If you use multi-exec the commands are buffered (ie: not executed) until the exec occurs, so if you have an exception before the exec then no commands are actually executed. If exec is called all the buffered commands are executed atomically, unless, of course, a watch variable has been changed since multi was first called. This latter mechanism is the optimistic locking part.

@whardier You're correct. Updated answer to reflect that memcached's cluster "support" is enabled by additional tools. Should have researched that better.

how about clustering with couchbase server? (memcached compatible)

caching - Memcached vs. Redis? - Stack Overflow

caching web-applications memcached redis
Rectangle 27 23

I'm using Visual Studio 2015 Preview to create ASP.NET 5 projects. I don't think that's difficult to deploy on IIS now. First publish your website by publishing it as file system in VS 2015 preview, then copy the published folder to your server, create an application in IIS and set the application folder to the wwwroot folder (rather than the root folder), that's all. Be aware, check if "Microsoft.AspNet.Server.IIS" exists in your website project.json before publishing it.

Edit: there should be a web.config in wwwroot folder, the content of web.config may be like this (with precompile option when publishing):

or like this (without precompile option):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <appSettings>
    <add key="kpm-package-path" value="..\approot\packages" />
    <add key="bootstrapper-version" value="1.0.0-beta1" />
    <add key="kre-package-path" value="..\approot\packages" />
    <add key="kre-version" value="1.0.0-beta1" />
    <add key="kre-clr" value="CoreCLR" />
    <add key="kre-app-base" value="..\approot\src\Rvc.PopUpSite" />
  </appSettings>
</configuration>

Please notice the value of kre-app-base. Occasionally its value is empty string after publishing in Visual Studio.

Hi Ricky, just tried as you said copy published folder to Amazon, so I have my site in C:\Inetpub\wwwroot\Blog on Amazon, created new site in IIS, pointed it to C:\Inetpub\wwwroot\Blog\wwwroot, but it's not working. I have "Microsoft.AspNet.Server.IIS" in roject json. Any Ideas what I'm doing wrong?

@Sergey just edited my post. I don't know if you can get notification of my edit, so added this comment. It would also be very helpful if you could post the error message you got, if any.

Hi Ricky, thx, now it's working. Faced another issue, after changing packages in project.json from *beta-1 to *rc-1, during publishing to local disk both versions are published, do you know how to remove old dependencies?

The kre-app-base value being empty was the root of all the problems I was having with this. Thanks!

ASP.NET 5 project hosting on IIS - Stack Overflow

asp.net asp.net-mvc iis amazon asp.net-core
Rectangle 27 10

I'm just starting to use EC2 myself so not an expert, but Amazon's own documentation says:

we recommend that you use the local instance store for temporary data and, for data requiring a higher level of durability, we recommend using Amazon EBS volumes or backing up the data to Amazon S3.

I do more data analysis than web hosting, so persistence doesn't matter as much to me as it might for a web site. Given the distinction made by Amazon itself, I wouldn't assume that EBS is right for everyone.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs
Rectangle 27 10

I'm just starting to use EC2 myself so not an expert, but Amazon's own documentation says:

we recommend that you use the local instance store for temporary data and, for data requiring a higher level of durability, we recommend using Amazon EBS volumes or backing up the data to Amazon S3.

I do more data analysis than web hosting, so persistence doesn't matter as much to me as it might for a web site. Given the distinction made by Amazon itself, I wouldn't assume that EBS is right for everyone.

amazon ec2 - Benefits of EBS vs. instance-store (and vice-versa) - Sta...

amazon-ec2 amazon-web-services amazon-ebs