For years we have been using SQL relational databases as the_ de facto_ standard backend for our applications, whether desktop apps or web apps, we were relying on relational databases as the only reliable solution for storing information that is searchable and consistent all the time. Enterprise-class database business was excelling because of the increasing need for a reliable consistent data store for mission critical businesses. But then everybody went to use those relational enterprise-grade databases without really studying whether the application requirements fits the features provided by the database or not.
Relational databases were originally used for their ACID promise then people started using it for everything as a general purpose data store. Impedance mismatch problems started to get more attention when developers started to focus more on object oriented design, then ORMs came to action to help integrating the relational SQL database model into object oriented application-layer and most of you will start saying; that's when the hell started :)
So, what went wrong?
As the web is becoming more and more the platform, more users are poured into the web to use the web-based applications as a replacement to the old desktop applications they used to have, the more people are coming online, the more challenging application development is getting, because not only people are expecting higher quality web apps that exceed the quality of the desktop world, but also because even small applications are getting more users everyday. Application developers started suffering from scalability issues and it all started on the database layer!
The database is the bottleneck for most of today's applications, that said, scaling a SQL database is a complete nightmare, cost of maintenance and administration increase dramatically when you start thinking about replication, sharding, partitioning, and etc. You also suffer because in most SQL databases, sharding/partitioning require application-layer awareness which means, changing your code!
There were many trials to scale the SQL databases and some of them were partially successful, but again, that wasn't enough for the massively increasing load and amounts of data generated every day on the web, developers came to a very important conclusion, that SQL databases don't scale!
Developers eventually started to realise that the whole idea was wrong! Why sticking to SQL databases in the first place? Do we really need ACID properties? The answer is Yes and No!
Yes, because some of our applications depend on the atomic consistency and can be disastrous if operations were not atomically consistent, think of financial applications as the most direct example used in every database administration or development course. And No, because in more that 95% of our applications we are overrating the need for atomic consistency without noticing. Relaxing the ACID properties led to interesting concepts like eventual consistency as a proposed solution for the CAP theorem problem.
So, developers started designing more relaxed databases that didn't try to achieve Consistency, Availability, and Partitioning at the same time by simply relaxing one or more of the CAP parameters in their database design, as a result the NoSQL movement kept growing. Developers started abandoning SQL databases in favour of NoSQL alternatives, as they can really handle an order of magnitude of more data and most of them are truly scalable.
Some of the NoSQL databases have very interesting features, like in-memory databases with disk swapping and document-based schema-free model, some have capitalised on being key-value stores. I'll illustrate more the differences between them in later posts.
A few months ago I gave a quick presentation on the NoSQL movement and that series is going to complete what I've started isA.