* Overestimating their need to scale - a powerful box with a cache-backed site can serve a LOT of hits. The vast majority of apps will never need to scale beyond this, don't kid yourself.

* Underestimating the amount of work and the cost of failure in maintaining data integrity at the application level - for all but the most simplistic applications you will end up writing a ton of code that replicates what a RDBMS does, except slower, with more bugs, and less generality.

* Underestimating the importance of data integrity - When your massive database with billions of records has subtle data integrity issues, how hard will it be to fix? My guess is that some truly nasty situations will arise and over time the hype will be tempered by the horror stories.

* Underestimating the constraints this puts on future development - Sure greenfield apps may seem like a great candidate for a document-oriented DB, but how often do apps look like the original version 1 year later, 2 years later, 5 years later. With a relational database your bets are hedged automatically. You can go in a million different directions with your data. The relational model gives you orders of magnitude more flexibility than complex algorithms designed for map-reduce-style scalability. If you don't need the scalability you are throwing an awful lot of flexibility out the window for nothing.

* Underestimating how relational their data really is - It's not just that developers understand the relational model, it's the fact that it's a rigorous theoretical model that actually models relationships in an academically complete way. Sure, real life RDBMSs don't live up to the theory, and sometimes things are too slow to be practical, in which case you fudge things as necessary or add layers of caching. But at the end of the day, the relational model can cover pretty much anything, where these hash-databases give you a small set of scalable functionality and some clever algorithms to accomplish a bunch of different things. The things you can accomplish are not comprehensive in a theoretical sense, they are just techniques that have proven useful for a number of applications recently. However the limitations are not as well defined as a RDBMS.

If I'm the first to explicate the underlying rule, can I name it after myself? Please? Pretty please with Lisp on it?

Doffing's Tenth Rule of Database Systems (with apologies to Phillip Greenspun):

"Any sufficiently complicated database management system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of a good RDBMS."

I think the scaling problems of relational databases a vastly overstated. Right now, using off the shelf kit, you could build a relational database handling 10,000 commits/sec and 100T of data, without using "sharding" or any nonsense like that. Too many people think MySQL and its limitations are representative of this technology; it isn't, not by a long way.

This is about as true as it is sad, but when your only exposure to relational databases is the least mature and featured one on the market, and everyone is too busy sharding and remaining oblivious about it, this is the kind of discussion you get.

I mean, when your DB can't join large result-sets quickly and efficiently in process, why care about the loss of joinability when you spread your data across several servers?

The best talk I ever saw in this area was by Paul Strong from eBay (they obviously have a very strong requirement for both scalability and consistency). He talked about how they eventually ripped out all transactions and stored procedures from the RDBMS layers and built their own giant, cross cluster, consistent system to make the zillion customer/auction problem finally managable.

For the less giant problems, I think Amazon could charge a premium for an API-managed RDBMS solution, they have all the tools for making one in place (as opposed to requiring you go build it yourself with EBS and EC2 nodes).

add-slave-node() ...

Scaling out to a dozen DB servers that master/slave their way to scalability is no fun, but it's solved. The problem is that you're currently required to do it yourself. It would rock to be able to outsource that to the cloud.

I want to toss my ASP.NET application up into the Microsoft Cloud, where it will figure out how many webservers it needs to spread itself across and how many database servers it needs to fire up to handle the load it's seeing. And I want it to pretend like it's a single webserver talking to a single DB instance on a single box.

Say what you will about Microsoft, but they have the skills to pull that off. I sure hope they're working on it.

Unfortunately brains don't scale, so you're better off using one huge brain, like Michael Stonebraker.

http://db.cs.yale.edu/hstore/

My comment: This makes me wonder if the solution shouldn't be looked for in a different direction. Instead of throwing out the RDBMs, why not give me one that gives me complete control of the query plan? That's the real stickler here, not the relations themselves, which are there whether you maintain them or not. If the DB can't guess up a good query plan, why not let me simply feed one to it?

Is there any DB that lets me have complete control over the query plan? (And even if there is, it's probably a half-hearted, unoptimized feature with little design effort poured behind it.)

Yes, even so you may have to give up 40-table joins, but it seems to me RDBMs, with more control given to the user, could still provide a lot of value without throwing out the baby too.

I think it's simply not the case that people are using key-value stores because the query plans that a typical DBMS optimizer chooses are suboptimal for the kinds of queries most web apps use.

You speak like good databases wont allow you to override the query-optimizer. Look up optimizer hints.

Most real database-systems worth using allows this, but it is discouraged because for most queries, most of the times, the statistics the DB has on existing indexes, data uniqueness, estimated size of result-sets etc etc provides a better basis for optimizing the query than anything you can come up with.

>> Is the Relational Database Doomed?

Ah, linkbait! Sorry, no sale. Using a (pre-compiled) stored procedure to do a key value lookup will likely be about as fast as a lookup in a key/value DB. In any case, it's easier to add a key/value lookup capability to an RDBMS than it is to add efficient query capabilities to a KVDB.

The article focuses on scaling, which is rarely an issue. KVDBs win on simplicity and flexibility (no need to pre-define a data model) and a better impedance match with OOPLs. RDBMSs crush on querying, which most (though not all) applications need.

Arrrgh! I meant that it might be accelerated by the pain of impedance matching to RDBMSs. Sorry, it was late at night.

These non-relational database structures are not a new or revolutionary idea. The same concepts have long been implemented in systems like LDAP, X.500, and document-oriented databases like Lotus Domino.

The real mystery is the recent trend towards completely reinventing the wheel. For instance, LDAP (the protocol) is incredibly efficient, standardized, and well supported on most development platforms. It provides a standard query / filter language, facilities for scoping, and data updates. There are standardized exchange formats, wide availability of OSS and commercial management tools, and multiple server implementations.

Yet, there seems to be a great interest in developing from scratch highly proprietary and non-portable alternatives that do exactly the same thing (or less). These systems ignore decades of research and lessons learned through practical implementation.

The real issue though is that direct comparison of the two models is inherently flawed. They each have their own strengths and weaknesses, each excelling in situations that the other falls short. Looking at them as complementary models is perhaps a better approach.

Mind you, the RDBMS could still make a comeback. The world is looking to non-relational because since 1998 or so, throwing multiple pieces of hardware and network capacity at the problem is cheaper than solving it all in one giant node or cluster. But maybe one day an off-the-shelf computer (perhaps with better concurrent programming techniques for multicore) will be able to handle all the queries you would ever want and have the bandwidth to match.

Also, just wondering; exactly how many organizations really need a distributed database? There are some top websites, like LinkedIn, that do it all in one giant node, and my guess is that even sites like Facebook have only a few hundred database shards, maybe a few thousand at most.

(i meant to say that i hope one word answers are doomed as well)

For a more complete list of RDBMS-killing science projects, try http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib... (discussion: https://hackernews.hn/item?id=440687 )

The relational model proposed by Codd[1] was formulated so that applications which adhere to it would continue to work and be free of inconsistencies when the underlying data is updated or reorganized. That may not sound like much but it's surprisingly useful when the data is important or valuable and needs to always be correct. This is also the whole point of all those irritating normal forms.

The relational model has a lot of shortcomings but it's not likely to be replaced by anything that doesn't address these issues.

That said, for some kinds of data it is definitely overkill. Religiously following the relational model for short lived stuff with little significance makes about as much sense as filling out your shopping lists in triplicate, storing permanent copies in a safe deposit box and requiring two signatures for every alteration the same way you might if you changing your will or a deed to your property.

[1] - "Relational Model of Data for Large Shared Data Banks" , http://www.cis.upenn.edu/~zives/03f/cis550/codd.pdf

"For all other requirements, you are probably best off with the good old RDBMS. So, is the relational database doomed? Clearly not. Well, not yet at least."

So the answer is no. The article doesn't do justice to the theoretical rigor of the relational database model. An online posting with some interesting discussion of technical trade-offs in popular terms and links to other posts can be found at

http://highscalability.com/paper-dynamo-amazon-s-highly-avai...

The reality is that 99% of web apps don't really want a RDBMS. There are no arbitrary queries to run, they just need to extract some objects from the database, interact with them, and render a web page. As of late, people have been using ORMs to make their relational database look like an object database. The problem is that the objects you get from an ORM don't work like real objects (try making a cyclical structure, or try `change-class`-ing the objects; doesn't work). This usually means you need to map the objects from your ORM into a better model of your app, adding complexity. (Some ORMs also screw up the relational part, making the data in the database nearly useless to any other apps. Oops.)

People are starting to realize that this model is a waste of their time, and they are using object databases instead. Now there is no ORM, they write their objects that the app interacts with, and those are persisted as needed. Suddenly half your app's code is gone, and it runs faster. That is why people are excited. Less code is always exciting.

This article is really heavy on key/value databases, probably because they are easy to understand. But really, key/value databases are like the assembly language of databases. You will have to do a lot of work to run your app on a key/value database. It is better to use something higher-level, like CouchDB or an object database. (For an OODB, I recommend KiokuDB. It has indexing (stolen from Postgres), and can store your data to BDB, a directory on disk, a RDBMS, Amazon SimpleDB, or CouchDB. This gives you a lot of flexibility.)

Well, I'll tell you what my reality has been. It's been that every time I started a project, it appeared like I just needed to extract some objects from the database. But very soon it turned out that there wasn't just one way to view that information. There were two or three important use cases that destroyed my preferred, supposedly "natural" view.

Now, I'm fully aware that sometimes one preferred view of the information you have is so dominant that you have to optimize for that case. The question, is how do you optimize? Do you hard code that particular view from the beginning, or do you normalize first and then layer an optimized view on top of it, e.g. in the form of caching or materialized views? RDBMS make it easy to do the latter.

You say "they really only handle one (small) problem space well -- modeling relational data."

I think being relational is not an a priori property of data and relational modelling is not a problem space. It's the other way around. First you have a problem space, then you decide how to model it, and then you get the data according to that model. Assuming that information already has a kind of "natural" model attached to it is a fallacy in my view.

Why not use a relational database for data that actually suits the relational model, and an object/hierarchical/keyvalue database for complete entities that are often mapped to objects in code? Just put each kind of data to a storage that makes sense.

If you start with a RDBMS, there are plenty of options of how to store the objects. You can just use plain files in the simplest case. Or a good dbm style database. Or if your objects are mostly read-only you can keep the data in the RDB and cache the big joins. Or if you're really stuck with the RDB only you can even store the key-value pairs in the RDB: just make a two-column table for run-time data and keep it separate from the actual relational data. Or keep everything in the RDB but make a live copy off the RDB when the user logs in to keep the runtime data accessible and then unpack it back to the RDB lazily or when the user logs out. There are pretty much endless possibilities depending on your business and search needs.

Ain't easy but non-trivial databases never are.

As a developer, I'd prefer one of two innovations: relational databases that scale better across machines or k/v databases that provide a familiar mechanism for ad-hoc querying and aggregation across keys.