This is actually a topic I love to talk about because I spent a lot of my time on S3A and the cloud FileSystem implementations. Fast metadata queries were actually a huge deal for query planning, and of course with performance there were a lot of potential surprises on S3. HBase was (unsurprisingly) heavily dependent on semantics that HDFS has but that are hard to get right on object storage, and required a couple of layers to be able to work properly on S3 (and even then - write-ahead logs were still on a small HDFS cluster last I heard). My biggest complaint about S3 was always eventual consistency (for which Hadoop developed a work-around - it originally employed a lot of worst-practices on S3 and suffered from eventual consistent A LOT) but now that S3 has much better consistency guarantees, I agree: it's incredibly hard to beat something that cheap.