In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.
We've been using it on HDP2 for about a month now. Everything works fairly well and it was super easy to setup because of YARN (and the work the Spark team put in).
You don't happen to work at Cloudera, do you? I noticed you have some submissions about Impala and Oracle being evil, which seems to be a pretty common view among the ex-Oracle DBAs there
I do happen to work at Cloudera (hence the Impala submissions), although I'm neither an ex-Oracle DBA nor a huge believer that they're evil. I really don't have a lot of first-hand experience with Oracle as a company - which as you'll see, is why my submission was actually a question about the community's perception and why that's a common view.
Hey, I've recently begun the interview process at Cloudera, would you mind sending me an email? My address is in my profile, and I'd love to ask you a couple questions.
Which areas would you say HDFS needs improvement the most? Just so you know, HDFS is still very actively developed, and keeps introducing features / improving functionality (e.g Native NFS, In-Memory Caching, Short Circuit Reads, High Availability, Namespace Federation, etc) on a pretty regular basis.
Feel free to suggest new features, or contribute to the project yourself.
I know of the JIRA, thanks. I'm aware that it is being actively developed, but I don't necessarily believe that activity is progress. Keeping a few different players in the mix helps keep everyone focused on progress.
