Yes, if you already have k8s cluster that you don't manage, and you can use for your Spark, I would like to say you have Spark serverless environment.
IMO, where actual spark executor runs (either on lambda or k8s, or somewhere else) doesn't much matter to say it is serverless or not, while user can access the cluster, without managing them.
In our case, we build Spark Serverless service through
- Fully managed Kubernetes cluster
- Isolation between Spark instances (network, storage access, etc)
- A dedicated Kubernetes node (VM) allocation for a executor, to provide container level security and better performance
- Secure tunneling between remote Driver and executors for interactive mode (spark client deploy mode)
- Various optimizations
The addition of angularjs is going to power some really nice dashboards. Thanks for the feature and the detailed demos (and comments on github). Looking forward to playing with this a lot!
I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface.
I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook.
Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is under Apache Software Foundation and it is being developed in Apache way, from copyrights, development process, decision making process to community development.
Zeppelin is focusing on providing analytical environment on top of Hadoop eco-system. I'm not sure about iPython's direction, but i don't think it's the same to Zeppelin.
I see many projects that has notebook interface. Not only iPython and Zeppelin, but also Databricks Cloud, Spark Notebook, Beaker and many others.
I'm sure they all have their own advantages. Hope see all softwares are beloved by users.
I find it concerning when project developers don't seem to know exactly WHY the product exists or how it compares to competition on the market.
If I'm spending a lot of time working on a product that has a clear competitor(s), I'd expect to know exactly why my product is going to be better than those competitors (ie. in which ways), and what their weaknesses are.
A lot of open source developers go to great lengths to avoid admitting that anything competes with anything else. I don't know if it's just conflict avoidance, rationalizing failure, or what.
I can spend hours and hours to explain why Zeppelin is good and what is different feature compare to the other.
for example, Zeppelin has pluggable architecture so it does support not only scala but also python with built in Spark integration. Zeppelin not only support data exchange between scala and python environment but also SparkContext sharing for spark cluster resource utilization. It's got ability create rich interactive analytics GUI inside of notebook. It's got customizable layout system, and so on..
But I wanted to address more fundamental difference.
Most competitors are opensource project. And as a opensource project, how community works and project direction are the most important things. That'll make huge difference in the end. And i think i explained and compared them.
IMO, where actual spark executor runs (either on lambda or k8s, or somewhere else) doesn't much matter to say it is serverless or not, while user can access the cluster, without managing them.
In our case, we build Spark Serverless service through