This is really good. Simple perhaps, but I'm guessing it's just early.
When I'm done with my MSc project you guys should take a look. It loads in near real-time (within seconds of a change) SharePoint data into a SQL database in a form that allows for the differently structured data to be reported on. It's only a few weeks away from being written up but I'm going to be putting the code out there for anyone to use once it's done.
Anyhow an issue that you're going to have is data storage.
Ignoring the fact that enterprises will be very reticent to letting you have data, SMEs in Europe are going to be very reticent to let that data leave the legal boundary of Europe. There are lots of good laws in Europe that protects data so long as it's kept in Europe.
You should look at putting in place something that allows you to offer Europe only hosting of the data to counter that, a silo for a major legal domain.
That's the big problem I see, but in your favour not all companies will think of it (though enough will).
Another question I would have is: Storage is cheap but ongoing cloud storage is still expensive (relative to buying a couple of hard drives and a cheap machine vs you holding the data for a couple of years). Have you not thought of just offering the charting as a product companies can install locally? This also solves the legal boundary problem of data protection.
Also you should think of adding KPI charts that can be embedded into Word and PowerPoint docs. Little smart widget things that go and get the chart data from you as it changes.
We will definitely add the ability to embed charts. It is all done in javascript which means it can be viewed pretty much everywhere.
We are looking into a Firewall version much like Github has for big businesses, but we really want to build the product with the small and medium business in mind.
Definitely stay tuned as we iterate. There is a lot more to come!
Cool. If you want any pointers let me know. I've basically built this twice before for medium and large enterprises. Unfortunately the company I worked for didn't appreciate fully the general applicability of the product (they thought it only solved project management problems, only 1 of our customers fully understood it solved nearly all of their "first 80%" reporting needs), but it did allow data to be fetched from multiple backends (Excel, MySQL, SQL Server, Oracle, etc) and to be fed into one database that added snapshots over time (configurable granularity from 1 day up to anything).
To companies that had trouble getting their data together the sale was always easy as they appreciated the problem. But most didn't appreciate that problem so it's hard to sell on that basis.
The real value and the whizz-bang that sold it to most companies was how I presented the charts. I used SharePoint to create chart blocks, and these could be connected to other charts via a very simple connection framework.
Example: Drag one chart onto the page, configure it to look at some data. Drag another chart onto the page and configure it to look at some other data. Drag a filter onto the page and connect Chart A to Chart B via the filter and specify that Chart B's data is restricted to the selected series in Chart A.
What you then get is a multi-level drill-down and sets of building blocks that allow you to build any dashboard you want and to have pages that show really rich related charts.
Unfortunately the company I worked for only ever considered this as a project management solution even though it worked for all structured data from any source.
My latest toy (in my spare time) is how to do the same thing for semi-structured data and data that may be structured but not consistently. This is just because what I saw as constraints to reporting within larger companies wasn't the lack of data, but the lack of common edges that could power reporting.
Anyhow, I'll open source my work on SharePoint in a month or so, feel free to use whatever you need. It shows how to fetch data from multiple SharePoint systems in an eventual consistent way. And it also allows for storing data from multiple tables in the same few tables whatever the structure whilst being extremely efficient at aggregated reporting queries.
Because I'm going to forget the points I'm thinking this morning remind me to talk about this stuff when we speak in a month or so time:
\1 Don't just copy their tables and expect them to navigate the front-end, store data in a way that relates to how they work with the data (if it says "Risks" in their UI, then they expect to see a 'table' called Risks in their reporting) and it's often much better to de-normalise lots of inputs into the one way that reflects how they think about the data.
\2 Don't try and understand their data or to apply any surprises to the data. It must be predictable. Most people don't comprehend stats properly and if you aggregate or do anything that they cannot compare to their raw data and immediately comprehend then they reject the chart/report. It's been more successful and easier to chart dumb data in a predictable way than it is to try and understand the data and chart it in an intelligent way. Leave domain specific problems unless that domain is lucrative enough and you know it well enough (risk management fits in here).
\3 Storage of data is interesting. Storing snapshots gives trends and trends are valuable, on the other hand storing snapshots is expensive (multiple copies of data) and so a delta is preferred, but deltas are expensive. Alternatively, for low maturity and 'free' accounts your could just do 'Now' data and nothing else as it has the lowest storage and processing requirements.
\4 Most business data is still in Excel within the SME world. Tech startups aside (where most stuff is in databases and Google Docs), just cleaning up the Excel so that it's usable is fun on it's own.
\5 Whilst everyone knows you can't chart TEXT fields, good reporting leads to the asking of more questions as one question begets another. Eventually this leads to root cause analysis and drilling down to the few exceptions. When they finally get down to knowing the n items that are the cause they really do want to see that full data in the one place (the reports). Unfortunately this means cleaning text as you wouldn't believe the weird and wonderful ways that people find to enter text (if it can be written in PowerPoint, pasted into Word and copied into whatever datastore that they are using - it will be).
\6 Relationships between data. The more these can be understood the better. Most places aren't good at this, tools which can analyse even just the names of fields and suggest possible relationships are good things. Using these relationships to build filters and drill-downs is really cool stuff that sells it easy.
\7 Tufte. Go read everything and excel at producing the cleanest and easiest to read diagrams and charts. Everyone wants their presentation to look cool, everyone wants to look at a sexy dashboard and understand the problem... the whole world is tired of Excel charting, but Google charting is also ugly... make the charts pretty as this the bit the user sees most of the time and is what they will most readily pay for.
Those are more talking points... an immediate brain dump. I'll try and give you a more structured brain dump in a month or so.
There are only a few key problems:
1) Aggregating multiple stores of data to create the high level reports.
2) Relationships between data.
3) Deciding on how to store stuff so you can trend.
Speaking as someone who's had to write these kinds of tools internally it's nice to see more offerings in this space.
That said, I think there is a reason all the players in the space gravitate to very expensive enterprise level offerings - it's the only place they can create real value.
Analytics is hard. It either requires a lot of smart code rolling up large amounts of data into manageable pieces, or a very big piece of iron that can deal with somewhat raw data. And in either case, everyone has their data in a different format in a different database on a different OS. The result is that you need a behemoth piece of enterprise software or a custom written solution.
Even if you build a really smart UI that lets you quickly build reports that slice and dice you data, one dimension is too large to quickly process, or the ID's in another set don't quite line up. Something. There is always a problem that comes up that makes those canned demo videos look comical.
The best tool I've used where I felt it was both easy to use and powerful is Tableau. But, the price tag ended up being a little too high to swallow.
Note that one space I see that is under served is outward facing dashboards. If you're an advertising company and you want to show the results of the campaign you ran for a client, you have to send them an excel file or write your own custom dashboard. A server that manages authentication, data versioning, chart presentation, regular email updates, browser compatability, etc. is adding a lot of value. GoodData is working on this problem, and I think they are on to something. The concept has a viral quality, as well, as every client they sign up exposes their technology to all that client's clients.
Tableau is an amazing piece of software but we agree about the price tag. We envision ourselves being Tableau on the web and eventually supporting native connections to Data APIs and then Intelligence.
Damn, we have been looking for something like this for so long. I even posted an Ask HN question about precisely this a little while ago (http://news.ycombinator.com/item?id=1393159). In the end, we keep stats nightly and export them to excel. Yeah, re-creating the graphs every time is what kills it. So we export a limited version to Roambi.
The issue is that Roambi (strangely) can simulate and such in flash, but only output to iphone/ipad. Granted, it's a gorgeus interface.
Now, I may be a bit old fashioned, but handing our database out is something that makes me very nervous and opens us up for liabilities if something were to happen.
So tempting, though ...
One possible idea would be to somehow derive a cooked version of our DB and use that. Not sure how to do it nicely.
Another option is to somehow license this as a gem or something?
This is exactly what we need for many needs and perhaps all needs if you actually managed to abstract all the needs in a general way.
Glad we can help you out! We take what we do seriously and we're like consultants under tight NDA. We'll connect to your data securely and never share it with anyone.
Large banks, tel cos and credit card companies hire consultants all the time. Plue we'll be so useful!
Does this require that your servers get access to our internal databases, i.e. do we have to expose our databases externally? Secondly, will you store none, any or all the data that you fetch locally at your servers? Temporary or infinitely?
I didn't find this information explicitly somewhere, so I thought I would ask.
Yes. We can securely connect via SSL and we're working on SSH tunneling. We recommend a db slave if possible.
We can also take nightly dumps to our servers right now.
We store metadata about your databases in order to provide a richer experience and we might eventually set up local caching but if we're remotely connecting we will not store any of your business data and we really don't want to.
Are you, or any of the other recent yc sites I've seen lately, finding Olark to be useful? Do you find that users are indeed using it to seek assistance?
Olark is amazing. Because I have customers all over the world, but only an English site, it lets me interact with people who don't fully understand what they are reading.
It really helps with my Japanese customers because we can communicate in their language and I can guide them through the registration process. In the 4 days since launching, I've had dozens of people interact with me on it.
The consensus is that we all find it really useful. We don't see a lot of volume via chat but almost all of the conversations started in olark are meaningful.
Our customers get a personal touch and we get live feedback. I would definitely recommend it.
I have a couple of retail stores. I have them push data into Google AppEngine nightly and have been using a Google Spreadsheet for the "dashboard". It is super cool that Chart.io is supporting AppEngine as a data source.
I have high hopes. I just wish I had checked this out sooner so that I was ahead of the TechCrunch rush in the beta account line.
At first glance this looks like a great bridge between engineering and marketing/ops/biz-dev: I can pull all these stats pretty easily from the console, but would much prefer if I weren't part of the process.
Does it support ODBC? I would love to pay for a local install that lets my users query a MS Navision database. I would pay the same as equivalent Tableau licenses.
When I'm done with my MSc project you guys should take a look. It loads in near real-time (within seconds of a change) SharePoint data into a SQL database in a form that allows for the differently structured data to be reported on. It's only a few weeks away from being written up but I'm going to be putting the code out there for anyone to use once it's done.
Anyhow an issue that you're going to have is data storage.
Ignoring the fact that enterprises will be very reticent to letting you have data, SMEs in Europe are going to be very reticent to let that data leave the legal boundary of Europe. There are lots of good laws in Europe that protects data so long as it's kept in Europe.
You should look at putting in place something that allows you to offer Europe only hosting of the data to counter that, a silo for a major legal domain.
That's the big problem I see, but in your favour not all companies will think of it (though enough will).
Another question I would have is: Storage is cheap but ongoing cloud storage is still expensive (relative to buying a couple of hard drives and a cheap machine vs you holding the data for a couple of years). Have you not thought of just offering the charting as a product companies can install locally? This also solves the legal boundary problem of data protection.
Also you should think of adding KPI charts that can be embedded into Word and PowerPoint docs. Little smart widget things that go and get the chart data from you as it changes.