I welcome this call to action and continue to believe that the open source approach provides the foundation for transparency ... though the red herring in this debate is the concept/notion of privacy.
The reasons why such systems are built and supported is that they check a lot of boxes for purely commercial reasons ... the scenarios that no one is talking about is the one where senators dip into the classified stream giving serious competitive advantage to actors in the free market.
Earlier efforts, such as ECHELON were just state sanctioned industrial espionage and what is going on now is just an extension of the same.
I am all for defending/upgrading the concepts of privacy and basic human rights, but as with many things ... the pragmatic solution is to make it very easy to 'follow the money' ...
just noticing the comment 'When a server is being stopped, we need to make sure we don’t lose any incoming data.'... the potential for data lose is already present with the use of redis server ?
I don't know all the details on the project and if there are some single point of failures, but as far as Circus is concerned, a properly working graceful shutdown is a must-have when your app is getting data to process.
e.g. 1/ notify the world you don't accept data anymore 2/ process what you have enqueued 3/ shutdown. In Fabien's use case 1 -> 3 can last for over 5 minutes.
my (perhaps snarky) comment was more to do with 'why be so careful' with the potential data failure (that circus handily addresses) when data is being placed in a 'leaky bucket' (redis) in the first place.
of course 'dont use xml'; unless your data mainly looks like documents, needs to handle mixed content, need to represent richer types and/or if you already have a robust, full featured, stable XML toolchain.
XML gets a lot of flack, and I think part of it is that "enterprise" solutions have mis-used and abused it so much (and part of it might be that people started out with only DTDs for specifying schemas).
Today, I think it's obvious that xml shouldn't ever really be written by a human, it should be used for machine generated interchange of data, much as html -- markdown and rst are reasonable languages to write in -- html "requires" a gui/wysiwig editor -- just as it really isn't pleasant to work with eg: xml config files directly. Or ant-scripts. It just isn't. Yes, you can hide the ugly with a smart editor, but it's still pretty ugly.
Now, html might be pleasant compared to writing in Postscript -- but it's not really pleasant to write (hence things like sparkup).
I believe the issue that many people have with XML goes a bit deeper ... using XML means you are creating a bit of distance between yourself and the host programming language. XML stands alone, apart, that distance comes with costs in terms of ease of use (which could have been addressed by proper tooling), as well as many benefits.
HTML may not be pleasant to write documents with and I would say exactly the same thing of json ... show me the equiv of a Word document in json and I would argue that only a madman would like to directly edit that.
Really, James Clark said it right, 'json for data, xml for documents'.
Today's 'ease of use' of manipulating a native data structure of the 'language du jour' will eventually be viewed a bit more negatively in the years, decades to come. But really this is a storm in a tea cup ... XML has embraced json and has lots of support for generating it so I am fine. For others, they will eventually find the boundaries of json and start redeveloping the same concepts.
However, all the above being said/done I do think json is the right 'over the wire' format for data today; I just think we should embrace heterogeneity instead of 'knee jerk' cargo cult advice of 'don't use XML'; that's the equivalent of throwing tools out of one's toolshed and that's just plain stupid.
The strength of xml is xsl. The ability to declaratively transform the content into another form. I'm not aware of any way to do this with json other than by hand-rolling code.
I would say that the strength of XML is XPath - forget the rest of XSL (XSLT and XSL-FO) - and I would still avoid it unless there was a real requirement for "document like" content.
I'm not entirely certain that is a plus... Is writing a compiler/translator in xsl really more pleasant/easier to maintain than writing it in a "proper" language?
ya, my eyes gloss over when I read 'rediscoveries' like this ...
I think the finer point is more about workflow then programming.
That is state based workflow is fundamentally harder for developers to grok, also maintaining state is a kind of fallacy when scaling out and/or up.
Flow based workflows, while probably can't cover easily the breadth of exotic corner cases that FSM can achieve, are much easier to understand and state is byproduct of the system, not to mention that a full record of mutation occurs with a copy of data at every step (similar to MVCC - Multi-value-consistency-control)
Its nice to see yet another XML technology, which of course is based on older established concepts in computing, being emulated/replicated in the JS universe ... shame about the visual programming antipattern/lunacy.
yes, a bit of a cheap shot that ... and the guy works at a company that makes MarkLogic server which spits out XML ... but it also spits out JSON (all at very large scale I may add) so choose your poison.
Until someone replies with the same experimental rigor to refute the findingsthen I think the observations this paper makes stands ... thats how peer reviewed journals work.
The paper is not saying 'XML is better' or even 'XML is faster' ... its just addressing the perception that XML is slow in certain scenarios which has become a default myth.
JSON has been accepted as a datatype in the Markup conferences of the world its great for data transfer, XML is a compromise on many different levels but tends to be good for mixed content and documents. I think we've all moved on.
as one who is owned by a dog ... I will categorize this post under the 'bleeding obvious' epiphany ... cue 3 mths from now a new startup whose main product is 'puppies for productivity'.
really folks, I mean this in the best possible way ... get a life (and a dog).
interesting commit, though its got a long way to go to match up to something like PostgreSQL (or at the other end of the scale, MarkLogic, for extremely mature structure/unstructured + text search).
from an impl pov I suspect the stemmer is the (logical) next step (of many) on this long road of implementing full text search features.
Agree with your sentiments, I won't be so nice though ... this article is stunning in its naivety and ignorance.
Perhaps the author realizes that for every successful startup there are thousands of failures; young people are being convinced (usually by clever older people with money) to use up that most precious commodity e.g. time.
An analogy I see is in the stock market; where hiring young (malleable) young guns to work 24/7 ... most of these people don't get wealthy, most of them burn out or used as responsibility fodder. Most of them don't enjoy their youth and end up regretting it.
as for the age bias in the article ..., lets hope a few decades from now that his article will be preserved for pleasurable reading later on in life as much wincing will ensue.
The reasons why such systems are built and supported is that they check a lot of boxes for purely commercial reasons ... the scenarios that no one is talking about is the one where senators dip into the classified stream giving serious competitive advantage to actors in the free market.
Earlier efforts, such as ECHELON were just state sanctioned industrial espionage and what is going on now is just an extension of the same.
I am all for defending/upgrading the concepts of privacy and basic human rights, but as with many things ... the pragmatic solution is to make it very easy to 'follow the money' ...