When I'm wrangling data, it's not necessarily that I don't want to have to deal with the type checker. It's that I have yet to see a statically typed language whose type system is designed to help solve the kinds of problems I have when I'm wrangling data.

The distressing example that nags at me here is how the headline feature of Apache Spark 2.0 is that they Greenspunned a dynamic typing mechanism into it. And, in doing so, realized improvements in ergonomics, performance, and memory consumption. Understanding why and how is an instructive lesson.

The most promising typing system for arbitrary data wrangling that I've seen so far is the one that's built into a Python package called Dagster. It seems reminiscent, as best I can tell, of dependent typing. (Although I haven't actually spent any time with a dependently typed language, so that could be a misunderstanding on my part.) The actual checks are done at run time, of course, but I don't see any particular reason, aside perhaps from the complexity involved, why a similar mechanism couldn't be implemented as a static type check.

What are the sorts of problems you have when wrangling data that type systems don't help with? I'm not a data scientist, but I'm a type system guy, and I'd like to learn about ways that type systems are failing users. I took a very quick peak at Dagster and the Google results for "apache spark 2.0 dynamic typing" (are you talking about "Datasets"?) and nothing immediately jumped out as a clear sort of elucidation of what you're saying.

In data science, types change all the times. E.g. when dealing with tabular data, the columns (which might be used as type parameters) change frequently. In Julia, for example, see DataFrames.jl vs. TypedTables.jl for some trade-offs.

In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.

It sounds like the structural typing in Typescript might help a lot here - you don’t need to formally define any types for a record, the language just infers and tracks fields/types for you. I don’t know what the current state of data science is for TS though.

But now you are talking about type annotations and not static typing. You can have static typing without any type annotations

Everything comes with trade offs. But I’d argue that one of the primary reasons folks avoid types is that it requires additional learning.

I’m speculating here, but as a software engineer who’s dealt with many different languages, I (personally) find types to increase my productivity, because it reduces errors at runtime. This is a huge benefit, and because I’ve spent a lot of time learning the target language’s type system, it doesn’t come at the cost of productivity, and like I said, I find it to improve my downstream productivity.

I’ve read that scientists have had significant mistakes in their findings in papers due to bugs, which I know is anecdotal, but it seems like adopting typed languages would reduce those errors and increase confidence in studies.

I wonder if they were able to recognize the downstream benefits of types, if they would shrug them off as irrelevant to their work.

I read about a guy who wanted to reproduce a neural net and he worked at it for a year only to find it missed a +1 in one place. Being a neural net it still worked but not as good. I don't think this kind of error would have been caught by a type system.

It depends. Off by one errors are common, but the context is important. Some languages for example prefer integrators to indexing, and if ‘a’ was an index that might have been avoided.

Languages with algebraic data types also allow you to express state in ways that can make it easier to identify state changes, possibly making the missing increment more obvious.

Some languages will warn when variables are being used or modified in scopes where they are referenced but not used.

But, even with all this, yes bugs will still happen.

It's true that many papers have mistakes, but the reasons for that go further than types, the main one being the prevalence of uneducated spreadsheet warriors.

I definitely agree that if you're making a big custom model you should use a typed language. But indeed, that requires a lot of learning, so the investment has to be worth it. The problem is often that scientists don't program all day, so remembering language intricacies is hard. Dynamic languages that can have their whole syntax written on a postcard make things much easier from that pov.

Haskell syntax would probably fit on a postcard too. I think type systems are orthogonal to syntactic complexity.

Right, the problem is that the semantics of statically typed languages are far more complex (since the program text conveys more information).

It’s not that it requires additional learning. It requires additional time when writing code. A nontrivial amount of time that isn’t generally worthwhile for me.

I tend to find the opposite - writing Python with or without type annotations is both more time consuming and a more fraught endeavour than writing Go, TypeScript or even Rust.

I suspect that is the case if you have written a lot in a typed languages. There may be both context switching and literally no issue dealing with types if you have mastered them.

Oh, I find that Haskell and Elm are much quicker to write code because I spend much less time on little programming mistakes and refactorings are much safer

Most static type checkers prove (to the limits of the soundness of the type system) that the types of the values at run time will always be compatible with the operations performed on those values. For these checkers, any program that's not provably sound (to the limits of the type system soundness) will be rejected. In other words, most static type checkers answer "is this definitely correct?"

However, it's possible to have a more lenient static type checker that only looks for cases where you're guaranteed to get an illegal operation at runtime. They answer the question "is this definitely wrong?"

For instance, if we have:

  def f(x):
      g(x)
      h(x)

  def g(x):
      x.a()

  def h(x):
      x.b()

If we're doing whole program analysis and there's no type that has both a() and b() methods, then f is guaranteed to call a non-existing method at one place or the other.

This sort of "this is clearly wrong" type checking is much less intrusive than the more common "I'm not positive this is correct" type checking.

If you're not doing whole-program analysis, then you may restrict the type search to the types imported into the transitive closure of the module and its dependencies. This makes type checking slightly more intrusive by sometimes forcing more module imports, but it's still much less intrusive than common type checkers.

> However, it's possible to have a more lenient static type checker that only looks for cases where you're guaranteed to get an illegal operation at runtime. They answer the question "is this definitely wrong?"

If you're thinking about Erlang's Dialyzer, almost everyone I've heard talking about it says that it's difficult to understand and doesn't provide as much confidence as a regular type system would.

right, but for languages that support monkey-patching, this is going to be hard. e.g.:

  setattr(type(x),sys.argv[1],x.a)

or

  setattr(type(x), 'b' if halts(sys.argv[1]) else 'c',x.a)

Yes, static analysis of highly dynamic languages like javascript or ruby is tremendously difficult. Sound results are nearly impossible because you need to havoc so frequently. But if you permit some error and ask for a little extra help from developers in cases outside normal style for a language you can still do a pretty good job.

Presumably, you'd have an escape hatch in the from of specially formatted comments to annotate each questionable call site. My understanding is that even strong proponents of monkey patching feel it should be used sparingly.

> Contra: (some) people doing exploratory programming in the small e.g. scientists don't want to have to deal with the type checker.

Which in at least some cases means ignoring bugs. Pretty much every system I've worked on that used a language that didn't to type checking (or was too forgiving about it) has had runtime bugs because of type incompatibility or type coercion issues.

> Which in at least some cases means ignoring bugs

Indeed, but simplicity is the name of the game for exploratory programming. It's difficult to explore using languages requiring a lot of ceremony.

But then, that's when the time and funding constraints come in and prevent rewriting in more type sound languages. Which is why we end up with buggy python (if speed not an issue) or buggy C (if speed is an issue).

There are alternate paths. For example, APL tries to be terse enough that you can have the whole thing on a single screen. That's another attempt at bug prevention.

For example, I do my data wrangling in J.

> languages requiring a lot of ceremony

Some statically-typed languages don't require a lot of ceremony.

Is it a bug if it never happens? Another way to say it, would you pay money to prevent a problem you don’t have?

> Is it a bug if it never happens? Another way to say it, would you pay money to prevent a problem you don’t have?

Let me rephrase that: Is it a bug if I don't know about it? Another way to say it, would you pay money to prevent a problem you're not (yet) aware of?

Saying the bugs "never happen" is basically saying "I always write bug-free code."

I feel languages that don't have type checking (or where it's sloppy), force developers perform semi-automated testing (i.e. unit tests to catch type issues) when they could have systematic, fully-automated testing (for type issues, at least)

Right, but some people are doing data wrangling by themselves with python in a jupyter notebook. These aren’t production systems, and static type checking slows you down without adding much value.

That’s all I’m saying, the data analyst that has some edge case “bug” that never gets executed probably shouldn’t pay the upfront overhead cost of type checking.

Is having buggy, unused code lying around a good thing?

It’s not always bad, and dealing with it has a cost. The cost is not always worth. It may be worth it most of the time, but not always.

Agreed. I am essentially an Excel Poweruser on steroids with the way I use Python.