Berlin is a bit odd because of the huge changes in the last few years in companies starting offices here and the influx of VC funding.
SoundCloud for example, had a bunch of engineers earning around that. Probably all laid off now, though.
There are companies on StackOverflow jobs openly advertising €80-90k for senior developers now, and occasionally over 100k for specialist roles like ML or performance sensitive development in C++.
yes i am sure those salaries are possible for certain specialties in high demand, but in this case we are talking about Senior Web Developers. I think for pure devs without a lead or other management role, it's hard to get more than 75k EUR or 90k USD, and in BCN i have the feeling it is at least 10k less.
Did you check stack overflow jobs? What people are currently earning and what people are openly offering now has diverged quite a bit just in 18 months or so. It’s hurting the smaller companies with less budget and it’ll only get worse when Google opens their big office.
I have recently talked to some HR people and been looking myself, so that is just the feeling that i got. You mean the Google Campus in Kreuzberg ? Will they really employ a lot of people there ? Sounded more like something similar to the Factory
Erlang has a really nice solution here by using a separate heap per Erland "process", and because they're owned by that process, can do a copying/compacting GC without having to take any kind of locking. Ruby's Guilds could adopt a similar idea and avoid STW GC there too.
Again, the term "stop the world" is misleading, there are a variety of compaction strategies that greatly minimize locking. There are even realtime GC implementations for the JVM.
Erlang's solution is interesting, but remember in Erlang processes can't share memory, they only have message passing. So you never have to worry about multiple threads writing to the same data structure at once, which is what makes compaction hard.
Yeah, it mostly happens when you have a server which runs an abnormally large task or payload, causes the heap to grow, and does it in a way that Go can't release the memory back to the OS due to heap fragmentation.
I haven't looked at this for a while but I believe it's still an issue.
I would also say the same if you host a Ruby or Python app, or anything using forking really.
Similar to the issues you had with Redis, the kernel change to THP on by default totally destroyed CoW sharing for forked Ruby processes, despite Koichi Sasada's change to make the GC more CoW friendly. Without disabling THP, a single run of GC marking can cause the entire heap to be copied for the child.
500MB per worker is totally standard. What happens is a job causes a huge array or hash to be allocated, and after it‘s finished the memory can’t be returned to the OS due to heap fragmentation. Java does some crazy stuff with compaction. C programs typically try and internally allocate into arenas to avoid it.
This was probably true 10 years ago, but these days it's basically FUD. Since then Ruby improved performance something like 5-80x from 1.8 to 2.5 and moved from an interpreted language to a VM nearly as fast as LuaJIT. Go only has 3-5x the throughput for tasks like: grab 100 rows from PostgreSQL, serialize to JSON, respond via HTTP. Lots of the hot code in Ruby around fetching rows from the DB, serializing to JSON, and HTTP parsing have also since been implemented as native functions in C, massively improving performance with zero effort required by the developer.
Do you mean: "take a few thousand rows and map them to a different data structure"? Because I benchmarked that recently and mapping 16,000 rows of GPS points using haversine distance in pure Ruby takes about 12ms, and about 5ms in Go. It's not that much slower. There are tasks where Ruby can be ~100x slower than Go, but a simple map isn't one of them.
This is basically my job at ChartMogul and we've pretty much solved this problem. The two biggest issues for us were: Ruby prefers to grow the heap really quickly rather than spend much time in garbage collection. You can turn this growth factor down at runtime using an enviroment variable.
The second problem is importing a huge chunk of rows at once means they have to exist in RAM at the same time. Use batched iterators to reduce peak memory usage. All GCed languages have this problem, Go included.
You'd think Go's GC was somehow revolutionary from the way they talk about it, but it's basically the same as the Ruby GC, plus a little more parallelism. What helps Go is that the compiler can optimise and use stack allocation and re-use temprory variables. If it fails, it causes a nightmare, and the Go standard library is full of tricks to convince the compiler to do stack allocation.
Java, OTOH, has compacting garbace collection, so after high peak memory usage, it can release the memory back to the OS. Aaron Patterson has been working on doing the same for Ruby. If you use JRuby, you'll get this right now, plus it's about 3x faster for shuffling data around.
Ruby is actually pretty good in this regard. If you define your module or class anonymously, but give it a name using a constant, Ruby will GC it when possible. The standard way of defining modules and classes obviously means they can never be GCed.
Java doesn't, or at least didn't, collect anonymous classes without an additional GC flag being enabled, which could bite you quite hard using JRuby with gems which made heavy use of anonymous classes.
In practice, modules and classes are not defined that way - likely not in your own application, and certainly not in all the gems you depend on, or in the Rails framework itself. That entire set of transitive dependencies can take up a lot of memory.
The class definitions will be a tiny fraction of memory usage. A template Rails app memory usage only has about 20% managed by Ruby. The rest is the VM, C libraries, maybe long strings etc. Definitely not class definitions pulled in from gems.
You should really only need one process per core, plus a few thread per process. Obviously, this is a problem on Heroku where they give you 8 "cores" but only 512mb of RAM per 1x dyno but on a DigitalOcean or AWS server you shouldn't be running out of RAM before maxing all cores.
