HN2new | past | comments | ask | show | jobs | submitlogin

I'm so happy this is (currently) the top comment and people are starting to realize measuring perf with these well tuned micro-benchmarks is a sham.


Yes. I was thinking exactly this when coming into the comments.

Real world benchmarks take more time to prepare, we get that, but let’s be honest: a bold claim _needs_ bold evidence.


Why is it a sham? It's useful to know that x is faster. As a user of x I don't really care if the reason it's faster is it's C++ under the hood. That's really an implementation detail for me.


Because it's measuring a scenario that's not representative of real usages of X and is usually tweaked for the framework/tool the author is trying to showcase.

For example: all the deno/bun benchmarks often use the Node.js frameworks in an unnecessarily slow way and don't measure apples to apples. (like using uWS in bun but not uWS for Node in the benchmark).

This one in particular just measures the speed of "hello world" in plaintext and JSON with very specific parameters.


What happens when you use X because it's fast at one thing in a microbenchmark, but it locks you into an environment that is slow at the rest of your program?


The real thing will be some mix of your code and the library's code, and in cases like this where you'd be writing all your logic in Python vs Golang, that will have a significant effect on the results.

A while back I implemented a game rules engine and MCTS in Torch (the Lua library). The training loop spent like half of its time running the rules engine. Writing it in Python would have been a disaster in comparison. To really make good use of the hardware and spend more time in the optimized machine learning code provided by libraries, one would have to write their rules engine in some language other than Lua or Python.


Because the moment you actually write any server or business logic in Python performance drops off a cliff and it doesn't differ significantly from any of the other Python web frameworks.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: