My opinion is that an optimization that costs less than a couple of hours of development time, if it doesn't harm code quality, is invariably worth doing. Even if there's no evidence (because the code is being constructed at the time) that this matters from a performance perspective, you should spend the hour or two and get that low-hanging fruit and not worry about it being "premature". If you don't do this when you're in-context, it'll cost an order of magnitude more time later on if the problem is discovered in profiling and needs to be addressed in the future.

When premature optimization is evil is when it damages code aesthetics and maintainability. There is a limit, in performance, to what can be achieved with aesthetically clean code. What awful programmers (who think they're being clever) tend to do is write the ugly, dirty code because it's super-fast, but no one can maintain it and even they don't understand it.

What good programmers do is write clean code on sound fundamentals first, so that people can understand what's going on. Then, if a performance problem's discovered and a certain module needs to be optimized, they put the critical region behind a clean interface and allow the dirtiness (which they document heavily) to be in the implementation.

TL;DR: spend time to make your code performant if it doesn't damage code health; be very cautious about replacing clean code with faster, dirty code.