Even there it really depends on the kind of game (and target platform). I would say that for a lot of indie games, worrying about low-level programming shouldn't be high on your list. In fact many indie-game developers are exactly the kinds of people who do love to worry about such things, when perhaps their time would be better spent making sure anybody wants to play their game, and/or finding a way to market it--- most indie games don't fail because of bad performance.

An exception might be if the game's core gameplay mechanic is fundamentally based around doing some trick on a console or handheld device that can't be done without careful optimization.