Too much, probably. For me, its not usually not related to reorganizing my code around. But, I'll nitpick over whitespace or brackets. I won't fix formatting up if I'm in the middle of working on something, but I'll make a mental note to come back to it.
My justification (to myself, at least) is that when I encounter code thats badly formatted, I find that the flow doesn't tend to make much sense (either to read, or as its actually executed) and it needs refactoring anyway. So, I'll fix the formatting up and read the code to see what it does. And then refactor parts, if necessary.
I do almost the exact opposite: I see that fixing minor flaws in the whole code base is a temptation, but I do not have enough time to do it. So I just polish everything that my edits touch (i.e. every function that is affected).
My justification (to myself, at least) is that when I encounter code thats badly formatted, I find that the flow doesn't tend to make much sense (either to read, or as its actually executed) and it needs refactoring anyway. So, I'll fix the formatting up and read the code to see what it does. And then refactor parts, if necessary.