She said goodbye too many times before doesn't seem confusing to me.

He's lamenting a romance that's been difficult for him. Before now, during the difficult relationship, she said goodbye or left him too many times, causing the difficulty and toll it has taken on him.