Recommend you also add "*" to your $HOME/.gitignore in this setup. Then you simply need to use `git add -f` to force-add files to your dotfiles store, so that you don't accidentally add files you don't intend to keep there.
If you put * in the gitignore file you don't have to deal with ignoring the downloads directory. When you want to add a new file / directory just add the -f flag (git add -f) and everything else works as normal.
cd ~
git init
git remote add origin [URL]
git fetch
git checkout -f master