I _think_ you might be able to run a tilling wm in X but that wouldn't exactly replace the wm. I sorta hate the OS X wm which is a major impetus for me to run debian where I can actually chose what wm to use instead of having to obey the almighty menu bar.

You can run stuff like XMonad, but it only works with things run inside X11, yeah. So there's pretty much no point to it -- might as well install Debian.