It couldn't? I'm by no means an expert on this, as my only experience with LLVM was for a undergraduate research project back when I was still in college, but I could swear that it allows custom calling conventions (would verify if I weren't on my phone).
Assuming I'm not mistaken about their existence, is there a reason Rust couldn't use one?
GHC and HiPE appear to have gotten themselves LLVM calling conventions, I guess.
But none of this can be customized further without modifying LLVM.
Besides, none of these choices seem to affect anything other than the (ordered) set of registers available for arguments, and the sets of caller/callee-save registers (i.e. what you expect from x86 calling conventions).
Once you get to the stack it's really all the same.
What I meant was that Rust could have picked any of those choices, which would make it incompatible with C (it already is for some argument types because of the simplistic handling around them), but it couldn't have used its own special one, not without adding it to all LLVM targets it wants to support.
Assuming I'm not mistaken about their existence, is there a reason Rust couldn't use one?