Generally they use the CCSDS specifications, CCSDS is an international body with the space agencies of each major country contributing. Google for CCSDS, they have a bunch of open standards. Space networks really aren't quite at the level of fully mature, layered, multi-hop packetized networks... It's a Hard Problem.
Basically, TCP/IP doesn't scale to meet the constraints of deep-space networks (high latency, asymmetric data rates, intermittent connectivity). It just breaks down. That's why there's a lot of active research in DTN (delay/disruption tolerant networking and ad-hoc networking protocols). ION (Interplanetary Overlay Network) is the JPL implementation of the DTN protocols, and has been flown in a demonstration capacity on some extended-mission/end-of-life Deep Space missions. Much of it has been open sourced at this point.
Basically, TCP/IP doesn't scale to meet the constraints of deep-space networks (high latency, asymmetric data rates, intermittent connectivity). It just breaks down. That's why there's a lot of active research in DTN (delay/disruption tolerant networking and ad-hoc networking protocols). ION (Interplanetary Overlay Network) is the JPL implementation of the DTN protocols, and has been flown in a demonstration capacity on some extended-mission/end-of-life Deep Space missions. Much of it has been open sourced at this point.