Some advantages of memory safety remain. When using pools, you don't get protection against use-after-free or leaks. It does somewhat mitigate the consequences of those, however: it's much harder for a use-after-free to become a security exploit in a pool in a GC'd language, for example.
Sounds like you have a specific kind of object pool implementation in mind? Invalid object refs don't have to break the memory safety, cf. weakrefs.
An object pool can also be implemented without unsafe bulding blocks or other language support, as a pool of objects that are manually recycled after the pool lifetime is up. Then a bug could let you confuse a previous generation object witha current-gen one, but would not break memory-safety either.
