Octal was far more common than hex back then, due to machines with a wordsize that was a multiple of 3 bits; hence Unix permissions, coming from the 18-bit PDP-7, also inheriting that convention.
"In 1969, Ken Thompson wrote the first UNIX system in assembly language on a PDP-7, then named Unics as a pun on Multics"
CDC "super" computers also had 60-bit words.
Moreover, the original ASCII standard had the 7-bit characters, the improvement over even less bits used for characters before that (6 bits == 2 octal digits). 8-bit bytes weren't something you could expect except when buying IBM (which had EBCDIC, not ASCII).
Separate issues, I think. Octal was used because that's what most everyone used at the time (a holdover from machines like the IBM 70xx and GE600 series which the early Unix folks were used to). Even in V6 & V7 and later, you see much more octal than hex. Unix perms just happened to be 3 groups of 3 bits. Said another way, we'd likely have used octal to represent them even if perms were 2 bits or 4 bits each.
