Oh interesting. The datasheet for the PCM5102A says it supports line-level output with a minimum load of 1kΩ [1]. I think most headphones typically have an impedance of less than 100Ω. I'd guess the excess current results in some distortion or low volumes? It works fine in practice though?
[1] https://www.ti.com/document-viewer/pcm5102a/datasheet