Well, we don't really know. Maybe there's some easy-to-guess text file in /misc/ that contains a password for something. We don't know what we don't know. We do know that there's considerably more information exposed here than zero - the question is whether any of that information could lead to sensitive information, not whether or not it constitutes sensitive information by itself.

How does someone on pentests not know it's the default robots.txt that comes with Drupal and hence does not leak anything except that it's Drupal?

Comparing it to Drupal's default robots.txt