Isn’t the Answer Obvious?
Finally, one might simply say: it’s obvious what defines the right direction for judging visually apparent direction. So, putting aside everything I’ve said so far (don’t appeal to body, world, head, etc.), what is that is obvious? I don’t know.
The ‘veridical’ in ‘how things visually appear in veridical perception’ depends in part on veridical direction. Things visually seem to lie in front of us. So, what determines the veridicality of that ‘in front’?