Measuring how effective immersive virtual environments (IVEs) are in reproducing sensations as in similar situations in the real world is an important task for many application fields. In this paper, we present an experimental setup which we call the virtual pole, in which we evaluated human responses to fear of heights. We conducted a set of experiments in which we analyzed correlations between subjective and physiological anxiety measures as well as the participant’s view direction. Our results show that the view direction plays an important role in subjective and physiological anxiety in an IVE due to the limited field of view (FOV), and that the subjective and physiological anxiety measures monotonically increase with the increasing height. In addition, we also found that participants recollected the virtual content they saw at the top height more accurately compared to that at the medium height. We discuss the results and provide guidelines for simulations aimed at evoking fear of heights responses in IVEs.