Virtual reality framework for design and evaluation of multispectral computer vision algorithms and augmented reality interfaces for enhancing situational awareness in degraded visual environments

Ross L. Bobb; Jeffry A. Coady; Virgil O. Barnard IV; Matthew A. Mueller; William D. Casebeer; Joseph P. Salisbury

doi:10.1117/12.2614699

31 May 2022 Virtual reality framework for design and evaluation of multispectral computer vision algorithms and augmented reality interfaces for enhancing situational awareness in degraded visual environments

Ross L. Bobb, Jeffry A. Coady, Virgil O. Barnard IV, Matthew A. Mueller, William D. Casebeer, Joseph P. Salisbury

Author Affiliations +

Proceedings Volume 12125, Virtual, Augmented, and Mixed Reality (XR) Technology for Multi-Domain Operations III; 1212505 (2022) https://doi.org/10.1117/12.2614699
Event: SPIE Defense + Commercial Sensing, 2022, Orlando, Florida, United States

Abstract

Advanced fire control technologies that utilize computer vision-guided target recognition will enable dismounted soldiers with augmented reality displays, such as the integrated visual augmentation system, enhanced situational awareness. Here we describe a virtual reality framework and environment for the design and evaluation of computer vision algorithms and augmented reality interfaces intended to enhance dismounted soldier situational awareness. For training models, synthetic image datasets of targets in virtual environments can be generated in tandem with neural network learning. To evaluate models under simulated operational environments, a dismounted soldier combat scenario was developed. Trained models are used to process input from a “virtual camera” in-line with a rifle-mounted telescopic sight. Augmented reality overlays are projected over the sight’s optics, modeling the function of current state-of-the-art holographic displays. To assess the impact of these capabilities on situational awareness, performance metrics and physiological monitoring were integrated into the system. To investigate how sensors beyond visible wavelength optical imaging may be leveraged to enhance this capability, particularly in degraded visual environments, the virtual camera framework was extended to introduce methods for simulating multispectral infrared imaging. Thus, this virtual reality framework provides a platform for evaluating multispectral computer vision algorithms under simulated operational conditions, as well as iteratively refining the design of augmented reality displays. Improving the design of these components in virtual reality provides a rapid and cost-effective method for refining specifications and capabilities toward a field-deployable system.

Conference Presentation

Citation Download Citation

Ross L. Bobb, Jeffry A. Coady, Virgil O. Barnard IV, Matthew A. Mueller, William D. Casebeer, and Joseph P. Salisbury "Virtual reality framework for design and evaluation of multispectral computer vision algorithms and augmented reality interfaces for enhancing situational awareness in degraded visual environments", Proc. SPIE 12125, Virtual, Augmented, and Mixed Reality (XR) Technology for Multi-Domain Operations III, 1212505 (31 May 2022); https://doi.org/10.1117/12.2614699

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available