Language of gleam: impressionism artwork automatic caption generation for people with visual impairments

Dongmyeong Lee; Hyegyeong Hwang; Muhammad Shahid Jabbar; Jun-Dong Cho

doi:10.1117/12.2588331

4 January 2021 Language of gleam: impressionism artwork automatic caption generation for people with visual impairments

Dongmyeong Lee, Hyegyeong Hwang, Muhammad Shahid Jabbar, Jun-Dong Cho

Proceedings Volume 11605, Thirteenth International Conference on Machine Vision; 1160512 (2021) https://doi.org/10.1117/12.2588331
Event: Thirteenth International Conference on Machine Vision, 2020, Rome, Italy

Abstract

User Experience Design (UX Design) comes from focusing on how products, in reality, affect the user's experience. In particular, the design of multi-modal interfaces for blind people facilitates the flexible and natural product or service capacity and improves blind people's interaction by overcoming the various existing constraints associated with any particular interaction. There have been various attempts to help visually impaired people appreciation of visual artwork, including multi-modal associations. However, these methods can only provide general information in terms of edge and pattern recognition by the sense of touch and restrained by the availability and number of specially developed artworks. We propose a novel method explaining visual artworks through image caption generation using artificial intelligence (AI) to improve artwork accessibility. This method can objectively describe any impressionism artwork used as a standalone description of art interpretation for blind people or can aide tactile-based methods. Based on end-to-end learning with a deep neural network, an encoder-decoder architecture model is adopted, and comprehensive experiments perform to confirm the stability of generated image captioning for stylized MS-COCO datasets with impressionism.

Citation Download Citation

Dongmyeong Lee, Hyegyeong Hwang, Muhammad Shahid Jabbar, and Jun-Dong Cho "Language of gleam: impressionism artwork automatic caption generation for people with visual impairments", Proc. SPIE 11605, Thirteenth International Conference on Machine Vision, 1160512 (4 January 2021); https://doi.org/10.1117/12.2588331

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available