The method of the optimization for a binary phase hologram used in a security system is proposed. The hologram is made by a joint Fourier transform method and a simulated annealing technique is used for the optimization. A hologram of a fingerprint image with a random key reference is successfully optimized and almost the same image quality as the original one is obtained as a decoded image by the proposed method. The usefulness of the method in an optical security system is also discussed. © 2001 Society of Photo-Optical Instrumentation Engineers.