Recognition of individuals using periocular images has become popular due to its ease of use in situations in which it is difficult to capture full-face regions or high-resolution iris images. Practical surveillance systems recognize individuals from images acquired in various imaging conditions. Recognition of individuals in cross-spectral environments is of practical interest as it can be used to match probe and gallery images that are captured in distinct wavelength domains. The process of matching probe near-infrared images with a gallery of visible images is significantly challenging due to variations in appearance and illumination conditions. In periocular regions, alleviating this wide appearance gap from such local regions is highly challenging. In this work, we propose a dual-spectrum network that considers both high-level deep features and low-level handcrafted attributes to match periocular images in cross-spectral scenarios. A mapping network is designed to establish the relationship between both types of features and generate mapped features that carry discriminative information relevant to both feature domains. The mapped features are passed through a distance layer and subsequent classification block to determine whether they are a genuine pair or an imposter pair. The network is evaluated on a publicly available PolyU dataset as well as two newly created in-house CASIA and IITBBS cross-spectral periocular datasets. The IITBBS dataset consists of visible and near-infrared periocular images from 100 subjects captured in unconstrained imaging conditions with pose and accessory variations. Extensive experiments indicate that the proposed network achieves verification accuracies of 97.96%, 96.48%, and 97.89% on the PolyU, CASIA, and IITBBS datasets, respectively. Performance analysis with existing methods confirms that our network achieves comparable results on the PolyU and CASIA datasets. In addition, the network outperforms existing works on the challenging IITBBS dataset by achieving improvements of 0.43% and 0.16% in terms of verification accuracy and genuine acceptance rate at a 0.1 false acceptance rate.