The persistent need for more qualified personnel in operating theatres exacerbates the remaining staff’s workload. This increased burden can result in substantial complications during surgical procedures. To address this issue, this research project works on a comprehensive operating theatre system. The system offers real-time monitoring of all surgical instruments in the operating theatre, aiming to alleviate the problem. The foundation of this endeavor involves a neural network trained to classify and identify eight distinct instruments belonging to four distinct surgical instrument groups. A novel aspect of this study lies in the approach taken to select and generate the training and validation data sets. The data sets used in this study consist of synthetically generated image data rather than real image data. Additionally, three virtual scenes were designed to serve as the background for a generation algorithm. This algorithm randomly positions the instruments within these scenes, producing annotated rendered RGB images of the generated scenes. To assess the efficacy of this approach, a separate real data set was also created for testing the neural network. Surprisingly, it was discovered that neural networks trained solely on synthetic data performed well when applied to real data. This research paper shows that it is possible to train neural networks with purely synthetically generated data and use them to recognize surgical instruments in real images.
For high-precision measurements through a inspection window, a 3D scanner based on fringe projection profilometry is being researched. The 3D scanner combines a micromirror array projector and two telecentric cameras. The affine camera model is commonly used to calibrate telecentric imaging systems, in which a single magnification factor is introduced and optimized for each lens. However, 3D reconstructions based on this model indicated that reconstruction uncertainties in the peripheral areas of the measuring volume are significanty affected by a possible inspection window. These uncertainties may occur due to the model-based determination of the magnification factor and the reduction in parallelism of the visible rays within the telecentric lens that can occur at larger working distances. To address this issue, a new method for calculating and identifying the influence of the magnification factor on the 3D point scaling for telecentric measuring systems is proposed. First, the measuring system is calibrated using an affine camera model. Then, the reconstructed 3D target points are used to estimate the magnification factor locally and assessing the influence of an inspection window in the optical path. In order to further investigate the influence of the inspection window on the imaging performance of the cameras the focus is estimated locally within the measuring volume. Initial measurements using these methods reveal that scale variations and the reduction of focus can be quantified locally and a model based correction as well as the removal of poorly reconstructed points is feasible.
The automation of inspection processes in aircraft engines comprises challenging computer vision tasks. In particular, the inspection of coating damages in confined spaces with hand-held endoscopes is based on image data acquired under dynamic operating conditions (illumination, position and orientation of the sensor, etc.). In this study, 2D RGB video data is processed to quantify damages in large coating areas. Therefore, the video frames are pre-processed by feature tracking and stitching algorithms to generate high-resolution overview images. For the subsequent analysis of the whole coating area and to overcome the challenges posed by the diverse image data, Convolutional Neural Networks (CNNs) are applied. In a preliminary study, it was found that the image analysis is advantageous when executed on different scales. Here, one CNN is applied on small image patches without down-scaling, while a second CNN is applied on larger down-scaled image patches. This multi-scale approach raises the challenge to combine the predictions of both networks. Therefore, this study presents a novel method to increase the segmentation accuracy by interpreting the network results to derive a final segmentation mask. This ensemble method consists of a CNN, which is applied on the predictions of the given patches from the overview images. The evaluation of this method comprises different pre-processing techniques regarding the logit outputs of the preceding networks as well as additional information such as RGB image data. Further, different network structures are evaluated, which include own structures specifically designed for this task. Finally, these approaches are compared against state-of-the-art network structures.
Numerical simulation to calculate the free spectral range scans (FSR scans) of laser resonators is a computationally intensive task. OSCAR is a well-established Matlab toolbox that enables for such simulations based on Fourier optics. Any arbitrary discrete complex electromagnetic input fields as well as misalignment or mismatching of resonators can be considered in the FSR simulation. Unfortunately, it currently only features CPU based calculations on one or more CPU cores. However, the computational cost increases exponentially with increasing lateral resolution of the complex electromagnetic fields. In addition, only a limited number of roundtrips can be carried out in an acceptable computation time, which limits the applicability only to low finesse resonators. Due to good parallelizability of the FSR scan calculation, this numerical computation is very well suited for modern graphics cards, which are outstanding in performing many calculations in parallel. This paper introduces the extension of FSR scan simulations on modern graphics cards (GPUs) within the OSCAR Toolbox. First, a statistical analysis is provided, that presents the massive performance improvement compared to CPU computations. Subsequently, the disadvantages in the form of memory limitations of GPUs are outlined. Therefore, generally valid data is presented, from which a trade-off between lateral resolution of the complex electromagnetic fields and the number of roundtrips to be performed can be derived. In conclusion, the great potentials of new applications are highlighted, which were previously not feasible. Any code of this GPU implementation discussed in this paper has been integrated into the OSCAR Matlab Toolbox and is made available open source on GitHub.
External Fabry-Perot resonators are widely used in the field of optics and are well established in areas such as frequency selection and spectroscopy. However, fine tuning and thus most efficient coupling of these resonators into the optical path is a time-consuming task, which is usually performed manually by trained personnel. The state of the art includes many different approaches for automatic alignment, which, however, are designed for special optical configurations and cannot be generalized. However, these approaches are only valid for individually designed optical systems and are not universally applicable. Moreover, none of these approaches address the identification of the spatial degrees of freedom of the resonator. Knowledge of this exact pose information can generally be integrated into the alignment process and has great potential for automation. In this work, convolutional neural networks (CNNs) are applied to identify the sensitive spatial degrees of freedom of a FabryPerot resonator in a simulation environment. For this purpose, well established CNN architectures, which are typically used for feature extraction, are adapted to this regression problem. The input of the CNNs was chosen to be the intensity profiles of the transversal modes, which can be obtained from the transmitted power behind the resonator. These modes are known to be highly correlated with the coupling quality and thus with the spatial location of resonators. To achieve an exact pose estimation, the CNN input consists of several images of mode profiles, which are propagated through an encoder structure followed by fully-connected layers providing the four spatial parameters as the network output. For training and evaluation, intensity images as well as resonator poses are obtained from a simulation of a free spectral range of a resonator. Finally, different encoder structures including a memory efficient, small self-developed network architecture are evaluated.
Within the aviation industry, considerable interest exists in minimizing possible maintenance expenses. In particular, the examination of critical components such as aircraft engines is of significant relevance. Currently, many inspection processes are still performed manually using hand-held endoscopes to detect coating damages in confined spaces and therefore require a high level of individual expertise. Particularly due to the often poorly illuminated video data, these manual inspections are susceptible to uncertainties. This motivates an automated defect detection to provide defined and comparable results and also enable significant cost savings. For such a hand-held application with video data of poor quality, small and fast Convolutional Neural Networks (CNNs) for the segmentation of coating damages are suitable and further examined in this work. Due to high efforts required in image annotation and a significant lack of broadly divergent image data (domain gap), only few expressive annotated images are available. This necessitates extensive training methods to utilize unsupervised domains and further exploit the sparsely annotated data. We propose novel training methods, which implement Generative Adversarial Networks (GAN) to improve the training of segmentation networks by optimizing weights and generating synthetic annotated RGB image data for further training procedures. For this, small individual encoder and decoder structures are designed to resemble the implemented structures of the GANs. This enables an exchange of weights and optimizer states from the GANs to the segmentation networks, which improves both convergence certainty and accuracy in training. The usage of unsupervised domains in training with the GANs leads to a better generalization of the networks and tackles the challenges caused by the domain gap. Furthermore, a test series is presented that demonstrates the impact of these methods compared to standard supervised training and transfer learning methods based on common datasets. Finally, the developed CNNs are compared to larger state-of-the-art segmentation networks in terms of feed-forward computational time, accuracy and training duration.
Optical triangulation systems based on fringe projection profilometry have emerged in recent years as a complement to traditional tactile devices. Due to the good scalability of the measurement approach, a highly compact novel sensor for maintenance and inspection in narrow spaces is realized by applying optical fiber bundles. Especially in the field of high-resolution and rapid maintenance in industrial environments, numerous applications arise. Endoscopic 3D measurements of gearing geometries are of particular technical relevance for detecting and quantifying damage. The measurement performance depends to a considerable extent on the technical surface to be inspected. Polished surfaces are particularly problematic due to specular reflections, but can still be partially reconstructed by using HDR imaging. However, if multiple reflections occur due to the specimen geometry and sensor arrangement in such a way that the optical path of each corresponding camera pixel can no longer be reconstructed unambiguously, a measurement is no longer feasible. In this study, the effects of surface roughness, sensor arrangement, and triangulation angle on measurement error are systematically investigated to describe possible application limits and provide guidance on sensor operation.
Benefitting from recent innovations in the smartphone sector, liquid optics in very compact designs have been cost-effectively introduced to the market. Without mechanical actuation, a focus variation can be adjusted within fractions of a second by curving a boundary layer between two liquids by applying a pulse width or amplitude modulated potential. Especially in the field of endoscopy, these innovative optical components open up many application possibilities. Conventional, mechanical zoom lenses are not very common in endoscopy and can only be miniaturized at considerable effort due to the necessary actuation and the complex design. In addition, the mechanical response is slow, which is a particular disadvantage in hand-held operation. A calibrated camera provides a two-dimensional camera pixel translated into a three-dimensional beam and, together with distortion correction enables the extraction of metric information. This approach is widely used in endoscopy, for example, to measure objects in the scene or to estimate the camera position and derive a trajectory accordingly. This is particularly important for triangulation-based 3D reconstruction such as photogrammetry. The use of liquid lenses requires a new data set with an adapted camera calibration for each focus adjustment. In practice, this is not feasible and would result in an extensive calibration effort. This paper therefore examines, on the basis of an experimental setup for automated endoscopic camera calibration, the extent to which certain calibration parameters can be modelled and approximated for each possible focal adjustment, and also investigates the influence of a liquid lens on the quality of the actual calibration.
Based the fringe projection profilometry, a compact and flexible positionable measuring head can be combined with optical fiber bundles to perform in-situ inspection tasks in industrial applications. Surfaces of complex geometries can be reconstructed and quantified in metric coordinates by means of a fast, non-contact and high-resolution measurement. Defect segmentation, on the other hand, is rather complex with three-dimensional point clouds, since reference data is required or a deviation determination is ambiguous and susceptible to errors. Due to each reconstructed object point corresponding to a camera pixel, it is possible to apply image processing algorithms or neural networks for defect segmentation. Since image based segmentation is more susceptible to poor illumination and deviating surface curvature or texture, a circular array of miniature LEDs has been coaxially arranged around the imaging optics of the camera’s fiber to provide different illumination directions. By utilizing a directional variable illumination sequence, the advantages of image-based segmentation can be combined with the unambiguousness and metric quantifiability of point cloud data.
A 3D measuring endoscope with a small measuring head and parallel arrangement of the fibers can be guided into forming plants and carry out precise measurements of geometries which are unreachable for most three-dimensional measuring systems. The data obtained can be used to quantify the wear of highly stressed structures and thus provide information for maintenance. Due to the compact sensor design and the required accuracy, optics with small working distance and a small measuring volume are used. In addition to in situ single measurements of highly stressed structures, over a hundred individual measurements are conceivable in order to convert large and complex geometries into point clouds. Besides the robust and accurate registration of all measurements, merging is one of the main causes of inaccurate measurement results. Conventional merging algorithms merge all points within a voxel into a single point. Due to the large overlap areas required for registration, points of diverse quality are averaged. In order to perform an improved adaptive merging, it is necessary to define metrics that robustly identify only the good points in the overlapping areas. On the one hand, the 2D camera sensor data can be used to estimate signal-based the quality of each point measured. Furthermore, the 3D features from the camera and projector calibration can evaluate the calibration of a triangulated point. Finally, the uniformity of the point cloud can also be used as a metric. Multiple measurements on features of a calibrated microcontour standard were used to determine which metrics provide the best possible merging.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.