0
Tutorials

Imaging system fundamentals

[+] Author Affiliations
Gerald C. Holst

JCD Publishing Company, 2932 Cove Trail, Winter Park, Florida 32789

Opt. Eng. 50(5), 052601 (February 14, 2011March 05, 2011March 07, 2011May 10, 2011May 10, 2011). doi:10.1117/1.3570681
History: Received February 14, 2011; Revised March 05, 2011; Accepted March 07, 2011; Published May 10, 2011; Online May 10, 2011
Text Size: A A A

Open Access Open Access

Point-and-shoot, TV studio broadcast, and thermal infrared imaging cameras have significantly different applications. A parameter that applies to all imaging systems is /d, where F is the focal ratio, λ is the wavelength, and d is the detector size. /d uniquely defines the shape of the camera modulation transfer function. When /d<2, aliased signal corrupts the imagery. Mathematically, the worst case analysis assumes that the scene contains all spatial frequencies with equal amplitudes. This quantifies the potential for aliasing and is called the spurious response. Digital data cannot be seen; it resides in a computer. Cathode ray tubes, flat panel displays, and printers convert the data into an analog format and are called reconstruction filters. The human visual system is an additional reconstruction filter. Different displays and variable viewing distance affect the perceived image quality. Simulated imagery illustrates different /d ratios, displays, and sampling artifacts. Since the human visual system is primarily sensitive to intensity variations, aliasing (a spatial frequency phenomenon) is not considered bothersome in most situations.

At “normal” viewing distance, the HVS attenuates the amplitudes of the spatial frequencies associated with the blockiness and the imagery appears continuous.

Point-and-shoot, TV studio broadcast, and thermal infrared imaging cameras have significantly different applications. Common to all is an optical system and detector array which are linked together by /d, where F is the focal ratio, λ is the wavelength of interest, and d is the detector size. In the frequency domain, it is the ratio of the detector cutoff to the optical cutoff. /d uniquely defines the shape of the camera modulation transfer function (MTF).

The MTF is the primary parameter used for system design, analysis, and specifications. It describes how sinusoidal patterns propagate through the system. Because any scene can be decomposed into a Fourier series, the MTF approach indicates how imagery will appear on the display. In general, images with higher MTFs are judged as having better image quality. However, there is no single ideal MTF shape that provides the best image quality.

Sampling is an inherent feature of all electronic imaging systems. The scene is spatially sampled in both directions by the discrete detector locations. It creates ambiguity in target edges and produces moiré patterns when viewing periodic targets. Aliasing becomes obvious when image features approach the detector size. It distorts the image and the amount of distortion is scene dependent. It is pronounced when viewing periodic structures and these are rare in nature. Aliasing is seldom reported when reproducing natural scenery.

Mathematically, worst case analysis assumes that the scene contains all spatial frequencies with equal amplitudes. This quantifies the potential for aliasing and is called the spurious response. However, real scenes have a limited spectrum and image quality is a subjective measure. This means there is no method of validating the theory with imagery. MTF theory and sampling issues are just two slices through the multidimensional image quality space. They provide guidance for camera design but do not uniquely quantify image quality.

Assuming the detectors are in a rectangular lattice, the fill factor (Fig. 1) is the ratio of areas Display Formula

1 Fill factor = FF =dHdVd CCH d CCV =ADA PIXEL .

The photosensitive area (AD) is dHdV. Larger detectors collect more photons (higher sensitivity). Unfortunately, the current trend is to make smaller detectors. The detector center-to-center spacing (pitch) defines the sampling frequency (uS = 1/dCCH and vS = 1/dCCV) and pixel area (APIXEL = dCCHdCCV). Note that the detector size and pixel size can be quite different and this leads to confusion when describing system performance.

For convenience, a one-dimensional (horizontal) approach is used with FF = 1. Then d = dH = dCC = dCCH. The equations and graphs are easily modified for finite fill factors.

As the world becomes digital, we tend to ignore linear system theory (developed for analog systems) and sampling theory (analog-digital-analog conversion). The analog output of each detector is immediately quantized. The digital data is processed (image processing), digitally transmitted, and then sent to a digital display. We cannot see digital data. It must be transformed into analog data. Each display medium [cathode ray tube (CRT), flat-panel display, or printer] modifies the image in a different way.

Grahic Jump LocationF1 :

Detector and pixel relationship.

Four conditions must be met to achieve a linear shift invariant (LSI) system: 1. the radiation is incoherent; 2. signal processing is linear; 3. the system mapping is single-valued; and 4. the image is spatially invariant. An LSI system only modifies the amplitude and phase of the target.

Nonlinear image processing, present in nearly every imaging system, violates the one-to-one mapping requirement. For convenience, image processing will be considered a linear process. Single-valued mapping only occurs with non-noisy and nonquantized systems. No system is truly noiseless, but can be approximated as one when the signal-to-noise ratio (SNR) is high.

A sampled-data system may be considered globally shift-invariant on a macroscale. As a target moves from the top to the bottom of the field of view, the image also moves from the top to the bottom. On a microscale, moving a point source across a single detector does not change the detector output. An imaging system system is not shift-invariant on a microscale.

In spite of these disclaimers, an imaging system is treated as quasilinear over a restricted operating region to take advantage of the wealth of mathematical tools available. For mathematical convenience, an electronic imaging system is characterized as a linear spatial-temporal system with respect to both time and two spatial dimensions. Although space is three-dimensional, an imaging system displays only two dimensions.

Linear System Theory

An object can be thought of as the sum of an infinite array of impulses located inside the object boundaries. Thus, an object can be decomposed into a two-dimensional array of weighted Dirac delta functions, δ(xx), δ(yy) Display Formula

2o(x,y)=x=y=o(x,y)δ(xx)δ(yy)ΔxΔy.
An optical system produces an image and the process is symbolically represented by the operator hSPATIAL{} Display Formula
3i(x,y)=x=y=h SPATIAL {o(x,y)δ(xx)δ(yy)ΔxΔy}.
For small increments, this becomes the convolution integral Display Formula
4i(x,y)=o(x,y)h SPATIAL {δ(xx)δ(yy)}dxdy,

and is symbolically represented by the two-dimensional convolution operator ** Display Formula

5i(x,y)=o(x,y)**h SPATIAL (x,y).

The function hSPATIAL(x, y) is the optical system's response to an input impulse. The resulting image is the point spread function (PSF). Equation 4 is simply the summation of all the impulse responses. If i(x, y) passes through another LSI system,

i(x,y)=i(x,y)**h SPATIAL (x,y)
. As the number of LSI systems increases, multiple convolution calculations become tedious.

Since convolution and multiplication are Fourier transform pairs, convolution in space becomes a multiplication in the frequency domain Display Formula

6I(u,v)=O(u,v)H SPATIAL (u,v).

HSPATIAL(u, v) is the optical transfer function and is usually labeled as OTF(u, v). The MTF is the magnitude and the phase transfer function (PTF) is the phase of the complex-valued OTF. Symbolically Display Formula

7 OTF SPATIAL (u,v)= MTF SPATIAL (u,v)ej PTF (u,v).

Spatial frequency can be defined in image-space (at the focal plane) with units of cycles/mm or in object-space (cycles/mrad). They are related by u = uo/fl. To maintain dimensionality, if uo is measured in cycles/mrad then the focal length, fl, must be measured in meters to obtain u in cycles/mm.

The MTF and PTF alter the image as it passes through the system. For LSI systems, the PTF is of no special interest since it only indicates a spatial or temporal shift with respect to an arbitrarily selected origin. An image where the MTF is drastically altered is still recognizable whereas large nonlinearities in the PTF can destroy recognizability.

The point spread function is assumed to be separable in Cartesian coordinates (taken as horizontal and vertical). Separability1 reduces the analysis so that complex calculations that include cross-terms are not required. Separability in Cartesian coordinates requires that Display Formula

8h SPATIAL (x,y)=h SPATIAL (x)h SPATIAL (y).

Separability in polar coordinates requires Display Formula

9h SPATIAL (r,θ)=h SPATIAL (r)h SPATIAL (θ).

The PSF of an aberration-free optical system can be characterized by a function that is separable in polar coordinates. The detector is assumed to be rectangular. Its PSF is separable in Cartesian coordinates, but is not separable in polar coordinates. The collective PSF of the detector and the optics is not separable in either polar or Cartesian coordinates. The errors associated with separability are usually small2 and therefore most analyses use the Cartesian separability approximation. The detector array is assumed to be composed of rectangular (or square) detectors spaced in a rectangular (or square) grid (Fig. 1). Any other spacing (e.g., hexagonal) can only be analyzed on a case-by-case basis.3

The electronic imaging system response consists of both an optical response and electronic response. Time and spatial coordinates are treated separately: hSYSTEM(x, y, t) = hSPATIAL(x, y)hELECTRICAL(t). This is reasonable. Optical elements do not generally change with time and therefore are characterized only by spatial coordinates. Similarly, electronic circuitry exhibits only temporal responses. The detector provides the interface between the spatial and temporal components, and its response depends on both temporal and spatial quantities. The conversion of two-dimensional optical information to a one-dimensional electrical response assumes a linear photodetection process. Implicit in the detector response is the conversion from input photon flux to output voltage (or amps).

The electronic circuitry is assumed to modify the horizontal signal only (although this depends on the system design). With appropriate scaling, the electronic frequencies can be converted into spatial frequencies. This is symbolically represented by feu: Display Formula

10H SYSTEM (u)=H SPATIAL (u)H ELECTRONICS (feu),
and Display Formula
11H SYSTEM (v)=H SPATIAL (v).
A system is composed of many components that respond to spatial and temporal signals. Here lies the advantage of working in the frequency domain. If multiple LSI components exist in the spatial and/or electronic domains, the individual MTFs can be multiplied together. Equivalently, multiple convolutions in space or time are equivalent to multiplications (or cascading) in the frequency domain. For independent MTFs Display Formula
12 MTF SYSTEM (u,v)=i=1nj=1m MTF SPATIAL (i,u,v)× MTF ELECTRONICS (j,feu).

When coupled with the three-dimensional noise parameters4 the MTF uniquely defines system performance. The MTF determines how the system responds to spatial frequencies. It does not contain any signal intensity information.

Image formation is straight forward. Over the region that linear system theory is valid, the scene is transformed into its frequency components O(u, v). Each frequency is then multiplied by MTFSYSTEM(u, v) to provide ISYSTEM(u, v). Then the inverse transform provides ISYSTEM(x, y).

While an imaging system is composed of many subsystems, generally the MTF is dominated by the optics, detector, electronic filters, digital filters, and display medium. Adding electronic and digital filters to the analysis obscures the fundamentals of image creation. Here, the electronic and digital filter MTFs are assumed to be unity over the spatial frequencies of interest. The basic MTF is Display Formula

13 MTF SYSTEM = MTF OPTICS MTF DETECTOR MTF DISPLAY .
Camera manufacturers have no control over how an observer will process the imagery and therefore their analyses generally omit the display MTF. The perceived MTF depends upon display characteristics and the human visual system (HVS) interpretation Display Formula
14 MTF PERCEIVED = MTF SYSTEM MTF HVS ,
and is used for predicting image quality metrics. These metrics will not be discussed here. However, MTFHVS plays an important role when viewing imagery (discussed in Sec. 5).

Optics MTF

A complex optical system is replaced with a simple lens that has the equivalent focal length. For an aberration-free, radially symmetric optical system, OTFOPTICS is the same in the horizontal and vertical directions. Since the OTF is positive, it is labeled as the MTF. In the horizontal direction, the diffraction-limited MTF for circular aperture (Fig. 2) is Display Formula

15 MTF OPTICS u=2πcos1uuCuuC1uuC2.
The image-space optics cutoff frequency is uC = D/(λfl) = 1/(), where D is the aperture diameter and F = fl/D. Because the cutoff frequency is wavelength dependent, Eq. 15 and Fig. 2 are only valid for noncoherent monochromatic light. The extension to polychromatic light is lens-specific. Most lens systems are color corrected (achromatized) and therefore there is no simple way to apply this simple formula to predict the MTF. As an approximation to the polychromatic MTF, the average wavelength is used to calculate the cutoff frequency: λAVE = (λMAX+ λMIN)/2.

Grahic Jump LocationF2 :

Optics MTF for a clear (unobscured) circular aperture.

Detector MTF

The detector OTF cannot exist by itself. Rather, the detector OTF must also have the optical MTF to make a complete imaging system. In the horizontal direction, the OTF of a single rectangular detector is Display Formula

16 OTF DETECTOR (u)=sin(πdu)πdu.

The OTF is equal to zero when u = k/d (Fig. 3). The first zero (k = 1) is considered the detector cutoff frequency, uD. It is customary to plot the OTF up to uD. This is probably done because it is unknown what effect the negative OTF has on overall image quality. Since the OTF is positive up to uD, it is called the MTF. Plotting the MTF up to the first zero erroneously suggests that the detector does not respond to frequencies greater than uD. The optical system cutoff limits the absolute highest spatial frequency that can be faithfully imaged and not uD. Nevertheless most analyses (and that considered here) consider the response up to uD only.

Display MTF

Display specifications are a mix of CRT terminology, video transmission standards, alphanumeric character legibility, and graphics terminology. CRTs are low cost with high resolution, wide color gamut, and high luminance. Flat panels do not have all these attributes. Nevertheless, flat panel displays will probably replace all CRTs in the near future.

Flat panel displays are assumed to have rectangular pixels. Usually the number of pixels matches the number of detectors. When referred to image space, Display Formula

17 OTF DISPLAY (u)=sinπdCCuπdCCu.

As with the detector, the display response is (erroneously) plotted up to the first zero. When FF = 1 the display MTF is identical to detector MTF.

Sampling is an inherent feature of all electronic imaging systems. The scene is spatially sampled in both directions by the discrete detector locations. Sampling theory states that the frequency can be unambiguously recovered for all input frequencies below Nyquist frequency. After aliasing, the original signal can never be recovered. The mathematics suggests that aliasing is an extremely serious problem. Objection-ability depends upon the scene, /d, dCC, display medium, and viewing distance.

Sampling Theorem

The sampling theorem as introduced by Shannon5 was applied to information theory. He stated that if a time-varying function v(t) contains no frequencies higher than fMAX (Hz), it is completely determined by giving its ordinates at a series of points spaced 1/(2fMAX) seconds apart. The original function can be reconstructed by an ideal low-pass filter. Shannon's work is an extension of others,6 and the sampling theorem is often called the Shannon–Whittaker theorem. Reference 7 provides an in-depth discussion on sampling effects.

If sampling occurs every T seconds, the sampling frequency is fS = 1/T. The resultant signal is Display Formula

18v SAMPLE (t)=v(t)s(t),
where s(t) is the sampling function is equal to δ(tnT). Since multiplication in one domain is represented as convolution in the other, the sampled frequency spectrum is Display Formula
19V SAMPLE (fe)=V(fe)*S(fe),
where V(fe) is the amplitude spectrum of the band-limited analog signal and S(fe) is the Fourier transform of the sampler. The transform S(fe) is a series of impulses at ±nfS and is called a comb function. When convolved with V(fe), the resultant is a replication of V(fe) about nfS (n = −∞ to +∞). Equivalently, the sampling frequency interacts with the signal to create sum and difference frequencies. Any input frequency, fo, will appear as nfS ± fo after sampling. Figure 4 illustrates a band-limited system with frequency components replicated by the sampling process. The base band (−fH to fH) is replicated about nfS. To avoid distortion, the lowest possible sampling frequency is that value where the first replicated spectrum just adjoins the base band. This leads to the sampling theorem that a band-limited system must be sampled at twice the highest frequency (fS ≥ 2fH). Nyquist frequency is defined as fN = fS/2.

Grahic Jump LocationF4 :

Replicated spectra. The ideal reconstruction filter response is unity up to fN and zero thereafter. It eliminates the replicated spectra leaving only the analog base band.

After digitization, the data reside in data arrays (e.g., a computer memory location). The signal must be converted (reconstructed) into an analog signal to be useful. If the original signal was oversampled (fS ≥ 2fH) and if the reconstruction filter limits frequencies to fN, then the reconstructed image can be identical to the original image.

Within an overlapping band (fS < 2 fH), there is an ambiguity in frequency. It is impossible to tell whether the reconstructed frequency resulted from an input frequency of fo or nfS ± fo (Fig. 5).

Grahic Jump LocationF5 :

An undersampled sinusoid will appear as a lower frequency after ideal reconstruction.

Aliasing

To apply the above sampling theory to imaging systems, let feu, fSuS, and TdCC. As the sampling frequency decreases, the first replicated spectrum starts to overlap the base band (Fig. 6). It is the summation of these spectra that distort the image.

Grahic Jump LocationF6 :

Overlapping spectra. The amount of aliasing (and hence image quality) is related to the amount of overlap.

A bar pattern consists of an infinite number of frequencies. While the fundamental may be less than the Nyquist frequency, higher-order terms will not. These higher-order terms are aliased and distort the signal. In Fig. 7 the input bar pattern fundamental is 1.68 uN and the aliased fundamental is 0.32 uN. Since higher-order frequencies are present, the reconstructed bars appear more triangular than sinusoidal.

Grahic Jump LocationF7 :

Aliasing. Input (left) and aliased output (right). An ideal reconstruction filter was used. Imagery created (Ref. 8) by MAVIISS.

Since aliasing occurs at the detector, the signal must be band-limited by the optical system to prevent it. This is achieved by designing a system with /d ≥ 2 or by using an optical low pass filter (OLPF).9 While monochrome aliasing is tolerable, color aliasing is bothersome. Single chip color arrays always have a window over the array. The first impression is that the window is designed to protect the array. This is an ancillary feature. The “window” is actually a birefringent crystal that acts as an OPLF. The OPLF reduces the MTF and reduces image contrast. Since color aliasing is unacceptable, the reduced MTF is a small penalty to pay.

Reconstruction

Digital data cannot be seen because it resides in a computer memory. Any attempt to view a digital image requires a reconstruction filter.10 Most imaging systems rely on the display medium and HVS to produce a perceived continuous image (discussed in Sec. 5). Display media include laser printers, half-toning, fax machines, CRTs, and flat panel displays. The display medium creates an image by painting a series of light spots on a screen or ink spots on paper. The spot acts as a low pass reconstruction filter. Each display medium has a different spot size and shape resulting in different frequency responses. The perceived imagery will be different on each display type.

A flat panel display is not an ideal reconstruction filter. It passes significant frequency components above uN and this makes the image blocky or pixelated. A CRT will remove the higher frequencies (above uN) but also attenuates the in-band frequencies to create a somewhat blurry image (Fig. 8). The ideal reconstruction filter abruptly drops to zero at uN. As illustrated in Fig. 9, a sharp drop in one-domain produces ringing in the other (Gibbs phenomenon). In these two figures, the optics and detector MTFs are unity over the spatial frequencies of interest. This emphasizes how different reconstruction filters affect image quality.

Grahic Jump LocationF8 :

Reconstruction with a flat panel display (left) and a CRT (right). See footnote on page 8.

Grahic Jump LocationF9 :

Reconstruction with an ideal reconstruction filter. See footnote on page 8.

Moiré patterns occur when periodic targets are viewed. Some published articles provide resolution charts. The sampling artifacts become more noticeable when viewing targets at an angle with respect to the array axis. Figure 10 illustrates various artifacts when viewing a TV resolution chart. It compares a Bayer pattern with the Kodak TrueSense color filter array (CFA) and their respective demosaicking algorithms. This figure illustrates the difficulty in quantifying resolution. It illustrates distorted edges and periodic structures.

Grahic Jump LocationF10 :

Moiré patterns vary according to the CFA type and demosaicking algorithm. Details are provided in Ref. 11. The heavy horizontal lines represent the estimated TV resolution afforded by each camera. The on-line PDF version is in color where color aliasing can be seen. This figure has been enlarged. At “normal” viewing distances, the individual pixels cannot be resolved. View this figure from several feet.

Resampling

Nearly all analyses focus on the spatial sampling created by the detector array. If the camera output is analog, then a frame capture board redigitizes the image for computer processing. The board can digitize the analog signal at a rate that is different than uS. Additional samplers in the overall system can add new sampling artifacts.

Imagery imported into word documents or other programs are automatically interpolated to provide a smooth continuous image. As the image is enlarged interpolation smoothes it. This is not so when the image is resampled without interpolation. In Fig. 11, an enlarged portion of a scene captured by a 10 Mpixel point-and-shoot camera was inserted as a desktop picture. The flat panel display resampled the scene. Since the flat panel elements (1280×1024) did not align with the camera pixels (3664×2748), resampling artifacts become obvious: straight lines appear as a stair step (jaggies).

Grahic Jump LocationF11 :

Original image (left) and image seen on the WindowXP desktop background scene (right). Each picture is 210×245 pixels (H×V). The arrows point to the most obvious sampling artifacts. Careful examination reveals numerous others. The on-line PDF version is in color.

Spurious Response

In one dimension, the reconstructed image12 is Display Formula

20I(u)= MTF POST (u)n=0 MTF PRE (nuS±u)O(nuS±u).

MTFPRE contains all the MTFs up to the sampler (the detector). For this paper, MTFPRE = MTFOPTICSMTFDETECTOR. MTFPOST represents all the filters after the sampler. For this paper MTFPOST = MTFDISPLAY. Equation 20 can be written as Display Formula

21I(u)= MTF POST (u) MTF PRE (u)O(u)+ MTF POST (u)n=1 MTF PRE (nuS±u)O(nuS±u).

The first term is the image spectrum when no sampling is present and is called the direct response. Sampling created the remaining terms and these may be considered an aliasing metric. If uS ≥ 2uH and the reconstruction filter response is zero for all frequencies greater than uSuH (see Fig. 4), the second term is zero.

Considering the first fold back frequency (n = 1) and assuming O(u) = 1, Schade13 defined the spurious response as Display Formula

22 Spurious response =0 MTF POST (u) MTF PRE (uSu)du0 MTF POST (u) MTF PRE (u)du.

The highest scene spatial frequency is limited by the optical cutoff. The upper limit in the denominator is uC. The highest spatial frequency in the aliased signal (numerator) is limited by the reconstruction filter. The assumption that the scene contains all frequencies [O(u) = 1] with equal amplitude is for mathematical convenience. Perhaps Schade's spurious response should be called the potential for aliasing metric. Figure 12 illustrates the spurious response when a practical post-reconstruction filter is used. The spurious response value depends critically upon the reconstruction filter used.

Grahic Jump LocationF12 :

Practical reconstruction filter. MTFPOST(u)MTFPRE(uSu) was created by the sampling process. It was not in the original scene.

Image metrics may be described in the spatial domain where the optical blur diameter is compared to the detector size or in the frequency domain where the detector cutoff is compared to the optics cutoff. Either comparison provides an image quality metric14 that is a function of /d. Table 1 summarizes the two limiting cases. Since the transition from one region to the other is gradual, it is difficult to select an /d value that separates the two regions. It is nominally set at /d = 1. From a sampling viewpoint, the important parameter15 is /dCC (also called /ρ and Q). When FF = 1, /d = /dCC. Figure 13 illustrates overall design space for visible, mid-wave infrared (MWIR) and long-wave infrared (LWIR) cameras. As listed in Table 2, the more popular designs are detector-limited.

Grahic Jump LocationF13 :

Design space for visible, MWIR, and LWIR cameras (FF = 1). There is no aliasing when /d ≥ 2.

Table Grahic Jump Location
Optics-limited versus detector-limited performance.
Table Grahic Jump Location
/d for F = 2.

The MTF at Nyquist frequency is often used as a measure of performance (Fig. 14). As MTF(uN) increases, image quality should increase. Unavoidably, as the MTF increases, aliasing also increases and image quality suffers. Figure 15 provides 256×256 pristine images. Figures 1618 illustrate the imagery for three different /d ratios with FF = 1. Figure 8 illustrates /d = 0.55. All images were reconstructed with a flat-panel display. As evident in these figures, image distortion is more obvious with periodic targets and straight lines. Low /d imagery exhibits the most aliasing. In Fig. 16 the 3-bar targets appear as one, two, or distorted 3 bars. High /d imagery (Fig. 18) has no aliasing or distortion. But the imagery is blurry because the MTF is so low.

Grahic Jump LocationF14 :

MTFOPTICS(uN)MTFDETECTOR(uN) when FF = 1.

Grahic Jump LocationF16 :

Imagery when /d = 0.2. See footnote on this page.

Grahic Jump LocationF17 :

Imagery when /d = 1.0. See footnote on this page.

Grahic Jump LocationF18 :

Imagery when /d = 2.0. See footnote on this page.

Whether the image is displayed on a CRT, flat-panel display, or printed, the viewing distance significantly affects perceived quality. As illustrated in Fig. 19, as the target frequency increases, the perceived modulation decreases. As the distance increases a small object will eventually subtend an angle smaller than that which can be resolved by the HVS. If several objects are close together, they are perceived as one where the saturation, hue, and brightness are an average of all the objects. In this context, the HVS is an additional reconstruction filter.

Grahic Jump LocationF19 :

Perceived modulation (right) when viewing a sweep frequency target (left). Very high spatial frequencies are perceived as a uniform gray.

Flat Panel Displays

For monochrome systems, the minimum viewing distance occurs when the individual display pixels are barely perceptible. At closer distances, the pixels become visible and this interferes with image interpretation. The flat panel pixel consists of three color elements (red, green, and blue). At normal viewing distances, the eye's MTF attenuates the spatial frequencies associated with the display pixels thus producing a smooth continuous image. Assuming a display pixel pitch of 0.26 mm and a typical viewing distance of 0.355 m (14 in.), each pixel subtends 0.73 mrad and each color element subtends 0.24 mrad. The HVS can resolve 0.29 mrad (equivalent to 20/20 vision). Thus at 0.355 m, each pixel is

resolved and each color element is not (desired result). At 0.154 m (6 in.), each color element subtends 0.56 mrad and can be just perceived. When too close, the individual color spots become visible and the impression of full color is lost.

For all the imagery in this paper, it is suggested that you move several feet away to achieve the same visual angle that would normally exist.

Prints

Color printing suffices with four colors (red, green, blue, and black). There are no grays. It is either ink or no ink. For lighter colors, a small ink spot is surrounded by a white area. To increase saturation, the ink spot is larger. For fixed pixel size the white area is smaller. The HVS blends the ink spot and white area to have a perceived saturation.

Printed imagery is considered “excellent” when there are more than 300 dots/in (dpi). Photos look good when there are about 300 pixels/in. To allow for color printing, the printer should provide about 3 times more dots, or at least 900 dpi. The values in Table 3 assume 300 pixels/in creates an excellent image. Larger images can be created. If pixel replication is used, at some point they will start to be blocky (you can see the individual pixels). However, blocky images are rarely seen because software always interpolates the data to create a smooth image. At small viewing distances this smoothed image may appear blurry. In contrast to wet-film cameras, image enlargement does not provide more resolution. Resolution is fixed by the pixel size and focal length.

Table Grahic Jump Location
Digital still camera prints (assuming 300 pixels per inch is acceptable).

Performance for all cameras (point-and-shoot, TV studio broadcast, and thermal infrared imaging) can be described by an MTF with /d being an important design metric. The difference between detector-limited and optics-limited is important. When detector-limited (/d < 1), changes in the optical system will have little effect on image quality. Likewise, in the optics-limited region (/d > 1), changing the detector size will have minimal effect on image quality. While Fig. 13 illustrates overall design space, most imaging systems today are detector-limited (Table 2).

The MTF will always be used for lens design. It will be used for scientific and military system optimization. It is of lesser concern for commercial applications. The MTF at Nyquist frequency is often used as a measure of performance (Fig. 14). As MTF(uN) increases, image quality should increase. Unavoidably, as the MTF increases, aliasing also increases and image quality suffers. This suggests there may be an optimum MTF, or, equivalently an optimum /d. This is not so. Cameras are designed for a specific application with /d being a secondary consideration.

There is no aliasing when /d ≥ 2. This limiting condition is known as “critical sampling” or “Nyquist sampling of the optical blur” in the astronomy community.16 The latter term is not good terminology since the blur diameter consists of all frequencies up to the optical cutoff. When /d ≥ 2 the imagery replicates the scene exactly. This may be an important consideration for medical imaging where a sampling artifact could be construed as a medical abnormality or a space probe where it is impossible to obtain ground truth.

Having /d ≥ 2 may be overly restrictive. In-band MTFs (frequencies less than the Nyquist frequency) are reduced in amplitude (Fig. 18). If the SNR is sufficiently high, a boost circuit can increase the MTF and sharpen the image. Note that if /d < 2, the boost circuit will also increase the aliased signal [(MTFPOST(u)MTFPRE(uS−u) in Fig. 12]. If too much boost is used, the amplified aliased signal can degrade image quality.

Sampling theory suggests that an ideal reconstruction filter could be used. Although unrealizable, it can be approximated by a high order Butterworth filter. This works well if there is no aliasing. With aliasing, the sharp cutoff filter creates undesirable ringing (Fig. 9). Flat panel displays are not ideal reconstruction filters and will not produce ringing.

Aliasing was quantified by the spurious response. The “information” in the higher frequencies has been aliased to lower frequencies. But it is not known how to interpret this information. Mathematically, worst case analysis assumes that the scene contains all spatial frequencies with equal amplitude. Real scenes have a limited spectrum and image quality is a subjective measure. This means there is no method of validating the theory with imagery. MTF theory and sampling issues are just two slices through the multidimensional image quality space. They provide guidance for camera design but do not uniquely quantify image quality. The perceived image quality depends upon /d, dCC, display medium, and viewing distance. Changing any one or all of these parameters affects the perceived image quality.

Nearly every visible digital camera aliases the scene (Fig. 13). Is this really bad? Sampling artifacts are seen routinely. The amount of aliasing is scene specific and may or may not be bothersome. It becomes apparent when viewing test patterns (Figs. 10,16), picket fences, plowed fields, railroad tracks, and Venetian blinds. In fact, while aliasing is present (Fig. 11), the imagery may be considered excellent. The color rendition on the flat panel display is excellent. The scene content is exciting and the few jagged lines are just ignored. On the other hand, a printed image is enlarged to 16×20 in. (see Table 2) and looks wonderful.

Why is aliasing acceptable? The eye is primarily sensitive to intensity variations and less so to frequencies. Therefore, sampling artifacts in imagery are usually tolerable. We have become accustomed to the aliasing. Simply watch television. Pay attention to the folks wearing clothing with narrow stripes or small patterns. Note the changing patterns as they move. Stripes will change in shape, width, and color.

In contrast, the ear is a frequency detector and any distortion is immediately obvious. The sampling theorem must be strictly followed for auditory-based processes but not for imagery.

Different display media create different images. Image quality is scene dependent. Go to any store selling TVs and observe the difference between “identical” TVs. Note the differences at different viewing distances. As the distance decreases, the differences become more apparent. Simply stated, all displays are designed for an assumed viewing distance. The printed image will never be the same as that seen on the flat panel display.

Finally, wet-film based cameras did not alias the scene. With technological advancements in digital cameras, image quality has decreased. This is the trade-off between instant imagery, ability to digitally transmit the imagery, and image quality. Current trends are to build smaller detectors because it leads to lower cost and lighter cameras. Staring array detectors create photoelectrons that are stored in charge wells. But the charge well capacity decreases as the pixel area decreases. The next generation camera will have a smaller dynamic range, smaller SNR, and poorer image quality.

Driggers  R. G., , Cox  P., , and Edwards  T.,  Introduction to Infrared and Electro-Optical Systems. , pp. 21–25 ,  Artech House ,  Norwood, MA  ((1999)).
Vollmerhausen  R. H., and Driggers  R. G.,  Analysis of Sampled Imaging Systems. , SPIE Tutorial Text TT39, pp. 28–31 ,  Bellingham, WA  ((2000)).
Hadar  O., , Dogariu  A., , and Boreman  G. D., “ Angular dependence of sampling modulation transfer function. ,” Appl. Opt.. 3628, , 7210–7216  ((1997)).
Holst  G. C.,  Electro-optical Imaging System Performance. , 5th ed., pp. 362–374 ,  JCD Publishing Company ,  Winter Park, FL  ((2008)).
Shannon  C. E., “ Communication in the Presence of Noise. ,”  Proceedings of the IRE.  Vol. 37, , pp. 10–21 , Institute of Electrical and Electronics Engineers, New York ((1949)).
Jerri  A. B., “ The Shannon Sampling Theorem – Its Various Extensions and Applications: A Review. ,”  Proceedings of the IEEE. , Vol. 85, (11 ), pp. 1565–1595 , Institute of Electrical and Electronics Engineers, New York ((1977)).
Holst  G. C.,  Sampling, Aliasing, and Data Fidelity. ,  JCD Publishing ,  Winter Park, FL  ((1998)).
MAVIISS (MTF based Visual and Infrared System Simulator) is an interactive software program available from JCD Publishing at www.JCDPublishing.com.
Holst  G. C., and Lomheim  T. L.,  CMOS/CCD Sensors and Camera Systems. , 2nd ed., pp. 287–289 ,  JCD Publishing Company ,  Winter Park, FL  ((2011)).
Holst  G. C., “ Are reconstruction filters necessary?. ,” In  Infrared Imaging Systems: Design, Analysis, Modeling, and Testing.  XVII, Holst  G. C., Ed., Proc. SPIE. 6207, , 62070K  ((2006)).
Tamburrino  D., , Speigle  J. M., , Tweet  D. J., , and Lee  J.-Jan, “ 2PFC (two pixels, full color): Image sensor demosaicing and characterization. ,” J. Electron. Imaging. 19, (2 ), 021103  ((2010)).
Holst  G. C.,  Electro-Optical Imaging System Performance. , 5th ed., pp. 197–199 ,  JCD Publishing Company ,  Winter Park, FL  ((2008)).
Schade  O. H.  Sr., “ Image reproduction by a line raster process. ,” in  Perception of Displayed Information. , Biberman  L. C., Ed., pp. 233–278 ,  Plenum Press ,  New York  ((1973)).
Holst  G. C., “ Imaging system performance based upon Fλ/d. ,” Opt. Eng.. 46, , 103204  ((2007)).
Fiete  R. D., “ Image quality and λFN/ρ for remote sensing systems. ,” Opt. Eng.. 38, , 1229–1240  ((1999)).
Clampin  M., “ Ultraviolet-optical charge-coupled devices for space instrumentation. ,” Opt. Eng.. 41, , 1185–1191  ((2002)).

Grahic Jump LocationImage not available.

Gerald C. Holst is an independent consultant for imaging system analysis and testing. His varied background includes serving as a technical liaison to NATO, research scientist for DoD, and a member of the Martin Marietta (now Lockheed Martin) senior technical staff. He has planned, organized, and directed the internationally acclaimed SPIE conference “Infrared Imaging Systems: Design, Analysis, Modeling and Testing” since 1990. He is author of over 30 journal articles and 6 books. He is a SPIE fellow and a member of OSA.

The 256×256 image (Fig. 15) was downsampled to 32×32 detectors. The imagery is enlarged so that your eye MTF and the printing do not significantly affect the image quality. This allows you to see the distortion created by sampling and the system MTF degradation. Image quality depends upon viewing distance. View the images at several feet to view to simulate “normal distance” (same visual angle).

© 2011 Society of Photo-Optical Instrumentation Engineers (SPIE)

Citation

Gerald C. Holst
"Imaging system fundamentals", Opt. Eng. 50(5), 052601 (February 14, 2011March 05, 2011March 07, 2011May 10, 2011May 10, 2011). ; http://dx.doi.org/10.1117/1.3570681


Access This Article

Figures

Grahic Jump LocationF1 :

Detector and pixel relationship.

Grahic Jump LocationF2 :

Optics MTF for a clear (unobscured) circular aperture.

Grahic Jump LocationF4 :

Replicated spectra. The ideal reconstruction filter response is unity up to fN and zero thereafter. It eliminates the replicated spectra leaving only the analog base band.

Grahic Jump LocationF5 :

An undersampled sinusoid will appear as a lower frequency after ideal reconstruction.

Grahic Jump LocationF6 :

Overlapping spectra. The amount of aliasing (and hence image quality) is related to the amount of overlap.

Grahic Jump LocationF7 :

Aliasing. Input (left) and aliased output (right). An ideal reconstruction filter was used. Imagery created (Ref. 8) by MAVIISS.

Grahic Jump LocationF8 :

Reconstruction with a flat panel display (left) and a CRT (right). See footnote on page 8.

Grahic Jump LocationF9 :

Reconstruction with an ideal reconstruction filter. See footnote on page 8.

Grahic Jump LocationF10 :

Moiré patterns vary according to the CFA type and demosaicking algorithm. Details are provided in Ref. 11. The heavy horizontal lines represent the estimated TV resolution afforded by each camera. The on-line PDF version is in color where color aliasing can be seen. This figure has been enlarged. At “normal” viewing distances, the individual pixels cannot be resolved. View this figure from several feet.

Grahic Jump LocationF11 :

Original image (left) and image seen on the WindowXP desktop background scene (right). Each picture is 210×245 pixels (H×V). The arrows point to the most obvious sampling artifacts. Careful examination reveals numerous others. The on-line PDF version is in color.

Grahic Jump LocationF12 :

Practical reconstruction filter. MTFPOST(u)MTFPRE(uSu) was created by the sampling process. It was not in the original scene.

Grahic Jump LocationF13 :

Design space for visible, MWIR, and LWIR cameras (FF = 1). There is no aliasing when /d ≥ 2.

Grahic Jump LocationF14 :

MTFOPTICS(uN)MTFDETECTOR(uN) when FF = 1.

Grahic Jump LocationF16 :

Imagery when /d = 0.2. See footnote on this page.

Grahic Jump LocationF17 :

Imagery when /d = 1.0. See footnote on this page.

Grahic Jump LocationF18 :

Imagery when /d = 2.0. See footnote on this page.

Grahic Jump LocationF19 :

Perceived modulation (right) when viewing a sweep frequency target (left). Very high spatial frequencies are perceived as a uniform gray.

Tables

Table Grahic Jump Location
Optics-limited versus detector-limited performance.
Table Grahic Jump Location
/d for F = 2.
Table Grahic Jump Location
Digital still camera prints (assuming 300 pixels per inch is acceptable).

References

Driggers  R. G., , Cox  P., , and Edwards  T.,  Introduction to Infrared and Electro-Optical Systems. , pp. 21–25 ,  Artech House ,  Norwood, MA  ((1999)).
Vollmerhausen  R. H., and Driggers  R. G.,  Analysis of Sampled Imaging Systems. , SPIE Tutorial Text TT39, pp. 28–31 ,  Bellingham, WA  ((2000)).
Hadar  O., , Dogariu  A., , and Boreman  G. D., “ Angular dependence of sampling modulation transfer function. ,” Appl. Opt.. 3628, , 7210–7216  ((1997)).
Holst  G. C.,  Electro-optical Imaging System Performance. , 5th ed., pp. 362–374 ,  JCD Publishing Company ,  Winter Park, FL  ((2008)).
Shannon  C. E., “ Communication in the Presence of Noise. ,”  Proceedings of the IRE.  Vol. 37, , pp. 10–21 , Institute of Electrical and Electronics Engineers, New York ((1949)).
Jerri  A. B., “ The Shannon Sampling Theorem – Its Various Extensions and Applications: A Review. ,”  Proceedings of the IEEE. , Vol. 85, (11 ), pp. 1565–1595 , Institute of Electrical and Electronics Engineers, New York ((1977)).
Holst  G. C.,  Sampling, Aliasing, and Data Fidelity. ,  JCD Publishing ,  Winter Park, FL  ((1998)).
MAVIISS (MTF based Visual and Infrared System Simulator) is an interactive software program available from JCD Publishing at www.JCDPublishing.com.
Holst  G. C., and Lomheim  T. L.,  CMOS/CCD Sensors and Camera Systems. , 2nd ed., pp. 287–289 ,  JCD Publishing Company ,  Winter Park, FL  ((2011)).
Holst  G. C., “ Are reconstruction filters necessary?. ,” In  Infrared Imaging Systems: Design, Analysis, Modeling, and Testing.  XVII, Holst  G. C., Ed., Proc. SPIE. 6207, , 62070K  ((2006)).
Tamburrino  D., , Speigle  J. M., , Tweet  D. J., , and Lee  J.-Jan, “ 2PFC (two pixels, full color): Image sensor demosaicing and characterization. ,” J. Electron. Imaging. 19, (2 ), 021103  ((2010)).
Holst  G. C.,  Electro-Optical Imaging System Performance. , 5th ed., pp. 197–199 ,  JCD Publishing Company ,  Winter Park, FL  ((2008)).
Schade  O. H.  Sr., “ Image reproduction by a line raster process. ,” in  Perception of Displayed Information. , Biberman  L. C., Ed., pp. 233–278 ,  Plenum Press ,  New York  ((1973)).
Holst  G. C., “ Imaging system performance based upon Fλ/d. ,” Opt. Eng.. 46, , 103204  ((2007)).
Fiete  R. D., “ Image quality and λFN/ρ for remote sensing systems. ,” Opt. Eng.. 38, , 1229–1240  ((1999)).
Clampin  M., “ Ultraviolet-optical charge-coupled devices for space instrumentation. ,” Opt. Eng.. 41, , 1185–1191  ((2002)).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

PubMed Articles
Polarization-based non-staining cell detection. Opt Express 2012;20(23):25378-90.
Advertisement

Buy this article ($18 for members, $25 for non-members).
Sign In