2.1.

Liver CT Data

2.2.

Reader Study

To track and record the reader’s gaze, an eye-tracking device (EyeLink1000, SR Research Ltd., Mississauga, Ontario, Canada) was positioned below the image display and calibrated to maintain the average gaze error below 1 deg. Fixations were detected using the default parameters: eye velocity and acceleration thresholds of $30\mathrm{deg}/\mathrm{s}$ and $9500\mathrm{deg}/{\mathrm{s}}^2$, respectively. The participants were seated in front of a 22 in. (56 cm) screen suited for medical image display in a reading room with low illuminance ($<50\mathrm{lux}$). The participant’s head position was fixed with a forehead- and chin-rest mount to improve accuracy in eye gaze measurements and control the visual angles. Before each reading session, a calibration procedure was applied to ensure a good eye-tracking accuracy. An additional eye-tracking drift check was performed between each trial. The cases were presented with a magnification factor of 2 with a window level of 50 HU and a width of 300 HU. Readers had no possibility to zoom or pan the images, neither to adjust the image contrast.

Using a mouse wheel, the readers could freely scroll forward and backward through all the slices and were instructed to mark a lesion with a mouse click. Before the actual trials, they were shown examples of the signal to be searched, and they were informed that each case contained at least one lesion to localize. No time limitation was imposed to encourage a thorough evaluation of each case.

In total, 20 readers took part in the experiment with reading expertise ranging from 1 to 17 years. In terms of demography, the reader group consisted of one undergraduate medical student, sixteen 1 to 5 year radiology residents, 3 fellows with 5 to 8 years of clinical experience in body imaging, and one radiologist with an experience of 17 years in abdominal CT imaging. However, for one participant (5 to 8 years of clinical experience in body imaging group), low calibration accuracy resulted in unreliable eye position data. These participant data were therefore removed from our study.

2.3.

Data Recording and Analysis

From the first scrolling wheel activation until the end of the trial, the eye gaze position in $x$, $y$ (within slice coordinates), and $z$ (slice number) was recorded at 60-Hz rate. The marker position was recorded when the readers localized a lesion.

From the raw gaze data and marker positions, we derived the following measurements: localization hit rate, perceptual and search error rate, search duration, saccades amplitude, liver coverage, and strategy quantification. A reader’s marking was considered a localization hit when it fell into a disk centered on the lesion’s “center of mass,” whose radius was twice the radius of the lesion. The hit rate was defined as the number of lesions correctly marked compared with the total number of lesions. A perceptual error refers to a missed lesion that was fixated.^14,15 A lesion was considered fixated if it was encompassed by a 2–deg-diameter circle centered on the recorded fixation locations. A search error refers to a missed lesion that was not fixated (encompassed by 2-deg–diameter circles around the fixations). The search time was considered the time measured between the first fixation onto the liver parenchyma and the moment the reader decided to terminate the trial. The saccade amplitude was defined as the distance between two consecutive fixations, measured in degrees.

The coverage was defined by the liver volume encompassed by a gaze cone defined by a 5-deg diameter disk centered on the gaze coordinate. Every point of the image that fell within the 5-deg gaze cone was considered as visible. We chose 5 deg to be consistent with the literature and the concept of the useful field of view (UFOV).¹⁵ For a $-50\text{-}\mathrm{HU}$ signal contrast, more than 70% of the detection saccades were within 5 deg of the previous fixation, and for a $-30\mathrm{HU}$ signal contrast more than 87% of the detection saccades were within 5 deg.

To classify the readers according to their strategy, we measured their EMI.⁶ This parameter had already been developed for the detection of lung nodules,⁶ and we extended it to focal liver lesions.

2.4.

Search Strategy Metrics

We first followed previous approaches to categorize drillers and scanners using the EMI.⁶ The EMI was derived from the summation of two components: (1) the saccadic amplitude, measured in degrees and (2) the time-averaged number of crossings over a line that delimits the left and right parts of the liver, measured in ${\mathrm{s}}^{-1}$ (Fig. 1). Before doing the summation, both quantities were normalized to the maximum value relative to the readers’ population. The only variation of the definition given by Drew et al.⁶ is that the eye movement crossovers in the lung CT study were defined across quadrants while in the current study we measured crossovers across the left and right part of the liver.

Fig. 1

Example of one liver slice from our study with colored overlay showing left and right parts of the liver. On the anatomical level, the left and right liver are defined differently but we chose to separate them according to the left and right parts of the screen. This makes it possible to have two almost similar volumes while anatomically the right liver represents the largest part of the organ. Having two of the same volumes is essential for the detection of crossovers because the number of crossovers is defined as the number of times a saccade crosses the line delimiting left and right liver during a trial. Furthermore, this separation facilitates the division of the organ at each test.

Drew et al.⁶ classified readers based on the EMI into two categories of search strategies: drillers go back and forth across slices during the trial, and each time they tend to fixate on a different area of the image. The few eye movements in the ($x,y$) plane are compensated by many back and forth scrolls across image slices ($z$). In contrast, the scanners scroll in one direction throughout the image stack and tend to explore each image slice one after the other through multiple fixations. The use of this search strategy for the scanners results in a high-EMI value.

Because the scanners might also perform fewer back and forth scrolling than the drillers, we decided to measure the number of courses, which we defined as the number of times a reader scrolled in a given direction during the test. For instance, a reader who scrolled through the image stack in one direction, then reversed through a couple of image slices and finally scrolled again in the original direction until the last slice would have performed three courses.

To evaluate a potential relationship between the EMI and the number of courses, we first computed the mean values of both parameters for each reader. We then labeled each reader as having either a high or a low EMI, and, respectively, a high or a low number of courses. The threshold between high and low categories was defined by the median value among all the readers. Therefore, a reader was labeled as a high EMI with a capital “E” if his or her mean EMI was above the median value computed among all the readers. Conversely, a reader with a mean EMI lower than the median was labeled with a lowercase “e.” We did similarly with the number of courses: a reader with a mean number of courses higher than the median of all the readers was labeled with a capital “C,” and a reader below the median was labeled with a lowercase “c.”

Because the images contained multiple lesions, we quantified the hit rate as the number of localized lesions divided by the total number of lesions across all cases ($N=49$ in $-50\mathrm{HU}$ image set, $N=45$ in $-30\text{-}\mathrm{HU}$ image set). Similarly, the miss rate was quantified as the number of missed lesions divided by the total number of all lesions across all cases. We also defined, as search errors, those lesions that were not fixated and missed (not localized).^8,14,16 Perceptual errors were defined as lesions that were fixated and missed.^8,14,16

To assess statistical significance, we used parametric $t$-tests. Independent samples $t$-tests were used when comparing separate groups of radiologists classified based on the EMI or number of course criteria. Paired $t$-tests were used when comparing the same individuals across signal contrast conditions. We used Pearson correlations to quantify the relationship between various variables. The analyses were performed with the Microsoft Excel 2016 Analysis ToolPak.

3.1.

Characterization of Reader Strategy and the Role of Scrolling Behavior

The number of courses was estimated by plotting the image slices (slice number in the $z$ direction) versus time for each trial and each reader. Figure 2 shows two archetypical examples of scrolling behavior: one with seven courses consistent with driller strategy (left) and another with a single course highly compatible with the behavior of a scanner, as described in Ref. 6.

Fig. 2

Depth (slice number in the $z$ direction) versus time plot example for typical (a) driller and (b) scanner. In this example, the number of courses per trial was 7 for the driller-like reader and 1 for the scanner-like reader. Colors indicate which part of the liver was analyzed by the reader. For driller, each part was individually fixated while scrolling, and for scanner both parts, left and right liver were alternatively fixated while scrolling.

Figure 3 shows the relationship between EMI and the mean number of courses for each of the readers in our study. We classified the readers based on the four reader categories delimited by the medians of the EMI and number of courses parameters (see Sec. 2). The first group is identified as “Ec” for high EMI and low number of courses, the second group is “EC” for high EMI and high number of courses, the third group is “ec” for low EMI and low number of course, and the fourth group is “eC” for low EMI and high number of courses. The most experienced radiologist (17 years) falls into group Ec in both higher and lower contrast case study.

Fig. 3

EMI versus the average number of courses over all trials for each signal contrast: (a) $-50$ and (b) $-30\mathrm{HU}$. Plain lines correspond to median value of EMI and number of courses across the entire sample of readers. Readers were grouped according to their EMI and number of courses in comparison to median values across the entire sample.

For both signal contrast values, readers with a high EMI tend to have a small number of courses and vice versa. However, while this is significant for the higher contrast ($r=-0.56$; $p=0.01$), it is not the case for the lower contrast ($r=-0.26$; $p=0.27$).

Comparing the behavior of the individual readers across contrasts (higher contrast versus lower contrast) showed that 7 readers out of 20 changed their EMI category (1 from Ec to ec, 3 to EC to eC, and 3 from eC to EC), and only 1 reader changed the number of courses (from ec to eC). No reader changed both EMI and number of courses. This finding suggests that a categorization of readers based on the number of courses might be more invariant across signal contrast conditions than using the EMI.

Figure 4 shows the dependence of the two quantities that define the EMI: the mean crossover per second versus the mean saccadic amplitude. There is a positive correlation between these two quantities, with $r=0.82$ ($p<0.01$) and $r=0.92$ ($p<0.01$) for $-50$ and $-30\mathrm{HU}$, respectively.

Fig. 4

Relationship between the two parameters that define the EMI. Mean crossover per second versus mean saccadic amplitude for (a) $-50$ and (b) $-30\mathrm{HU}$.

The fact that the correlation between the two variables composing the EMI (the crossovers and the saccade amplitude) is higher than the correlation of the EMI and the number of courses suggests that the latter adds additional information to characterize the reader’s search strategy.

3.2.

Search Performance

In this section, we investigated how radiologists categorized along the various search strategy characteristics (EMI and number of courses), differ in basic visual search measures such as the UFOV (mean covered volume) and the trial decision time (mean trial duration). Figure 5 shows the relationship between the liver volume coverage and the mean trial duration. Readers with a high number of courses (eC and EC) tend to cover more volume [at $-50\mathrm{HU}$ ($p=0.03$) and at $-30\mathrm{HU}$ ($p=0.01$)] in more time [at $-50\mathrm{HU}$ ($p<0.01$) and at $-30\mathrm{HU}$ ($p<0.01$)] than readers with a low number of courses (Ec and ec). No trend was observed in terms of covered volume or trial duration when we look at the readers’ EMI (all $p$-values $>0.4$).

Fig. 5

Liver volume coverage with respect to trial duration for (a) $-50$ and (b) $-30\mathrm{HU}$.

The covered volume is positively correlated with trial duration for the high contrast ($r=0.68$, $p<0.01$) and for the low-contrast signal ($r=0.58$, $p<0.01$). As expected, the decrease in signal contrast tends to increase the need to thoroughly search the volume (the coverage; $p<0.01$) and the duration of trials ($p<0.01$).

Figure 6 shows the relationship between the localization hit rate and the liver volume coverage. The results show a ceiling effect at the $-50\mathrm{HU}$ signal contrast and thus the hit rate varies with the number of courses or EMI ($p=0.6$ for the number of courses and $p=0.3$ for EMI). However, the hit rate at $-30\mathrm{HU}$ signal contrast is significantly higher for readers with higher number of courses (difference in mean hit $\mathrm{rate}=0.44$; $p=0.04$) than for these with a lower number of courses [Fig. 6(b)]. This was not the case when we categorized the readers on the basis of a high and low EMI (difference in mean $\mathrm{EMI}=0.1$; $p=0.3$).

Fig. 6

Liver volume coverage with respect to localization hit rate for (a) $-50$ and (b) $-30\mathrm{HU}$.

Figure 7 shows the search error rate (missed lesion that were not fixated) versus the trial duration. As expected, the $-50\mathrm{HU}$ contrast images [Fig. 7(a)] led to shorter observation times and significantly lower search errors than $-30\mathrm{HU}$ [Fig. 7(b)] for all groups. For the $-30\mathrm{HU}$ signal contrast, readers with a high number of courses (eC and EC) tended to have longer trial durations [$\text{mean difference}=42\mathrm{s}$; $p<0.01$; CI95% (100,131) compared to CI95% (64,91)] and lower search error rates [$\text{mean difference}=0.13$, $p<0.01$; CI95% (0.06,0.12) compared to CI95% (0.16,0.21)] than readers with a low number of courses (Ec and EC). However, a change of EMI did not result in a significant difference neither for trial durations [$\text{mean difference}=18\mathrm{s}$; $p=0.2$; CI95% (78,125) for low e; CI95% (78,104) for E] nor for search error rates [$\text{mean difference}=0.04$; $p=0.2$; CI95% (0.08,0.18) for low e; CI95% (0.12,0.19) for E].

Fig. 7

Search error rate versus trial duration for (a) $-50$ and (b) $-30\mathrm{HU}$.

Figure 8 shows the perceptual error rate (missed lesions that were fixated) versus the trial duration. The results show that the $-50\mathrm{HU}$ contrast images led to significantly fewer errors than $-30\mathrm{HU}$ contrast images ($p<0.01$). The perceptual error rate was not dependent on the number of courses ($\text{mean difference}=0.31$ ; $p=0.5$ for $-50\mathrm{HU}$ and $\text{mean difference}=0.04$, $p=0.8$ for $-30\mathrm{HU}$) or the EMI ($\text{mean difference}=0.01$, $p=0.3$ for -50 HU and $\text{mean difference}=0.04$, $p=0.7$ for $-30\mathrm{HU}$).

Fig. 8

Perceptual error rate versus trial duration for (a) $-50$ and (b) $-30\mathrm{HU}$.

3.3.

Effect of Signal Contrast on Search Strategy

To highlight the effect of a lower signal contrast on the search strategy, we estimated the difference in EMI and the mean number of courses when the signal contrast changed from $-50$ to $-30\mathrm{HU}$. To understand the EMI variation, we also estimated the variation ($\mathrm{\Delta }$) of its two components when the contrast was decreased: the saccadic amplitude and crossover per second. Figure 9(a) shows $\mathrm{\Delta }\mathrm{EMI}$ versus $\mathrm{\Delta }\text{course}$. Figure 9(b) shows $\mathrm{\Delta }\text{crossover}$ per second versus $\mathrm{\Delta }\text{saccadic}$ amplitude for each reader, where $\mathrm{\Delta }$ is the difference of the considered parameter from $-50$ to $-30\mathrm{HU}$. For all readers, $\mathrm{\Delta }\text{course}$ is positive, whereas for most readers, $\mathrm{\Delta }\mathrm{EMI}$ is negative. This means that when the task becomes more difficult, the EMI tends to decrease and the number of courses to increase. In other words, as the signal contrast lowers, the readers tend to drill more. The fact that $\mathrm{\Delta }\text{saccadic}$ amplitude and $\mathrm{\Delta }\text{crossover}$ per second tend to be negative means that both parameters contribute to the decrease of EMI with decreasing signal contrast.

Fig. 9

Effect of the signal contrast on EMI and number of courses ($\mathrm{\Delta }$ is the difference of the considered parameter from $-50$ to $-30\mathrm{HU}$). (a) $\mathrm{\Delta }\mathrm{EMI}$ versus $\mathrm{\Delta }\text{course}$. (b) Decomposition of the EMI in its two components: $\mathrm{\Delta }\text{saccadic}$ amplitude versus $\mathrm{\Delta }\text{crossover}$ per second. Note: the symbols correspond to the four reader categories defined in Fig. 3(a).