CapContact

More images

Figure 2: CapContact’s high-resolution reconstruction of the contact area between the user’s finger and the surface produces a more accurate input location of the touch event compared to the center of mass from the capacitive values.

Figure 3: Our generator is a VGG-style fully-convolutional network. Five residual blocks extract the relevant features from the low-resolution capacitive input. Three consecutive sub-pixel convolution layers then upsample it 8×.

Figure 4: Our critic is based on a VGG-style convolutional neural network. Instead of batch normalization, we apply layer normalization to match the WGAN-GP loss. Our network uses ten convolutional layers to reduce the dimensions of the input image, before it reduces the input to a single dimension using average pooling.

Figure 5: Our apparatus integrates mutual-capacitance touch sensing and optical touch sensing using frustrated total internal reflection to capture accurate contact areas. We flipped the capacitive sensor to expose the ITO layer to the top. A sheet of Plexiglas atop completed the FTIR setup.

Figure 6: In our data capture study, participants produced a series of touch input and drag events using each of these finger combinations (black circles indicate touches), once at 0° yaw rotation and once at 90° yaw rotation. The figure includes representative capacitive images and their corresponding contact area images from the FTIR sensor for each pose.

Figure 7: We compare predictions to the ground-truth contact mask using these metrics. For offsets, we first adjust for yaw. Not shown: percentage difference in aspect ratios.

Figure 8: Exp.1—Tested across blocks within participants, CapContact and the learned threshold achieved a much higher IoU score than the baseline, improving it by factors of 3.5 and 3.2, respectively. On average, CapContact reduced the centroid error by ∼30% and the learned threshold by 6%.

Figure 9: Exp. 2—Tested across participants, CapContact and the learned threshold achieved a much higher IoU score than the baseline, improving it by factors of 3 and 2.8, respectively. CapContact reduced the average centroid error by 20%, whereas the learned threshold reduced it by 2.5%.

Figure 10: Exp.4—Half-resolution performance on touch metrics: CapContact and the learned threshold achieved an average IoU score that is 3.7× and 3.1× higher than the baseline, respectively. CapContact lowered the average centroid error by 22%, while the learned threshold was 3% worse.

Figure 11: Exp.6—CapContact running at half resolution (Project Zanzibar mat, 7 mm pitch). Solely trained on down-sized data from our capture study, CapContact robustly detects individual contact areas while all threshold-based methods suffer from noise and missed touch events.

Figure 12: Sample image of hand side resting on the surface when writing with a pen. Since our dataset to-date contains combinations of finger touches at various angles, but does not include more general and complex shapes, the fidelity of shape approximation is limited. While CapContact reconstructs shapes that are more accurate than the bicubic baseline, which produces an expansive imprint, our method still misses the fine crevasse between parts of the palm.