In the wake of COVID-19, the digital fitness market combining health equipment and ICT technologies is experiencing unexpected high growth. A smart trampoline fitness system is a new representative home exercise equipment for muscle strengthening and rehabilitation exercises. Recognizing the motions of the user and evaluating user activity is critical for implementing its self-guided exercising system. This study aimed to estimate the three-dimensional positions of the user’s foot using deep learning-based image processing algorithms for footprint shadow images acquired from the system. The proposed system comprises a jumping fitness trampoline; an upward-looking camera with a wide-angle and fish-eye lens; and an embedded board to process deep learning algorithms. Compared with our previous approach, which suffered from a geometric calibration process, a camera calibration method for highly distorted images, and algorithmic sensitivity to environmental changes such as illumination conditions, the proposed deep learning algorithm utilizes end-to-end learning without calibration. The network is configured with a modified Fast-RCNN based on ResNet-50, where the region proposal network is modified to process location regression different from box regression. To verify the effectiveness and accuracy of the proposed algorithm, a series of experiments are performed using a prototype system with a robotic manipulator to handle a foot mockup. The three root mean square errors corresponding to X, Y, and Z directions were revealed to be 8.32, 15.14, and 4.05 mm, respectively. Thus, the system can be utilized for motion recognition and performance evaluation of jumping exercises.