This paper proposes an automatic cast product surface defect detection system based on deep learning artificial intelligence technology. Application of deep learning is difficult because of the uneven surface and small defects of the cast product which are easily affected by the lighting position and angle. Therefore, three channel fusion data from an optical system that simultaneously acquires a 2D surface image and 3D shape information of the target object were obtained and used for deep learning. The mean average precision (mAP) of the proposed defect detection model using the three-channel fusion data is about 77%. And this result is greater than the 60% mAP of a defect detection model that uses single-channel data. For further optimization, we investigate a deep learning model that employs a deep learning network with multiple models, where each model trains and detects only a single type of defect. The experimental results demonstrate that the mAP of the model was improved to 88%.