Detection and segmentation of multi-class artifacts in endoscopy

Detection and segmentation of multi-class artifacts in endoscopy 内窥镜视频中多类别人造物的检测与分割 1 first-author 2 corresp ‡Corresponding Author ‡Corresponding Author †E-mail:xiedi@hikvision.com †E-mail:xiedi@hikvision.com xiedi@hikvision.com Department of Psychology, the Children’s Hospital, School of Medicine, Zhejiang University, Hangzhou 310003, China Department of Psychology, the Children’s Hospital, School of Medicine, Zhejiang University, Hangzhou 310003, China Hikvision Research Institute, Hangzhou 310052, China Hikvision Research Institute, Hangzhou 310052, China Endoscopy may be used for early screening of various cancers, such as nasopharyngeal cancer, esophageal adenocarcinoma, gastric cancer, colorectal cancer, and bladder cancer, and performing minimal invasive surgical procedures, such as laparoscopy surgery. During this procedure, an endoscope is used; it is a long, thin, rigid, or flexible tube having a light source and a camera at the tip, which facilitates visualization inside the affected organs on a screen and helps doctors in diagnosis. 为准确定位内窥镜视频中的人造物,帮助医生提升诊断准确率,引入深度神经网络检测与分割模型,采用特征金字塔与级联R-CNN相互结合的框架,并使用PSPNet结合分类器链的思想,从而解决分割及数据匮乏问题,有效提升性能,并在EAD 2019数据集上取得领先的性能. Endoscopic diagnosis Detection and segmentation Multi-class artifacts Cascade R-CNN PSPNet 内窥镜 检测与分割 多类别人造物 级联R-CNN PSPNet Endoscopy may be used for early screening of various cancers, such as nasopharyngeal cancer, esophageal adenocarcinoma, gastric cancer, colorectal cancer, and bladder cancer, and performing minimal invasive surgical procedures, such as laparoscopy surgery. During this procedure, an endoscope is used; it is a long, thin, rigid, or flexible tube having a light source and a camera at the tip, which facilitates visualization inside the affected organs on a screen and helps doctors in diagnosis. Artifact detection in endoscopic imaging is not only essential but also poses a significant challenge to accurately diagnose hollow organ diseases. Artifacts can be classified into many categories. In endoscopic environments, there are several variants of artifacts that may be a cause of concern, including pixel saturations, motion blur, defocus, specular reflection, bubbles, and artificial devices (Fig. 1 1 ). The importance of precisely detecting these artifacts is essential to perform high-quality endoscopic frame restoration and crucial for realizing reliable computer-assisted endoscopy tools for improved patient care. Since artifacts are superimposed on the organ surfaces and have a great impact on the diagnosis, they may also interfere with other post-processing steps (video stitching and video frame retrieval). The yellow border regions are with high saturation, the purple border regions represent the bubbles, the green border regions represent the mirror reflection (a), and the blue border region represents an artificial device (b) Because most artifacts have irregular edges, they require two kinds of tasks depending on the different granularities: artifact detection and artifact region segmentation. In this paper, for the detection task, we extended Faster region-based convolutional neural network (R-CNN) (Ren et al., 2015 2015 ) into a multi-stage cascaded R-CNN framework (Cai and Vasconcelos, 2018 2018 ). The purpose of cascaded R-CNN is to train a series of bounding-box regressors which could improve the intersection over union (IoU) between the candidate outputs and the ground-truth boxes. Furthermore, cascaded R-CNN also considers the high negative correlation between detector performance and IoU under a given threshold (higher threshold raises the requirement of IoU and results in lower detection performance). In this paper, we have used a phased approach to gradually raise the IoU threshold when training cascaded R-CNN. To resolve the multi-label issue in the segmentation task, we constructed a classifier chain (CC) method (Read et al., 2011 2011 ), in which the input of each classifier is the original image along with the outputs of all previous classifiers. The “one-vs-rest” training strategy is incapable of modeling the internal association between labels; on the contrary, our proposed CC strategy could elucidate the relationship between different label types to a certain extent and address this issue. Several studies have reported on the various detection and segmentation performance of medical devices using endoscopy (Bouget et al., 2017 2017 ; Münzer et al., 2018 2018 ). Early detection and segmentation of medical devices typically depend on the unique color or shape of devices, particularly resorting to circle detection (Münzer et al., 2013 2013 ). However, the accuracy attained is not reliable when the medical devices deal with complex conditions such as blurs, smoke fog, blood immersion, and bubbles. Classic support vector machine (SVM) and deformable part model (DPM) have also been introduced into endoscopic scenes to improve detection (Bouget et al., 2015 2015 ). With the rise of deep learning in recent years, data-driven feature representation learning has demonstrated better performance on numerous tasks than manual design features. However, due to the lack of labeled endoscopic datasets, deep learning methods could not be applied regularly. By considering the increasing amount of endoscopic data, a number of deep models performing well in other visual tasks, gradually become the mainstream processing methods for such images. In the 2017–2018 segmentation competition for medical devices (Allan et al., 2019 2019 ), deep models such as UNet and fully convolution neural network (FCN) overtook the most competitive methods (Shvets et al., 2018 2018 ). Unlike previous endoscopic scene datasets, endoscopy artifact detection (EAD 2019) detects and segments not only medical devices, but also areas of over-exposure, reflection, and special bubbles (Ali et al., 2019 2019 ). Furthermore, in the segmentation task, one single pixel could be labeled for multiple categories, which is rather different from segmentation tasks previously. Our proposed approach achieves better results than previous methods in dealing with these new challenges. Except for general artifact analysis, specific methods have been proposed by several researchers. Kalalembang et al. ( 2009 2009 ) used various size block-based discrete cosine transform (DCT) calculations on the distorted image to detect blurring. Loukas and Georgiou ( 2015 2015 ) performed smoke detection to ensure successful video retrieval. Leibetseder et al. ( 2017 2017 ) used pre-trained CNNs and threshold-based saturation analysis for the smoke detection task. In the multi-class artifact detection task, it is possible that we may encounter the problem of domain gaps between datasets; alternatively, the data have been derived from videos recorded using different endoscopes (Fig. 2 2 ). (a) and (b) are obtained from the same endoscopic device, while (c) is obtained from a different endoscope Fig. 2 2 demonstrates different categories and sizes of ground-truth bounding-boxes, which indicates that there is a profound discrepancy among the scales of different classes. Additionally, objects from different classes may appear similar, e.g., red boxes and blue boxes in Fig. 2b 2b ; thus, it overlaps in some areas, which also means that even the manual labeling standards of one single object class may not be the same in different pictures. To resolve the abovementioned problems, we proposed a deep neural network structure combining feature pyramid network (FPN) and cascaded R-CNN (Fig. 3 3 ). To detect small object areas, we used input images with 1024×1024 resolution; various data augmentation techniques were adopted to avoid over-fitting due to the small amount of training data. Meanwhile, we introduced the FPN to extract and fuse multi-scale feature map information in a “top-down” mode, which generated a large number of high-quality candidate boxes through the regional proposal network (RPN), and significantly raised the detection recall rate. Finally, the cascade structure designed by us further improved the precision of candidate detection boxes via a layer-by-layer correction mechanism. All the regional proposal phases, including Bbox1, Bbox2, and Bbox3, are refined with increasing intersection over union (IoU) value, for the motivation of retaining proposals with low IoU but high quality. R-CNN: region-based convolutional neural network In comparison with single-scale detection methods (including Faster R-CNN), the proposed FPN structure could generate a large number of candidate boxes (Fig. 4 4 ), which greatly increased the recall rate in mean average precision (mAP). Although increased numbers of incorrect boxes would simultaneously reduce the precision, which may result in a lower mean IoU (mIoU); however, the overall performance could be improved by adjusting the output confidence and non-maximum suppression (NMS) threshold. There are boxes within boxes, which show false positives In data augmentation, we introduced noise interference to hue-saturation-value (HSV) color space and randomly alternated the channels of red-green-blue (RGB) image, which enlarged the domain scope by introducing color divergence. Randomly cropping or enlarging an image, which simulates local focusing, was used as well to counteract the effects of a difference in scale. We also performed the geometric transformation on the images, including horizontal and vertical flips, to produce variations in the geometric structure. We employed a technique to transfer learning experience in training. Firstly, the model was pre-trained on Microsoft Common Objects in Context (MS COCO) dataset (Lin et al., 2014 2014 ). Consequently, an output layer of the model was initialized according to EAD class numbers. Lastly, the model was re-trained with EAD data. Regardless of the large-scale domain gap between natural and medical images, the underlying representational features continued to share certain commonalities. Due to the small volume of the EAD data, our transfer strategy had a considerable advantage than the “train-from-scratch” method when training on EAD dataset. In this task, we explore a variety of training strategies to resolve the multi-label classification problem at pixel level. Some of the pixels in an image may be associated with multiple labels. Therefore, it is not appropriate to use classical softmax with cross-entropy loss function ( L ) to resolve this problem. Denote the finite set of labels as: L ={ l 0 , l 1 , …, l K ; l k }; M ={ l i , …, l j ; i , j =0, 1, …, K and i ≤ j } is a subset of L , where K is the number of classes in L . Dataset consisting of N training samples is denoted as {( x 0 , M 0 ), ( x 1 , M 1 ), …, ( x n , M n ); n =1, 2, …, N }, where x denotes training samples, and then the objective function can be expressed as follows: $$\mathop {\min }\limits_\theta \sum\limits_{n = 0}^N {L(f({x_n};\;\theta ),\;{M_n})}$$ , where f is the network and θ represents the model parameters that need to be optimized. We can decompose this multi-label classification problem into a set of binary classification tasks { τ 0 , τ 1 , …, τ k ; k =0, 1, …, K }. We can resolve each sub-task independently if there is no correlation between τ k . However, this assumption is not valid considering the task. Essentially, the probability that a certain pixel is classified into classes i or j may not be equal, but there is a certain degree of co-occurrence. To model the more flexible co-occurrence relationship, we constructed a CC method: each binary classifier B k uses the image and prediction results of all previous stages as input to produce results of the current stage. Characteristically, the predicted results of i th stage Ο i can be expressed as: Ο i ← B k ( x , ( Ο 0 , Ο 1 , …, Ο i −1 )), where Ο 0 , Ο 1 , …, Ο i −1 represent the outputs of the previous k −1 classifiers. The order of classifiers is fixed, related to the segmentation difficulty in each category (order from easy to difficult is medical devices, over-exposed areas, bubbles, specular reflection, and other artifacts). In terms of network architecture, we investigated three mainstream segmentation structures: UNet (Ronneberger et al., 2015 2015 ), PSPNet (Zhao et al., 2017 2017 ), and DeepLabv3 (Chen et al., 2018 2018 ). We noted that PSPNet achieved relatively optimal performance. EAD2019 included three subsets, each corresponding to three tasks: artifact detection, semantic region segmentation, and detection generalization. Regarding the artifact detection and generalization tasks, the training set contained 2613 images. We randomly selected 2322 images for training and 291 images for validation. K-fold cross validation was not considered because of the limited data size which may bring performance fluctuation in the process. We will elaborate on the same if more data are collected in a future study. The sample numbers of test sets for detection and generalization tasks were 195 and 51, respectively. In the segmentation task, 474 samples from 589 images were included in the training, and the rest was used as a test set. We trained on four graphic processing units (GPUs) with a batch-size of 8, and augmented training data by using random scaling, random flipping, and color normalization. Stochastic gradient descent algorithm with momentum was used in the optimization process; the initial learning rate was set to 0.02 with a scheduling strategy of multistep. Cascaded R-CNN had three stages, and each stage was trained in sequence. The thresholds of RPN were set to 0.5, 0.6, and 0.7 for each training session, respectively. Table 1 1 lists the improvements in the performance using different techniques sequentially. Notably, data augmentation effectively prevents a model from over-fitting, while pre-training further enhances the generalization of the final model. NMS: non-maximum suppression Technique Score_d (detection) Score_g (generalization) Baseline 0.1800 +Data augmentation 0.3105 0.3600 +Validation set 0.3215 0.3600 +COCO pre-training 0.3395 0.3900 +Adjust NMS parameters 0.3429 0.3500 To illustrate the effectiveness of our cascade detection method, we compared the detection performance of networks with/without the cascade structure. The results demonstrated that the cascade structure could significantly improve the detection score of different types of artifacts (Table 2 2 ). Model Score_d  YOLOv3 0.2200  Cascade R-CNN 0.3429 Fig. 5 5 shows the results of our proposed method on EAD dataset qualitatively. The size variances of different classes are significantly large. This also reflects on the specialization of medical image detection, which differs from general object detection. The red boxes represent specular reflection, the light blue boxes represent bubbles, the dark blue boxes represent supersaturated regions, the green boxes represent blurs, and the purple red boxes represent other defects We evaluated several backbone networks for the segmentation task, including UNet with VGG-11 and VGG-16, PSPNet with residual structure, and DeepLabv3 with residual structure. PSPNet with a structure of ResNet-34 achieved the best results. We divided the entire dataset into training and test sets according to a ratio of 4:1. To perform data augmentation, the original image was first resized into five different spatial scales (0.7, 0.9, 1.0, 1.2, and 1.5 times of the original size); then, the scaled images were randomly cropped to 224×224. Next, the processed images were randomly flipped horizontally or vertically and rotated at a certain angle. Thereafter, the augmented images were transferred to a CC for training, in which each classifier was initialized with pre-trained weights. Finally, the predicted results at current resolution were mapped back to the original image by interpolation from the nearest neighbor. Similar to the detection task, we used a batch size of 8 to train the four GPUs. Stochastic gradient descent algorithm with momentum was used for optimizing, where the weight decay was set to 5×10 −4 , and the initial learning rate was 0.01. In the first 100 epochs, five epochs were deemed as a cycle to adjust the learning rate cyclically. Table 3 3 compares the performance between DeepLabv3 and PSPNet. DeepLabv3 is trained with five independent binary classifiers, while PSPNet corresponds to the CC structure. Model Overlap F2 score Score_s DeepLabv3 0.5775 0.6136 0.5865 PSPNet 0.5986 0.6225 0.6046 Fig. 6 6 shows some results of our proposed method on segmentation set. We observe that the model learns the apparent characteristics of various artifact classes well, and can outline the region contours accurately. The yellow region represents the supersaturated region, the blue region represents the artificial devices, the green region represents the specular reflection, the purple region represents the bubble, and the red region represents other artifacts In this paper, we propose two methods, based on deep neural network, to perform the multi-class artifacts detection and segmentation in endoscopic video imaging environment. For the detection task, we integrate the FPN with neural network, and introduce the idea of cascade detection. For the segmentation task, we revise the classical PSPNet network and combine it with CC. We have achieved inspiring results on both tasks. This technology is expected to be applied to automated or semi-automated endoscopic organ examination in the near future, so as to improve the efficiency of medical workers. References S Ali , F Zhou , C Daul , 等 . Endoscopy artifact detection (EAD 2019) challenge dataset . 2019 . arXiv preprint, arXiv: 1905.03209 . DOI: 10.17632/c7fjbxcgj9.1 http://doi.org/10.17632/c7fjbxcgj9.1 . Endoscopy artifact detection (EAD 2019) challenge dataset M Allan , A Shvets , T Kurmann , 等 . 2017 Robotic instrument segmentation challenge . 2019 . arXiv preprint, arXiv: 1902.06426 . . 2017 Robotic instrument segmentation challenge D Bouget , R Benenson , M Omran , 等 . Detecting surgical tools by modelling local appearance and global shape . IEEE Transact Med Imag , 2015 . 34 ( 12 ): 2603 - 2617 . DOI: 10.1109/TMI.2015.2450831 http://doi.org/10.1109/TMI.2015.2450831 . Detecting surgical tools by modelling local appearance and global shape D Bouget , M Allan , D Stoyanov , 等 . Vision-based and marker-less surgical tool detection and tracking: a review of the literature . Med Image Anal , 2017 . 35 633 - 654 . 27744253 DOI: 10.1016/j.media.2016.09.003 http://doi.org/10.1016/j.media.2016.09.003 . Vision-based and marker-less surgical tool detection and tracking: a review of the literature ZW Cai , N Vasconcelos . Cascade R-CNN: delving into high quality object detection . 2018 . IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6154 - 6162 . DOI: 10.1109/CVPR.2018.00644 http://doi.org/10.1109/CVPR.2018.00644 . Cascade R-CNN: delving into high quality object detection LC Chen , YK Zhu , G Papandreou , 等 . Encoder-decoder with atrous separable convolution for semantic image segmentation . 2018 . Proceedings of the European Conference on Computer Vision . 833 - 851 . DOI: 10.1007/978-3-030-01234-2_49 http://doi.org/10.1007/978-3-030-01234-2_49 . Encoder-decoder with atrous separable convolution for semantic image segmentation E Kalalembang , K Usman , IP Gunawan . DCT-based local motion blur detection . 2009 . International Conference on Instrumentation, Communication, Information Technology, and Biomedical Engineering . 1 - 6 . DOI: 10.1109/icici-bme.2009.5417252 http://doi.org/10.1109/icici-bme.2009.5417252 . DCT-based local motion blur detection A Leibetseder , MJ Primus , S Petscharnig , 等 . Real-time image-based smoke detection in endoscopic videos . 2017 . Proceedings of the on Thematic Workshops of ACM Multimedia . 296 - 304 . DOI: 10.1145/3126686.3126690 http://doi.org/10.1145/3126686.3126690 . Real-time image-based smoke detection in endoscopic videos TY Lin , M Maire , S Belongie , 等 . Pajdla T Fleet D , et al Schiele B . Microsoft COCO: common objects in context . Computer Vision–ECCV 2014 , 2014 . Springer , 740 - 755 . DOI: 10.1007/978-3-319-10602-1_48 http://doi.org/10.1007/978-3-319-10602-1_48 . Microsoft COCO: common objects in context C Loukas , E Georgiou . Smoke detection in endoscopic surgery videos: a first step towards retrieval of semantic events . Int J Med Robot Comput Assist Surg , 2015 . 11 ( 1 ): 80 - 94 . DOI: 10.1002/rcs.1578 http://doi.org/10.1002/rcs.1578 . Smoke detection in endoscopic surgery videos: a first step towards retrieval of semantic events B Münzer , K Schoeffmann , L Böszörmenyi . Detection of circular content area in endoscopic videos . 2013 . Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems . 534 - 536 . DOI: 10.1109/CBMS.2013.6627865 http://doi.org/10.1109/CBMS.2013.6627865 . Detection of circular content area in endoscopic videos B Münzer , K Schoeffmann , L Böszörmenyi . Content-based processing and analysis of endoscopic images and videos: a survey . Multimed Tools Appl , 2018 . 77 ( 1 ): 1323 - 1362 . DOI: 10.1007/s11042-016-4219-z http://doi.org/10.1007/s11042-016-4219-z . Content-based processing and analysis of endoscopic images and videos: a survey J Read , B Pfahringer , G Holmes , 等 . Classifier chains for multi-label classification . Mach Learn , 2011 . 85 ( 3 ): 333 - 359 . DOI: 10.1007/s10994-011-5256-5 http://doi.org/10.1007/s10994-011-5256-5 . Classifier chains for multi-label classification SQ Ren , KM He , R Girshick , 等 . Faster R-CNN: towards real-time object detection with region proposal networks . 2015 . Advances in Neural Information Processing Systems . 91 - 99 . DOI: 10.1109/tpami.2016.2577031 http://doi.org/10.1109/tpami.2016.2577031 . Faster R-CNN: towards real-time object detection with region proposal networks O Ronneberger , P Fischer , T Brox . U-Net: convolutional networks for biomedical image segmentation . 2015 . International Conference on Medical Image Computing and Computer-assisted Intervention . 234 - 241 . DOI: 10.1007/978-3-319-24574-4_28 http://doi.org/10.1007/978-3-319-24574-4_28 . U-Net: convolutional networks for biomedical image segmentation AA Shvets , A Rakhlin , AA Kalinin , 等 . Automatic instrument segmentation in robot-assisted surgery using deep learning . 2018 . 17th IEEE International Conference on Machine Learning and Applications . 624 - 628 . DOI: 10.1109/ICMLA.2018.00100 http://doi.org/10.1109/ICMLA.2018.00100 . Automatic instrument segmentation in robot-assisted surgery using deep learning HS Zhao , JP Shi , XJ Qi , 等 . Pyramid scene parsing network . 2017 . Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 6230 - 6239 . DOI: 10.1109/cvpr.2017.660 http://doi.org/10.1109/cvpr.2017.660 . Pyramid scene parsing network We thank Tao SONG, Hai-ming SUN, Qi YAO, Liang-jun ZHANG, and Jian ZHOU (Hikvision Research Institute, Hangzhou, China) for helpful discussion and insightful advices for this paper. Compliance with ethics guidelines Yan-yi ZHANG and Di XIE declare that they have no conflict of interest. All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008 (5). Informed consent was obtained from all patients for being included in the study. Additional informed consent was obtained from all patients for whom identifying information is included in this article. 10.1631/jzus.B1900340 , 2019-06-25 2019-09-12 2022-01-05 Copyright © Zhejiang University and Springer-Verlag GmbH Germany, part of Springer Nature 2019 2019 Journal of Zhejiang University-SCIENCE B (Biomedicine and Biotechnology) 12 20