Segmentation of Overlapping Tree Images in the Digital Photographs of Forest Areas. C. 126-140

Complete text of the article:

Download article (pdf, 2.8MB )

UDС

004.896; 630*96

DOI:

10.37482/0536-1036-2024-1-126-140

Abstract

The use of decision support systems based on computer vision and artificial intelligence significantly improves the working conditions for the operators of technological machines in the timber sector, whose work implies high intensity and psycho-emotional overload. By means of computer vision and artificial intelligence the operator can quickly and easily obtain the data on the state of the cutting area and adopt the optimal solution for holding the working operation. This facilitates his work and reduces the time spent searching and analyzing the data on the cutting area. Meanwhile, one of the key elements of such a system is a subsystem for automatic segmentation of objects in the photograph. We have explored the possibility of segmenting overlapping objects in the photographs of forest areas using a convolutional neural network based on the Mask R-CNN architecture. Unlike in most works on similar topics, the objects of this study are color photographs taken by an RGB camera rather than a lidar. This creates the prospect for reducing the cost of hardware and software systems used to support decision-making by the operators of logging machines. The images of the stems and crowns of coniferous and deciduous trees overlapping each other are the segmented objects under consideration. Using the GIMP graphic editor, we have manually marked the color photographs depicting a total of 134 trees of 4 different species: spruce, aspen, birch and pine. Utilizing the developed database, we have carried out an experiment to further train the Mask R-CNN convolutional neural network for segmentation of overlapping parts of the trees in the digital photographs of forest areas. The neural network has been pre-trained using the Microsoft COCO dataset containing more than 200,000 images of 80 different classes of objects such as people, cars, animals and various items. While training the neural network, the images supplied to its input were subjected to a series of various linear and nonlinear geometric transformations, which made it possible to increase the volume of training data by 11 times. As a result, the accuracy of segmentation of the images of the stems and crowns of coniferous and deciduous trees overlapping each other has reached 79 %, which allows the use of neural networks based on a similar architecture in decision support systems for logging machine operators.

Authors

Igor V. Petukhov*, Doctor of Engineering, Prof.; ResearcherID: A-9472-2014,
ORCID: https://orcid.org/0009-0000-2365-4857
Konstantin O. Ivanov, Candidate of Engineering; ResearcherID: A-6724-2014,
ORCID: https://orcid.org/0000-0002-6395-4085
Dmitry M. Vorozhtsov, Candidate of Engineering; ResearcherID: JAN-6772-2023,
ORCID: https://orcid.org/0009-0005-8396-2778
Alexey A. Rozhentsov, Doctor of Engineering, Prof.; ResearcherID: AAU-8039-2020,
ORCID: https://orcid.org/0000-0003-0043-0070
Nataliya I. Rozhentsova, Candidate of Engineering; ResearcherID: JMD-0172-2023,
ORCID: https://orcid.org/0009-0006-4024-7886
Ludmila A. Steshina, Candidate of Engineering, Assoc. Prof.;
ResearcherID: JMC-9977-2023,
ORCID: https://orcid.org/0009-0006-1526-991X

Affiliation

Volga State University of Technology, Lenina pl., 3, Yoshkar-Ola, 424000, Russian Federation; PetuhovIV@volgatech.net*, IvanovKO@volgatech.net, VorozhtsovDM@volgatech.net, RozhencovAA@volgatech.net, RozhencovaNI@volgatech.net, SteshinaLA@volgatech.net

Keywords

deep learning, artificial intelligence, convolutional neural network, segmentation of overlapping objects, decision support system for logging machine operators

For citation

Petukhov I.V., Ivanov K.O., Vorozhtsov D.M., Rozhentsov A.A., Rozhentsova N.I., Steshina L.A. Segmentation of Overlapping Tree Images in the Digital Photographs of Forest Areas. Lesnoy Zhurnal = Russian Forestry Journal, 2024, no. 1, pp. 126–140. (In Russ.). https://doi.org/10.37482/0536-1036-2024-1-126-140

References

1. Ivanov K.O. Classification of Electroencephalogram Segments Based on Pathological Significance Using Sequential Neural Networks. Vestnik Povolzhskogo gosudarstvennogo tekhnologicheskogo universiteta. Seriya: Radiotekhnicheskie i infokommunikatsionnye sistemy = Vestnik of Volga State University of Technology. Series: Radio Engineering and Infocommunication Systems, 2022, no. 4(56), pp. 24–37. (In Russ.). https://doi.org/10.25686/2306-2819.2022.4.24
2. Nikolenko S., Kadurin A., Arkhangelskaya E. Deep Learning. Dive into the World of Neural Networks. St. Petersburg, Piter Publ., 2022. 480 p. (In Russ).
3. Portnov A.M., Ivanova N.V., Shashkov M.P. Experience of Using the DeepForest Neural Network to Detect Trees in a Broad-Leaved Forest. Proceedings of the International Conference “Mathematical Biology and Bioinformatics”. Ed. V.D. Lakhno. Pushchino, IMPB RAS, 2022, vol. 9, art. no. e45. (In Russ). https://doi.org/10.17537/icmbb22.12
4. Timofeev V.S., Sivak M.A. Robust Neural Network with Simple Architecture. Sibirskiy zhurnal industrialnoy matematiki = Journal of Applied and Industrial Mathematics, 2021, vol. 24, no. 4, pp. 126–138. (In Russ). https://doi.org/10.33048/SIBJIM.2021.24.409
5. Callan R. The Essence of Neural Network. Upper Saddle River: Prentice Hall Europe, 1998. 232 p.
6. Chollet F. Deep Learning with Python: Second Edition. New York, Manning Publications Co., 2021. 504 p.
7. COCO: Common Objects in Context. Available at: https://cocodataset.org/#home (accessed 17.01.23).
8. Daubechies I., DeVore R., Foucart S., Hanin B., Petrova G. Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation, 2022, vol. 55, pp. 127–172. https://doi.org/10.1007/s00365-021-09548-z
9. Diez Y., Kentsch S., Fukuda M., Caceres M.L.L., Moritake K., Cabezas M. Deep Learning in Forestry Using UAV-Acquired RGB Data: A Practical Review. Remote Sensing, 2021, vol. 13, no. 14, art. no. 2837. https://doi.org/10.3390/rs13142837
10. Girshick R. Fast R-CNN. 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.
11. Goodfellow I., Benigo Y., Courville A. Deep Learning. Cambridge, MIT Press Publ., 2017. 785 p.
12. Grondin V., Fortin J.-M., Pomerleau F., Giguère P. Tree Detection and Diameter Estimation Based on Deep Learning. Forestry: An International Journal of Forest Research, 2023, vol. 96, iss. 2, pp. 264–276. https://doi.org/10.1093/forestry/cpac043
13. Haykin S. Neural Networks: A Comprehensive Foundation (3rd Edition). Hoboken: Prentice-Hall Inc., 2007. 938 p.
14. He K., Gkioxari G., Dollar P., Girshick R. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988.
15. Imgaug: a Library for Image Augmentation in Machine Learning Experiments. Available at: https://imgaug.readthedocs.io/en/latest/ (accessed 17.07.23).
16. Ivanov K.O., Kazarinov A.A., Rozhentsov A.A., Dubrovin V.N., Baev A.A., Evdokimov A.O. An Algorithm for Segmentation of Kidney Tissues on CT Images Based on a U-Net Convolutional Neural Network. Biomedical Engineering, 2023, vol. 56, pp. 424–428. https://doi.org/10.1007/s10527-023-10249-z
17. Mask R-CNN for Object Detection and Segmentation. Available at: https://github.com/matterport/Mask_RCNN (accessed 17.07.23).
18. Moussaid A., El Fkihi S., Zennayi Y. Tree Crowns Segmentation and Classification in Overlapping Orchards Based on Satellite Images and Unsupervised Learning Algorithms. Journal of Imaging, 2021, vol. 7, no. 241. https://doi.org/10.3390/jimaging7110241
19. Ochoa C.G., Caruso P., Ray G., Deboodt T., Jarvis T.W., Guldan S.J. Ecohydrologic Connections in Semiarid Watershed Systems of Central Oregon USA. Water, 2018, vol. 2, no. 10, art. no. 181. https://doi.org/10.3390/w10020181
20. Pointer I. Programming PyTorch for Deep Learning: Creating and Deploying Deep Learning Applications. O’Reilly Media Inc., 2019. 217 p.
21. Qassim H., Verma A., Feinzimer D. Compressed Residual-VGG16 CNN Model for Big Data Places Image Recognition. 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), 2018, pp. 169–175.
22. Qianwei L., Weifeng M., Jianpeng Z., Yicheng L., Dongfan X., Jinliang W. Point-Cloud Segmentation of Individual Trees in Complex Natural Forest Scenes Based on a Trunk Growth Method. Journal of Forestry Research, 2021, vol. 32, pp. 2403–2414. https://doi.org/10.1007/s11676-021-01303-1
23. Rao D., McMahan B. Natural Language Processing with PyTorch: Build Intelligent Language Applications Using Deep Learning. O’Reilly Media Inc., 2019. 254 p.
24. Ren S., Kaiming H., Ross G., Jian S. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NIPS’15: Proceedings of the 28th International Conference on Neural Information Processing Systems. Great Britain, Oxford, December, 2015, vol. 1, pp. 91–99.
25. Rojas R. The Backpropagation Algorithm. Neural Networks, 1996, pp. 149–182. https://doi.org/10.1007/978-3-642-61068-4_7
26. Tarasova A., Strielkowski W., Larionova N. Digital Relations and Innovative Ecosystem for Sustainable Engineering Education: A Case of Russia. Increasing Supply Chain Performance in Digital Society. IGI Global, 2022, pp. 304–318.
27. Wang D., Liang X., Mofack II G., Martin-Ducup O. Individual Tree Extraction from Terrestrial Laser Scanning Data via Graph Pathing. Forest Ecosystems, 2021, vol. 8, no. 67. https://doi.org/10.1186/s40663-021-00340-w
28. Weinstein B.G., Graves S.J., Marconi S., Singh A., Zare A., Stewart D. A Benchmark Dataset for Canopy Crown Detection and Delineation in Co-Registered Airborne RGB, LiDAR and Hyperspectral Imagery from the National Ecological Observation Network. PLoS Computational Biology, 2021, vol. 7, no. 17. https://doi.org/10.1371/journal.pcbi.1009180
29. Windrim L., Bryson M. Detection, Segmentation, and Model Fitting of Individual Tree Stems from Airborne Laser Scanning of Forests Using Deep Learning. Remote Sensing, 2020, vol. 12, no. 9, art. no. 1469. https://doi.org/10.3390/rs12091469
30. Xu X., Iuricich F., Floriani L.D. A Topology-Based Approach to Individual Tree Segmentation from Airborne LiDAR Data. GeoInformatica, 2023, vol. 27, pp. 759–788. https://doi.org/10.1007/s10707-023-00487-4
31. Yang Z., Dong R., Xu H., Gu J. Instance Segmentation Method Based on Improved Mask R-CNN for the Stacked Electronic Components. Electronics, 2020, vol. 9, no. 6, art. no. 886. https://doi.org/10.3390/electronics9060886