Abstract. Matching images acquired from different sensing modalities, in particular infrared and visible-spectrum imaging, with three-dimensional geometric models of a scene is highly challenging because of fundamental differences in the physical mechanisms of data formation. This problem is particularly relevant to unmanned aerial vehicle (UAV) navigation, where information from heterogeneous onboard sensors must be fused to ensure reliable positioning and orientation. Modern UAVs are often equipped with visible-spectrum optical cameras, thermal imagers, and LiDAR systems, creating a need for efficient cross-modal matching algorithms. In this study, a modality is understood as a specific type of acquired information (in particular, RGB images, infrared imagery, and depth maps), and each of these types captures specific scene characteristics: texture features, thermal field distributions, surface reflectance properties, or the spatial geometry of objects. Since each sensing modality produces a fundamentally different set of features-textural, thermal, reflectance-related, or geometric-direct correspondence estimation between heterogeneous modalities is a nontrivial research problem. A methodological approach based on transforming multimodal data into a unified depth space, treated as an invariant representation of the scene, is proposed. By recovering depth information from source images of different modalities or mapping them to a common spatial geometric structure, cross-modal discrepancies can be reduced and correspondence estimation between heterogeneous data types can be substantially simplified. This approach can be applied in autonomous UAV navigation systems, where accurate and robust matching of multisensor information is important for safe mission execution. The architecture of the developed method is described in detail, the results of experimental validation of its performance and effectiveness on real multispectral datasets are presented, and the identified limitations of the proposed approach are critically analyzed together with a discussion of promising directions for its further improvement and development.

Keywords: image processing, computer vision, neural networks, image matching

For citation: Gessen P. A., Lizin A. I., Pavlova V. A., Saenko V. D., Tupikov V. A. Matching visible and infrared images with 3D models by mapping to a unified depth space. Aerospace Engineering and Technology.
2026. Vol. 4, no. 2, pp. 372–388. (In Russ.) https://doi.org/10.52467/2949-401X-2026-4-2-372-388; https://elibrary.ru/twyrky 

References

1. Patin M. V., Korobov D. V. Comparative analysis of methods for detecting keypoints and descriptors in image grouping by similar content. Molodoy Uchenyy [Young Scientist]. 2016. Vol. 115, no. 11, pp. 214–221. (In Russ.) https://moluch.ru/archive/115/31188
2. Isola P., Zhu J.-Y., Zhou T., Efros A. A. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA. 2017, pp. 5967–5976, https://doi.org/10.1109/CVPR.2017.632
3. Nazhimova N. A., Kiselev A. A. Comparative analysis of methods for determining reference points in images. Scientific Review. Technical Science. 2024. No. 1. Pp. 23–28. (In Russ.) https://doi.org/10.17513/srts.1454
4. Berthold K. P. Horn, Brian G. Schunck Determining optical flow. Artificial Intelligence. 1981. Vol. 17, Iss. 1–3, Pp. 185–203. https://doi.org/10.1016/0004-3702(81)90024-2
5. Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., Bengio Y. Generative adversarial nets. Advances in Neural Information Processing Systems 27 (NIPS 2014), 8–11^th December 2014, Montreal, Canada. NIPS, 2014, 9 p. https://papers.nips.cc/paper_files/paper/2014/hash/f033ed80deb0234979a61f95710dbe25-Abstract.html
6. Zhu J.-Y., Park T., Isola P., Efros A. A. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV), 22-29 October 2017 Venice, Italy. IEEE, 2017, pp. 2242–2251, https://doi.org/10.1109/ICCV.2017.244
7. Liu M.-Y., Breuel T., Kautz J. Unsupervised Image-to-Image Translation Networks. Advances in Neural Information Processing Systems30 – Proceedings of the 2017 Conference, 4^th–9^th December, 2017, Long Beach. NIPS 2017, pp. 700–708.
8. Huang X., Liu M.-Y., Belongie S., Kautz J., Multimodal Unsupervised Image-to-Image Translation. Computer Vision – ECCV 2018. Lecture Notes in Computer Science, Springer, Cham., 2018. Vol. 11207. pp.172–189. https://doi.org/10.1007/978-3-030-01219-9_11
9. Hui , Li X., Chen J., He H., Yang J. Unsupervised Multi-Domain Image Translation with Domain-Specific Encoders/Decoders. 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 2018, pp. 2044–2049, https://doi.org/10.1109/ICPR.2018.8545169
10. Radford A., Kim J. W., Hallacy C., Ramesh A., Goh G., Agarwal S., Sastry G., Askell A., Mishkin P., Clark J., Krueger G., Sutskever I. Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, 18-24 July 2021. PMLR, 2021. Vol. 139. Pp. 8748–8763. https://proceedings.mlr.press/v139/
11. Eigen D., Puhrsch C., Fergus R. Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. Advances in Neural Information Processing Systems 27 (NIPS 2014) 8–11th December 2014, Montreal, Canada. https://papers.nips.cc/paper_files/paper/2014/hash/91c56ce4a249fae5419b90cba831e303-Abstract.html
12. Yang L., Kang B., Huang Z., Zhao Z., Xu X., Feng J., Zhao H. Depth Anything V2. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). https://doi.org/10.52202/079017-0688
13. Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., DehghaniM., Minderer M., Heigold G., Gelly S., Uszkoreit J., Houlsby N. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations, Vienna, Austria, May 03–07, 2021. ICLR, 2021. https://doi.org/10.48550/arXiv.2010.11929
14. Appel A. Some Techniques for Shading Machine Renderings of Solids. AFIPS ’68 (Spring): Proceedings of the joint computer conference (Atlantic City New Jersey, April 30–May 2, 1968). New York, Association for Computing Machinery, 1968. Pp. 37–45. https://doi.org/10.1145/1468075.1468082
15. DeTone D., Malisiewicz T., Rabinovich A. SuperPoint: Self-Supervised Interest Point Detection and Description. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 2018, pp. 337–33712, https://doi.org/10.1109/CVPRW.2018.00060
16. Lindenberger P., Sarlin P.-É., Pollefeys M. LightGlue: Local Feature Matching at Light Speed. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France. 2023. Pp. 1234–1243. https://doi.org/10.1109/ICCV51070.2023.01616
17. Fischler M. A., Bolles R. C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. ACM. 1981. Vol. 24, no. 6, pp. 381–395. https://doi.org/10.1145/358669.358692
18. Lowe D. G. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision. 2004. Vol. 60, no. 2, pp. 91–110, https://doi.org/10.1023/B:VISI.0000029664.99615.94
19. Rublee E., Rabaud V., Konolige K., Bradski G. ORB: An efficient alternative to SIFT or SURF. 2011 International Conference on Computer Vision, Barcelona, Spain. 2011. Pp. 2564–2571, https://doi.org/10.1109/ICCV.2011.6126544
20. Sasaki Y. The truth of the F-measure. URL: https://www.researchgate.net/publication/268185911_The_truth_of_the_F-measure#fullTextFileContent (accessed: 07.04.2025)