Design of three-mode free-form nanostructured optical fibers: comparison of dense and convolutional neural networks...

We report a numerical study on the inverse design of the internal structure of weakly coupled three-mode fibers. We explored a new class of optical fibers - free-form nanostructured fibers (FFNFs) - operating at 1550 nm for potential application in Mode-Division Multiplexing (MDM) systems. The fiber geometries were generated and optimized within the Generative Inverse Design Networks (GIDNs) framework using convolutional neural networks (CNNs) and fully connected dense neural networks (DNNs). The objective of the optimization was to maximize the minimal effective refractive index separation Min|Δn_eff| between supported modes, ensuring weak intermodal coupling. The proposed free-form nanostructured fiber designed with the CNN achieved a minimum modes separation of Min|Δn_eff| = 2.15 × 10^- 3, exceeding that of a reference three-mode elliptical-core fiber (Min|Δn_eff| = 2.085 × 10^- 3). In contrast, the best DNN-optimized structure reached Min|Δn_eff| = 1.99 × 10^- 3. The results demonstrate that the CNN-based inverse design yields fiber geometries outperforming conventional designs. The proposed methodology can be extended to higher-order mode systems and can include additional fiber properties crucial for telecommunication purposes.