The photonic integration technology based on lithium niobate thin films has become increasingly prominent in the field of high-speed optoelectronics, and is widely used for various on-chip functions, such as electro-optical modulation, optical frequency comb, filter, nonlinear optical frequency converter, nonlinear quantum light source, laser etc. In the development of lithium niobate film photonic integration technology, there is an important technical bottleneck which is the effective coupling of lithium niobate film nanowaveguides and single-mode fibers, which is also the key to hinder the practical application of lithium niobate thin film photonic devices. On-chip mode size converter is widely used in mode field transformation to realize waveguide mode field transformation. Although the existing researches have improved the coupling efficiency by using bilayer tapered waveguides or composite structures, they are all coupled with tapered fiber or thin diameter fiber, which still cannot achieve effective coupling with single-mode fiber. To solve this problem, a mode size converter based on SiO₂ waveguide, SiON tapered waveguide and bilayer LN tapered waveguide is designed to achieve efficient mode and energy transfer and conversion between lithium niobate film nanowaveguide and single-mode fiber. The structure of the mode size convertor composed of SiO₂ waveguide, SiON tapered waveguide and bilayer LN tapered waveguide is simulated by using the three-dimensional finite difference beam propagation method, and the structural parameters of each section are sequentially optimized through optical pattern matching design and adiabatic mode transmission design, and the optical coupling efficiency and adiabatic mode conversion efficiency of each section are simulated. The research results show that when the refractive index difference between the core layer and cladding layer of the SiO₂ waveguide is 0.75% and the size of SiO₂ waveguide is 6 μm×6 μm, the coupling efficiency between SiO₂ waveguide and single-mode fiber is about 93% (Fig.6). When the mode field size of the wide end of SiON tapered waveguide is 2.5 μm×2.5 μm-3.5 μm×3.5 μm, the refractive index of the corresponding core layer is 1.48-1.51, the length of the SiON tapered waveguide (L₁) is greater than 250 μm and the width of the tapered tip W₃ is 0.1-0.3 μm, the optical mode is gradually converted from the SiO₂ waveguide to the SiON waveguide, and the conversion efficiency of the SiON tapered waveguide is 93%-97.2% (Fig.8). The bilayer LN tapered waveguide includes the LN tapered planar waveguide and the LN tapered ridge waveguide. In the LN tapered planar waveguide, when the tapering length (L₂) changes in the range of 200-300 μm, the width of the tapered tip W₄ changes within 0.1-0.15 μm, and the width of the wide end (W₅) changes in the range of 0.8-1.4 μm, the optical mode profile in LN tapered planar waveguide increases with the increase of the inverse taper width of LN tapered planar waveguide, while that in SiON layer decreases, and the conversion efficiency of the LN tapered planar waveguide is 96%-98.5% (Fig.9). In the LN tapered ridge waveguide, when the length of LN ridge tapered waveguide (L₃) varies from 40 to 100 μm, and the width of the tapered tip of LN ridge tapered waveguide W₆ varies from 0.1 μm to 0.3 μm, the optical mode is gradually converted into LN ridge waveguide optical mode, and conversion efficiency of the LN tapered ridge waveguide exceeds 99% (Fig.10). Through the above design, effective coupling with lithium niobate film waveguide and single-mode fiber can be realized, and the coupling efficiency is 82.2%-89.0% (Fig.11). At the same time, ± 1.8 μm fiber coupling alignment tolerance is obtained (Fig.12). The proposed mode size converter based on SiO₂ waveguide, SiON tapered waveguide and bilayer LN tapered waveguide provides a new method for the coupling and integration of lithium niobate thin film photonic devices, which can provide a reference for the next step of preparing highly efficient coupling lithium niobate thin film photonic devices, and is beneficial to further realize the integrated application of lithium niobate devices.