This time it's definitly working! The good thing is that thanks to my XR framework (which is on github by the way), I can use the Vive Focus without any change in my prefabs or gameplay/settings code.
Now I've to optimize all scenes for mobile.
Basically I've one scene with all gameplay code/elements and another scene for the static structure, the lighting and the collisions. When mobile is enabled, I load a specific scene that contains less objects. In this scene, most of meshes are merged and materials are merged too when possible (aka atlasing).