Electronic structure and topological phase transition in multi-orbital triangular lattice with rotation and mirror symmetry breaking

Triangular lattice is one of the most generic structures in the two-dimensional (2D) limit with numerous exotic properties, such as magnetic frustration and quantum spin liquid. Here, we investigate the electronic properties of a multi-orbital (p(x), p(y), p(z)) triangular lattice while focusing on the role of rotation and mirror symmetry breaking in regulating the topological phase diagram. The tight-binding modeling reveals that rotation symmetry (C-2z) breaking splits the Dirac point and preserves the nodal loop. A further inclusion of broken horizontal mirror symmetry (M-h) destroys the nodal loop. Consequently, with spin-orbit coupling (SOC), the (p(x), p(y), p(z))-orbital triangular lattice is topologically non-trivial with only C-2z or M-h symmetry breaking. With both symmetries breaking, it becomes trivial, except when the SOC is large enough to initiate band inversion. For real materials, we propose that the recently grown AgSe/AgTe monolayer in a binary honeycomb structure is an ideal system to realize the multi-orbital triangular lattice. The first-principles calculations demonstrate that free-standing AgSe and AgTe are Z(2) non-trivial. On the Ag(111) substrate, AgSe becomes trivial because of the breaking of both C-2z and M-h symmetries, whereas AgTe remains non-trivial because of the strong SOC of Te. Our work not only provides physical insights into the experimentally synthesized 2D binary honeycomb structures but also highlights the manipulation of electronic and topological properties through symmetry/orbital physics, which are valuable for designing new quantum materials and devices.