The first-principle calculations is performed to search for the candidate stable structures for Li-In-Se and Li-In-Te system below 100 GPa for exploring nonlinear optical materials. A new structure with semiconductor properties has been successfully predicted, namely R-3m LiInSe2, may be a potential nonlinear optical material. At the same time, the electronic and optical properties confirm that the band gap of Pna21 LiInSe2 is increasing under pressure, indicating that the optical damage threshold is increasing, which provides a new idea for improving the nonlinear optical properties of materials. In addition, the study of bonding properties shows that In atom exhibits negative valence state in the structure with high pressure or increased Li atom content, which is similar to Ga atom.