This sign language becomes an integral part of communicating with deaf people who are mute. otherwise, it is a long way barrier to interacting with other people who are unfamiliar with this way of communication. So the sign language recognition system is a way to remove this long-way barrier to better and proper communication. As there are quite several sign languages in use across the globe, it is not practically possible for everyone to learn these languages. This paper proposes the technique derived from deep learning to sign language recognition using custom datasets for two different languages. Each dataset has 35 classes with each class having 1200 different images. Hereby, in this work, we further present a process of describing the datasets of ISL and ASL in this respect by image classes followed by labeling the same. The TensorFlow model is trained with these datasets to achieve a system to identify sign language. High accuracy can still be obtained even though the dataset size is small.