Fisheye cameras got their name from fisheye lenses, used by them to capture extremely wide-angled images, often providing a hemispherical or panoramic view. However, the fisheye lens is characterized by its distinctive visual distortion, where straight lines appear curved, and objects at the edges of the frame are elongated or stretched. Fisheye cameras have played an irreplaceable component in vision-first detection approaches in autonomous driving due to their ability to capture a complete view of objects that are extremely close-range to the ego vehicle, making them most critical to detect from the safety point of view. Fisheye cameras are also very specialized, with limited but critical usage in robots like autonomous vehicles (AVs). This review paper fills the gap in our literature by walking across how to leverage the pros of fisheye cameras by minimizing their cons. This paper also provides a good one-stop destination of literature for anyone willing to deploy a fisheye camera in the autonomy stack of a robot. Finally, we also discuss future research directions to enlighten the progress of the fisheye camera perception domain.