The Internet of Things (IoT) is a grand vision of connecting billions of devices with heterogeneous types and capabilities. Even though it is an attractive environment that could change the way we interact with the devices, the real-life and large-scale implementation of it is greatly impeded by the potential security risks that it is susceptible to. The promise is great but the security challenges are also daunting indeed. The IoT security can be addressed from different angles but one of the key issues is the access control model. This would eventually determine which device would be given access to the IoT systems and which would be denied access. In this work, we have done a systematic and thorough survey on the state-of-the-art access control models in IoT. We have covered conventional as well as the advanced access control models taking the crucial period of various studies in this particular area. A number of critical questions have been answered and key works have been summarized. Based on our investigation, some insights into the area, its requirements, scale, and future challenges and prospects have been discussed. We have also provided our rationale and motivation for this work and compared it with previous surveys, none of which has captured this security aspect of IoT with such a depth and detail.