Infractions other than collisions are also a crucial factor in Autonomous driving since other infractions can result in an Accident. Most of the existing work has been tested for Navigation, collisions and least tested for other infractions such as off-road driving and not obeying road signs. We present an imitation learning based Base Model called CILDO and an optimized model of the base model optimized using an and additional traffic light detection branch and Deep Deterministic Policy Gradient based Reinforcement Learning called CILDOLI-RL. The proposed base model is designed to detect dynamic objects by developing the vision for semantic features, depth and motion which is crucial in Autonomous driving. The CILDO-RL model presented in this paper achieves highest score for the newly introduced No-OTHERINFRACTION benchmark ensuring safe autonomous driving while the base CILDO model achieves the best performance in Navigation under Urban or rural dense traffic environments. Further, this work makes a comparison on the computational complexities of the driving models.