Monitoring performance in organizations can act as a catalyst for (i) fostering a culture of continuous improvement over-time, and (ii) achieving their overarching business goals. To this aims, organizations employ Key Performance Indicators (KPIs). However, a common pitfall is the collection of excessive data which is partially disconnected from the business goals. In this study, we propose a KPI framework in the form of a model, an associated template, and a checklist with the aim of providing guidance to practitioners for creating sound KPIs. We examine KPI development and monitoring through four case studies in the context of software systems in two organizations. Through these case studies, we evaluate our KPI framework for its utility in practice and gain insights to uncover related challenges.