ENGAGE is in development as a web-based observation system with core features we believe will facilitate its use as a scalable assessment-to-action coaching and instructional support system in preschool classrooms. ENGAGE assesses adult interaction practices and children’s active engagement such that classroom teams receive data to inform their intentional design and delivery of embedded instruction that can be made more effective for children’s developmental needs through differentiation and intensification. For this study, we describe important precursors to evaluating other psychometric properties within an argument-based approach to validation. We used iterative cycles to gather evidence to refine and validate the content of our measurement targets (i.e., adult interaction practices and child active engagement) as well as constructs (i.e., groupings of practices by theorized mechanisms). Following two iterative rounds using online questionnaires, responses from participants representing intended users of ENGAGE generally demonstrated consistency with our definitions and conceptualizations, with more varied responses for child engagement. We discuss our procedures and results in terms of reducing measurement error that may be attributable to measurement targets in advance of future examinations of observation conditions (e.g., observer training, session duration) and situational variance (e.g., instructional pedagogy, adult-child ratios) that are often influential within observation-based assessments.