Crowd-based hydrological observations can supplement existing monitoring networks and allow data collection in regions where otherwise no data would be available. In the citizen science project CrowdWater, repeated water level observations using a virtual staff gauge approach result in time series of water level classes. To investigate the quality of these observations, we compared the water level class data for a number of locations where water levels were also measured and assessed when these observations were submitted. We analysed data for nine locations where citizen scientists reported multiple observations using a smartphone app and stream level data were also available. At twelve other locations, signposts were set up to ask citizens to record observations on a form that could be left in a letterbox. The results indicate that the quality of the data collected with the app was higher than for the forms. A possible explanation is that for each app location, most contributions were made by a single person, whereas at the locations of the forms almost every observation was made by a new contributor. On average, more contributions were made between May and September than during the other months. Observations were submitted for a range of flow conditions, with a higher fraction of high flow observations for the data collected with the app. Overall, the results are encouraging for citizen science approaches in hydrology and demonstrate that the smartphone application with its virtual staff gauge is a promising approach for crowd-based water level class observations.