Insect herbivory is a critical top-down force structuring plant communities, and quantifying the factors that mediate damage caused by herbivores is fundamental to understanding biodiversity. As herbivory is the result of numerous ecological and evolutionary processes, including complex population dynamics and the evolution of plant defense, it has been difficult to predict variation in herbivory across meaningful spatial scales. In the present work, we characterized patterns of herbivory on plants in a speciose and abundant tropical understory genus (Piper) across forests spanning 44° of latitude in the Neotropics. We modeled the effects of geography, climate, resource availability, species richness and top-down pressure from parasitoids on the mean, dispersion, and skew of generalist and specialist herbivory. By examining these multiple moments of the distribution of herbivory, we were able to determine factors that increase biologically meaningful herbivory at the upper ends of its distribution. The strongest pattern that emerged at a large spatial scale was a roughly two-fold increase in herbivory in humid relative to seasonal forests. Site level variables such as latitude, seasonality and maximum Piper richness explained variation in herbivory at the local scale (plot level) better for communities of Piper congeners than for a single species. Predictors that varied between local communities, such as resource availability and diversity, best explained the distribution of herbivory within sites, dampening any broad patterns across latitude and climate and demonstrating why generalizations about gradients in herbivory have been elusive. The estimated population means, skew, and dispersion of herbivory respond differently to abiotic and biotic factors, demonstrating the need for careful studies to explore the distributions of herbivory and their effects on forest diversity.