Publication bias
Following Nakagawa et al. (2022), we relied on two complementary
approaches to assess small study effects, which may result from
publication bias. First, we visualized the relationship between effect
sizes and precision (SE) using funnel plots. To do this, we re-fitted
selected models as random effect models and computed residual effect
sizes conditional on experiment, observation, and factor level, for
factors included as moderators in the main analyses. These conditional
residuals have the advantage of taking some within-experiment
non-independence into account, but they still make unlikely assumptions
about sampling variances (Nakagawa et al. 2022).
We, therefore, complemented funnel plots with a two-step, modified
Egger’s test for multilevel meta-analysis (Nakagawa et al. 2022).
In the first step of this test, SE of effect sizes is included as the
only moderator in a meta-regression with the same random effect
structure as our main MLMA analyses. A significant slope of this
moderator means that studies with low precision tended to report either
more negative or more positive effects than studies with higher
precision. Therefore, if the SE slope is different from zero, the second
step of the test is to fit a meta-regression with the variance of effect
sizes as the only moderator. The intercept of this second
meta-regression is then a more appropriate estimate of the overall
meta-analytic effect (Stanley & Doucouliagos 2014). Because we
uncovered evidence consistent with publication bias in Q1 and Q2, we
tested the robustness of the meta-analytic effects of moderators by
fitting a multi-level meta-regression (MLMR) with variance in addition
to the moderators of interest for each question in our study (see
Supporting Material).