Discussion
External validity, target populations, generalizability,
transportability, and target validity were not explicit priorities when
designing and constructing these distributed networks. As the networks
have developed, however, differences in results between nodes of the
distributed networks have forced them to confront external validity and
related concepts, and each has chosen to adapt to this challenge in
their own way.
Sentinel uses US-based data networks that cover a large portion of the
population to respond to FDA queries about target populations treated
with a given medication in the US. As a result, it has primarily
interfaced with a lack of generalizability and transportability as
explanations for differences in effect estimates between study sites.
Its primary method for increasing representability has been adding new
data (e.g., Medicaid claims). While CNODES is similarly designed to
answer government queries, the fact that it targets a Canadian
population (despite including non-Canadian data partners) has resulted
in more consideration of how health systems can shape the
generalizability and transportability of study results earlier during
the study design process. PCORnet, in turn, was built for both the
enrollment of pragmatic trials and observational research on cohorts in
a wide array of disease states. PCORnet research questions and target
populations vary more widely than with the other two targeted networks,
making representativeness a major consideration for projects conducted
within the network. PCORnet is also uniquely situated to explore
external validity in these pragmatic trials, since confounding is very
unlikely to be the origin of any differences in estimated treatment
effects across nodes of the network. Of course, these are not the only
noteworthy distributed networks. Exploring how AsPEN, ConcePTION,
DARWIN-EU, and any other networks that may start performing research
differ in their approaches to external validity can only further
understanding of the utility of such analyses.
Despite these differences, it is reassuring to see that all three
networks consider external validity during the study design and
enrollment and interpretation of the final results. Still, none of the
three networks have established analytic tools for standardizing,
generalizing, or transporting partner- or site-specific estimates to
specific target populations (even if some ad hoc programming could
technically be used within some of the systems). This is not surprising
because these methods are still in the early stages of development and
may not yet be sufficiently formalized to be trusted by regulatory
bodies. Researchers need to adapt analytic methods for generalizing and
transporting study results to these distributed settings and establish
how they can improve the overall interpretability of findings. Such
methods are also likely to play a key role in cross-network comparisons
that contrast results from Sentinel, CNODES, PCORnet, or other
distributed networks with one another.