Discussion
External validity, target populations, generalizability, transportability, and target validity were not explicit priorities when designing and constructing these distributed networks. As the networks have developed, however, differences in results between nodes of the distributed networks have forced them to confront external validity and related concepts, and each has chosen to adapt to this challenge in their own way.
Sentinel uses US-based data networks that cover a large portion of the population to respond to FDA queries about target populations treated with a given medication in the US. As a result, it has primarily interfaced with a lack of generalizability and transportability as explanations for differences in effect estimates between study sites. Its primary method for increasing representability has been adding new data (e.g., Medicaid claims). While CNODES is similarly designed to answer government queries, the fact that it targets a Canadian population (despite including non-Canadian data partners) has resulted in more consideration of how health systems can shape the generalizability and transportability of study results earlier during the study design process. PCORnet, in turn, was built for both the enrollment of pragmatic trials and observational research on cohorts in a wide array of disease states. PCORnet research questions and target populations vary more widely than with the other two targeted networks, making representativeness a major consideration for projects conducted within the network. PCORnet is also uniquely situated to explore external validity in these pragmatic trials, since confounding is very unlikely to be the origin of any differences in estimated treatment effects across nodes of the network. Of course, these are not the only noteworthy distributed networks. Exploring how AsPEN, ConcePTION, DARWIN-EU, and any other networks that may start performing research differ in their approaches to external validity can only further understanding of the utility of such analyses.
Despite these differences, it is reassuring to see that all three networks consider external validity during the study design and enrollment and interpretation of the final results. Still, none of the three networks have established analytic tools for standardizing, generalizing, or transporting partner- or site-specific estimates to specific target populations (even if some ad hoc programming could technically be used within some of the systems). This is not surprising because these methods are still in the early stages of development and may not yet be sufficiently formalized to be trusted by regulatory bodies. Researchers need to adapt analytic methods for generalizing and transporting study results to these distributed settings and establish how they can improve the overall interpretability of findings. Such methods are also likely to play a key role in cross-network comparisons that contrast results from Sentinel, CNODES, PCORnet, or other distributed networks with one another.