The Atlantic Coastal Plain is comprised of thick, unconsolidated sediments that span much of the East Coast of the United States. Seismic waves passing through this shallow, loosely compacted layer of sedimentary rock display increased ground motion amplitudes as measured on broadband seismometers deployed during the Southeastern Suture of the Appalachian Margin Experiment (SESAME) from 2010–2014. This amplification can lead to greater damage during earthquakes, making site response analysis, and constraints on the associated shallow shear velocity structure, important for informing ground motion modeling in the region. Site amplification is most noticeable in the horizontal direction and, therefore, can be quantified at single seismic stations by dividing the horizontal spectrum by the vertical spectrum of ambient seismic noise. The resulting horizontal-to-vertical spectral ratio (HVSR) provides a simple estimate of the fundamental frequency of a given site; however, more holistic modeling of the complete HVSR curve to constrain shallow elastic structure remains challenging. We leverage the diffuse field assumption (DFA) to interpret HVSR in terms of the ratios of the imaginary parts of the Green’s function components, which allows for the inversion of HVSR curves for elastic structure. We measure HVSR at 91 broadband stations of the Southeastern Suture of the Appalachian Margin Experiment (SESAME) and apply the HV-Inv tool to model the Atlantic Coastal Plain sediment structure beneath each station. While we observe clear lateral variability in HVSR across the array due to variations in sediment structure, we generally find little azimuthal variability of HVSR at most stations, allowing local simplification of the structure to horizontal layers. To resolve tradeoffs in the elastic structure due to fitting HVSR alone, we plan to incorporate additional observations of high-frequency Rayleigh wave ellipticity and multimode phase velocity estimated from ambient noise cross-correlations. Together, these complementary datasets will contribute to a robust framework for subsurface characterization in complex geological settings.