\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[greek,ngerman,english]{babel}










\begin{document}

\title{Assessing Conformer Energies using Electronic Structure and Machine
Learning Methods}



\author[1]{Dakota Folmsbee}%
\author[1]{Geoffrey Hutchison}%
\affil[1]{Department of Chemistry, University of Pittsburgh}%


\vspace{-1em}



  
  \date{May 02, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
We have performed a large-scale evaluation of current computational
methods, including conventional small-molecule force fields,
semiempirical, density functional,~\emph{ab initio} electronic structure
methods, and current machine learning (ML) techniques to evaluate
relative single-point energies. Using up to 10 local minima geometries
across \textasciitilde{}700 molecules, each optimized by B3LYP-D3BJ with
single-point DLPNO-CCSD(T) triple-zeta energies, we consider over 6,500
single points to compare the correlation between different methods for
both relative energies and ordered rankings of minima. We find
promise~from current ML methods and recommend methods at each tier of
the accuracy-time tradeoff, particularly the recent GFN2 semiempirical
method, the B97-3c density functional approximation, and RI-MP2 for
accurate conformer energies. The ANI family of ML methods shows promise,
particularly the ANI-1ccx variant trained in part on coupled-cluster
energies. Multiple methods suggest continued improvements should be
expected in both performance and accuracy.%
\end{abstract}%



\sloppy


\textbf{Keywords:} conformers, thermochemistry, machine learning,
density functional, semiempirical, DFTB, coupled-cluster\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.77\columnwidth]{figures/ToC/ToC}

\end{center}
\end{figure}

\section*{Introduction}

{\label{861857}}

For almost all molecules, multiple geometrically-distinct conformers
exist. Understanding and predicting thermodynamically accessible
ensembles of molecular conformers is a key task underlying much of
computational chemistry.\textsuperscript{\hyperref[csl:1]{[1]},\hyperref[csl:2]{[2]},\hyperref[csl:3]{[3]}} In principle, for each
rotatable bond, the number of possible minima increases exponentially.
Consequently, most conformer sampling methods\textsuperscript{\hyperref[csl:4]{[4]}} use
classical small-molecule force fields to evaluate energies because of
their fast performance, despite potentially poor correlation with
quantum mechanical methods.\textsuperscript{\hyperref[csl:5]{[5]}}

Multiple efforts have evaluated the success of wavefunction and density
functional first-principles methods to compare the energetics of
different conformers.\textsuperscript{\hyperref[csl:6]{[6]},\hyperref[csl:7]{[7]},\hyperref[csl:8]{[8]},\hyperref[csl:9]{[9]},\hyperref[csl:10]{[10]},\hyperref[csl:11]{[11]},\hyperref[csl:12]{[12]}} While experimental crystal
structures and bioactive docked conformers are not always the lowest
energy conformer, recent efforts have demonstrated only small energy
differences when using quantum chemical methods instead of force
fields.\textsuperscript{\hyperref[csl:13]{[13]},\hyperref[csl:14]{[14]}}

Even for simple molecules such as~1,1'-biphenyl, use of large basis set
coupled cluster methods are needed to accurately place the dihedral
angle and barrier.\textsuperscript{\hyperref[csl:15]{[15]}} Other works have documented the
need for accurate treatment of non-covalent interactions to model
conformers in \selectlanguage{greek}π-\selectlanguage{english}conjugated oligomers.\textsuperscript{\hyperref[csl:16]{[16]}}

One common assumption is the presumed balance between increasing desired
thermochemical accuracy and increased computational time. That is, more
computationally intensive methods produce more accurate geometries and
thermochemical properties. For example, the rise of composite~\emph{ab
initio} thermochemical recipes such as G3,\textsuperscript{\hyperref[csl:17]{[17]}}
G4,\textsuperscript{\hyperref[csl:18]{[18]}} and W1\textsuperscript{\hyperref[csl:19]{[19]},\hyperref[csl:20]{[20]}} to W4\textsuperscript{\hyperref[csl:21]{[21]}}
seeks to provide highly accurate thermochemical predictions by separate
estimates of basis set extrapolation and electron correlation. Still,
such methods are largely limited to small molecules due to the high
computational cost.\textsuperscript{\hyperref[csl:22]{[22]}}~As mentioned above, efforts for
conformer sampling have often focused on classical force fields or
multi-level approaches using semiempirical methods.\textsuperscript{\hyperref[csl:4]{[4]},\hyperref[csl:23]{[23]},\hyperref[csl:24]{[24]}}

In our previous paper,\textsuperscript{\hyperref[csl:5]{[5]}}~we considered both the
single-point energies and geometry optimizations of a range of common
computational chemistry methods, including classical force fields,
semiempirical quantum chemistry, and dispersion-corrected density
functional methods. In general, due to the large differences in the
potential energy surfaces predicted by force fields and quantum methods,
we found poor correlation between both single point energies at the same
geometry and optimized geometries using different methods.~

In this work, in order to expand our range of computational methods, we
only consider the relative single~ point energies from the same set of
density-functional optimized geometries, comparing multiple current
methods to a high-quality coupled cluster baseline.~We consider the mean
absolute relative errors in energies (MARE), as well as the correlation
of relative energies, reflected in the R\textsuperscript{2} coefficient
of determination, and the ranking of single-point energies reflected in
the Spearman \selectlanguage{greek}ρ \selectlanguage{english}correlation. The~use of correlation coefficients and the
Spearman correlation intend to consider whether methods exhibit
systematic errors that may not affect linear correlation or ranking of
energetic stabilities.~

{While we find increased accuracy typically still requires exponential
increases in computational time, several methods stand out as widely
useful methods for ranking conformer energies. Future improvements in
standard computational methods and machine learning surrogates suggest
that both increased accuracy and efficiency are expected from further
method development.}

\section*{Computational Methods}

{\label{143611}}

Calculations were performed using Open Babel version
3.0\textsuperscript{\hyperref[csl:25]{[25]}} for all force field calculations
(MMFF94\textsuperscript{\hyperref[csl:26]{[26]},\hyperref[csl:27]{[27]},\hyperref[csl:28]{[28]},\hyperref[csl:29]{[29]},\hyperref[csl:30]{[30]}} and
UFF\textsuperscript{\hyperref[csl:31]{[31]},\hyperref[csl:32]{[32]}})~\texttt{OpenMOPAC}~for
PM7,\textsuperscript{\hyperref[csl:33]{[33]}}~\texttt{xtb} version 6.2\textsuperscript{\hyperref[csl:34]{[34]}} for
GFN0\textsuperscript{\hyperref[csl:35]{[35]}} GFN1\textsuperscript{\hyperref[csl:36]{[36]}} and GFN2
calculations,\textsuperscript{\hyperref[csl:37]{[37]}}~and~\texttt{Orca}
4.0.1\textsuperscript{\hyperref[csl:38]{[38]}} for all density functional and~\emph{ab initio}
calculations, unless otherwise indicated. For density functional
methods, the D3(BJ)\textsuperscript{\hyperref[csl:39]{[39]},\hyperref[csl:40]{[40]},\hyperref[csl:41]{[41]},\hyperref[csl:42]{[42]}} dispersion correction scheme was
used as indicated, except
for~\(\omega\)B97X-D3\textsuperscript{\hyperref[csl:43]{[43]}} which uses a similar
approach. For \emph{ab initio} methods, \texttt{Orca} 4.0.1 was used for
MP2\textsuperscript{\hyperref[csl:44]{[44]}} and~{DLPNO-CCSD(T)}\textsuperscript{\hyperref[csl:45]{[45]},\hyperref[csl:46]{[46]}}~with
``TightPNO'' using the cc-pVTZ basis set.\textsuperscript{\hyperref[csl:47]{[47]},\hyperref[csl:48]{[48]}} Energies
are read from all output files using
the~\texttt{cclib}\textsuperscript{\hyperref[csl:49]{[49]}}~version 1.6.2, and \texttt{pybel}
version 3.0.\textsuperscript{\hyperref[csl:50]{[50]}}

Machine learning methods included ``bag-of-features'' representations
and ANI-1x\textsuperscript{\hyperref[csl:51]{[51]}}, ANI-1ccx\textsuperscript{\hyperref[csl:52]{[52]}}, and
ANI-2x\textsuperscript{\hyperref[csl:53]{[53]}} models. The Bag-of-Features representations
chosen~were Bag of Bonds\textsuperscript{\hyperref[csl:54]{[54]}}(BOB),~ Bond Angle
Torsion~\textsuperscript{\hyperref[csl:55]{[55]}}(BAT), and Bond Angle Torsion Typed (BATTY).
BOB represents atoms and pair-wise interactions into sorted bags with
BAT being a many-body expansion to include angles and torsions. Both of
these representations were
implemented~using~\texttt{chemreps}.\textsuperscript{\hyperref[csl:56]{[56]}} The BATTY
representation takes inspiration from BAT in order to include minimal
atom typing in all bond, angle, and torsion bags while excluding
nonbonding interaction and nuclear charge bags in the final
representation, as discussed
below.~\texttt{scikit-learn}\textsuperscript{\hyperref[csl:57]{[57]}} was used for kernel
ridge regression of Bag-of-Features representations.

For this work, all timings are single-core CPU times using a 2.60 GHz
Intel Skylake CPU (Intel Xeon Gold 6126) with 192GB RAM per node,
through the University Pittsburgh Center for Research Computing.

Python scripts and Jupyter notebooks were used to compile all data
into~\texttt{pandas}\textsuperscript{\hyperref[csl:58]{[58]}}~ data frames,
using~\texttt{numpy}\textsuperscript{\hyperref[csl:59]{[59]}}
and~\texttt{scipy}\textsuperscript{\hyperref[csl:60]{[60]}} functions for analysis. 3DMol.js
was used for interactive molecular visualization of
conformers.\textsuperscript{\hyperref[csl:61]{[61]}} Plotly was used for interactive
plots.\textsuperscript{\hyperref[csl:62]{[62]}}~

All scripts and data, including molecular geometries, are provided
through GitHub (\url{https://github.com/ghutchis/conformer-benchmark})
with the intent that additional computational methods can be added to
these benchmark comparisons.

\section*{Test Set Selection}

As in our previous work,\textsuperscript{\hyperref[csl:5]{[5]}} a dataset consisting of
experimental crystal structures of 700 small molecules capable of
multiple conformer geometries was provided to us by
Ebejer\textsuperscript{\hyperref[csl:63]{[63]}} and were derived from the work of Hawkins et
al.\textsuperscript{\hyperref[csl:23]{[23]}} along with ligands from the Astex Diverse
Set.\textsuperscript{\hyperref[csl:64]{[64]}} These compounds have been repeatedly used to
evaluate the quality of conformer generation.\textsuperscript{\hyperref[csl:23]{[23]},\hyperref[csl:63]{[63]}}
Approximately half (320 molecules) consist solely of carbon, hydrogen,
nitrogen, and oxygen (CHON) atoms, while the remainder are more complex
drug-like compounds and ligands from the Protein Data Bank
(PDB).\textsuperscript{\hyperref[csl:23]{[23]}} A list of Simplified Molecular Input Line Entry
Specification (SMILES)\textsuperscript{\hyperref[csl:65]{[65]}} for all 700 molecules can be
found in the Supporting Information.

For~\emph{ab initio~}calculations using the cc-pVTZ basis sets,
relativistic effective core potentials were not available for molecules
containing iodine. Thus, for comparisons with DLPNO-CCSD(T) and RI-MP2
methods, such species were omitted. Similarly, the ANI-1x and ANI-1cxx
methods only support molecules containing CHON atoms and evaluations
were only performed on the subset of molecules supported. The ANI-2x
method supports additional elements, but not bromine or iodine and thus
evaluations were similarly only performed on the supported subset for
that method.

For bag of feature ML testing, the training set was five conformers of
each molecule, with the remaining conformers as test/validation. Any
molecule with fewer than five conformers had the conformers added to the
training set and was omitted from the test set.

\section*{Results}

{\label{538076}}

In this work, we focus on the evaluation of single point atomization
energy calculations on a subset of \textasciitilde{}700 organic
molecules. Conformers were initially created from a set of 250 diverse
poses with maximal heavy-atom root mean squared deviation (RMSD) using
Open Babel, and at most 10 poses were selected based on the lowest heat
of formation calculated by PM7, followed by full geometry optimization
using B3LYP-D3BJ with the def2-SVP basis set.\textsuperscript{\hyperref[csl:5]{[5]}}

Using this set of DFT-optimized minima, in this work, single point
atomization energies were computed using the
DLPNO-CCSD(T)\textsuperscript{\hyperref[csl:45]{[45]},\hyperref[csl:46]{[46]}} method using the cc-pVTZ basis
set.\protect\hyperlink{Dunning_1989}{(Dunning
1989,}~\protect\hyperlink{Kendall_1992}{Kendall 1992)} This approach has
been found to be a highly accurate method for calculating thermochemical
properties and with a significantly lower computational cost for medium
to large organic molecules, compared to canonical CCSD(T)
methods.\textsuperscript{\hyperref[csl:66]{[66]},\hyperref[csl:67]{[67]},\hyperref[csl:45]{[45]}} U{sing only~the set of molecules in which all
standard (i.e., not machine-learning based) methods completed leaves
6511 entries. Of those, 9 molecules (out of 690) had 2 or fewer poses
and were also removed, leaving 681 unique molecules and
\textasciitilde{}6500 entries for comparison.}

To our knowledge, this is the most extensive computational validation
set, both in terms of the number of compounds, geometries, and
computational methods for studying low energy molecular conformers. We
provide all data and analysis scripts as open data and open source to
allow future reuse via
a~\href{https://github.com/ghutchis/conformer-benchmark/}{GitHub
repository}.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=1.00\columnwidth]{figures/figure-1/figure-1}
\caption{{Example analysis of \selectlanguage{greek}ω\selectlanguage{english}B97X-D3 and GFN2 methods, starting with (A)
correlation between~\selectlanguage{greek}ω\selectlanguage{english}B97X-D3 and DLPNO-CCSD(T) energies for a single
molecule, (B) histogram of R\textsuperscript{2} correlations across all
molecules, (C) correlation between GFN2 and DLPNO-CCSD(T) energies, and
(D) corresponding histogram of R\textsuperscript{2} correlations across
all molecules.
{\label{423464}}%
}}
\end{center}
\end{figure}

As illustrated in Figure~{\ref{423464}}, each method is
correlated with DLPNO-CCSD(T) / cc-pVTZ energies for each molecule
(e.g.,~\texttt{astex\_1hwi} in~Figure~{\ref{423464}}).
Since each molecule has several conformers, three metrics are compiled,
the mean absolute relative energy (MARE) compared to the DLPNO-CCSD(T)
atomization energies, the Pearson R\textsuperscript{2} correlation, and
the Spearman correlation \selectlanguage{greek}ρ. \selectlanguage{english}The MARE metric gives an absolute measure of
the energetic errors, but since different methods use different energy
scales (e.g., heats of formation for PM7 and force fields), the
statistical correlations use linear regression (R\textsuperscript{2})
and relative ordering (Spearman \selectlanguage{greek}ρ\selectlanguage{english}) to remove sources of systematic
energy differences. For each metric across each method, the median value
was compiled as illustrated in~Figure~{\ref{423464}},
to represent the overall quality of a given method.

Since the metrics are unlikely to reflect normal distributions
(e.g.,~Figure~{\ref{423464}} shows highly non-Gaussian
distributions), determining confidence intervals cannot be established
from analytical formulas. Consequently, we used bootstrap sampling to
establish 95\% confidence values for the medians, as reported below. For
ease of discussion, we have given the confidence ranges in all tables
and figures, but indicate \selectlanguage{ngerman}± errors using the average of the upper and
lower bounds. In general, the asymmetry between upper and lower bounds
are smalll.

By considering a large number of diverse organic molecules with many
poses per molecule, we seek to sample a wide variety of conformer energy
preferences (e.g., intramolecular hydrogen and halogen bonding, \selectlanguage{greek}π-π
\selectlanguage{english}stacking, electrostatic interactions, etc.). While using optimized
low-energy conformers may under-estimate the accuracy of methods for
high-energy structures,\textsuperscript{\hyperref[csl:7]{[7]}} we believe the current work is
a challenging but useful comparison. In general, such high-energy
geometries reflect steric repulsion more than the diverse types of
interactions driving low-energy geometries.~

Moreover, many computational predictions rely on Boltzmann-weighted
averages of multiple thermally accessible conformers, including NMR~
prediction,\textsuperscript{\hyperref[csl:2]{[2]},\hyperref[csl:1]{[1]}} reactions, and~{even understanding the
effects of dipole moments on solvent viscosity.}\textsuperscript{\hyperref[csl:68]{[68]}}
Consequently, deriving accurate relative energies of molecular
conformers is a crucial task, as discussed below.

\subsection*{Comparison of single points vs.
DLPNO-CCSD(T)}

{\label{405566}}

For comparison, we considered a wide variety of currently available
computational methods:

\begin{itemize}
\tightlist
\item
  \textbf{Common classical organic force fields}:
  MMFF94,\textsuperscript{\hyperref[csl:69]{[69]},\hyperref[csl:27]{[27]},\hyperref[csl:28]{[28]},\hyperref[csl:29]{[29]},\hyperref[csl:30]{[30]}} UFF,\textsuperscript{\hyperref[csl:31]{[31]}}
  GAFF\textsuperscript{\hyperref[csl:70]{[70]}}
\item
  \textbf{Semiempirical wave function:~}PM7\textsuperscript{\hyperref[csl:33]{[33]}}
\item
  \textbf{Density functional tight binding}: GFN0,\textsuperscript{\hyperref[csl:35]{[35]}}
  GFN1,\textsuperscript{\hyperref[csl:71]{[71]}} GFN2\textsuperscript{\hyperref[csl:37]{[37]}}
\item
  \textbf{Low-cost density functional approximation}s:
  PBEh-3c,\textsuperscript{\hyperref[csl:72]{[72]}} B97-3c\textsuperscript{\hyperref[csl:73]{[73]}}
\item
  \textbf{Dispersion-corrected density functionals}:
  B3LYP,\textsuperscript{\hyperref[csl:74]{[74]},\hyperref[csl:75]{[75]},\hyperref[csl:76]{[76]},\hyperref[csl:77]{[77]}} PBE\textsuperscript{\hyperref[csl:78]{[78]},\hyperref[csl:79]{[79]}},
  \selectlanguage{greek}ω\selectlanguage{english}B97X-D\textsuperscript{\hyperref[csl:43]{[43]}} with dispersion correction (using def2-TZVP
  basis set\textsuperscript{\hyperref[csl:80]{[80]},\hyperref[csl:81]{[81]}})
\item
  \selectlanguage{ngerman}\textbf{Møller-Plesset RI-MP2}\textsuperscript{\hyperref[csl:44]{[44]}}~with the cc-pVTZ
  basis set\textsuperscript{\hyperref[csl:47]{[47]},\hyperref[csl:48]{[48]}}
\end{itemize}

In the case of B3LYP and PBE dispersion-corrected functionals, we also
considered both the commonly-used double-zeta def2-SVP and triple-zeta
def2-TZVP basis sets to understand the effects of basis set size. For
B3LYP, PBE, and \selectlanguage{greek}ω\selectlanguage{english}B97X, we also considered the accuracy with and without
dispersion correction.\selectlanguage{english}
\begin{table}[h!]
\centering
\small\begin{tabulary}{1.0\textwidth}{CCCCCCCCC}
Method & MARE &  & R\textsuperscript{2}
 &  & Spearman \selectlanguage{greek}ρ &  & CPU Time &  \\
DLPNO-CCSD(T) & 0 &  & 1 &  & 1 &  & 21901.38 & 276.95 \\
RI-MP2 & 0.11 & [0.11-0.13] & 0.96 & [0.96-0.97] & 0.95 & [0.95-0.96] & 2118.83 & 42.26 \\
\selectlanguage{greek}ω\selectlanguage{english}B97X-D3 & 0.16 & [0.15-0.17] & 0.93 & [0.91-0.94] & 0.92 & [0.9-0.93] & 2524.83 & 35.67 \\
B3LYP (TZ) & 0.17 & [0.15-0.19] & 0.92 & [0.91-0.93] & 0.92 & [0.9-0.93] & 1672.79 & 20.67 \\
B97-3c & 0.2 & [0.18-0.22] & 0.9 & [0.89-0.92] & 0.9 & [0.88-0.92] & 137.45 & 2.16 \\
PBE (TZ) & 0.21 & [0.19-0.23] & 0.88 & [0.87-0.9] & 0.89 & [0.88-0.9] & 358.65 & 6.94 \\
PBEh-3c & 0.21 & [0.18-0.23] & 0.88 & [0.86-0.9] & 0.88 & [0.87-0.9] & 453.04 & 9.46 \\
B3LYP (SVP) & 0.23 & [0.21-0.26] & 0.87 & [0.84-0.89] & 0.88 & [0.87-0.89] & 330.94 & 4.35 \\
PBE (SVP) & 0.26 & [0.24-0.3] & 0.83 & [0.81-0.86] & 0.85 & [0.84-0.88] & 149.03 & 2.24 \\
ANI-1ccx & 0.44 & [0.36-0.52] & 0.64 & [0.57-0.71] & 0.71 & [0.64-0.77] & 1.45 & 0.0 \\
GFN2 & 0.39 & [0.33-0.43] & 0.64 & [0.59-0.68] & 0.72 & [0.68-0.75] & 2.6 & 0.07 \\
GFN1 & 0.35 & [0.31-0.41] & 0.62 & [0.58-0.66] & 0.7 & [0.66-0.73] & 2.66 & 0.05 \\
ANI-2x & 0.41 & [0.36-0.48] & 0.62 & [0.56-0.69] & 0.68 & [0.65-0.72] & 3.45 & 0.01 \\
ANI-1x & 0.45 & [0.38-0.54] & 0.59 & [0.52-0.66] & 0.65 & [0.57-0.72] & 1.46 & 0.0 \\
BATTY/n & 0.42 & [0.38-0.48] & 0.47 & [0.41-0.54] & 0.5 & [0.4-0.6] & 0.14 & 2.16e-05 \\
GFN0 & 0.44 & [0.39-0.49] & 0.4 & [0.35-0.48] & 0.53 & [0.46-0.56] & 0.07 & 0.0 \\
GAFF & 1.64 & [1.42-1.83] & 0.35 & [0.29-0.41] & 0.48 & [0.44-0.54] & 0.01 & 5.73e-05 \\
MMFF94 & 0.7 & [0.58-0.85] & 0.33 & [0.29-0.38] & 0.47 & [0.43-0.52] & 0.0 & 4.4e-05 \\
BOB & 1.92 & [1.72-2.16] & 0.32 & [0.28-0.39] & 0.1 & [0.0-0.2] & 0.14 & 3.92e-05 \\
PM7 & 0.62 & [0.56-0.71] & 0.32 & [0.27-0.36] & 0.33 & [0.27-0.41] & 0.06 & 0.0 \\
BAT & 1.18 & [1.03-1.3] & 0.31 & [0.28-0.38] & 0.2 & [0.1-0.3] & 0.18 & 1.32e-05 \\
UFF & 5.03 & [4.4-5.61] & 0.29 & [0.24-0.34] & 0.32 & [0.24-0.41] & 0.0 & 8.61e-06 \\
\end{tabulary}
\caption{{Overall statistics across all molecules studied and all methods. Columns
indicate median mean absolute relative error (MARE in kcal/mol), median
R\textsuperscript{2} correlation, median Spearman correlation, and
median single-core CPU time in seconds. MARE, R\textsuperscript{2}, and
Spearman correlation are relative to the DLPNO-CCSD(T)/cc-pVTZ baseline.
Ranges indicate 95\% confidence intervals for the median metrics
established by bootstrap sampling.
{\label{310223}}%
}}
\end{table}\subsection*{Basis Set Effects}

{\label{457487}}

For the frequently-used B3LYP-D3BJ and PBE-D3BJ density functional
methods, we considered both the def2-SVP and def2-TZVP basis sets. In
both cases, the triple zeta basis set significantly improved correlation
with the DLPNO-CCSD(T)/cc-pVTZ baseline, for example, the median
R\textsuperscript{2} scores improved from 0.868\selectlanguage{ngerman}±0.064 to 0.920±0.025 for
B3LYP-D3BJ and from 0.835±0.025 to 0.885±0.018 for PBE-D3BJ. There were
comparable improvements in median Spearman rank correlation and reduced
mean absolute relative errors, all statistically significant (i.e.
p-values far below 0.001).~{The increased basis sets also roughly
doubled the CPU time required.~}

While the PBE method is still significantly faster than B3LYP, the newer
B97-3c proves to be faster than either with comparable accuracy (i.e.,
roughly intermediate to the TZ results for B3LYP-D3BJ and PBE-D3BJ).
Additionally, the time required for B3LYP-D3BJ/def2-TZVP calculations is
only slightly less than RI-MP2/cc-pVTZ results, which exhibit
significantly improved accuracy relative to DLPNO-CCSD(T)/cc-pVTZ (i.e.,
median R\textsuperscript{2} = 0.964±0.006 and median MARE of 0.115
±0.011 kcal/mol for RI-MP2).

Thus increasing basis set size for these density functional methods, at
least from double zeta to triple zeta, does improve accuracy, albeit at
a significant computational cost. In general, the B97-3c method provides
accuracy comparable to popular dispersion-corrected DFT methods such as
B3LYP-D3BJ with faster performance, and RI-MP2 provides greater accuracy
at a very similar speed.

\subsection*{Dispersion Corrections}

{\label{555266}}

Since the bonding is consistent across multiple conformers, the ranking
of small energy differences is known to be dominated by non-bonding
interactions.\textsuperscript{\hyperref[csl:82]{[82]},\hyperref[csl:83]{[83]}} Density functional methods are known to
incorrectly account for dispersion interactions, which has led to a
variety of empirical corrections.\textsuperscript{\hyperref[csl:39]{[39]},\hyperref[csl:40]{[40]},\hyperref[csl:41]{[41]},\hyperref[csl:42]{[42]},\hyperref[csl:84]{[84]},\hyperref[csl:85]{[85]},\hyperref[csl:86]{[86]},\hyperref[csl:87]{[87]}}~Comparing
un-corrected PBE, B3LYP, and \selectlanguage{greek}ω\selectlanguage{english}B97X single-point energies to
DLPNO-CCSD(T) illustrate a significant effect. The uncorrected median
R\textsuperscript{2} values drop by \textasciitilde{}0.12, and the
median Spearman correlations drop by \textasciitilde{}0.08. For example,
the median R\textsuperscript{2} of B3LYP / TZ drops from 0.920\selectlanguage{ngerman}±0.012 to
0.706±0.050 without the D3BJ dispersion correction.

\par\null\selectlanguage{english}
\begin{table}[h!]
\centering
\small\begin{tabulary}{1.0\textwidth}{CCCCCCCCC}
Method & Median R\textsuperscript{\textbf{2}}
 &  &  &  & Median Spearman \selectlanguage{greek}ρ &  &  &  \\
 & Dispersion &  & No Dispersion &  & Dispersion &  & No Dispersion &  \\
DLPNO-CCSD(T) & 1 &  & --- &  & 1 &  & --- &  \\
\selectlanguage{greek}ω\selectlanguage{english}B97X & 0.93 & [0.91-0.94] & 0.88 & [0.86-0.9] & 0.92 & [0.9-0.93] & 0.89 & [0.87-0.9] \\
B3LYP (TZ) & 0.92 & [0.91-0.93] & 0.71 & [0.66-0.76] & 0.92 & [0.9-0.93] & 0.78 & [0.75-0.81] \\
PBE (TZ) & 0.89 & [0.87-0.9] & 0.75 & [0.71-0.79] & 0.89 & [0.88-0.9] & 0.81 & [0.77-0.83] \\
B3LYP (SVP) & 0.87 & [0.84-0.89] & 0.73 & [0.67-0.76] & 0.88 & [0.87-0.89] & 0.78 & [0.76-0.81] \\
PBE (SVP) & 0.84 & [0.81-0.86] & 0.75 & [0.7-0.79] & 0.86 & [0.84-0.88] & 0.81 & [0.77-0.83] \\
\end{tabulary}
\caption{{Effect of dispersion correction for DFT methods. Values in brackets
indicate 95\% confidence intervals from bootstrap sampling.
{\label{498669}}%
}}
\end{table}On the time-scale of a density functional calculation, these empirical
dispersion corrections require only a minuscule time, yet significantly
improve the accuracy of the relative energies. Thus, even though this
work is concerned with intramolecular interactions in conformers,
dispersion-corrected density functional calculations should always be
used. Continued efforts, such as the improved D3
methods\textsuperscript{\hyperref[csl:88]{[88]}} or the new D4 method\textsuperscript{\hyperref[csl:85]{[85]},\hyperref[csl:84]{[84]}} will
hopefully improve their accuracy further.

\subsection*{Comparison of Timing}

{\label{975412}}

As discussed above, a{~frequent concern for conformer screening is the
relative computational performance. I}{n general, classical molecular
force field methods have been preferred since they allow the generation
of hundreds of conformers per compound in seconds. While traditional
high-level~}\emph{ab initio}{ methods are considered a ``gold standard''
for thermochemical energies, the time required for a single point energy
evaluation may be high. For this work, all timings are single-core CPU
times using a 2.60 GHz Intel Skylake CPU (Intel Xeon Gold 6126) with
192GB RAM per node.}

As indicated in Figure~{\ref{413433}}, hybrid density
functional methods such as B3LYP-D3BJ require significant
single-computational time for single-point energies of medium-sized
organic molecules (median 26\selectlanguage{ngerman}±0.3 minutes) compared to GGA methods such
as PBE or approximate density functional tight binding methods such as
GFN1 / GFN2 (median 2.6±0.06 s yields \textasciitilde{}600x speedup).
Conventional density~functional methods nevertheless represent a
meaningful mid-point relative to DLPNO-CCSD(T) method, which may be
faster than traditional coupled cluster methods but are still five to
ten times slower than B3LYP (i.e., hours per single point energy).\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/Timings/Timings}
\caption{{Histograms of relative timings for key methods considered, normalized to
B3LYP-D3BJ single points on the same molecule, using ORCA 4.0.1. Median
relative times and median wall clock times for single-core runs are
included for reference.
{\label{413433}}%
}}
\end{center}
\end{figure}

Consequently, an important consideration is also the typical trade-off
in computational chemistry between thermochemical accuracy and
computational time. Since traditional MP2 and coupled-cluster methods
exhibit high computational complexity, much research ignored them for
medium to large organic molecules due to the time required. Particularly
in computational screening and conformer generation, fast molecular
force fields such as MMFF94 and UFF, as well as semiempirical quantum
chemical methods such as AM1,\textsuperscript{\hyperref[csl:89]{[89]}} PM3,\textsuperscript{\hyperref[csl:90]{[90]}}
PM6,\textsuperscript{\hyperref[csl:91]{[91]}} and PM7\textsuperscript{\hyperref[csl:33]{[33]}} were considered ``good
enough'' to generate structures for further refinement with density
functional and other methods. More recent methods, particularly the ANI
machine learning methods and the GFN family of density functional tight
binding appear to significantly improve on accuracy with only modest
increases in the time required.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/AccuracyTime/R^2}
\caption{{Comparison of single-core computational time required for energy
evaluation (in log scale) to median R\textsuperscript{2} found when
compared to DLPNO-CCSD(T) energies. Error bars indicate 95\% confidence
intervals of time and medain R\textsuperscript{2} from bootstrap
sampling.\textsuperscript{~}Dashed line indicates approximate ``best
current method'' threshold defined from force fields through RI-MP2
methods.
{\label{444292}}%
}}
\end{center}
\end{figure}

We find that consistent with common assumptions, even recent methods
roughly adhere to the requirement of significant increases in
computational (time) cost to gain increased thermochemical accuracy, as
illustrated in Figure~{\ref{444292}} with
R\textsuperscript{2}. Similar trends are found for MARE and Spearman \selectlanguage{greek}ρ
\selectlanguage{english}metrics. Since multiple studies have demonstrated the need for accurate
treatment of noncovalent interactions including intramolecular
electrostatic and dispersion effects for understanding conformer
relative energies, it is not surprising that this benchmark illustrates
the significant accuracy advantage of modern dispersion-corrected
density functional and wavefunction methods.

\subsection*{Use of Machine Learning Methods as Surrogates: ANI and
Bag-of-Features}

{\label{376256}}

One possible solution to the trade-off between accuracy and
computational cost would be the growing use of machine learning (ML)
methods in chemistry, particularly as a surrogate for thermochemical
parameters such as atomization energies.\textsuperscript{\hyperref[csl:92]{[92]},\hyperref[csl:93]{[93]},\hyperref[csl:94]{[94]}} Typically,
these ML methods use deep neural networks (DNN) and have been trained to
density functional calculations, particularly hybrid B3LYP or \selectlanguage{greek}ω\selectlanguage{english}B97X
atomization energies\textsuperscript{\hyperref[csl:95]{[95]}}\textsuperscript{\hyperref[csl:96]{[96]}} although recent
efforts have included training on coupled-cluster quality data as
well.\textsuperscript{\hyperref[csl:52]{[52]}}

In~principle, since the evaluation of the DNN is fast, the time required
for the prediction of~an ML method is dominated by the time to generate
the input descriptors -- still~only a small fraction of that required
for a quantum calculation. Therefore, if~an ML method could reproduce
density functional or coupled-cluster energies at semiempirical or force
field computational cost, it would dramatically change the conventional
accuracy/time tradeoff.

While evaluation of DNN methods would be significantly faster on
graphics processing units (GPUs), and may not be optimized for CPU
evaluation, we note that many quantum chemistry methods are also
accelerated on GPUs. Thus we retain the single-core CPU timings in
Table~{\ref{310223}} and
Figure~{\ref{444292}} but note that the actual speed of
ML methods such as ANI would be faster when evaluated on a modern GPU.

\subsubsection*{ANI methods}

{\label{796693}}

Table~{\ref{310223}} and
Figure~{\ref{444292}} show the ANI family ML methods,
ANI-1x, ANI-1ccx, and ANI-2x, performing similarly to GFN tight binding
semiempirical methods in both accuracy and speed. ANI-1cxx outperforms
the ANI-1x model that does not contain dispersion corrections while
performing slightly better than the ANI-2 model. The inclusion of
dispersion correction for DFT methods is clearly beneficial as they
improve upon their non-dispersion corrected counterparts, as seen in
Table~{\ref{498669}}.

In principal, it is possible to perform~\emph{post hoc} addition of a D3
dispersion correction to both ANI-1x and ANI-2x.
Table~{\ref{201193}} shows potentially improved
performance over their non-dispersion corrected counterparts, although
the differences are not statistically significant. Moreover, since the
D3 dispersion correction for \selectlanguage{greek}ω\selectlanguage{english}B97X-D3~cannot be calculated by standard
tools, applying such a~\emph{post hoc} correction is challenging. For
our set, one could calculate the dispersion correction from the~\selectlanguage{greek}ω\selectlanguage{english}B97X-D3
calculations performed on the same molecule, but without such density
functional calculations, applying dispersion correction would be
impossible.

While the newer D4 correction\textsuperscript{\hyperref[csl:85]{[85]},\hyperref[csl:84]{[84]}} can be calculated using
the DFTD4 program,\textsuperscript{\hyperref[csl:97]{[97]}} we find adding D4 corrections
worsen the median R\textsuperscript{2}~and Spearman metrics, although
again the differences are not statistically significant. The variance of
applying D3 and D4 corrections to the ANI models illustrates the
challenge in current machine learning methods. Since they inherently add
some error on top of the underlying data used for training the model,
use of coupled-cluster or other highly accurate dispersion-corrected
training is needed.\selectlanguage{english}
\begin{table}[h!]
\centering
\small\begin{tabulary}{1.0\textwidth}{CCCCCCCCCCCCC}
Method & Median R\textsuperscript{\textbf{2}}
 &  &  &  &  &  & Median Spearman \selectlanguage{greek}ρ &  &  &  &  &  \\
 & No Dispersion &  & D3 &  & D4 &  & No Dispersion &  & D3 &  & D4 &  \\
ANI-1ccx & 0.64 & [0.57-0.7] & -- &  & -- &  & 0.71 & [0.64-0.77] & -- &  & -- &  \\
ANI-1x & 0.59 & [0.52-0.66] & 0.63 & [0.57-0.71] & 0.57 & [0.48-0.67] & 0.65 & [0.57-0.72] & 0.71 & [0.65-0.75] & 0.62 & [0.56-0.71] \\
ANI-2x & 0.62 & [0.56-0.68] & 0.66 & [0.61-0.7] & 0.6 & [0.54-0.66] & 0.69 & [0.64-0.72] & 0.71 & [0.67-0.73] & 0.66 & [0.62-0.7] \\
\end{tabulary}
\caption{{Comparison of post hoc dispersion correction for ANI machine learning
methods.~Values in brackets indicate 95\% confidence intervals from
bootstrap sampling.
{\label{201193}}%
}}
\end{table}\subsubsection*{Bag of Feature methods}

{\label{417766}}

The performance of the bag-of-features models, while faster than the ANI
symmetry function models, were more comparable to the accuracy of force
field methods. The inclusion of additional information to the descriptor
such as three and four-body interactions and atom typing were beneficial
to the bag-of-features models, the accuracy pales in comparison to the
ANI symmetry function models.

Standard bag-of-features have at minimum a bag of nuclear charges and a
bag of two-body interactions as seen in BOB and further bags are added
that contain additional information such as angles and torsions with
BAT. This approach was taken for the BATTY representation with the
modification of using minimal atom typing (i.e., sp,
sp\textsuperscript{2}, sp\textsuperscript{3} hybridization) to sort
bags. Unlike other bag-of-features representations, the performance of
BATTY was increased by removing the bags of nuclear charges and
excluding the nonbonding interactions from the two-body interactions bag
to create a bag of simple bonds. Since relative conformer energies are
strongly dominated by non-bonded interactions, this finding is
surprising, although perhaps separating bonding and two-body non-bonded
interactions facilitate ML training. A recent example{, BAND-NN, took
the approach of separating the bonding and nonbonding information
similarly to classical force fields and finds an improvement in
performance.}\textsuperscript{\hyperref[csl:98]{[98]}}

ML commonly employs techniques to normalize the data, improving the
model's training.\textsuperscript{\hyperref[csl:99]{[99]},\hyperref[csl:100]{[100]}}~In this work, we used
physically-motivated normalization techniques for the bag-of-features
representations. Four molecular properties, the number of atoms, bonds,
electrons, and the molecular mass, were chosen for normalizing the
atomization energy. BATTY saw improvements in performance when
normalizing by the number of atoms (i.e., BATTY/n) and the number of
bonds (BATTY/b) across Spearman, R\textsuperscript{2}, and MARE. The
other bag-of-feature representations experienced a slight improvement in
R\textsuperscript{2} when normalizing by the number of atoms but not an
improvement in the MARE. Normalizing the atomization energy for
bag-of-features methods does provide minor improvements, but not enough
to compete with the ANI-1 and ANI-2 methods.

ML methods, despite training on density functional and coupled-cluster
energies, are still not as accurate as conventional quantum methods for
predicting conformer energies. At present, the ANI family is comparable
to the semiempirical GFN methods for accuracy on this task.\selectlanguage{english}
\begin{table}[h!]
\centering
\normalsize\begin{tabulary}{1.0\textwidth}{CCCCCCCC}
Method & Normalization & MARE &  & R\textsuperscript{2}
 &  & Spearman \selectlanguage{greek}ρ &  \\
BOB & --- & 1.92 & [1.73-2.16] & 0.32 & [0.27-0.39] & 0.1 & [0.0-0.2] \\
BOB & Atoms & 1.94 & [1.76-2.17] & 0.36 & [0.32-0.42] & 0.1 & [0.0-0.2] \\
BOB & Mass & 2.2 & [1.93-2.41] & 0.32 & [0.27-0.37] & 0.1 & [-0.1-1.0] \\
BOB & Electrons & 2.06 & [1.75-2.28] & 0.32 & [0.28-0.37] & 0 & [-0.1-1.0] \\
BOB & Bonds & 5.09 & [4.46-5.78] & 0.27 & [0.24-0.32] & 0 & [-0.1-1.0] \\
BAT & --- & 1.18 & [1.05-1.3] & 0.31 & [0.28-0.37] & 0.2 & [0.1-0.3] \\
BAT & Atoms & 1.36 & [1.19-1.49] & 0.34 & [0.28-0.4] & 0.1 & [0.1-0.3] \\
BAT & Mass & 1.4 & [1.27-1.55] & 0.31 & [0.27-0.37] & 0.2 & [0.1-0.3] \\
BAT & Electrons & 1.28 & [1.16-1.45] & 0.32 & [0.28-0.38] & 0.15 & [0.1-0.3] \\
BAT & Bonds & 1.62 & [1.45-1.81] & 0.35 & [0.3-0.4] & 0.1 & [0.0-0.2] \\
BATTY & --- & 0.51 & [0.47-0.6] & 0.4 & [0.34-0.44] & 0.4 & [0.3-0.5] \\
BATTY & Atoms & 0.42 & [0.38-0.48] & 0.47 & [0.4-0.54] & 0.5 & [0.4-0.6] \\
BATTY & Mass & 0.69 & [0.61-0.75] & 0.41 & [0.35-0.48] & 0.4 & [0.3-0.5] \\
BATTY & Electrons & 0.63 & [0.55-0.71] & 0.42 & [0.35-0.48] & 0.4 & [0.3-0.5] \\
BATTY & Bonds & 0.42 & [0.37-0.5] & 0.48 & [0.41-0.54] & 0.5 & [0.4-0.6] \\
\end{tabulary}
\caption{{Effects of normalization descriptors on machine learning methods (e.g.
BATTY/n refers to BATTY with number of atom normalization). Numbers in
brackets indicate 95\% confidence intervals for the median MARE,
R\textsuperscript{2}, and Spearman \selectlanguage{greek}ρ \selectlanguage{english}metrics.
{\label{358296}}%
}}
\end{table}\section*{Discussion}

{\label{761846}}

\subsection*{Effects of Conformer Energy Ranges on Accuracy
Metrics}

{\label{935333}}

Previous work has suggested that the poor correlations found between
force field and semiempirical methods are derived from the small number
of low-energy conformers considered in this
benchmark.\textsuperscript{\hyperref[csl:7]{[7]}}~ Certainly, one might imagine that when
considering multiple geometries with only small differences in energies,
random errors are magnified. Figure~{\ref{361711}}
illustrates a histogram of the ranges in DLPNO-CCSD(T) energies across
the molecules considered. Despite the small ranges in energies, there is
little correlation between the energy range of a molecule and the
accuracy metrics of a particular method. This suggests no bias from the
small energy windows used in this benchmark set.

Figure~{\ref{584948}}~ indicates there is no
correlation between R\textsuperscript{2} and the energy window of the
conformers. The ML methods have a relatively even distribution of
R\textsuperscript{2} across the energy window indicating that random
errors in the model may have more of an impact on performance than the
size of the energy window.~\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/Energy-Window/energy-window}
\caption{{Histogram of relative DLPNO-CCSD(T) energy ranges across multiple
conformers.
{\label{361711}}%
}}
\end{center}
\end{figure}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/rsq-ewindow/rsq-ewindow}
\caption{{Examples of the relation of energy windows to R\textsuperscript{2} for
the ML methods (A) ANI-1x, (B) ANI-1ccx, (C) BOB, and (D) BATTY/\#
atoms.
{\label{584948}}%
}}
\end{center}
\end{figure}

\subsection*{\texorpdfstring{Connection Between Accuracy Metrics: MARE,
R\textsuperscript{2},
Spearman}{Connection Between Accuracy Metrics: MARE, R2, Spearman}}

{\label{984720}}

In principle, the mean absolute relative errors in energies (MARE)
consider both random and systematic errors of a method, while the
R\textsuperscript{2} and Spearman correlation metrics remove systematic
errors through linear correlation (R\textsuperscript{2}) or ranking
(Spearman \selectlanguage{greek}ρ\selectlanguage{english}). However, for the comparisons here, there is a strong
connection between all three metrics, as illustrated in
Figure~{\ref{964988}}. Methods with smaller MARE have
almost a linear correlation with increased median R\textsuperscript{2}.
The three classical force field methods have essentially the same median
R\textsuperscript{2} metric despite differences in MARE, likely due to
systematic errors in the methods. Similarly, while increasing the data
in the bag-of-features descriptors from BOB to BAT decreases the median
MARE from 1.92 kcal/mol to 1.18 kcal/mol, the accuracy as judged by the
median R\textsuperscript{2}~remains essentially constant (0.31 and 0.32,
respectively).\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/MARE-R2/MARE-R2}
\caption{{Correlation between mean absolute relative energies (MARE) and median
R\textsuperscript{2} correlation. Since the R\textsuperscript{2} metric
minimizes systematic errors, the high degree of correlation between the
two metrics indicate most methods exhibit relatively random /
non-systematic errors. Error bars indicate 95\% confidence intervals
from bootstrap sampling.
{\label{964988}}%
}}
\end{center}
\end{figure}

\subsection*{Dipole Moment Ranges}

{\label{210891}}

Since we generally find very small energy differences between the
conformers considered in this work, one might wonder whether such
differences have meaningful consequences. Due to Boltzmann statistics,
many properties are dominated by the lowest energy geometry, even~with
small energy windows to other geometries. One recent example comes
from~understanding the effects of dipole moments on solvent
viscosity.\textsuperscript{\hyperref[csl:68]{[68]}} Finding all conformers with proper
weighting is thus crucial to predicting the dipole moment of an ensemble
of different conformers.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/Dipole-Ranges/dipole-range}
\caption{{Histogram of the range of B3LYP-computed dipole moments in Debye across
the conformers considered in this work. While most molecules show only
small differences in polarity across conformers, many have over 3-4
Debye ranges.
{\label{127138}}%
}}
\end{center}
\end{figure}

We find over the set of molecules considered, over 140 molecules have a
range of 3 D or more, and 75 molecules have a range of 4 Debye or above
across multiple conformers in the study.
Figure~{\ref{423369}} illustrates the example
of~\texttt{omegacsd\_CNBPCT}, with two conformers that are close in
energy yet span dramatically different dipole moments. Using~ B3LYP-D3BJ
(TZ), the computed dipole moments range from~1.41D to 9.78D. The
molecule contains two carbonyl bonds, either parallel (high dipole
moment) or anti-parallel (low dipole moment) depending on the rotation
of several bonds and the more polar conformer is predicted to be more
stable by B3LYP-D3BJ, possibly due to an intramolecular hydrogen bond.
On the other hand, u{sing DLPNO-CCSD(T) cc-pVTZ, the conformers differ
by only 0.3 kcal/mol, with the anti-parallel, less polar conformer more
stable than the other.}\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.97\columnwidth]{figures/3dmol/Molecules}
\caption{{Example of conformational diversity in dipole moment in the molecule
omegacsd\_CNBPCT reflecting anti-parallel carbonyl (\emph{left} -
rmsd45) or parallel carbonyl groups (\emph{right} - rmsd92), with
B3LYP-D3BJ def2-TZVP computed dipole moments ranging from 1.41D to
9.78D,~ respectively.~ The two geometries differ by only 1.3 kcal/mol~
at the B3LYP-D3BJ def2-TZVP level, with the more polar conformer (right)
stabilized by an intramolecular hydrogen bond. Using DLPNO-CCSD(T)
cc-pVTZ, the less polar conformer (left) is more stable by 0.3 kcal/mol.
{\label{423369}}%
}}
\end{center}
\end{figure}

Such polarity differences are examples in which small differences in
conformer energies can have significant effects on molecular
properties.~Since experimental properties reflect a Boltzmann-weighted
average of multiple thermally accessible conformers, even small
differences in conformer energies have large effects on populations
involved in property prediction,~ as recently discussed with conformer
and polarity effects on~{solvent viscosity.}\textsuperscript{\hyperref[csl:68]{[68]}}

\textbf{Machine Learning Batch Evaluation~}

An advantage for ML and force field predictions is the ability to batch
evaluate by loading all conformers of a molecule at once and evaluating
them as a batch opposed to evaluating one at a time, as with
conventional quantum chemistry methods.
Table~{\ref{362588}} indicates the median sequential
times from Table~{\ref{310223}} and median time per
single point in batch evaluation. Speedups range \textasciitilde{}70-170
times faster for both force field and ANI methods. We note that while
the ANI methods improve performance in batch evaluation, traditional
force field methods do as well, with similar speedups.\selectlanguage{english}
\begin{table}[h!]
\centering
\small\begin{tabulary}{1.0\textwidth}{CCCCCC}
Method & Median Time &  & Median Batch Time &  & Speedup \\
MMFF94 & 0.0 & 4.4e-05 & 5.05e-05 & 6.02e-07 & 70.89 \\
GAFF & 0.01 & 5.73e-05 & 3.29e-05 & 4.49e-07 & 160.64 \\
UFF & 0.0 & 8.61e-06 & 4.32e-05 & 4.46e-07 & 21.88 \\
ANI-1x & 1.46 & 0.0 & 0.01 & 0.0 & 113.15 \\
ANI-1ccx & 1.45 & 0.0 & 0.01 & 0.0 & 111.5 \\
ANI-2x & 3.45 & 0.01 & 0.02 & 9.7e-05 & 172.44 \\
\end{tabulary}
\caption{{Comparison of single-core median sequential time to median batch
time~(in seconds), and relative speedups for batch evaluation.
{\label{362588}}%
}}
\end{table}\section*{Conclusions}

{\label{582758}}

The current work extends previous efforts to consider the accuracy of
modern computational chemistry methods to rank the energies of drug-like
conformers. Since such energy differences are small, this poses a
challenging benchmark even for density functional methods. Use of
dispersion-corrections for density functionals are required -- the slim
time required is offset with dramatically increased accuracy. While
triple-zeta and larger basis sets also provide higher accuracy, likely
because of better treatment of non-covalent interactions, the large
number of possible conformers forces trade-offs in accuracy and
computational time required.

Current ML methods show great promise, particularly the ANI-1ccx method
trained in part on coupled-cluster energies,\textsuperscript{\hyperref[csl:52]{[52]}} since
they provide accuracy comparable to the semiempirical GFN2 method and
can be performed in batch and accelerated on GPUs. Despite claims of
reaching and exceeding DFT accuracy, we do not find these methods yet
meet the accuracy of modern dispersion-corrected methods. Nevertheless,
we expect these methods will provide increased accuracy in the future.
An important caveat is the need to train on accurate data, such as
dispersion-corrected density functional, MP2, or coupled-cluster
calculations.

We expect continued improvement from other methods,
particularly~{multiple efforts to improve classical force
fields,}\textsuperscript{\hyperref[csl:101]{[101]},\hyperref[csl:102]{[102]},\hyperref[csl:103]{[103]},\hyperref[csl:104]{[104]}}{~inclusion of polarizable atomic
charges,}\textsuperscript{\hyperref[csl:105]{[105]},\hyperref[csl:106]{[106]},\hyperref[csl:107]{[107]},\hyperref[csl:108]{[108]},\hyperref[csl:109]{[109]},\hyperref[csl:110]{[110]},\hyperref[csl:111]{[111]},\hyperref[csl:112]{[112]}} novel force fields from experimental data,
density functional and other quantum methods,\textsuperscript{\hyperref[csl:113]{[113]},\hyperref[csl:114]{[114]},\hyperref[csl:115]{[115]},\hyperref[csl:116]{[116]},\hyperref[csl:117]{[117]},\hyperref[csl:118]{[118]}} and
continued development of approximate semiempirical quantum
methods.\textsuperscript{\hyperref[csl:37]{[37]}}

At present, we can highly recommend methods at each tier of the
accuracy-time tradeoff, particularly the recent GFN2 semiempirical
method, the B97-3c density functional approximation, and RI-MP2 for
accurate conformer energies. Previous efforts to use a hierarchy of
methods are still useful, for example, the use of GFN2 methods to refine
initial conformer ensembles, followed by refinement of a smaller set of
low-energy geometries with more accurate methods. Batch evaluation with
ANI methods are also efficient, although they do not yet span the range
of elements supported by semiempirical methods such as GFN2 or density
functional methods.

The current benchmark reflects conformational preferences in a vacuum as
judged by enthalpy differences alone. Since free energy differences
drive experimental conformers, introducing entropic considerations will
be needed for further work.\textsuperscript{\hyperref[csl:15]{[15]}} Moreover, much chemistry
is performed in solution, thus work on understanding conformer energies
in solvation is also critical.\textsuperscript{\hyperref[csl:119]{[119]},\hyperref[csl:120]{[120]}}

\section*{Acknowledgments}

{\label{552681}}

GRH and DLF acknowledge the~National Science Foundation (CHE-1800435)
for support and the University of Pittsburgh Center for Research
Computing through the computational resources provided. The authors
thank~Olexandr Isayev and Justin Smith for access to the ANI-2x model.

\section*{Supporting Information}

{\label{196787}}

Additional supporting information may be found at the GitHub repository
for this article:~\url{https://github.com/ghutchis/conformer-benchmark}

\selectlanguage{english}
\FloatBarrier
\section*{References}\sloppy
\phantomsection
\label{csl:1}[1]S. Grimme, in \textit{Reviews in Computational Chemistry}, John Wiley {\&} Sons Inc., \textbf{2004}, pp. 153–218.

\phantomsection
\label{csl:2}[2]M. W. Lodewyk, M. R. Siebert, D. J. Tantillo, \textit{Chemical Reviews} \textbf{2011}, \textit{112}, 1839–1862.

\phantomsection
\label{csl:3}[3]N. E. Jackson, B. M. Savoie, K. L. Kohlstedt, T. J. Marks, L. X. Chen, M. A. Ratner, \textit{Macromolecules} \textbf{2014}, \textit{47}, 987–992.

\phantomsection
\label{csl:4}[4]P. C. D. Hawkins, \textit{Journal of Chemical Information and Modeling} \textbf{2017}, \textit{57}, 1747–1756.

\phantomsection
\label{csl:5}[5]I. Y. Kanal, J. A. Keith, G. R. Hutchison, \textit{International Journal of Quantum Chemistry} \textbf{2017}, \textit{118}, e25512.

\phantomsection
\label{csl:6}[6]M. Habgood, T. James, A. Heifetz, \textit{{Conformational Searching with Quantum Mechanics}}, Springer {US}, \textbf{2020}.

\phantomsection
\label{csl:7}[7]D. I. Sharapa, A. Genaev, L. Cavallo, Y. Minenkov, \textit{{ChemPhysChem}} \textbf{2018}, DOI 10.1002/cphc.201801063.

\phantomsection
\label{csl:8}[8]M. K. Kesharwani, A. Karton, J. M. L. Martin, \textit{Journal of Chemical Theory and Computation} \textbf{2015}, \textit{12}, 444–454.

\phantomsection
\label{csl:9}[9]J. {\v{R}}ez{\'{a}}{\v{c}}, D. B{\'{\i}}m, O. Gutten, L. Rul{\'{\i}}{\v{s}}ek, \textit{Journal of Chemical Theory and Computation} \textbf{2018}, \textit{14}, 1254–1266.

\phantomsection
\label{csl:10}[10]V. K. Prasad, A. Otero-de-la-Roza, G. A. DiLabio, \textit{Scientific Data} \textbf{2019}, \textit{6}, DOI 10.1038/sdata.2018.310.

\phantomsection
\label{csl:11}[11]Y. K. Kang, H. S. Park, \textit{Chemical Physics Letters} \textbf{2018}, \textit{702}, 69–75.

\phantomsection
\label{csl:12}[12]Y. Yuan, M. J. L. Mills, P. L. A. Popelier, F. Jensen, \textit{The Journal of Physical Chemistry A} \textbf{2014}, \textit{118}, 7876–7891.

\phantomsection
\label{csl:13}[13]B. K. Rai, V. Sresht, Q. Yang, R. Unwalla, M. Tu, A. M. Mathiowetz, G. A. Bakken, \textit{Journal of Chemical Information and Modeling} \textbf{2019}, \textit{59}, 4195–4208.

\phantomsection
\label{csl:14}[14]N. Foloppe, I.-J. Chen, \textit{Future Medicinal Chemistry} \textbf{2019}, \textit{11}, 97–118.

\phantomsection
\label{csl:15}[15]M. P. Johansson, J. Olsen, \textit{Journal of Chemical Theory and Computation} \textbf{2008}, \textit{4}, 1460–1471.

\phantomsection
\label{csl:16}[16]N. E. Jackson, B. M. Savoie, K. L. Kohlstedt, M. O. de la Cruz, G. C. Schatz, L. X. Chen, M. A. Ratner, \textit{Journal of the American Chemical Society} \textbf{2013}, \textit{135}, 10475–10483.

\phantomsection
\label{csl:17}[17]L. A. Curtiss, K. Raghavachari, P. C. Redfern, V. Rassolov, J. A. Pople, \textit{The Journal of Chemical Physics} \textbf{1998}, \textit{109}, 7764–7776.

\phantomsection
\label{csl:18}[18]L. A. Curtiss, P. C. Redfern, K. Raghavachari, \textit{The Journal of Chemical Physics} \textbf{2007}, \textit{126}, 084108.

\phantomsection
\label{csl:19}[19]J. M. L. Martin, G. de Oliveira, \textit{The Journal of Chemical Physics} \textbf{1999}, \textit{111}, 1843–1856.

\phantomsection
\label{csl:20}[20]S. Parthiban, J. M. L. Martin, \textit{The Journal of Chemical Physics} \textbf{2001}, \textit{114}, 6014–6029.

\phantomsection
\label{csl:21}[21]A. Karton, E. Rabinovich, J. M. L. Martin, B. Ruscic, \textit{The Journal of Chemical Physics} \textbf{2006}, \textit{125}, 144108.

\phantomsection
\label{csl:22}[22]M. M. Ghahremanpour, P. J. van Maaren, J. C. Ditz, R. Lindh, D. van der Spoel, \textit{The Journal of Chemical Physics} \textbf{2016}, \textit{145}, 114305.

\phantomsection
\label{csl:23}[23]P. C. Hawkins, A. G. Skillman, G. L. Warren, B. A. Ellingson, M. T. Stahl, \textit{J Chem Inf Model} \textbf{2010}, \textit{50}, 572–84.

\phantomsection
\label{csl:24}[24]J. Ju{\'{a}}rez-Jim{\'{e}}nez, X. Barril, M. Orozco, R. Pouplana, F. J. Luque, \textit{The Journal of Physical Chemistry B} \textbf{2014}, \textit{119}, 1164–1172.

\phantomsection
\label{csl:25}[25]N. M. O'Boyle, M. Banck, C. A. James, C. Morley, T. Vandermeersch, G. R. Hutchison, \textit{J Cheminform} \textbf{2011}, \textit{3}, 33.

\phantomsection
\label{csl:26}[26]T. A. Halgren, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 490–519.

\phantomsection
\label{csl:27}[27]T. A. Halgren, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 520–552.

\phantomsection
\label{csl:28}[28]T. A. Halgren, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 553–586.

\phantomsection
\label{csl:29}[29]T. A. Halgren, R. B. Nachbar, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 587–615.

\phantomsection
\label{csl:30}[30]T. A. Halgren, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 616–641.

\phantomsection
\label{csl:31}[31]A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard, W. M. Skiff, \textit{Journal of the American Chemical Society} \textbf{1992}, \textit{114}, 10024–10035.

\phantomsection
\label{csl:32}[32]C. J. Casewit, K. S. Colwell, A. K. Rappe, \textit{Journal of the American Chemical Society} \textbf{1992}, \textit{114}, 10035–10046.

\phantomsection
\label{csl:33}[33]J. J. P. Stewart, \textit{Journal of Molecular Modeling} \textbf{2012}, \textit{19}, 1–32.

\phantomsection
\label{csl:34}[34].

\phantomsection
\label{csl:35}[35]P. Pracht, E. Caldeweyher, S. Ehlert, S. Grimme, \textbf{2019}, DOI 10.26434/chemrxiv.8326202.v1.

\phantomsection
\label{csl:36}[36]S. Grimme, C. Bannwarth, P. Shushkov, \textit{J Chem Theory Comput} \textbf{2017}, \textit{13}, 1989–2009.

\phantomsection
\label{csl:37}[37]C. Bannwarth, S. Ehlert, S. Grimme, \textbf{2018}, DOI 10.26434/chemrxiv.7246238.v2.

\phantomsection
\label{csl:38}[38]F. Neese, \textit{Wiley Interdisciplinary Reviews: Computational Molecular Science} \textbf{2011}, \textit{2}, 73–78.

\phantomsection
\label{csl:39}[39]S. Grimme, S. Ehrlich, L. Goerigk, \textit{Journal of Computational Chemistry} \textbf{2011}, \textit{32}, 1456–1465.

\phantomsection
\label{csl:40}[40]A. D. Becke, E. R. Johnson, \textit{The Journal of Chemical Physics} \textbf{2005}, \textit{123}, 154101.

\phantomsection
\label{csl:41}[41]E. R. Johnson, A. D. Becke, \textit{The Journal of Chemical Physics} \textbf{2005}, \textit{123}, 024101.

\phantomsection
\label{csl:42}[42]E. R. Johnson, A. D. Becke, \textit{The Journal of Chemical Physics} \textbf{2006}, \textit{124}, 174104.

\phantomsection
\label{csl:43}[43]J.-D. Chai, M. Head-Gordon, \textit{Physical Chemistry Chemical Physics} \textbf{2008}, \textit{10}, 6615.

\phantomsection
\label{csl:44}[44]S. Kossmann, F. Neese, \textit{Journal of Chemical Theory and Computation} \textbf{2010}, \textit{6}, 2325–2338.

\phantomsection
\label{csl:45}[45]D. G. Liakos, F. Neese, \textit{Journal of Chemical Theory and Computation} \textbf{2015}, \textit{11}, 4054–4063.

\phantomsection
\label{csl:46}[46]Y. Guo, C. Riplinger, U. Becker, D. G. Liakos, Y. Minenkov, L. Cavallo, F. Neese, \textit{The Journal of Chemical Physics} \textbf{2018}, \textit{148}, 011101.

\phantomsection
\label{csl:47}[47]T. H. Dunning, \textit{The Journal of Chemical Physics} \textbf{1989}, \textit{90}, 1007–1023.

\phantomsection
\label{csl:48}[48]R. A. Kendall, T. H. Dunning, R. J. Harrison, \textit{The Journal of Chemical Physics} \textbf{1992}, \textit{96}, 6796–6806.

\phantomsection
\label{csl:49}[49]N. M. O{\textquotesingle}boyle, A. L. Tenderholt, K. M. Langner, \textit{Journal of Computational Chemistry} \textbf{2008}, \textit{29}, 839–845.

\phantomsection
\label{csl:50}[50]N. M. O{\textquotesingle}Boyle, C. Morley, G. R. Hutchison, \textit{Chemistry Central Journal} \textbf{2008}, \textit{2}, DOI 10.1186/1752-153x-2-5.

\phantomsection
\label{csl:51}[51]J. S. Smith, B. Nebgen, N. Lubbers, O. Isayev, A. E. Roitberg, \textit{The Journal of Chemical Physics} \textbf{2018}, \textit{148}, 241733.

\phantomsection
\label{csl:52}[52]J. S. Smith, B. T. Nebgen, R. Zubatyuk, N. Lubbers, C. Devereux, K. Barros, S. Tretiak, O. Isayev, A. Roitberg, \textbf{2019}, DOI 10.26434/chemrxiv.6744440.v2.

\phantomsection
\label{csl:53}[53]C. Devereux, J. Smith, K. Davis, K. Barros, R. Zubatyuk, O. Isayev, A. Roitberg, \textit{ChemRxiv} \textbf{2020}, DOI 10.26434/chemrxiv.11819268.v1.

\phantomsection
\label{csl:54}[54]K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O. A. von Lilienfeld, K.-R. Müller, A. Tkatchenko, \textit{The Journal of Physical Chemistry Letters} \textbf{2015}, \textit{6}, 2326–2331.

\phantomsection
\label{csl:55}[55]B. Huang, O. A. von Lilienfeld, \textit{The Journal of Chemical Physics} \textbf{2016}, \textit{145}, 161102.

\phantomsection
\label{csl:56}[56]D. Folmsbee, S. Upadhyay, A. Dumi, D. Hiener, D. Mulvey, \textbf{2019}, DOI 10.5281/zenodo.3333856.

\phantomsection
\label{csl:57}[57]F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., \textit{Journal of Machine Learning Research} \textbf{2011}, \textit{12}, 2825–2830.

\phantomsection
\label{csl:58}[58]W. McKinney, in \textit{Proceedings of the 9th Python in Science Conference} (Eds.: S. van der Walt, J. Millman), \textbf{2010}, pp. 51–56.

\phantomsection
\label{csl:59}[59]S. van der Walt, S. C. Colbert, G. Varoquaux, \textit{Computing in Science {\&} Engineering} \textbf{2011}, \textit{13}, 22–30.

\phantomsection
\label{csl:60}[60]P. Virtanen, and Ralf Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al., \textit{Nature Methods} \textbf{2020}, \textit{17}, 261–272.

\phantomsection
\label{csl:61}[61]N. Rego, D. Koes, \textit{Bioinformatics} \textbf{2014}, \textit{31}, 1322–1324.

\phantomsection
\label{csl:62}[62]P. T. Inc., \textbf{2015}.

\phantomsection
\label{csl:63}[63]J. P. Ebejer, G. M. Morris, C. M. Deane, \textit{J Chem Inf Model} \textbf{2012}, \textit{52}, 1146–58.

\phantomsection
\label{csl:64}[64]M. J. Hartshorn, M. L. Verdonk, G. Chessari, S. C. Brewerton, W. T. Mooij, P. N. Mortenson, C. W. Murray, \textit{J Med Chem} \textbf{2007}, \textit{50}, 726–41.

\phantomsection
\label{csl:65}[65]D. Weininger, \textit{Journal of Chemical Information and Modeling} \textbf{1988}, \textit{28}, 31–36.

\phantomsection
\label{csl:66}[66]E. Paulechka, A. Kazakov, \textit{The Journal of Physical Chemistry A} \textbf{2017}, \textit{121}, 4379–4387.

\phantomsection
\label{csl:67}[67]D. G. Liakos, Y. Guo, F. Neese, \textit{The Journal of Physical Chemistry A} \textbf{2019}, DOI 10.1021/acs.jpca.9b05734.

\phantomsection
\label{csl:68}[68]M. N. Vo, M. Call, C. Kowall, J. K. Johnson, \textit{Industrial {\&} Engineering Chemistry Research} \textbf{2019}, DOI 10.1021/acs.iecr.9b03699.

\phantomsection
\label{csl:69}[69]T. A. Halgren, \textit{Journal of Computational Chemistry} \textbf{1996}, \textit{17}, 490–519.

\phantomsection
\label{csl:70}[70]J. Wang, R. M. Wolf, J. W. Caldwell, P. A. Kollman, D. A. Case, \textit{Journal of Computational Chemistry} \textbf{2004}, \textit{25}, 1157–1174.

\phantomsection
\label{csl:71}[71]S. Grimme, C. Bannwarth, P. Shushkov, \textit{Journal of Chemical Theory and Computation} \textbf{2017}, \textit{13}, 1989–2009.

\phantomsection
\label{csl:72}[72]S. Grimme, J. G. Brandenburg, C. Bannwarth, A. Hansen, \textit{The Journal of Chemical Physics} \textbf{2015}, \textit{143}, 054107.

\phantomsection
\label{csl:73}[73]J. G. Brandenburg, C. Bannwarth, A. Hansen, S. Grimme, \textit{The Journal of Chemical Physics} \textbf{2018}, \textit{148}, 064104.

\phantomsection
\label{csl:74}[74]C. Lee, W. Yang, R. G. Parr, \textit{Physical Review B} \textbf{1988}, \textit{37}, 785–789.

\phantomsection
\label{csl:75}[75]A. D. Becke, \textit{Physical Review A} \textbf{1988}, \textit{38}, 3098–3100.

\phantomsection
\label{csl:76}[76]P. J. Stephens, F. J. Devlin, C. F. Chabalowski, M. J. Frisch, \textit{The Journal of Physical Chemistry} \textbf{1994}, \textit{98}, 11623–11627.

\phantomsection
\label{csl:77}[77]S. H. Vosko, L. Wilk, M. Nusair, \textit{Canadian Journal of Physics} \textbf{1980}, \textit{58}, 1200–1211.

\phantomsection
\label{csl:78}[78]J. P. Perdew, K. Burke, M. Ernzerhof, \textit{Physical Review Letters} \textbf{1997}, \textit{78}, 1396–1396.

\phantomsection
\label{csl:79}[79]J. P. Perdew, K. Burke, M. Ernzerhof, \textit{Physical Review Letters} \textbf{1996}, \textit{77}, 3865–3868.

\phantomsection
\label{csl:80}[80]F. Weigend, R. Ahlrichs, \textit{Physical Chemistry Chemical Physics} \textbf{2005}, \textit{7}, 3297.

\phantomsection
\label{csl:81}[81]F. Weigend, \textit{Physical Chemistry Chemical Physics} \textbf{2006}, \textit{8}, 1057.

\phantomsection
\label{csl:82}[82]J. M. L. Martin, \textit{The Journal of Physical Chemistry A} \textbf{2013}, \textit{117}, 3118–3132.

\phantomsection
\label{csl:83}[83]D. Gruzman, A. Karton, J. M. L. Martin, \textit{The Journal of Physical Chemistry A} \textbf{2009}, \textit{113}, 11974–11983.

\phantomsection
\label{csl:84}[84]E. Caldeweyher, C. Bannwarth, S. Grimme, \textit{The Journal of Chemical Physics} \textbf{2017}, \textit{147}, 034112.

\phantomsection
\label{csl:85}[85]E. Caldeweyher, S. Ehlert, A. Hansen, H. Neugebauer, S. Spicher, C. Bannwarth, S. Grimme, \textit{The Journal of Chemical Physics} \textbf{2019}, \textit{150}, 154122.

\phantomsection
\label{csl:86}[86]S. Grimme, \textit{Wiley Interdisciplinary Reviews: Computational Molecular Science} \textbf{2011}, \textit{1}, 211–228.

\phantomsection
\label{csl:87}[87]E. R. Johnson, I. D. Mackie, G. A. DiLabio, \textit{Journal of Physical Organic Chemistry} \textbf{2009}, \textit{22}, 1127–1135.

\phantomsection
\label{csl:88}[88]J. Witte, N. Mardirossian, J. B. Neaton, M. Head-Gordon, \textit{Journal of Chemical Theory and Computation} \textbf{2017}, \textit{13}, 2043–2052.

\phantomsection
\label{csl:89}[89]M. J. S. Dewar, E. G. Zoebisch, E. F. Healy, J. J. P. Stewart, \textit{Journal of the American Chemical Society} \textbf{1985}, \textit{107}, 3902–3909.

\phantomsection
\label{csl:90}[90]J. J. P. Stewart, \textit{Journal of Computational Chemistry} \textbf{1989}, \textit{10}, 209–220.

\phantomsection
\label{csl:91}[91]J. J. P. Stewart, \textit{Journal of Molecular Modeling} \textbf{2007}, \textit{13}, 1173–1213.

\phantomsection
\label{csl:92}[92]M. Rupp, A. Tkatchenko, K.-R. Müller, O. A. von Lilienfeld, \textit{Physical Review Letters} \textbf{2012}, \textit{108}, DOI 10.1103/physrevlett.108.058301.

\phantomsection
\label{csl:93}[93]K. Hansen, G. Montavon, F. Biegler, S. Fazli, M. Rupp, M. Scheffler, O. A. von Lilienfeld, A. Tkatchenko, K.-R. Müller, \textit{Journal of Chemical Theory and Computation} \textbf{2013}, \textit{9}, 3404–3419.

\phantomsection
\label{csl:94}[94]F. A. Faber, L. Hutchison, B. Huang, J. Gilmer, S. S. Schoenholz, G. E. Dahl, O. Vinyals, S. Kearnes, P. F. Riley, O. A. von Lilienfeld, \textit{Journal of Chemical Theory and Computation} \textbf{2017}, \textit{13}, 5255–5264.

\phantomsection
\label{csl:95}[95]R. Ramakrishnan, P. O. Dral, M. Rupp, O. A. von Lilienfeld, \textit{Scientific Data} \textbf{2014}, \textit{1}, DOI 10.1038/sdata.2014.22.

\phantomsection
\label{csl:96}[96]J. S. Smith, O. Isayev, A. E. Roitberg, \textit{Chemical Science} \textbf{2017}, \textit{8}, 3192–3203.

\phantomsection
\label{csl:97}[97].

\phantomsection
\label{csl:98}[98]S. Laghuvarapu, Y. Pathak, U. D. Priyakumar, \textit{Journal of Computational Chemistry} \textbf{2019}, \textit{41}, 790–799.

\phantomsection
\label{csl:99}[99]S. Ioffe, C. Szegedy, \textit{arXiv} \textbf{2015}, \textit{abs/1502.03167}.

\phantomsection
\label{csl:100}[100]G. Klambauer, T. Unterthiner, A. Mayr, S. Hochreiter, in \textit{Advances in Neural Information Processing Systems 30} (Eds.: I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett), Curran Associates, Inc., \textbf{2017}, pp. 971–980.

\phantomsection
\label{csl:101}[101]J. Wahl, J. Freyss, M. von Korff, T. Sander, \textit{Journal of Cheminformatics} \textbf{2019}, \textit{11}, DOI 10.1186/s13321-019-0371-6.

\phantomsection
\label{csl:102}[102]D. van der Spoel, M. M. Ghahremanpour, J. A. Lemkul, \textit{The Journal of Physical Chemistry A} \textbf{2018}, \textit{122}, 8982–8988.

\phantomsection
\label{csl:103}[103]K. Roos, C. Wu, W. Damm, M. Reboul, J. M. Stevenson, C. Lu, M. K. Dahlgren, S. Mondal, W. Chen, L. Wang, et al., \textit{Journal of Chemical Theory and Computation} \textbf{2019}, \textit{15}, 1863–1874.

\phantomsection
\label{csl:104}[104]E. Harder, W. Damm, J. Maple, C. Wu, M. Reboul, J. Y. Xiang, L. Wang, D. Lupyan, M. K. Dahlgren, J. L. Knight, et al., \textit{Journal of Chemical Theory and Computation} \textbf{2015}, \textit{12}, 281–296.

\phantomsection
\label{csl:105}[105]F.-Y. Lin, A. D. MacKerell, in \textit{Biomolecular Simulations: Methods and Protocols} (Eds.: M. Bonomi, C. Camilloni), Springer New York, New York, NY, \textbf{2019}, pp. 21–54.

\phantomsection
\label{csl:106}[106]V. S. S. Inakollu, D. P. Geerke, C. N. Rowley, H. Yu, \textit{Current Opinion in Structural Biology} \textbf{2020}, \textit{61}, 182–190.

\phantomsection
\label{csl:107}[107]A. Warshel, M. Kato, A. V. Pisliakov, \textit{Journal of Chemical Theory and Computation} \textbf{2007}, \textit{3}, 2034–2045.

\phantomsection
\label{csl:108}[108]Z. Jing, C. Liu, S. Y. Cheng, R. Qi, B. D. Walker, J.-P. Piquemal, P. Ren, \textit{Annual Review of Biophysics} \textbf{2019}, \textit{48}, 371–394.

\phantomsection
\label{csl:109}[109]C. Zhang, C. Lu, Z. Jing, C. Wu, J.-P. Piquemal, J. W. Ponder, P. Ren, \textit{Journal of Chemical Theory and Computation} \textbf{2018}, \textit{14}, 2084–2108.

\phantomsection
\label{csl:110}[110]J. A. Rackers, Q. Wang, C. Liu, J.-P. Piquemal, P. Ren, J. W. Ponder, \textit{Physical Chemistry Chemical Physics} \textbf{2017}, \textit{19}, 276–291.

\phantomsection
\label{csl:111}[111]J. W. Ponder, C. Wu, P. Ren, V. S. Pande, J. D. Chodera, M. J. Schnieders, I. Haque, D. L. Mobley, D. S. Lambrecht, R. A. DiStasio, et al., \textit{The Journal of Physical Chemistry B} \textbf{2010}, \textit{114}, 2549–2564.

\phantomsection
\label{csl:112}[112]C. Liu, J.-P. Piquemal, P. Ren, \textit{Journal of Chemical Theory and Computation} \textbf{2019}, \textit{15}, 4122–4139.

\phantomsection
\label{csl:113}[113].

\phantomsection
\label{csl:114}[114]K. A. Beauchamp, J. M. Behr, A. S. Rustenburg, C. I. Bayly, K. Kroenlein, J. D. Chodera, \textit{The Journal of Physical Chemistry B} \textbf{2015}, \textit{119}, 12912–12920.

\phantomsection
\label{csl:115}[115]C. Zanette, C. C. Bannan, C. I. Bayly, J. Fass, M. K. Gilson, M. R. Shirts, J. D. Chodera, D. L. Mobley, \textit{Journal of Chemical Theory and Computation} \textbf{2018}, \textit{15}, 402–423.

\phantomsection
\label{csl:116}[116]B. Waldher, J. Kuta, S. Chen, N. Henson, A. E. Clark, \textit{Journal of Computational Chemistry} \textbf{2010}, NA--NA.

\phantomsection
\label{csl:117}[117]F. Zahariev, N. D. Silva, M. S. Gordon, T. L. Windus, M. Dick-Perez, \textit{Journal of Chemical Information and Modeling} \textbf{2017}, \textit{57}, 391–396.

\phantomsection
\label{csl:118}[118]S. Grimme, \textit{Journal of Chemical Theory and Computation} \textbf{2014}, \textit{10}, 4497–4514.

\phantomsection
\label{csl:119}[119]Y. Basdogan, A. M. Maldonado, J. A. Keith, \textit{{WIREs} Computational Molecular Science} \textbf{2019}, \textit{10}, DOI 10.1002/wcms.1446.

\phantomsection
\label{csl:120}[120]Y. Basdogan, J. A. Keith, \textit{Chemical Science} \textbf{2018}, \textit{9}, 5341–5346.

\end{document}

