Annual North Atlantic tropical cyclone (TC) counts are frequently modeled as a Poisson process with a state-dependent rate. We provide a lower bound on the forecasting error of this class of models. Remarkably we find this bound is already saturated by a simple linear model that explains roughly 50\% of the annual variance using three climate indices: El Ni\ no/Southern Oscillation (ENSO), average sea surface temperature (SST) in the main development region (MDR) of the North Atlantic and the North Atlantic oscillation (NAO) atmospheric circulation index \cite{Kozar2012}. As expected under the bound, increased model complexity does not help: we demonstrate that allowing for quadratic and interaction terms, or using an Elastic Net to forecast TC counts using global SST maps, produces no detectable increase in skill. We provide evidence that observed TC counts are consistent with a Poisson process, limiting possible improvements in TC modeling by relaxing the Poisson assumption.