We describe a method for the creation of a compact model for local layout effects (LLE) using pixelated images of the layers of physical layouts as input features. We incorporate these together with electrical measurements of devices on an integrated circuit as input to a neural network. The neural network combines a variational autoencoder (VAE) with a regression network and is used to identify important features of the layout. Then, using a parallel feature neural network to model the effects of those features, we can determine the functional dependence of those effects. Finally, we incorporate those functions into our existing compact model. Because the VAE takes the full design as input, it represents every possible relevant feature. The model visually reveals which features are important, and these visual features can be converted to numerical features. Existing compact models rely on engineers to determine a priori which features are important, but important dimensions on novel devices can easily be missed. We also show that the constraints of the latent space of the VAE allow us to determine during inference whether a particular device geometry is within the domain of the training data. In addition, we describe a process for training the VAE that improves its ability to interpolate more accurately.