Annotation plays an essential and important role in the performance of machine learning approaches. However, manually labeling objects accurately is often time-consuming, especially when processing large image data. To improve this process, we developed a deep learning-based interactive annotation approach with application to hyperspectral imagery. Specifically, a deep network with a metric-based loss function will be pre-trained on a large training set. Encoding layers in the network are used for dimensionality reduction as well as boosting discriminability among pixels belonging to different classes in the embedding space. Then, when new images are provided, the user can provide (fast, easy, weak) annotations to indicate objects of interest and fine-tune the pre-trained network to the task at hand. In this way, the fine-tuned network can adapt for the new category and task. Annotations can be iteratively modified and corrected. Once the model is fine-tuned by quickly annotating a few images, it can then be generalized to a large image dataset of similar objects. Experiments on the MUUFL Gulfport hyperspectral imaging dataset show that our approach outperforms classical active learning methods based on support vector machine in average Intersect of Union (from around 65% to around 80%) with significant improvement on computing speeds (from 30-200 seconds to less than 10 seconds)