Solving combinatorial optimization problems can be challenging due to their inherent complexity, large search spaces, and NP-hard nature. Traditional solvers often require problem-specific heuristics or exact algorithms, which may not generalize well across domains. Large language models (LLMs) offer a novel approach by leveraging their vast knowledge, heuristic reasoning, and compositional capabilities to explore solutions dynamically and interactively, potentially complementing or enhancing classical methods. This paper analyzes the capabilities of large language models (LLMs) in solving combinatorial optimization problems (both convex and non-convex), and evaluates their correctness against traditional optimization solvers over several benchmark problems. The paper then explores the use of LLMs beyond solving classical combinatorial optimization problems. Specifically, it investigates the strengths of LLMs in generating novel and enhanced heuristic solutions to efficiently solve NP-hard problems. The findings in this article highlight the unique advantages and limitations of LLMs in optimization tasks, shedding light on how they can enhance existing methodologies and contribute to solving real-world problems. Furthermore, it discusses implications for future research and practical applications of LLMs in fields such as domain-agnostic resource allocation, where efficient optimization solutions will have a significant impact. The codes used to generate the results in this paper are available at: https://github.com/laythhamad/LLM_in_OPT/.