Rising shares of renewable generation raises uncertainty and thus the number of possible power flow scenarios in the power system, which in turn increases the possibility for unforeseen contingencies, such as power line or generator failures and their combinations. Therefore, operators cannot longer rely only on operational experience to deal with every contingency. Our proposed method involves identifying the most effective countermeasures to minimize the impact of contingencies on the power system. We take into account various options, such as load shedding, adjusting phase-shifting transformer angles, and injecting active power using fast elements. The proposed approach considers primary control of generators and its limitations in order to compensate for power imbalances in the system. The problem is formulated as a mixed-integer linear optimization problem, employing DC power flow equations. The applicability of the approach is evaluated on the IEEE 39-bus system, and the scalability of the approach is shown on five systems with up to 6470 buses.