Due to the inherent complexity arising from powertrain, aerodynamic, and unseen driving scenarios, efficient coordination of connected hybrid electric vehicles (C-HEVs) platooning encounters complex challenges. Existing studies primarily do not consider thermal load impacts. However, this overlooks the interconnection between the C-HEVs related to powertrain power and thermal loads, which can hinder effective optimization and collaboration across C-HEV platoon members. To fully address this gap, a multi-agent deep reinforcement learning (DRL)-based joint cooperative adaptive cruise control (CACC) with an integrated thermal and energy management (ITEM) framework is proposed. Additionally, to decrease the sample complexity and accelerate policy learning, the vehicle-level (string stability and time headway) and powertrain-level (energy consumption and health indicator) information are integrated into the reward function to design a safety-and physics-informed DRL framework. To meticulously assess the mutual impact of thermal load and management on the platooning optimization problem, air drag and airflow reductions related to inter-vehicle distance (IVD) are accounted for with a high-fidelity model provided by Autonomie. Numerical and hardware-in-the-loop (HIL) experiments demonstrate that the proposed CACC+ITEM algorithm outperforms the classic DRL version in stability, energy-saving, driving comfort, and battery longevity.