We propose a hierarchical control framework for a bilateral hip exoskeleton to provide the adaptive optimal hip joint assistance with a control objective of imposing the desired gait symmetry during walking. Three control levels are included in the hierarchical framework, including the high-level control to tune three control parameters based on a policy iteration reinforcement learning approach, the middle-level control to define the desired assistive torque profile based on a delayed output feedback control method, and the low-level control to achieve a good torque trajectory tracking performance. To evaluate the feasibility of the proposed control framework, five healthy young participants are recruited for treadmill walking experiments, where an artificial gait asymmetry is imitated as the hemiparesis post-stroke, and only the 'paretic' hip joint is controlled with the proposed framework. The pilot experimental studies demonstrate that the hierarchical control framework for the hip exoskeleton successfully and efficiently achieved the desired gait symmetry by providing adaptive optimal assistance on the 'paretic' hip joint. Our next step will validate the control framework's effectiveness on individuals post-stroke during treadmill walking and overground walking.