The vertical distribution of soil properties is crucial in accurately representing various environmental processes such as freeze-thaw cycles and diurnal variations. In this paper, considering the complexity of the multi-parameter features of layered soil, we evaluate the potential to retrieve the vertical distribution of the moisture and temperature of soil using multi-channel passive microwave observations. To enhance the inversion rate and accuracy, a novel Physics-Driven Artificial Neural Network (P-ANN) inversion algorithm combining multi-angle (30 to 50 degree), multi-frequency (L-, C-, and X-band), and multi-polarization (horizontal and vertical polarization) passive observations is proposed. In this approach, the multi-channel physical brightness temperature simulations corresponding to the predicted soil state parameters are integrated into the loss function of a standard fully connected neural network, enabling efficient convergence with limited sampling data in the training process. Testing results exhibit that the inversion performance of P-ANN is superior than that of conventional neural network approaches which only adopts errors in soil states in the loss function to train the network. Test also shows the proposed P-ANN approach outperforms traditional optimization algorithms in dealing with layered soil retrieval. In order to further improve the retrieval accuracy, an advanced local optimization scheme is also proposed, where the output from P-ANN is further treated as the initial value to a local optimization algorithm, achieving even closer results to the true values without excessive computational resources. In addition, to estimate the reliability of the model predictions, this paper also establishes a statistical relationship between the soil inversion error and the error of corresponding brightness temperatures from the testing process. When the trained neural network is in operation, the error of brightness temperature is calculated through the physical model, and the reliability of retrieval soil results is then acquired by putting the calculated brightness temperature errors into the pre-established statistical relationship. The proposed concepts and approaches have demonstrated the feasibility of using P-ANN model with multidimensional observations to invert the layered soil structure. The proposed approach holds great potentials for various remote sensing applications as well as a wide range of inverse problem challenges.