Internal erosion is one of the most common causes of failure in hydraulic engineering structures, such as embankments and levees. It also plays a vital role in the geohazards (such as landslides and sinkhole developments) and more importantly, the earth landscape evolution, which has a broad environmental and ecosystem impacts. The groundwater seepage is multi-directional, and its multi-dimensional nature could affect the initiation and the progression of internal erosion. With a newly developed apparatus, we carry out nine internal erosion experiments under five different seepage directions. The results reveal that the critical hydraulic gradient increases as the seepage direction varies from the horizontal to the vertical. After a global erosion is triggered, preferential erosion paths distribute randomly from the bottom to the top of the specimen. If the seepage direction is not vertical, small preferential erosion paths merge into a large erosion corridor, in which the loss of fine particles is significant but negligible outside. Results of experiments manifest that the erosion is heterogeneous and three-dimensional, even in the unidirectional seepage flow. The particles are rapidly eroded at the early stage of the erosion, indicating a high erosion rate. With the erosion time increasing, the particle loss slows down and even ceases if the time is long enough. The erosion rate increases if the seepage direction approaches a vertical direction. Overall, the erosion rate approximately decreases with erosion time exponentially. We proposed exponential equations to illustrate the variation of the erosion rate in the erosion process.