Abstract: In order to simulate the free surface flow problem,a 3D Navier-Stokes equation based on the tri-prism grid is established,which has the hydrodynamic pressure model and the semi-implicit numerical discrete format. At the outflow boundary,we add the linear attenuation terms of sponge layer processing to the right-hand of the momentum equations to reduce the wave energy.We use the k-ε turbulent equation to solve the eddy viscosity coefficient in order to close the model. The numerical tests include the simulation of regular period wave propagation over a bar and the compound open channel situation. While tackling the short-wave high frequency problem,we found that the results are in good agreement with the analytic solutions or the measured values,which shows the accuracy and validity of this model.