The study of modeling and optimization methods of the highway soil slope stability has been carried out based on the reliability theory in this paper, because of the extensive uncertainties of the cost and safety factors in highway slope design. First, considering the overall construction cost and risk cost, using the cost-benefit analysis method, taking the minimum total expected cost as the objective function, using the slope-type and slope-ratio as design variables and the reliability index of the slope stability as the main constraint conditions, and then the economical risk analysis model of the soil slope is established. Considering high nonlinearity of the model, genetic algorithm is used to iteratively solve and the optimum program in slope engineering design is developed based on the platform of MATLAB. The program was used to design slope engineering in some highway and the optimum slope-type and slope-ratios were obtained. The result shows that the program has good stability, high computational speed, high precision and practical value.