Hot Mix Asphalt (HMA) is a composite material that consists of mineral aggregates, asphalt binders and air voids. A Discrete Element Model (DEM) of the HMA microstructure was developed to study the stiffness behavior in both two dimensions (2D) and three dimensions (3D). Image analysis techniques were used to capture the HMA microstructure. The HMA microstructure was divided into two phases: aggregates phase (i.e., aggregates larger than 1.18mm sieve) and mastic phase (i.e., with binder and aggregates smaller than 1.18 mm). Air voids were modeled within the DEM to meet a specific air void level (i.e. 0%, 4%, and 7%) by using a random algorithm. The 3D microstructure of the asphalt mixture was obtained by using a number of parallel 2D images. The input data on the models included not only the aggregate and mastic properties, but also the microstructure of the aggregate skeleton and mastic distribution. Both 2D and 3D models were used to compute the stress-strain response under compressive loads. The moduli of the specimens were computed from the stress-strain curve in the DEM simulation. The moduli of the 2D and 3D models were then compared with the experimental measurements. It was found that the 3D discrete element models were able to predict the mixture dynamic modulus across a range of temperatures and loading frequencies. The 3D model predictions were much better than that of the 2D model. In addition, the effect of different air void percentages was discussed in this paper. As air void increased, the predicted modulus decreased.