How can I maintain a constant volume during the simulation period?
 

Boundary Conditions:
Xmin:wave
Xmax: outflow
front ,back & bottom:wall
top: outflow

How can I maintain a constant volume during the simulation period?

Not easy. The degree of conservation depends on mesh resolution, VOF scheme used and velocity advection. Split Lagrangian VOF method may work better.

To add to MuxaB's response, you can also make the domain long enough in the direction of wave propagation that the transit time required for three waves to reach the downstream outflow boundary is sufficient to find the results you need. The wave-propagation mass conservation issue at the outflow boundary usually takes three full wave fronts to start to significantly appear.