Thanks Mark!
I was able to convert the audio to 32 bit and mix it together and convert it back to 16 bit. The audio quality has improved greatly.
There is one issue when two people talk at the same time though. When both streams mix, you hear this horrible screeching sound.