the reason for mixing in 32 bit instead of 16 bit is headroom. If you mix in 16 bit, then you are adding together Int16s, which could overflow, causing distortion. However, when you go to 32 bit IEEE, you make 1.0 the max scale, so if you mix together and the total goes above 1.0, you can simply scale the overall output level back down before converting back into 16 bit to send to the speakers.
NAudio includes adapter classes to turn 16 bit audio into 32 bit (and vice versa if you need it)