.Net is plenty fast enough to do a project like that, *IF* you correctly optimize the code (using the profilers as guides).
That said, I would recommend only opening a single output stream; This uses less handles and allows you to mix on the fly in your own code. I'd also recommend that you make sure all sources use the same sample rate as the output stream (we'll assume all of them are 32-bit float internally).
No matter which language you use, this will be a large project... take your time and make sure you are always using the best algorithm for each piece of the system.
Good Luck!