New Post: Convert WaveStream to double-array (or apply FFT directly)

I've answered your question on StackOverflow. Also, unless you are using a really old NAudio, no need for the WaveFormatConversionStream. Simply reading out of the Mp3FileReader will give you PCM. It is most likely 16 bit stereo, so you'd want to split out the channels before you do an FFT.

WaveBuffer saves you using BitConverter. Simply bind a WaveBuffer to your byte array containing PCM samples, and then you can access each sample using the ShortBuffer property. But your FFT probably wants float/double anyway, so you still need to divide by 32,768 and put it into your separate left and right floating point arrays.