There are a few issues to consider with latency and NAudio.
First is simply the performance of the NAudio code. I have tried to write it with performance in mind, but it has never been through a round of serious performance tuning (too many other priorities unfortunately), so there is undoubtedly scope for improvement. One big annoyance with .NET is that there is no way to cast from a byte[] to a float[] or short[] without using either unsafe code or doing a manual buffer copy. Obviously the latter is slower, but the former results in code that can't be run in a sandboxed environment. There is a mixture of techniques in the existing NAudio codebase.
Second is the wide variety of different performance of the various audio APIs. With WaveOut you will be lucky to go below 50ms, but Wasapi and AsioOut can often go lower.
Third is the way latency is measured. For the purposes of playback with two buffers (one being filled while the other is played), this is often reported as the size of one buffer. But there can be complications, such as WaveOut with more than two buffers, or DirectSoundOut which uses a circular buffer technique.
Finally, there is garbage collection itself. The trouble with .NET garbage collection is that when it runs it stops everything, on every thread. The ideal scenario is for the garbage collector not to run while you are playing audio, and I have tried to write the code so as not to create any new objects during playback (buffers are reused etc). However, it is likely that if you have an application with a GUI, the garbage collector will be triggered. If it takes too long, you can be left without enough time to fill the next buffer, or even miss a buffer. Hopefully MS have improved the GC enough for it to not cause noticable glitches, but I suspect at the sort of low latencies required in applications like DAWs (lower than 20ms), you would get the occasional glitch.
Anyway, will be interested to hear your findings.