I found the source code to the control and saw what my problem is. The control is expecting 16 bit data and I was feeding it 8 bit data. I used WaveFormatConversionStream plotStream = new WaveFormatConversionStream(new WaveFormat(8000, 16, 1), rawStream) to do the conversion and now it plots correctly. I also saw the control is hard coded to use a black pen so I will probably modify the control to use the forecolor.
I am very pleased with NAudio, it does everything I need, just wish there was more documentation available for newbies like myself.