When I use Visual Select, it brings up my select screen at the same size I have it set at...yet when I run MUGEN, for some reason, everything looks terrible. My MUGEN looks very pixelated compared to the Visual Select screen. It's probably always been that way, but I never really noticed it until I had something much sharper to juxtapose it with.
It's hard to speculate without screenshots. I suspect you're running Mugen at a low resolution. If you're using WinMugen, that has a native resolution of 320x240. You can change the screen dimensions but this doesn't change the resolution, it just means the sprites will be scaled. VSelect on the other hand runs at your desktop resolution, which is presumably much higher. So, say you're running WinMugen at 640x480: when WinMugen displays the select screen it will render it at 320x240, scaled by 2. VSelect will render it at 640x480. In effect, it will be twice as detailed.
I could make VSelect mimic the way Mugen scales the sprites, and so more accurately reflect how the select screen looks in-game, but to be honest I don't want to
It would amount to more work for lower quality results.