I noticed that when you have a VirtualBox guest, no matter what kind of OS, making a lot of connection to the Internet, the incoming connection using NAT port forwarding will became very slow. In my case the inbound SSH connection froze after a few minutes and the HTTP connection has a very huge latency. The load on both guest and host will be low as usual, so no bottleneck could be found.
It seems like a VirtualBox problem because, as said before, it happens with different guests and hosts.
The workaround is very simple: just use two NATs! Use the first for the many outbound connections (configure it as guest’s default gateway), the second for the few inbound connection (configure port forwarding for this one).