If you find that connecting via SSH is slow you might want to disable DNS lookup done by the SSH Server to the Client by disabling reverse DNS. I have noticed this behaviour with Ubuntu 13 and 14.
According to Ask Ubuntu, the reason is due to time-out when doing reverse DNS lookup. This worked to resolve slow connection to one of my Ubuntu 14.04 servers. However, connecting from same client machine to Ubuntu 12.04 had no issues.
In terms of security audit loss I think it is reasonable to turn this off.