If you find that connecting via SSH is slow you might want to disable DNS lookup done by the SSH Server to the Client by disabling reverse DNS. I have noticed this behaviour with Ubuntu 13 and 14.

echo '# Disable reverse DNS lookup to prevent slow login' | sudo tee -a /etc/ssh/sshd_config
echo 'UseDNS no' | sudo tee -a /etc/ssh/sshd_config
 
# I don't think you need to restart the ssh service (will update this comment once I have to do it again)

According t the I had to do this website, the reason is due to time-out when doing reverse DNS lookup. This worked to resolve slow connection to my Ubuntu 14.04 server. Strangely, connecting from my same client machine had not problems with Ubuntu 12.04. So I am not included to believe the reason. If somebody figures this out please share or add additional observations.

In terms of security audit loss I think it is reasonable to turn this off.

References

Discusses reasoning - http://askubuntu.com/questions/246323/why-does-sshs-password-prompt-take-so-long-to-appear