If you find that connecting via SSH is slow you might want to disable DNS lookup done by the SSH Server to the Client by modifying the /etc/ssh/sshd_config and adding "UseDNS no" to the bottom of the file,

echo '# Disable reverse DNS lookup to prevent slow login' | sudo tee -a /etc/ssh/sshd_config
echo 'UseDNS no' | sudo tee -a /etc/ssh/sshd_config

According t the I had to do this website, the reason is due to time-out when doing reverse DNS lookup. This worked to resolve slow connection to my Ubuntu 14.04 server. Strangely, connecting from my same client machine had not problems with Ubuntu 12.04. So I am not included to believe the reason. If somebody figures this out please share or add additional observations.

References

Discusses reasoning - http://askubuntu.com/questions/246323/why-does-sshs-password-prompt-take-so-long-to-appear