...

According t the I had to do this website, the reason is due to time-out when doing reverse DNS lookup. This worked to resolve slow connection to my Ubuntu 14.04 server. Strangely, connecting from my same client machine had not problems with Ubuntu 12.04. So I am not included to believe the reason. If somebody figures this out document hereplease share or add additional observations.

References

Discusses reasoning - http://askubuntu.com/questions/246323/why-does-sshs-password-prompt-take-so-long-to-appear