Setting up Two Factor Auth for SSH

One of my colleagues got me interested in setting up 2FA on my VPS, so finally got around to it, and seems to be working very well.

Here are the steps I used to get it setup:

apt-get install libpam-google-authenticator

In /etc/pam.d/sshd I have changed/added the following lines (at the top):

# @include common-auth    
auth required pam_google_authenticator.so

And in /etc/ssh/sshd_config:

ChallengeResponseAuthentication yes    
UsePAM yes    
AuthenticationMethods publickey,keyboard-interactive
PasswordAuthentication no

You will then need to the authenticator setup:

google-authenticator

This will then create a QR code for you to scan and generate 5 scratch codes that you can use to login if you lost or are without your phone.

Works well and I now receive a "Verification code" prompt after authentication with a public key. I am not sure how I would allow authentication with password+token OR key+token, as I have now effectively removed the password authentication method from PAM.