When you add your pubkey to an openssh server’s config in ~/.ssh/authorized_keys permissions are important or it will ignore your login attempt.

Lock down .ssh directory

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Lock down home directory

chmod g-w,o-w /home/USERNAME

Should let you back in.