Whilst trying to deploy a new Django project from a private Github repo using Ansible (see my post series), I hit quite a few snags all related to how SSH keys are loaded, but the error messages didn’t always give a clear indication of the problem. Here’s are four issues I encountered which might give you a hint on where to start debugging your own errors.

 

Rookie Error 1. Not all VPS are created equal (especially AWS – Root user login is disabled!)

One of the reasons I like to use Anisble is that I’m not tied into one provider and I can easily switch to a different one if there is a pricing change or other reason. But I discovered there are some differences which can be a bit irritating. On other VPS, its possible to SSH in as a root user, but for this project I was testing AWS Lightsail, and immediately running the Ansible playbook get the error:

Please login as the user \"ubuntu\" rather than the user \"root\"

Ok, thats probably best security practice, but I’m using secure SSH keys, and I want my setup to work this way! There were no details of how to change this but fortunately a bit of searching turns up that this message is given in the authorized keys file on the remote server

To enable root login, SSH into the server and remove the following test from `/root/.ssh/authorized_keys` – but keep the SSH key that follows it:

no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"ubuntu\" rather than the user \"root\".';echo;sleep 10"

That fixed, the first part of the Ansible playbook ran as normal….

 

Rookie Error 2. Github access via SSH uses a different address format

… until trying to git pull of the private Github repo on the remote server, which just hung silently with no error messages. I added the correct config in Ansible to forward SSH keys, and the SSH keys were defeintely loaded, so this was a strange error. But with this being the first time I’ve accessed Github over SSH, I just stuck the HTTP address in the Ansible config, but, duh, it doesn’t work – and it’s actually obvious there’s a different one if you look at the Github web interface. You need to use: `git@github.com:github_username/repo_name.git`, not: `https://github.com/github_username/repo_name.git` when using SSH access. Not easy to work this out when there are no error messages though…

 

 

Rookie Error 3. Github repos must have unique SSH keys, AND only the key for the repo you are accessing should be loaded.

Correct SSH Github address entered, and running the playbook again, the git pull of the private repo from Github doesn’t hang but now I get an SSH public key error from Github – now this is more than a rookie error!

Knowing my co-worker won’t want to find his SSH keys every time, I had decided to put my .zshrc skills into practice and have it load all the SSH keys whenever a new shell is loaded, so I know the correct key was loaded. Mistake!

Despite knowing the correct ssh key was loaded, I unloaded all ssh keys:

ssh-add -D

Added the single one back temporarily:

ssh-add ssh_keys/another_ssh_key

Checked it was loaded with:

ssh-add -l

This time it works – but why? Well, the other keys I had loaded in my original .zshrc configuration included and SSH key for a different Github repo, so I think what could be happening is that it tries to authenticate with one key and finds the user has permission but not for that repo, so it fails. If I only have one SSH key loaded for the specific repo I’m using, it seems to work (even with a second different SSH key which is for the remote server only). Could be an issue if you are pulling from multiple private repos each needing its own SSH key though!

To debug the most helpful command was testing the SSH login to Github to tell you which user and repo you are authenticated for:

SSH -T git@github.com

For convenience I decided to add an alias command to ~/.zshrc to easily remove previous keys and load the specific key for each project:

  alias myproject-ssh='
  ssh-add -D
  ssh-add ~/my_project/ssh_keys/my_ssh_key

So that’s it, all working nicely, until …

 

Rookie Error 4. SSH key forwarding gets disabled if the remote host identification changes (e.g. rebuilding a server from scratch)

… rebuilding a server. The beauty of Ansible is that you can mess things up, start a new server from scratch and run your playbook again. On AWS Lightsail you have to delete and make a brand new server with a new IP address, but for other VPS providers (I use Hetzner and Vultr at the moment), you can rebuild the server with the same IP address. After doing this, the dreaded `Permission denied (publickey)` error is back when authenticating to the private Github repo, although at least it was obvious this time what change had cause the issue. SSHing directly into the remote host gave the following warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:??????????????????????????????????????.
Please contact your system administrator.
Add correct host key in /Users/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/username/.ssh/known_hosts:4
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Agent forwarding is disabled to avoid man-in-the-middle attacks.
UpdateHostkeys is disabled because the host key is not trusted.
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-88-generic x86_64)

So, just exiting back to my local machine and `rm Users/username/.ssh/known_hosts` fixed this one. Job done.

 

Conclusion

Hopefully I’m less of a rookie with SSH authentication now, and it’s made me realise I need to write good error messages!

Now I have a new error – why is the port I’ve specifically opened in UFW not accessible… ah, AWS Lightsail has a default firewall in the web front end. Back to Step 1. Not All VPS Are Created Equal ….