Remotely Initiated Reverse SSH Tunnel

This article lays out the steps to set up an SSH tunnel[1] so that you can connect to a server hidden behind a firewall/router. Cron is used on the server to maintain a connection from the hidden remote server to my local server, that I can use to connect to the remote server.

Problem

To highlight what this setup does lets pretend we have the following scenario; Mr. Bond wants to connect to a server running on Dr. No’s private island:

Figure 1 – Blocked SSH Connection

When Mr. Bond tries to connect, he hits the firewall put up by Dr. No and can’t see the SSH Server [Figure 1].

Goal

In order to get pass this and connect any time Mr. Bond is in the mood, he needs to sneak onto Dr. No’s island and set up an SSH tunnel back to MI6 [Figure 2].

Figure 2 – SSH Tunnel Established

With this established, Mr. Bond can connect to the SSH tunnel open on a port in his local machine. This will then pass traffic back to Dr. No’s server [Figure 3].

Figure 3 – Connected through SSH Tunnel

Solution Setup

These instructions consist of:

Create an account for SSH to connect with
Set up key based authentication for the tunnel
Use cron / shell script to re-establish the connection if it drops.

With these steps, we make sure that Dr. No’s server is only able to open a port for SSH on the MI6 computer and doesn’t have access to run any commands on Mr. Bonds machine.

Using key based authentication means bots crawling the net wont be able to guess a password, but more vitally, will allow us to imitate the SSH tunnel from cron without having to pass through a password.

Cron will periodically run a shell script that opens the SSH tunnel (if there isn’t already one open). An alternative here could be to use autossh if that is available [7].

The command to connect from the remote server is fairly straight forward:

ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

Breakdown:

ssh : Here we are using OpenSSH. More details ‘man ssh’[2].
-R : Indicates that the next value is the connection to send to the remote server.
62222: This is the port that we want to set up on our local machine for the tunnel to appear on.
localhost: This is the loopback of the remote machine – could open a tunnel to a different machine on the remote network if we wanted to.
22: The port on the remote machine where ssh is running.
drno-ssh: This is the account we will connect to our local machine with.
mi6.example.com: This is the address/IP where SSH can connect to our local machine.
-N : Do not execute a remote command. This is useful for just forwarding ports (protocol version 2 only).

Connection Test

To start with, we can check that this works by running it on the remote server (substitute drno-ssh@mi6.example.com with your username and hostname):

drno@island$ ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

This should open the SSH tunnel. Now from your local server:

bond@mi6$ ssh localhost -p 62222

This should let you connect past the firewall. The next steps are to create a locked down user for the connection and to set up the automated connection.

User Setup

In this scenario, we are going to create an account on the local (mi6) server for the remote server to log in with. We will name this user “drno-ssh”:

bond@mi6$ sudo adduser drno-ssh --system --shell=/bin/false
Adding system user `drno-ssh' (UID 111) ...
Adding new user `drno-ssh' (UID 111) with group `nogroup' ...
Creating home directory `/home/drno-ssh' ...
bond@mi6$ sudo passwd -l drno-ssh
passwd: password expiry information changed.

The first command adduser creates the account; the ‘–system’ option indicates that this is a ‘system account’ and the ‘–shell=/bin/false’ prevents commands from being run by preventing a shell from loading after a login[3].

After this, the ‘passwd’ command is used with the ‘-l’ option to disable the accounts password.

We can check this worked by running the flollowing commands:

bond@mi6$ less /etc/passwd | grep drno-ssh
drno-ssh:x:111:65534::/home/drno-ssh:/bin/false
bond@mi6$ sudo less /etc/shadow | grep drno-ssh
drno-ssh:!*:17540:0:99999:7:::

The result of the first command shows us that the shell is set to ‘/bin/false’ and the password for the account is managed in the shadow file (the ‘x’)[4].

In the result from the second command you can see the row in the shadow file. The shadow file stores user passwords and the key thing that we want to see is the ‘!*’. This means that the users password is disabled[5].

As an additional check, we can try using root permission to jump into the account. If we try the following, we shouldn’t get anywhere and be bounced back to our current user:

bond@mi6$ sudo su drno-ssh
bond@mi6$

Setup Key-Based Authentication

These steps store the SSH key in the home directory of the user we created above. It’s possible to use directives in ‘ssh_config’ to avoid having a home directory as well as using a user other than root for the connection but I am not covering these here. By setting up key-based authentication instead of a password, we are able to create the ssh tunnel form the remote server without having to set a password for the account we are connecting to on our local machine. This is a bit more secure as well as allowing us to create the ssh connection from a cron job easily.

Create a folder to store the private key on your local server:

bond@mi6$ sudo mkdir -p /home/drno-ssh/.ssh
bond@mi6$ sudo chmod 0777 /home/drno-ssh/.ssh/

Important: don’t do this if you already have a private key you are using for root as it will destroy the existing key:

The next step is to generate a key pair on the remote server (press return to accept each default).

drno@island$ sudo ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
9b:57:26:d0:cb:0e:a0:2a:c6:b1:a5:d0:c6:1e:a5:58 root@mi6
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|         .       |
|  E . . . .      |
| = o . . o .     |
|o.*..   S + o    |
|oo=o     = +     |
|.=o     o o      |
|..       .       |
|                 |
+-----------------+

After this, we need to send the public key that was generated to the remote server. This will recognise our private key when we try to connect and let us in[6].

drno@island$ sudo cat /root/.ssh/id_rsa.pub | ssh drno@island "cat > /home/drno-ssh/.ssh/authorized_keys"

On your local server, we will update the public SSH key to include some options limiting what the connection is able to do. In this case, only open port 62222 on the localhost. Its also possible to limit the connecting IP here by adding something like from=”1.2.3.4″.

bond@mi6$ echo -n "command=\"echo 'This account can only be used to create an SSH tunnel'\",no-agent-forwarding,no-X11-forwarding,permitopen=\"localhost:62222\" " > /home/drno-ssh/.ssh/ssh_options && cat /home/drno-ssh/.ssh/authorized_keys >> /home/drno-ssh/.ssh/ssh_options && mv /home/drno-ssh/.ssh/ssh_options /home/drno-ssh/.ssh/authorized_keys

Tidy up the file permissions and restart:

bond@mi6$ sudo chmod 700 /home/drno-ssh/.ssh && sudo chmod 640 /home/drno-ssh/.ssh/authorized_keys
bond@mi6$ sudo chown drno-ssh:nogroup -R /home/drno-ssh/.ssh/
bond@mi6$ sudo systemctl restart ssh.service

Cron Connection

With the key based login setup, we should be able to open a connection from the remote machine with the following (password may be required for sudo but should be needed for drno-ssh@mi6.example.com):

drno@island$ sudo ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

Now from the local machine:

bond@mi6$ ssh localhost -p 62222
The authenticity of host '[localhost]:62222 ([::1]:62222)' can't be established.
ECDSA key fingerprint is SHA256:A7lSIQz/J9N2l7kvlKxz4QxxxF+vi8uhx2lvxdxxQ.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:62222' (ECDSA) to the list of known hosts.
bond@islands's password:
** Unauthorised use of this computer  **
** Will result in being fed to sharks **
Last login: Wed Jan 10 15:51:13 2018

The next step is to set up a cron job to keep this connection alive. Here is a shell script to do this. I will check for a running process containing our SSH username before opening the connection:
Create this by running the following on the remote server:

drno@island$ sudo nano /home/drno-ssh/ssh_tunnel.sh

Paste in the following script (after updating the ‘drno-ssh’s and mi6.example.com):

#!/bin/bash
# Set up an SSH Tunnel if it isn't running
PROCESS_NUM=$(ps -ef | grep "drno-ssh" | grep -v "grep" | wc -l)
if [ "$PROCESS_NUM" == "0" ];
then
 ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N
fi

Press CTRL+x to exit and ‘Y’ to save.
Make the script executable:

drno@island$ sudo chmod +x /home/drno-ssh/ssh_tunnel.sh

Set the script to run hourly using cron:

drno@island$ sudo crontab -e

Add the following line:

@hourly /home/drno-ssh/ssh_tunnel.sh #Set up an SSH Tunnel to MI6 every hour if it isn't running.

Checking that it works

With the cron job above enabled, the tunnel will be checked / initiated every hour on the hour. We can check if the tunnel is active with the following command on the remote server:

drno@island$ ps -ef | grep "drno-ssh"
root     18945 18917  0 21:00 ?        00:00:00 ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N
bond     22081 20036  0 23:15 pts/5    00:00:00 grep --color=auto drno-ssh

And on the local server with:

bond@mi6$ ssh localhost -p 62222
bond@islands's password:

This will let us log in with one of our normal SSH users over the SSH tunnel.

Mission Accomplished!

References

(last checked 2018-01-10)

Mark Gardiner

Solution Architect