Recent Posts – Page 2 – Mark's Website

M 10th January 201824th June 2019

Remotely Initiated Reverse SSH Tunnel

This article lays out the steps to set up an SSH tunnel[1] so that you can connect to a server hidden behind a firewall/router. Cron is used on the server to maintain a connection from the hidden remote server to my local server, that I can use to connect to the remote server.

Problem

To highlight what this setup does lets pretend we have the following scenario; Mr. Bond wants to connect to a server running on Dr. No’s private island:

Figure 1 – Blocked SSH Connection

When Mr. Bond tries to connect, he hits the firewall put up by Dr. No and can’t see the SSH Server [Figure 1].

Goal

In order to get pass this and connect any time Mr. Bond is in the mood, he needs to sneak onto Dr. No’s island and set up an SSH tunnel back to MI6 [Figure 2].

Figure 2 – SSH Tunnel Established

With this established, Mr. Bond can connect to the SSH tunnel open on a port in his local machine. This will then pass traffic back to Dr. No’s server [Figure 3].

Figure 3 – Connected through SSH Tunnel

Solution Setup

These instructions consist of:

Create an account for SSH to connect with
Set up key based authentication for the tunnel
Use cron / shell script to re-establish the connection if it drops.

With these steps, we make sure that Dr. No’s server is only able to open a port for SSH on the MI6 computer and doesn’t have access to run any commands on Mr. Bonds machine.

Using key based authentication means bots crawling the net wont be able to guess a password, but more vitally, will allow us to imitate the SSH tunnel from cron without having to pass through a password.

Cron will periodically run a shell script that opens the SSH tunnel (if there isn’t already one open). An alternative here could be to use autossh if that is available [7].

The command to connect from the remote server is fairly straight forward:

ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

Breakdown:

ssh : Here we are using OpenSSH. More details ‘man ssh’[2].
-R : Indicates that the next value is the connection to send to the remote server.
62222: This is the port that we want to set up on our local machine for the tunnel to appear on.
localhost: This is the loopback of the remote machine – could open a tunnel to a different machine on the remote network if we wanted to.
22: The port on the remote machine where ssh is running.
drno-ssh: This is the account we will connect to our local machine with.
mi6.example.com: This is the address/IP where SSH can connect to our local machine.
-N : Do not execute a remote command. This is useful for just forwarding ports (protocol version 2 only).

Connection Test

To start with, we can check that this works by running it on the remote server (substitute drno-ssh@mi6.example.com with your username and hostname):

drno@island$ ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

This should open the SSH tunnel. Now from your local server:

bond@mi6$ ssh localhost -p 62222

This should let you connect past the firewall. The next steps are to create a locked down user for the connection and to set up the automated connection.

User Setup

In this scenario, we are going to create an account on the local (mi6) server for the remote server to log in with. We will name this user “drno-ssh”:

bond@mi6$ sudo adduser drno-ssh --system --shell=/bin/false
Adding system user `drno-ssh' (UID 111) ...
Adding new user `drno-ssh' (UID 111) with group `nogroup' ...
Creating home directory `/home/drno-ssh' ...
bond@mi6$ sudo passwd -l drno-ssh
passwd: password expiry information changed.

The first command adduser creates the account; the ‘–system’ option indicates that this is a ‘system account’ and the ‘–shell=/bin/false’ prevents commands from being run by preventing a shell from loading after a login[3].

After this, the ‘passwd’ command is used with the ‘-l’ option to disable the accounts password.

We can check this worked by running the flollowing commands:

bond@mi6$ less /etc/passwd | grep drno-ssh
drno-ssh:x:111:65534::/home/drno-ssh:/bin/false
bond@mi6$ sudo less /etc/shadow | grep drno-ssh
drno-ssh:!*:17540:0:99999:7:::

The result of the first command shows us that the shell is set to ‘/bin/false’ and the password for the account is managed in the shadow file (the ‘x’)[4].

In the result from the second command you can see the row in the shadow file. The shadow file stores user passwords and the key thing that we want to see is the ‘!*’. This means that the users password is disabled[5].

As an additional check, we can try using root permission to jump into the account. If we try the following, we shouldn’t get anywhere and be bounced back to our current user:

bond@mi6$ sudo su drno-ssh
bond@mi6$

Setup Key-Based Authentication

These steps store the SSH key in the home directory of the user we created above. It’s possible to use directives in ‘ssh_config’ to avoid having a home directory as well as using a user other than root for the connection but I am not covering these here. By setting up key-based authentication instead of a password, we are able to create the ssh tunnel form the remote server without having to set a password for the account we are connecting to on our local machine. This is a bit more secure as well as allowing us to create the ssh connection from a cron job easily.

Create a folder to store the private key on your local server:

bond@mi6$ sudo mkdir -p /home/drno-ssh/.ssh
bond@mi6$ sudo chmod 0777 /home/drno-ssh/.ssh/

Important: don’t do this if you already have a private key you are using for root as it will destroy the existing key:

The next step is to generate a key pair on the remote server (press return to accept each default).

drno@island$ sudo ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
9b:57:26:d0:cb:0e:a0:2a:c6:b1:a5:d0:c6:1e:a5:58 root@mi6
The key's randomart image is:
+---[RSA 2048]----+
|                 |
|         .       |
|  E . . . .      |
| = o . . o .     |
|o.*..   S + o    |
|oo=o     = +     |
|.=o     o o      |
|..       .       |
|                 |
+-----------------+

After this, we need to send the public key that was generated to the remote server. This will recognise our private key when we try to connect and let us in[6].

drno@island$ sudo cat /root/.ssh/id_rsa.pub | ssh drno@island "cat > /home/drno-ssh/.ssh/authorized_keys"

On your local server, we will update the public SSH key to include some options limiting what the connection is able to do. In this case, only open port 62222 on the localhost. Its also possible to limit the connecting IP here by adding something like from=”1.2.3.4″.

bond@mi6$ echo -n "command=\"echo 'This account can only be used to create an SSH tunnel'\",no-agent-forwarding,no-X11-forwarding,permitopen=\"localhost:62222\" " > /home/drno-ssh/.ssh/ssh_options && cat /home/drno-ssh/.ssh/authorized_keys >> /home/drno-ssh/.ssh/ssh_options && mv /home/drno-ssh/.ssh/ssh_options /home/drno-ssh/.ssh/authorized_keys

Tidy up the file permissions and restart:

bond@mi6$ sudo chmod 700 /home/drno-ssh/.ssh && sudo chmod 640 /home/drno-ssh/.ssh/authorized_keys
bond@mi6$ sudo chown drno-ssh:nogroup -R /home/drno-ssh/.ssh/
bond@mi6$ sudo systemctl restart ssh.service

Cron Connection

With the key based login setup, we should be able to open a connection from the remote machine with the following (password may be required for sudo but should be needed for drno-ssh@mi6.example.com):

drno@island$ sudo ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N

Now from the local machine:

bond@mi6$ ssh localhost -p 62222
The authenticity of host '[localhost]:62222 ([::1]:62222)' can't be established.
ECDSA key fingerprint is SHA256:A7lSIQz/J9N2l7kvlKxz4QxxxF+vi8uhx2lvxdxxQ.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:62222' (ECDSA) to the list of known hosts.
bond@islands's password:
** Unauthorised use of this computer  **
** Will result in being fed to sharks **
Last login: Wed Jan 10 15:51:13 2018

The next step is to set up a cron job to keep this connection alive. Here is a shell script to do this. I will check for a running process containing our SSH username before opening the connection:
Create this by running the following on the remote server:

drno@island$ sudo nano /home/drno-ssh/ssh_tunnel.sh

Paste in the following script (after updating the ‘drno-ssh’s and mi6.example.com):

#!/bin/bash
# Set up an SSH Tunnel if it isn't running
PROCESS_NUM=$(ps -ef | grep "drno-ssh" | grep -v "grep" | wc -l)
if [ "$PROCESS_NUM" == "0" ];
then
 ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N
fi

Press CTRL+x to exit and ‘Y’ to save.
Make the script executable:

drno@island$ sudo chmod +x /home/drno-ssh/ssh_tunnel.sh

Set the script to run hourly using cron:

drno@island$ sudo crontab -e

Add the following line:

@hourly /home/drno-ssh/ssh_tunnel.sh #Set up an SSH Tunnel to MI6 every hour if it isn't running.

Checking that it works

With the cron job above enabled, the tunnel will be checked / initiated every hour on the hour. We can check if the tunnel is active with the following command on the remote server:

drno@island$ ps -ef | grep "drno-ssh"
root     18945 18917  0 21:00 ?        00:00:00 ssh -R 62222:localhost:22 drno-ssh@mi6.example.com -N
bond     22081 20036  0 23:15 pts/5    00:00:00 grep --color=auto drno-ssh

And on the local server with:

bond@mi6$ ssh localhost -p 62222
bond@islands's password:

This will let us log in with one of our normal SSH users over the SSH tunnel.

Mission Accomplished!

References

(last checked 2018-01-10)

M 6th January 201813th December 2019

Hide OpenSSH Version Banner

Here is a quick and dirty method for hiding some of the banner information advertised by OpenSSH[1]. This article focuses on how to use hexedit[2] to update your sshd binary to reduce information leakage.

Hiding the version information from OpenSSH is not supported by configuration[3]. This may have been added in some versions[4] and could be achieved by compiling your own version from source or switching to port knocking – these approaches are not covered. Be aware that this is only a temporary change to obfuscate what you are running and the following change will revert whenever OpenSSH is updated.

Problem

SSH Servers such as OpenSSH advertise information such as protocol support, build version and host operating system. For example:

$ nc example.local 22
 SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u2

Would be attackers scan the internet[5] and create large databases to search for servers running software with known vulnerabilities[6]. While this may have somewhat legitimate uses such as research into market share of SSH servers or for system admins monitoring their network for machines requiring patches; its generally not a good idea to give away such information freely to the Internet.

You can also use telnet or nmap to snoop on the same information e.g:

$ nmap -A -T4 -p 22 example.local

Starting Nmap 7.60 ( https://nmap.org ) 
Nmap scan report for example.local (1.2.3.4)
Host is up (0.0044s latency).
rDNS record for 1.2.3.4: 1-2-3-4.kram.nz

PORT STATE SERVICE VERSION
22/tcp open ssh (protocol 2.0)
| fingerprint-strings:
| NULL:
|_ SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u2

$ telnet example 22
Trying ::1...
Connected to example.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u2

Removing the Banner

Warning: Doing this while connected via SSH is risky as you can lock yourself out. If you are, set up a way to recover the original binary (i.e. set up another way to connect to the machine or a cron job to restore a copy of the original).

These steps use the hexedit[2] tool which is light weight and should be in your package manager. The goal is to write over the existing version string with text of your choice. The instructions are written for Debian / Systemd from steps followed on a Raspberri Pi running Raspbian.

Check the current banner:
```
$ echo "Hello" | nc localhost 22
SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u2
Protocol mismatch.
```
In this case, the part of the banner we want to hide is “OpenSSH_7.4p1 Raspbian-10+deb9u2” which is broadcasting the versions of my SSH server and operating system. Hiding the protocol is a bit harder and not covered here.
We can see this in the binary too:
```
$ strings /usr/sbin/sshd | grep Rasp
OpenSSH_7.4p1 Raspbian-10+deb9u2
```
Escalate to a root session:
```
$ sudo su
```

Install hexedit:

# apt-get update && apt-get install hexedit

Back up your sshd binary and create an editable working copy (as root):

# cp /usr/sbin/sshd /tmp/sshd.backup
# cp /tmp/sshd.backup /tmp/sshd.new

Update the binary with hexedit:
```
# hexedit /tmp/sshd.new
```
Press TAB to switch from the HEX are to the ASCII area
Use CTRL+S to bring up the search prompt and search for the text in your banner than you want to hide e.g. ‘OpenSSH_7.4’. You should see something like:
```
0007DA54   61 67 65 6E  74 00 00 00  4F 70 65 6E  agent...Open
0007DA60   53 53 48 5F  37 2E 34 70  31 20 52 61  SSH_7.4p1 Ra
0007DA6C   73 70 62 69  61 6E 2D 31  30 2B 64 65  spbian-10+de
0007DA78   62 39 75 32  00 00 00 00  4F 70 65 6E  b9u2....Open
```
Use the arrow keys to highlight the start of the string that you want to update and type your replacement. Be careful to stay within the bounds of the length of the original banner. You can also press TAB to switch back to the HEX area if you wanted to just null out the string setting each word to ’00’.
Your change should look something like:
```
0007DA54   61 67 65 6E  74 00 00 00  48 65 72 65  agent...Here
0007DA60   20 62 65 20  64 72 61 67  6F 6E 73 2E   be dragons.
0007DA6C   20 54 75 72  6E 20 42 61  63 6B 00 00   Turn Back..
0007DA78   00 00 00 00  00 00 00 00  4F 70 65 6E  ........Open
```
Save your changes with CTRL+x and a Y.
Check if there are any instances that we missed (we expect no output now):
```
# strings /tmp/sshd.new | grep Rasp
```

Update sshd and restart the service for good measure:

# rm /usr/sbin/sshd
# cp /tmp/sshd.new /usr/sbin/sshd
# systemctl restart ssh.service

Check that you can still SSH in (otherwise restore the backup or reinstall OpenSSH from your package manager!):
```
# ssh user@localhost
```

This change will only be temporary as any time you update OpenSSH, the binary will be replaced.

Result

The process above will result in something like the following:

$ nc localhost 22
SSH-2.0-Here be dragons. Turn Back

Where before this would be along the lines of:

$ nc localhost 22 
SSH-2.0-OpenSSH_7.4p1 Raspbian-10+deb9u2

Which makes the version you are running just a little bit more obscure and and also adds bit of fun.

References

(last checked 2018-01-06)

M 30th October 201113th January 2018

Gerbil Time Tracker

Hamster is a cool little Linux tool that I have recently started to use and its made my time tracking a breeze.

In the past, at the end of each week, I would have to try and remember how much time I spent doing different activities at work, so that I could record my time and bill the correct customers. This was a painful chore and sometimes required a bit of email archaeology, Skype log trawling and black-magic. Hamster allowed me to track my current activity with easy integration to my desktop and proactively record time instead of using retroactive sorcery.

I love the tool so much I thought I’d create my own PHP web based version which would allow me to use it from my phone and create all sorts of graphs. I have begun development on my tool and it is already usable. When I have a comparable set of features to Hamster, I plan to migrate my data across and open it up for other people to use!

Mark Gardiner

Solution Architect

Remotely Initiated Reverse SSH Tunnel

Problem

Goal

Solution Setup

Connection Test

User Setup

Setup Key-Based Authentication

Cron Connection

Checking that it works

References

Hide OpenSSH Version Banner

Problem

Removing the Banner

Result

References

Gerbil Time Tracker