# Fun with bots – SSH tarpitting – hackanooga

June 24, 2024

# Fun with bots – SSH tarpitting

For those of you who aren’t familiar with the concept of a network tarpit it is a fairly simple concept. Wikipedia defines it like this:

> A tarpit is a service on a computer system (usually a server) that purposely delays incoming connections. The technique was developed as a defense against a computer worm, and the idea is that network abuses such as spamming or broad scanning are less effective, and therefore less attractive, if they take too long. The concept is analogous with a tar pit, in which animals can get bogged down and slowly sink under the surface, like in a swamp.

> https://en.wikipedia.org/wiki/Tarpit_(networking)

If you run any sort of service on the internet then you know as soon as your server has a public IP address and open ports, there are scanners and bots trying to get in constantly. If you take decent steps towards security then it is little more than an annoyance, but annoying all the less. One day when I had some extra time on my hands I started researching ways to mess with the bots trying to scan/attack my site.

It turns out that this problem has been solved multiple times in multiple ways. One of the most popular tools for tarpitting ssh connections is endlessh. The way it works is actually pretty simple. The SSH RFC states that when an SSH connection is established, both sides MUST send an identification string. Further down the spec is the line that allows this behavior:

>        The server MAY send other lines of data before sending the version
>        string.  Each line SHOULD be terminated by a Carriage Return and Line
>        Feed.  Such lines MUST NOT begin with "SSH-", and SHOULD be encoded
>        in ISO-10646 UTF-8 [RFC3629] (language is not specified).  Clients
>        MUST be able to process such lines.  Such lines MAY be silently
>        ignored, or MAY be displayed to the client user.  If they are
>        displayed, control character filtering, as discussed in [SSH-ARCH],
>        SHOULD be used.  The primary use of this feature is to allow TCP-
>        wrappers to display an error message before disconnecting.

> SSH RFC

Essentially this means that their is no limit to the amount of data that a server can send back to the client and the client must be able to wait and process all of this data. Now let’s see it in action.

    git clone https://github.com/skeeto/endlessh.git
    cd endlessh
    make
    ./endlessh &

By default this fake server listens on port 2222. I have a port forward set up that forwards all ssh traffic from port 22 to 2222. Now try to connect via ssh:

    ssh -vvv localhost -p 2222

If you wait a few seconds you will see the server send back the version string and then start sending a random banner:

    $:/tmp/endlessh$ 2024-06-24T13:05:59.488Z Port 2222
    2024-06-24T13:05:59.488Z Delay 10000
    2024-06-24T13:05:59.488Z MaxLineLength 32
    2024-06-24T13:05:59.488Z MaxClients 4096
    2024-06-24T13:05:59.488Z BindFamily IPv4 Mapped IPv6
    2024-06-24T13:05:59.488Z socket() = 3
    2024-06-24T13:05:59.488Z setsockopt(3, SO_REUSEADDR, true) = 0
    2024-06-24T13:05:59.488Z setsockopt(3, IPV6_V6ONLY, true) = 0
    2024-06-24T13:05:59.488Z bind(3, port=2222) = 0
    2024-06-24T13:05:59.488Z listen(3) = 0
    2024-06-24T13:05:59.488Z poll(1, -1)
    ssh -vvv localhost -p 2222
    OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022
    debug1: Reading configuration data /home/mikeconrad/.ssh/config
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
    debug1: /etc/ssh/ssh_config line 21: Applying options for *
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/mikeconrad/.ssh/known_hosts'
    debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/mikeconrad/.ssh/known_hosts2'
    debug2: resolving "localhost" port 2222
    debug3: resolve_host: lookup localhost:2222
    debug3: ssh_connect_direct: entering
    debug1: Connecting to localhost [::1] port 2222.
    debug3: set_sock_tos: set socket 3 IPV6_TCLASS 0x10
    debug1: Connection established.
    2024-06-24T13:06:08.635Z = 1
    2024-06-24T13:06:08.635Z accept() = 4
    2024-06-24T13:06:08.635Z setsockopt(4, SO_RCVBUF, 1) = 0
    2024-06-24T13:06:08.635Z ACCEPT host=::1 port=43696 fd=4 n=1/4096
    2024-06-24T13:06:08.635Z poll(1, 10000)
    debug1: identity file /home/mikeconrad/.ssh/id_rsa type 0
    debug1: identity file /home/mikeconrad/.ssh/id_rsa-cert type 4
    debug1: identity file /home/mikeconrad/.ssh/id_ecdsa type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ecdsa-cert type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ecdsa_sk type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ecdsa_sk-cert type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ed25519 type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ed25519-cert type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ed25519_sk type -1
    debug1: identity file /home/mikeconrad/.ssh/id_ed25519_sk-cert type -1
    debug1: identity file /home/mikeconrad/.ssh/id_xmss type -1
    debug1: identity file /home/mikeconrad/.ssh/id_xmss-cert type -1
    debug1: identity file /home/mikeconrad/.ssh/id_dsa type -1
    debug1: identity file /home/mikeconrad/.ssh/id_dsa-cert type -1
    debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.7
    2024-06-24T13:06:18.684Z = 0
    2024-06-24T13:06:18.684Z write(4) = 3
    2024-06-24T13:06:18.684Z poll(1, 10000)
    debug1: kex_exchange_identification: banner line 0: V
    2024-06-24T13:06:28.734Z = 0
    2024-06-24T13:06:28.734Z write(4) = 25
    2024-06-24T13:06:28.734Z poll(1, 10000)
    debug1: kex_exchange_identification: banner line 1: 2I=ED}PZ,z T_Y|Yc]$b{R]

This is a great way to give back to those bots and script kiddies. In my research into other methods I also stumbled across this brilliant program fakessh. While fakessh isn’t technically a tarpit, it’s more of a honeypot but very interesting nonetheless. It creates a fake SSH server and logs the ip address, connection string and any commands executed by the attacker. Essentially it allows any username/password combination to connect and gives them a fake shell prompt. There is no actual access to any file system and all of their commands basically return gibberish.

Here are some logs from an actual server of mine running fakessh

    2024/06/24 06:51:20 [conn] ip=183.81.169.238:40430
    2024/06/24 06:51:22 [auth] ip=183.81.169.238:40430 version="SSH-2.0-Go" user="root" password="0"
    2024/06/24 06:51:23 [conn] ip=183.81.169.238:40444
    2024/06/24 06:51:25 [auth] ip=183.81.169.238:40444 version="SSH-2.0-Go" user="root" password="eve"
    2024/06/24 06:51:26 [conn] ip=183.81.169.238:48408
    2024/06/24 06:51:27 [auth] ip=183.81.169.238:48408 version="SSH-2.0-Go" user="root" password="root"
    2024/06/24 06:51:28 [conn] ip=183.81.169.238:48434
    2024/06/24 06:51:30 [auth] ip=183.81.169.238:48434 version="SSH-2.0-Go" user="root" password="1"
    2024/06/24 06:51:30 [conn] ip=183.81.169.238:48448
    2024/06/24 06:51:32 [auth] ip=183.81.169.238:48448 version="SSH-2.0-Go" user="root" password="123"
    2024/06/24 06:51:32 [conn] ip=183.81.169.238:48476
    2024/06/24 06:51:35 [auth] ip=183.81.169.238:48476 version="SSH-2.0-Go" user="root" password="admin"
    2024/06/24 06:51:35 [conn] ip=183.81.169.238:39250
    2024/06/24 06:51:37 [auth] ip=183.81.169.238:39250 version="SSH-2.0-Go" user="root" password="123456"
    2024/06/24 06:51:38 [conn] ip=183.81.169.238:39276
    2024/06/24 06:51:40 [auth] ip=183.81.169.238:39276 version="SSH-2.0-Go" user="root" password="123123"
    2024/06/24 06:51:40 [conn] ip=183.81.169.238:39294
    2024/06/24 06:51:42 [auth] ip=183.81.169.238:39294 version="SSH-2.0-Go" user="root" password="test"
    2024/06/24 06:51:43 [conn] ip=183.81.169.238:39316
    2024/06/24 06:51:45 [auth] ip=183.81.169.238:39316 version="SSH-2.0-Go" user="root" password="123456789"
    2024/06/24 06:51:45 [conn] ip=183.81.169.238:35108
    2024/06/24 06:51:47 [auth] ip=183.81.169.238:35108 version="SSH-2.0-Go" user="root" password="12345"
    2024/06/24 06:51:48 [conn] ip=183.81.169.238:35114
    2024/06/24 06:51:50 [auth] ip=183.81.169.238:35114 version="SSH-2.0-Go" user="root" password="password"
    2024/06/24 06:51:50 [conn] ip=183.81.169.238:35130
    2024/06/24 06:51:52 [auth] ip=183.81.169.238:35130 version="SSH-2.0-Go" user="root" password="12345678"
    2024/06/24 06:51:52 [conn] ip=183.81.169.238:35146
    2024/06/24 06:51:54 [auth] ip=183.81.169.238:35146 version="SSH-2.0-Go" user="root" password="111111"
    2024/06/24 06:51:55 [conn] ip=183.81.169.238:58490
    2024/06/24 06:51:57 [auth] ip=183.81.169.238:58490 version="SSH-2.0-Go" user="root" password="1234567890"
    2024/06/24 06:51:57 [conn] ip=183.81.169.238:58528
    2024/06/24 06:51:59 [auth] ip=183.81.169.238:58528 version="SSH-2.0-Go" user="root" password="1234"
    2024/06/24 06:52:00 [conn] ip=183.81.169.238:58572
    2024/06/24 06:52:02 [auth] ip=183.81.169.238:58572 version="SSH-2.0-Go" user="root" password="password123"
    2024/06/24 06:52:02 [conn] ip=183.81.169.238:58588
    2024/06/24 06:52:04 [auth] ip=183.81.169.238:58588 version="SSH-2.0-Go" user="root" password="ubuntu"
    2024/06/24 06:52:05 [conn] ip=183.81.169.238:37198
    2024/06/24 06:52:07 [auth] ip=183.81.169.238:37198 version="SSH-2.0-Go" user="Antminer" password="root"
    2024/06/24 06:52:07 [conn] ip=183.81.169.238:37214
    2024/06/24 06:52:09 [auth] ip=183.81.169.238:37214 version="SSH-2.0-Go" user="Antminer" password="admin"
    2024/06/24 06:52:10 [conn] ip=183.81.169.238:37238
    2024/06/24 06:52:11 [auth] ip=183.81.169.238:37238 version="SSH-2.0-Go" user="root" password="innot1t2"
    2024/06/24 06:52:12 [conn] ip=183.81.169.238:37258
    2024/06/24 06:52:14 [auth] ip=183.81.169.238:37258 version="SSH-2.0-Go" user="root" password="t1t2t3a5"
    2024/06/24 06:52:14 [conn] ip=183.81.169.238:55658
    2024/06/24 06:52:16 [auth] ip=183.81.169.238:55658 version="SSH-2.0-Go" user="root" password="blacksheepwall"
    2024/06/24 06:52:17 [conn] ip=183.81.169.238:55670
    2024/06/24 06:52:19 [auth] ip=183.81.169.238:55670 version="SSH-2.0-Go" user="root" password="envision"
    2024/06/24 06:52:19 [conn] ip=183.81.169.238:55708
    2024/06/24 06:52:21 [auth] ip=183.81.169.238:55708 version="SSH-2.0-Go" user="root" password="bwcon"
    2024/06/24 06:52:22 [conn] ip=183.81.169.238:55776
    2024/06/24 06:52:23 [auth] ip=183.81.169.238:55776 version="SSH-2.0-Go" user="admin" password="root"
    2024/06/24 06:52:24 [conn] ip=183.81.169.238:46646
    2024/06/24 06:52:26 [auth] ip=183.81.169.238:46646 version="SSH-2.0-Go" user="baikal" password="baikal"
    2024/06/24 06:52:26 [conn] ip=180.101.88.197:44620
    2024/06/24 06:52:27 [conn] ip=180.101.88.197:44620 err="ssh: disconnect, reason 11: "
    2024/06/24 06:53:35 [conn] ip=218.92.0.76:50610
    2024/06/24 06:53:36 [conn] ip=218.92.0.76:50610 err="ssh: disconnect, reason 11: "
    2024/06/24 07:02:28 [conn] ip=218.92.0.27:64676
    2024/06/24 07:02:30 [conn] ip=218.92.0.27:64676 err="ssh: disconnect, reason 11: "
    2024/06/24 07:10:05 [conn] ip=218.92.0.76:57601
    2024/06/24 07:10:07 [conn] ip=218.92.0.76:57601 err="ssh: disconnect, reason 11: "
    2024/06/24 07:14:05 [conn] ip=193.201.9.156:63056
    2024/06/24 07:14:05 [auth] ip=193.201.9.156:63056 version="SSH-2.0-Go" user="ubnt" password="ubnt"
    2024/06/24 07:14:05 [conn] ip=193.201.9.156:63056 err="read tcp 10.10.10.107:2222->193.201.9.156:63056: read: connection reset by peer"
    2024/06/24 07:24:53 [conn] ip=218.92.0.31:25485
    2024/06/24 07:24:54 [conn] ip=218.92.0.31:25485 err="ssh: disconnect, reason 11: "
    2024/06/24 07:24:54 [conn] ip=218.92.0.112:39270
    2024/06/24 07:24:56 [conn] ip=218.92.0.112:39270 err="ssh: disconnect, reason 11: "
    2024/06/24 07:26:42 [conn] ip=218.92.0.34:59993
    2024/06/24 07:35:46 [conn] ip=218.92.0.34:59993 err="read tcp 10.10.10.107:2222->218.92.0.34:59993: read: connection reset by peer"
    2024/06/24 07:41:28 [conn] ip=218.92.0.107:62285
    2024/06/24 07:41:31 [conn] ip=218.92.0.107:62285 err="ssh: disconnect, reason 11: "
    2024/06/24 07:43:27 [conn] ip=218.92.0.29:34556
    2024/06/24 07:43:28 [conn] ip=218.92.0.29:34556 err="ssh: disconnect, reason 11: "
    2024/06/24 07:44:15 [conn] ip=218.92.0.118:37047
    2024/06/24 07:44:22 [conn] ip=218.92.0.118:37047 err="ssh: disconnect, reason 11: "
    2024/06/24 07:56:10 [conn] ip=157.245.98.245:6116
    2024/06/24 07:56:11 [conn] ip=157.245.98.245:6116 err="ssh: unexpected message type 20 (expected 21)"
    2024/06/24 07:57:57 [conn] ip=218.92.0.112:28326
    2024/06/24 07:57:58 [conn] ip=218.92.0.112:28326 err="ssh: disconnect, reason 11: "
    2024/06/24 08:00:01 [conn] ip=218.92.0.24:24948
    2024/06/24 08:00:02 [conn] ip=218.92.0.24:24948 err="ssh: disconnect, reason 11: "
    2024/06/24 08:06:19 [conn] ip=193.201.9.156:46865
    2024/06/24 08:06:20 [auth] ip=193.201.9.156:46865 version="SSH-2.0-Go" user="root" password="xc3511"
    2024/06/24 08:06:20 [conn] ip=193.201.9.156:46865 err="read tcp 10.10.10.107:2222->193.201.9.156:46865: read: connection reset by peer"
    2024/06/24 08:14:26 [conn] ip=180.101.88.197:48347
    2024/06/24 08:14:28 [conn] ip=180.101.88.197:48347 err="ssh: disconnect, reason 11: "
    2024/06/24 08:16:28 [conn] ip=218.92.0.56:18064
    2024/06/24 08:16:32 [conn] ip=218.92.0.56:18064 err="ssh: disconnect, reason 11: "
    2024/06/24 08:30:55 [conn] ip=180.101.88.196:40495
    2024/06/24 08:30:57 [conn] ip=180.101.88.196:40495 err="ssh: disconnect, reason 11: "
    2024/06/24 08:32:20 [conn] ip=85.209.11.227:15493
    2024/06/24 08:32:21 [auth] ip=85.209.11.227:15493 version="SSH-2.0-Go" user="telecomadmin" password="admintelecom"
    2024/06/24 08:32:21 [conn] ip=85.209.11.227:15493 err="read tcp 10.10.10.107:2222->85.209.11.227:15493: read: connection reset by peer"
    2024/06/24 08:33:19 [conn] ip=218.92.0.34:59804
    2024/06/24 08:33:21 [conn] ip=218.92.0.34:59804 err="ssh: disconnect, reason 11: "
    2024/06/24 08:41:00 [conn] ip=218.92.0.27:45567
    2024/06/24 08:41:02 [conn] ip=218.92.0.27:45567 err="ssh: disconnect, reason 11: "
    2024/06/24 08:47:15 [conn] ip=180.101.88.196:17032
    2024/06/24 08:47:16 [conn] ip=180.101.88.196:17032 err="ssh: disconnect, reason 11: "
    2024/06/24 08:49:51 [conn] ip=218.92.0.29:26360
    2024/06/24 08:49:57 [conn] ip=218.92.0.29:26360 err="ssh: disconnect, reason 11: "
    2024/06/24 08:58:27 [conn] ip=193.201.9.156:49525
    2024/06/24 08:58:28 [auth] ip=193.201.9.156:49525 version="SSH-2.0-Go" user="admin" password="1234"
    2024/06/24 08:58:28 [conn] ip=193.201.9.156:49525 err="read tcp 10.10.10.107:2222->193.201.9.156:49525: read: connection reset by peer"
    2024/06/24 08:58:44 [conn] ip=218.92.0.31:11835
    2024/06/24 08:58:46 [conn] ip=218.92.0.31:11835 err="ssh: disconnect, reason 11: "
    2024/06/24 09:03:38 [conn] ip=218.92.0.107:57758
    2024/06/24 09:03:40 [conn] ip=218.92.0.107:57758 err="ssh: disconnect, reason 11: "
    2024/06/24 09:07:36 [conn] ip=218.92.0.56:21354
    2024/06/24 09:07:39 [conn] ip=218.92.0.56:21354 err="ssh: disconnect, reason 11: "

Those are mostly connections and disconnections. They probably connected, realized it was fake and disconnected. There are a couple that tried to execute some commands though:

    :~$ sudo grep head /var/log/fakessh/fakessh.log
    2024/06/23 15:48:02 [shell] ip=184.160.233.163:45735 duration=0s bytes=15 head="ls 2>/dev/null\n"
    2024/06/24 03:55:11 [shell] ip=14.46.116.243:43656 duration=20s bytes=0 head=""

Fun fact: Cloudflare’s Bot Fight Mode uses a form of tarpitting:

> Once enabled, when we detect a bad bot, we will do three things: (1) we’re going to disincentivize the bot maker economically by tarpitting them, including requiring them to solve a computationally intensive challenge that will require more of their bot’s CPU; (2) for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; and (3) we’re going to plant trees to make up for the bot’s carbon cost.

> https://blog.cloudflare.com/cleaning-up-bad-bots

In

Cloudflare , Networking , Open Source , Security , SSH
Blog Post