Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/assafmo/xioc
Extract indicators of compromise from text, including "escaped" ones.
https://github.com/assafmo/xioc
command-line command-line-tool data-mining defang escaping extract extraction indicators-of-compromise ioc iocs regex regexp text-mining text-processing
Last synced: 13 days ago
JSON representation
Extract indicators of compromise from text, including "escaped" ones.
- Host: GitHub
- URL: https://github.com/assafmo/xioc
- Owner: assafmo
- License: mit
- Created: 2019-01-16T18:38:35.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2020-04-19T17:42:40.000Z (over 4 years ago)
- Last Synced: 2024-08-02T18:40:02.303Z (3 months ago)
- Topics: command-line, command-line-tool, data-mining, defang, escaping, extract, extraction, indicators-of-compromise, ioc, iocs, regex, regexp, text-mining, text-processing
- Language: Go
- Homepage:
- Size: 64.5 KB
- Stars: 162
- Watchers: 10
- Forks: 13
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# xioc
Extract indicators of compromise from text, including "escaped" ones like `hxxp://banana.com`, `1.1.1[.]1` and `phish at malicious dot com`.
[](https://circleci.com/gh/assafmo/xioc)
[](https://coveralls.io/github/assafmo/xioc?branch=master)
[](https://goreportcard.com/report/github.com/assafmo/xioc)
[](https://godoc.org/github.com/assafmo/xioc/xioc)
## Installation
- Download a precompiled binary from https://github.com/assafmo/xioc/releases
- Or... Use `go get`:
```bash
go get -u github.com/assafmo/xioc
```
- Or... Use snap install (Ubuntu):
```bash
snap install xioc
```
- Or use Ubuntu PPA:
```bash
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add -
sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list
sudo apt update
sudo apt install xioc
```
## Features
- Extract IOCs (indicators of compromise) from an input text:
- IPv4
- IPv6
- Domain
- URL
- Email
- MD5
- SHA1
- SHA256
- Translate some kinds of "escaping"/"defanging" techniques:
- `(dot)`, `[dot]`, `(.)`, `[.]`, `{.}` to `.`.
- `(at)`, `[at]`, `(@)`, `[@]`, `{@}` to `@`.
- `hxxp`, `hzzzp`, `hxxxp`, `hXXp`, `h__p`, `h**p` to `http`.
- Command line interface
- Go library
## Command line usage
```bash
$ xioc -h
Usage of xioc:
-o string
Extract only specified types.
Types must be comma seperated. E.g: xioc -o "ip4,domain,url,md5"
Available types:
- ip4
- ip6
- domain
- url
- email
- md5
- sha1
- sha256
-v Print version and exit
```
```bash
$ REPORT="https://unit42.paloaltonetworks.com/digital-quartermaster-scenario-demonstrated-in-attacks-against-the-mongolian-government/"
$ lynx -dump "$REPORT" | xioc
sha256 5beb50d95c1e720143ca0004f5172cb8881d75f6c9f434ceaff59f34fa1fe378
domain energy.gov.mn
email [email protected]
sha256 10090692ff40758a08bd66f806e0f2c831b4b9742bbf3d19c250e778de638f57
# ...
```
```bash
$ REPORT="https://unit42.paloaltonetworks.com/digital-quartermaster-scenario-demonstrated-in-attacks-against-the-mongolian-government/"
$ lynx -dump "$REPORT" | xioc -o email,sha256
sha256 5beb50d95c1e720143ca0004f5172cb8881d75f6c9f434ceaff59f34fa1fe378
email [email protected]
sha256 10090692ff40758a08bd66f806e0f2c831b4b9742bbf3d19c250e778de638f57
email [email protected]
# ...
```
## Library usage
Full API:
[](https://godoc.org/github.com/assafmo/xioc/xioc)
```golang
package main
import (
"fmt"
"github.com/assafmo/xioc/xioc"
)
func main() {
input := `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
banana.com
hxxp://i.robot.com/robots.txt
1.2.3.4
1.1.1[.]1
info at gmail dot com
hxxps://m.twitter[dot]com/`
fmt.Println(xioc.ExtractDomains(input)) // => [i.robot.com m.twitter.com gmail.com banana.com]
fmt.Println(xioc.ExtractSHA256s(input)) // => [e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]
fmt.Println(xioc.ExtractMD5s(input)) // => []
fmt.Println(xioc.ExtractIPv4s(input)) // => [1.2.3.4 1.1.1.1]
fmt.Println(xioc.ExtractURLs(input)) // => [http://i.robot.com/robots.txt https://m.twitter.com/]
fmt.Println(xioc.ExtractEmails(input)) // => [[email protected]]
}
```
## Sources
- Test email address: http://codefool.tumblr.com/post/15288874550/list-of-valid-and-invalid-email-addresses
- Domains can start with a number: https://serverfault.com/a/638270
- IPv6 Examples: http://www.gestioip.net/docu/ipv6_address_examples.html
- Fang and defang IOCs: https://github.com/ioc-fang/ioc_fanger
- Indicator of Compromise (De)Fanging Project: https://ioc-fang.hightower.space/
- InQuest/python-iocextract test data: https://github.com/InQuest/python-iocextract/tree/master/test_data
- Email address can be treated as case-insensitive: https://stackoverflow.com/a/9808332