{"id":22096107,"url":"https://github.com/perimeterx/bot-tools","last_synced_at":"2025-07-18T10:08:44.895Z","repository":{"id":70060464,"uuid":"72262007","full_name":"PerimeterX/bot-tools","owner":"PerimeterX","description":null,"archived":false,"fork":false,"pushed_at":"2017-05-23T06:50:49.000Z","size":51,"stargazers_count":22,"open_issues_count":0,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-09T23:10:31.721Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PerimeterX.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-29T03:11:15.000Z","updated_at":"2023-12-17T15:49:49.000Z","dependencies_parsed_at":"2023-04-04T08:02:26.111Z","dependency_job_id":null,"html_url":"https://github.com/PerimeterX/bot-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PerimeterX/bot-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerimeterX%2Fbot-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerimeterX%2Fbot-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerimeterX%2Fbot-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerimeterX%2Fbot-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PerimeterX","download_url":"https://codeload.github.com/PerimeterX/bot-tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PerimeterX%2Fbot-tools/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265738554,"owners_count":23820166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-01T04:09:41.580Z","updated_at":"2025-07-18T10:08:44.873Z","avatar_url":"https://github.com/PerimeterX.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bot-Tools\n\nThis repository will host all the resources used and shared for the tutorial session in [O'Reilly Security NY](http://conferences.oreilly.com/security/network-data-security-ny/public/schedule/detail/52989) and [O'Reilly Security EU](http://conferences.oreilly.com/security/network-data-security-eu/public/schedule/detail/53016).\n\nThis readme file has the instructions and requirements on what should participants download and preinstall prior to the session.\nThe specific tools, test site and instructions are also included in this repository under the `Lab N` sections and will be covered in the session itself.\n\n## Please install the following prior to the session:\n### Required packages and tools\nPlease install the following packages and tools prior to the session:\n\n* Install Docker. You can download and find instructions on how to install docker for your OS here: https://www.docker.com/products/overview\n  * Docker for Windows requirements are 64bit Windows 10 Pro, and the Hyper-V package enabled. \n  * If you have a different windows version - see https://docs.docker.com/engine/installation/windows/ on how to install docker with Docker Toolbox.\n  * In case you can't install docker, you will need to install node.js for your OS. You can download node.js here https://nodejs.org/en/download/\n* Install python 2.7.\n  * Windows: download the latest python version at: https://www.python.org/downloads/\n  * Mac:\n    * Check if you already have python installed and its version by running `python -V` on the command line\n    * If you have brew - run `brew install python`\n    * or go to https://www.python.org/downloads/ to download python for MacOSX\n* Install Chrome\n* Install ModHeader Chrome extension - https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj\n\n### Docker images\nVerify that Docker is running, and from the command line install the following images:\n\n* Scrapy\n```bash\ndocker pull aciobanu/scrapy\n```\n* CasperJS\n```bash\ndocker pull zopanix/casperjs\n```\n* Selenium\n```bash\ndocker pull selenium/hub\ndocker pull selenium/node-chrome\n```\n\n### Node packages (if you didn't install docker)\n* Install PhantomJS: http://phantomjs.org/download.html\n* Install CasperJS:\n```bash\nnpm install casperjs\nnode_modules/casperjs/bin/casperjs selftest\n```\n\n# The following will be done and used during the session itself:\n## Usage\n\nClone this repository to your selected folder\n```\ngit clone https://github.com/PerimeterX/bot-tools.git\n```\n\n## Build Docker\nFrom the repository directory do the following:\n```bash\ncd demoApplication\ndocker build -t bot-lab-demo:latest .\n```\nto run application natively on your machine using python see instructions below\n\n## Run Docker\n`docker run -ti -p 5000:5000 bot-lab-demo`\n\n## Run application locally\ninstead of running it in a docker container, the application can run locally using python. To do that do the following:\n```bash\ncd demoApplication\npip install -r requirements.txt\npython app.py\n```\n\n## Docker commands\nIn case you aren't familiar with docker - here are a few commands that can help you get started, and should be sufficient for this tutorial:\n\nDownload an image, and all its parents, from the registry: \n```\ndocker pull \u003cimage name\u003e\n```\nStart and stop a container: \n```\ndocker start/stop \u003ccontainer name\u003e\n```\nTo check the running containers execute: \n```\ndocker ps\n```\n\nTo show last 50 lines and follow the log output of the container execute: \n```\ndocker logs --tail=50 -f \u003ccontainer name\u003e\n```\nTo check the running and stopped containers execute: \n```\ndocker ps -a\n```\nTo show the list of the images run: \n```\ndocker images\n```\nTo run a command inside the container namespace you run the following docker command: \n```\ndocker exec -ti \u003ccontainer name\u003e \u003ccommand\u003e\n```\nThis is very useful when you want to connect to a running container and work inside of the container. \n```\ndocker exec -ti \u003ccontainer name\u003e bash\n```\n\n## Notes and instructions for the session\nEach lab folder contains a README.md containing a short description, instructions, and any supporting files required.\n\n## [Lab 1](https://github.com/PerimeterX/bot-tools/tree/master/lab1) - Checking User Agent for Bad Bots\n\n## [Lab 2](https://github.com/PerimeterX/bot-tools/tree/master/lab2) - Blocking Users from Cloud Hosting and Malicious IP Addresses\n\n## [Lab 3](https://github.com/PerimeterX/bot-tools/tree/master/lab3) - Rate Limiting\n\n## [Lab 4](https://github.com/PerimeterX/bot-tools/tree/master/lab4) - Catching Scrapers with Hidden Links\n\n## [Lab 5](https://github.com/PerimeterX/bot-tools/tree/master/lab5) - Identifying Good Crawlers\n\n## [Lab 6](https://github.com/PerimeterX/bot-tools/tree/master/lab6) - Challenging with Javascript\n\n## [Lab 7](https://github.com/PerimeterX/bot-tools/tree/master/lab7) - Checking User Agent Take 2 (Navigator)\n\n## [Lab 8](https://github.com/PerimeterX/bot-tools/tree/master/lab8) - Finding Traces of PhantomJS\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperimeterx%2Fbot-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperimeterx%2Fbot-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperimeterx%2Fbot-tools/lists"}