{"id":26236911,"url":"https://github.com/estebanpdl/tik-spyder","last_synced_at":"2025-10-24T18:08:34.327Z","repository":{"id":250065376,"uuid":"829583009","full_name":"estebanpdl/tik-spyder","owner":"estebanpdl","description":"A Python command-line tool designed to collect TikTok data using SerpAPI for Google search results and Apify for TikTok data extraction.","archived":false,"fork":false,"pushed_at":"2025-09-03T17:13:04.000Z","size":257,"stargazers_count":55,"open_issues_count":0,"forks_count":11,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-09T15:11:54.015Z","etag":null,"topics":["command-line-tool","open-source","osint","osint-python","osint-tool","python","tiktok"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/estebanpdl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-07-16T18:17:08.000Z","updated_at":"2025-10-04T05:40:57.000Z","dependencies_parsed_at":"2024-08-02T21:26:58.392Z","dependency_job_id":"b629e7f1-3d33-4b42-b1a0-e43dc614fd45","html_url":"https://github.com/estebanpdl/tik-spyder","commit_stats":null,"previous_names":["estebanpdl/tikspyder","estebanpdl/tik-spyder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/estebanpdl/tik-spyder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estebanpdl%2Ftik-spyder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estebanpdl%2Ftik-spyder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estebanpdl%2Ftik-spyder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estebanpdl%2Ftik-spyder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/estebanpdl","download_url":"https://codeload.github.com/estebanpdl/tik-spyder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/estebanpdl%2Ftik-spyder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280841755,"owners_count":26400499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","open-source","osint","osint-python","osint-tool","python","tiktok"],"created_at":"2025-03-13T04:19:57.278Z","updated_at":"2025-10-24T18:08:34.297Z","avatar_url":"https://github.com/estebanpdl.png","language":"Python","funding_links":["https://buymeacoffee.com/estebanpdl"],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# **TikSpyder**\n\n\u003c/div\u003e\n\n\u003cbr /\u003e\n\n`TikSpyder` is a command-line tool designed to collect TikTok data using SerpAPI for Google search results and Apify for TikTok data extraction. The tool supports video downloading via yt-dlp and uses Python's asynchronous capabilities and multithreading for efficient data collection.\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![GitHub forks](https://img.shields.io/github/forks/estebanpdl/tik-spyder.svg?style=social\u0026label=Fork\u0026maxAge=2592000)](https://GitHub.com/estebanpdl/tik-spyder/network/)\n[![GitHub stars](https://img.shields.io/github/stars/estebanpdl/tik-spyder?style=social)](https://github.com/estebanpdl/tik-spyder/stargazers)\n[![Open Source](https://badges.frapsoft.com/os/v1/open-source.svg?v=103)](https://x.com/estebanpdl)\n[![Made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)\n[![Twitter estebanpdl](https://badgen.net/badge/icon/twitter?icon=twitter\u0026label)](https://x.com/estebanpdl)\n[![Buy Me A Coffee](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg)](https://buymeacoffee.com/estebanpdl)\n\n\u003c/div\u003e\n\n\u003chr /\u003e\n\n## 🔧 **Companion Tools**\n\n| Tool | Description | Access |\n|------|-------------|--------|\n| 🎙️ Audio Transcription | Transcribe audio files from TikTok videos | [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1qMcMsS2YI9btXGfFN1sCviQeB7RSKqUH) |\n\n\u003cbr /\u003e\n\n## 🖥️ **User Interface Options**\n\nTikSpyder provides **two ways** to interact with the tool:\n\n### **1. 🎨 Streamlit Web Interface (Recommended for Non-Technical Users)**\nA modern, user-friendly web interface with TikTok-inspired dark theme that makes data collection accessible to everyone.\n\n![TikSpyder Streamlit Interface](images/streamlit-interface.png)\n\n**Features:**\n- 🎯 **Intuitive Configuration**: Easy search setup with tabbed interface for keywords, users, or hashtags\n- 📅 **Visual Date Filters**: Calendar widgets for precise date range selection  \n- 🚀 **Apify Integration**: Simple toggle to enable enhanced data collection\n- ⚙️ **Advanced Options**: Collapsible section for Google search parameters\n- 📥 **Download Settings**: Visual controls for video downloads and Tor network usage\n- 📂 **File Browser**: Point-and-click directory selection\n- 📊 **Real-time Progress**: Live progress tracking with step-by-step status updates\n\n**Launch the Interface:**\n\n**Method 1 (Recommended):**\n```sh\n# Using package installation\ntikspyder --app\n\n# Using standard installation  \npython main.py --app\n```\n\n**Method 2 (Direct):**\n```sh\nstreamlit run app.py\n```\n\n### **2. ⌨️ Command Line Interface (For Advanced Users)**\nFull-featured command-line tool for automation and scripting scenarios.\n\n## 🔍 **Description**\n\nTikSpyder offers two main methods of data collection:\n1. **Google Search Results**: Using SerpAPI to find TikTok videos based on search queries\n2. **Apify Data collection**: Using Apify to collect videos directly from TikTok profiles or keywords\n\nThe tool supports various filtering options, including date ranges and content types, and can download both videos and thumbnails. Data is stored in a SQLite database and can be exported to CSV files for further analysis.\n\nGiven the dynamic nature of search results and the constantly evolving landscape of TikTok's platform, it's important to note that the data collected by TikSpyder represents a sample rather than a comprehensive dataset. However, this sample can still be valuable for monitoring trends and identifying emerging narratives in the information ecosystem.\n\nTo get the most out of TikSpyder, **it is recommended to test your query using Google's advanced search features. This can help refine your search query, improve the relevance of your results, and test specific keywords more effectively**. By taking advantage of these features, you can ensure that you're collecting the most relevant data for your research or analysis.\n\n\u003cbr /\u003e\n\n## 🚀 **Features**\n\n### **Core Functionality**\n- 🔍 Collects TikTok video links using SerpAPI and Apify\n- 🖼️ Collects and downloads thumbnails for TikTok videos  \n- 🔗 Collects related content to the search query\n- 💾 Stores collected data in SQLite database\n- 📊 Exports data to CSV files for analysis\n- 📹 Downloads TikTok videos using yt-dlp\n- 🎞️ Extracts keyframes from downloaded videos\n- ⚡ Supports asynchronous and multithreaded downloading for improved performance\n- 🔒 Supports Tor network for enhanced privacy and rate limiting avoidance\n\n### **User Interfaces**\n- 🎨 **Modern Streamlit Web Interface**: User-friendly GUI with TikTok-inspired dark theme\n- ⌨️ **Command Line Interface**: Full-featured CLI for automation and advanced users\n- 🎯 **Search Types**: Support for keywords, user profiles, and hashtag searches\n- 📅 **Date Range Filtering**: Precise temporal data collection controls\n\n\u003cbr /\u003e\n\n## ⚙️ **Requirements**\n\n### **System Requirements**\n- [Python](https://www.python.org/) \u003e= 3.11.7\n- [ffmpeg](https://ffmpeg.org/) (for video processing and keyframe extraction)\n\n### **API Keys \u0026 Services**  \n- [SerpAPI key](https://serpapi.com/) (required for Google search functionality)\n- [Apify API token](https://apify.com/) (optional, for direct TikTok profile scraping)\n\n### **Optional Components**\n- [Tor Browser](https://www.torproject.org/) (optional, for enhanced privacy during downloads)\n\n### **Platform-Specific Requirements**\n- **All Platforms**: Python libraries listed in `requirements.txt`\n- **Streamlit Interface**: Automatically installed with requirements\n- **Linux Users**: For GUI components, install tkinter: `sudo apt-get install python3-tk` (Ubuntu/Debian)\n\n\u003cbr /\u003e\n\n## 🔧 **Installation**\n\n### **Method 1: Standard Installation**\n\n1. Clone the repository\n\n```sh\ngit clone https://github.com/estebanpdl/tik-spyder.git\ncd tik-spyder\n```\n\n2. Install the required packages\n\n```sh\npip install -r requirements.txt\n```\n\nor\n\n```sh\npip3 install -r requirements.txt\n```\n\n### **Method 2: Package Installation (Recommended)**\n\nThis method installs TikSpyder as a package, making the `tikspyder` command available from anywhere on your system.\n\n1. Clone the repository\n\n```sh\ngit clone https://github.com/estebanpdl/tik-spyder.git\ncd tik-spyder\n```\n\n2. Install the package in editable mode\n\n```sh\npip install -e .\n```\n\nor\n\n```sh\npip3 install -e .\n```\n\nAfter installation, you can use `tikspyder` directly from any directory instead of `python main.py`.\n\n### **Configuration**\n\n3. Once you obtain an API key from SerpAPI and Apify, populate the config/config.ini file with the described values. Replace `api_key_value` and `apify_token_value` with your API key and token.\n\n```ini\n\n[SerpAPI Key]\napi_key = your_serp_api_key\n\n[Apify Token]\napify_token = your_apify_token\n```\n\n\u003cbr /\u003e\n\n## 📚 **Usage**\n\nTikSpyder offers two interface options to suit different user preferences and use cases:\n\n## 🎨 **Streamlit Web Interface Usage**\n\nThe Streamlit interface provides an intuitive, visual way to configure and run data collection tasks.\n\n### **Launch the Interface**\n\n```sh\n# Navigate to TikSpyder directory\ncd tik-spyder\n\n# Launch the Streamlit app\nstreamlit run app.py\n```\n\nThe interface will automatically open in your default web browser at `http://localhost:8501`\n\n### **Using the Interface**\n\n1. **🎯 Configure Search**: Choose between keyword, user profile, or hashtag search\n2. **📅 Set Date Filters**: Use calendar widgets to define your collection timeframe  \n3. **🚀 Enable Apify** (Optional): Toggle for enhanced direct TikTok data collection\n4. **⚙️ Adjust Advanced Options**: Fine-tune Google search parameters if needed\n5. **📥 Configure Downloads**: Set video download preferences and worker counts\n6. **📂 Choose Output Directory**: Select where your data will be saved\n7. **🚀 Start Collection**: Click the centered \"Start Data Collection\" button\n\n---\n\n## ⌨️ **Command Line Interface Usage**\n\nFor advanced users and automation scenarios, TikSpyder provides a full-featured CLI.\n\n### **Using Package Installation (Method 2)**\n\n```sh\ntikspyder [OPTIONS]\n```\n\n### **Using Standard Installation (Method 1)**\n\n```sh\npython main.py [OPTIONS]\n```\n\n### **Command Line Arguments**\n\n```sh\n# Package installation\ntikspyder --help\n\n# or\ntikspyder -h\n\n# Standard installation\npython main.py --help\n\n# or\npython main.py -h\n```\n\n```\nCommand Line Arguments.\n\nHelp options:\n  -h, --help            Show this help message and exit.\n\nSerpAPI options:\n  --q                   The search term of phrase for which to retrieve TikTok data.\n  --user                Specify a TikTok user to search for videos from.\n  --tag                 Specify a TikTok tag to search for videos from.\n  --google-domain       Defines the Google domain to use. It defaults to google.com.\n  --gl                  Defines the country to use for the search. Two-letter country code.\n  --hl                  Defines the language to use for the search. Two-letter language code.\n  --cr                  Defines one or multiple countries to limit the search to.\n  --safe                Level of filtering for adult content. Options: active (default), off\n  --lr                  Defines one or multiple languages to limit the search to.\n  --depth               Depth of iterations to follow related content links.\n\nGoogle advanced search options:\n  --before              Limit results to posts published before the specified date. Format: YYYY-MM-DD.\n  --after               Limit results to posts published after the specified date. Format: YYYY-MM-DD.\n\nOptional Apify arguments:\n  --apify               Specify whether to use Apify integration.\n  --oldest-post-date    Filter posts newer than the specified date. Format: YYYY-MM-DD.\n  --newest-post-date    Filter posts older than the specified date. Format: YYYY-MM-DD.\n  --number-of-results   Specify the number of results to return from Apify. Default: 25\n\nOptional arguments and parameters:\n  --app                 Launch the Streamlit web interface instead of using CLI mode.\n  --use-tor             Specify whether to use Tor for downloading TikTok videos.\n  -d, --download        Specify whether to download TikTok videos from SerpAPI and Apify.\n  -w , --max-workers    Specify the maximum number of threads to use for downloading TikTok videos and extracting keyframes.\n  -o , --output         Specify output directory path. If not provided, data is saved in the current working directory in a folder named `tikspyder-data`\n```\n\n### **Example Usage**\n\n1. Search-based collection:\n\n```sh\n# Using package installation (Method 2)\ntikspyder --q \"F-16 AND Enemy AND (Ukraine OR Russia)\" --gl us --hl en --after 2024-02-01 --before 2024-05-31 --output {output_directory}/ --download\n\n# Using standard installation (Method 1)\npython main.py --q \"F-16 AND Enemy AND (Ukraine OR Russia)\" --gl us --hl en --after 2024-02-01 --before 2024-05-31 --output {output_directory}/ --download\n\n# Note: Replace '{output_directory}' with the desired output path.\n```\n\n2. Profile-based collection:\n\n```sh\n# Using package installation (Method 2)\ntikspyder --q Trump --user username --output {output_directory}/ --download --apify --oldest-post-date 2025-01-01\n\n# Using standard installation (Method 1)\npython main.py --q Trump --user username --output {output_directory}/ --download --apify --oldest-post-date 2025-01-01\n\n# Note: Replace '{output_directory}' with the desired output path.\n```\n\n3. Tag-based collection\n```sh\n# Using package installation (Method 2)\ntikspyder --tag sinaloa --apify --oldest-post-date 2025-08-01 --number-of-results 50 --output {output_directory}/ --download\n\n# Using standard installation (Method 1)\npython main.py --tag sinaloa --apify --oldest-post-date 2025-08-01 --number-of-results 50 --output {output_directory}/ --download\n\n# Note: Replace '{output_directory}' with the desired output path.\n```\n\n### Tor Integration\nYou can use Tor network for downloading TikTok videos to enhance privacy and avoid rate limiting. To use this feature:\n\n1. Make sure Tor Browser is installed and running\n2. Configure your torrc file with:\n\n```\n## Enable SOCKS proxy\nSocksPort 9050\n\n## Enable Control Port for IP rotation\nControlPort 9051\nCookieAuthentication 1\n```\n\n3. Use the `--use-tor` flag when running the script. If Tor connection fails, the script will automatically fall back to a normal connection.\n\n\n\u003cbr /\u003e\n\n## ☕ Support\n\nIf you find TikSpyder helpful, please consider buying me a coffee to support ongoing development and maintenance. Your donation will help me continue to improve the tool and add new features.\n\n[![Buy Me A Coffee](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow.svg?style=for-the-badge\u0026logo=buy-me-a-coffee\u0026logoColor=white)](https://buymeacoffee.com/estebanpdl)\n\n\u003cbr /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festebanpdl%2Ftik-spyder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Festebanpdl%2Ftik-spyder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Festebanpdl%2Ftik-spyder/lists"}