{"id":21434394,"url":"https://github.com/serguun42/social-picker-api","last_synced_at":"2025-08-21T18:17:57.118Z","repository":{"id":128965887,"uuid":"450684419","full_name":"serguun42/Social-Picker-API","owner":"serguun42","description":"Extract media from various posting platforms like Twitter, Reddit, Pixiv, Youtube, Tiktok, Osnova and many others","archived":false,"fork":false,"pushed_at":"2024-02-15T19:26:03.000Z","size":139,"stargazers_count":20,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-02-15T20:32:56.200Z","etag":null,"topics":["coub","ffmpeg","instagram","joyreactor","osnova","parser","pixiv","reddit","tiktok","tumblr","twitter","ugoira","youtube-dl","yt-dlp"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/serguun42.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-01-22T01:05:46.000Z","updated_at":"2024-01-18T06:19:03.000Z","dependencies_parsed_at":"2023-07-08T14:01:05.727Z","dependency_job_id":"9b6d72c6-7622-4f5e-be43-07b989c909f0","html_url":"https://github.com/serguun42/Social-Picker-API","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serguun42%2FSocial-Picker-API","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serguun42%2FSocial-Picker-API/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serguun42%2FSocial-Picker-API/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/serguun42%2FSocial-Picker-API/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/serguun42","download_url":"https://codeload.github.com/serguun42/Social-Picker-API/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225980892,"owners_count":17554919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coub","ffmpeg","instagram","joyreactor","osnova","parser","pixiv","reddit","tiktok","tumblr","twitter","ugoira","youtube-dl","yt-dlp"],"created_at":"2024-11-22T23:35:37.872Z","updated_at":"2024-11-22T23:35:38.447Z","avatar_url":"https://github.com/serguun42.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Social Picker API\n\nExtract media from various posting platforms like Twitter, Reddit, Pixiv, Youtube, Tiktok, Osnova and many others. This project written in Typescript for Node.js and it works as REST service for other services, e.g. [Anime Ultra Bot](https://github.com/serguun42/Anime-Ultra-Bot) or [Social Picker Vue](https://github.com/serguun42/Social-Picker-Vue).\n\n## Usage\n\nStart a server, make HTTP-request to it with a link to the post and receive that post's content back.\n\n## Configs\n\nThere are some configuration files:\n\n- [`service.json`](./config/service.json) – service port, base URL of external service for viewing some and proxy for some platforms – see [_Platform specific/Proxies_](#proxies)\n- [`tokens.json`](./config/tokens.json) – tokens for platforms and other filepaths. See [_Platform specific_](#platform-specific) and [types with comments](./src/types/configs.d.ts)\n- [`pm2.production.json`](./config/pm2.production.json) – config for `pm2` daemon\n- Other configuration files for particular platforms – see [_Platform specific_](#platform-specific)\n\nDevelopment config files can be created and placed along production ones (e.g. `tokens.dev.json` or `service.dev.json`). You can also install all npm modules (including dev) one with `npm install`. `npm run dev` will run service in dev-environment, `npm run lint` will use `eslint`.\n\n## How to run\n\n1. Install necessary dependencies – `npm i --omit=dev`\n2. [_Install necessary binaries needed for some platforms_](#binaries)\n3. Compile Typescript – `npm run compile`\n4. Start server – `npm run production`\n\nAfter launching you can access Picker with fetching it like `curl http://localhost:8080/?url=__LINK_TO_ANY_POST__` (change `8080` to real port you specified in [`service.json`](./config/service.json)).\n\n#### Open access\n\nIf you're planning opening your Picker instance to the world, I'd suggest creating your own middleware service for handling user access, downloading combined videos, etc. ([something like that](https://social.serguun42.ru/docs/redoc.html)). After creating such service set `CUSTOM_IMG_VIEWER_SERVICE` at [`service.json`](./config/service.json) in this app pointing to that service (_you can delete `proxy` search param from queries – see [Platform specific/Proxies](#proxies) for more_).\n\n## List of supported platforms\n\n- _Twitter_ – images, videos, GIFs and direct media from `*.twimg.com`. [_More details_](#twitter)\n- _Pixiv_ – images, single direct media from `*.pximg.net` and _Ugoira_-GIFs. [_More details_](#pixiv)\n- _Reddit_ – images, videos, GIFs and galleries. [_More details_](#reddit)\n- _Youtube_ – video with response in [default type](./src/types/social-post.d.ts) containing all streams (via `yt-dlp`)\n- _Tiktok_ – Any video in multiple formats. [_More details_](#tiktok)\n- _Instagram_ – images, videos, galleries and Reels. [_More details_](#instagram)\n- _Osnova_ – images, videos, GIFs and galleries. [_More details_](#osnova)\n- _Coub_ – looped videos with linear audio. [_More details_](#coub)\n- _Joyreactor_ – images and GIFs from multiple subdomains and direct links to media files. [_More details_](#joyreactor)\n- _Tumblr_ – images and galleries. [_More details_](#tumblr)\n- _Kemono_ – images and galleries. [_More details_](#kemono)\n- _Danbooru_ – images (unstable)\n- _Gelbooru_ – images (unstable)\n- _Konachan_ – images (unstable)\n- _Yandere_ – images (unstable)\n- _Eshuushuu_ – images (unstable)\n- _Sankaku_ – images (unstable)\n- _Zerochan_ – images (unstable)\n- _AnimePictures_ – images (unstable)\n\n### Platform specific\n\n#### Binaries\n\nSome platform modules utilizes binary executable, here is the list:\n\n- `Social Picker Twitter Scrapper` – read more about [_Twitter specific_](#twitter)\n- `ffmpeg` – for media stream merging and convering. Follow instructions on [the official site](https://ffmpeg.org/download.html). After installing make sure you added its location to [`$PATH`](\u003chttps://en.wikipedia.org/wiki/PATH_(variable)\u003e)\n- `yt-dlp` – for picking some video-platforms. Download [built release](https://github.com/yt-dlp/yt-dlp/releases) for your platform, make it executable and ensure its location is in the [`$PATH`](\u003chttps://en.wikipedia.org/wiki/PATH_(variable)\u003e)\n\n#### Proxies\n\nSome platforms have very harsh limitation of allowed IP addresses/[AS](\u003chttps://en.wikipedia.org/wiki/Autonomous_system_(Internet)\u003e). Therefore [`service.json`](./config/service.json) contains two fields for setting up host and port of SOCKS5 proxy to some remote server (`PROXY_HOSTNAME` and `PROXY_PORT`); you can set them to `null` if accessing such platforms from your IP address is not being limited. For the same reason base URL of external service hash template for proxy flag in its search query (see [Open access](#open-access)).\n\n#### Twitter\n\nSupports images, videos, GIFs (temporary unstable) and direct media from `*.twimg.com`.\n\nThis project uses [Social Picker Twitter Scrapper](https://github.com/serguun42/Social-Picker-Twitter-Scrapper) – wrapper [for this Go lib](https://github.com/n0madic/twitter-scraper) – to fetch tweets without API. Thus if you want to get Tweets, before running this app you should consider reading that project README and placing its compiled/created executable and config-files into these folders:\n\n1. Move executable file `Social-Picker-Twitter-Scrapper` into [`bin`](./bin/) folder\n2. Move generated `cookies.json` into [`config`](./config/), choose a explicit name for it, e.g. `twitter-scrapper-cookies.json`\n   - For convenience you can move `credentials.json` next to `cookies.json` (this app and executable binary use only cookies file).\n3. Check and edit [`tokens.json`](./config/tokens.json) – give a look to properties `TWITTER_SCAPPER.binary_file_path` and `TWITTER_SCAPPER.cookies_file_path`: paths are relative to project's root, default ones are in accordance to this example.\n\n#### Pixiv\n\nSupports images, single direct media from `*.pximg.net` and _Ugoira_-GIFs.\n\nUses external service for end-user viewing of high-res images due to Referer Header issues. Uses [`ugoira-builder`](./src/util/ugoira-builder.ts) for creating mp4 video from Ugoira ZIP with PNGs (via `ffmpeg`).\n\n#### Reddit\n\nSupports images, videos, GIFs and galleries. Because Reddit's videos are divided from audios, this app utilizes [`video-audio-merge`](./src/util/video-audio-merge.ts) for merging separated streams via `ffmpeg`.\n\nRequires setup of the site's cookies in [`tokens.json`](./config/tokens.json)\n\n#### Tiktok\n\nAny video in multiple formats. Does not support image galleries.\n\nUses `yt-dlp` for extracting metadata and then [`video-codec-convert`](./src/util/video-codec-convert.ts) to convert original highest quality HEVC video without watermark to H264 file (via `ffmpeg`) for compatibility. Still prioritizes HEVC one.\n\n#### Instagram\n\nSupports images, videos, galleries and Reels.\n\n_To pick default regular post_ (square photos, videos and galleries) set property `INSTAGRAM_COOKIE_ONE_LINE_FOR_POSTS` in [`tokens.json`](./config/tokens.json). Place there contents of cookies from your web-browser's DevTools (Press `F12` —\u003e `Console` tab —\u003e type `document.cookie` —\u003e press `Enter`).\n\n\u003e _Question_: Are those cookies enough?\u003cbr\u003e _Answer_: Works well for me! If they're not sufficient anymore, copy from DevTools tabs `Network` or `Application`\n\n_To pick Reels_:\n\n1. Install [this browser extension](https://github.com/kairi003/Get-cookies.txt-LOCALLY) – it's open source and it supports Chrome and Firefox\n2. Go to [instagram.com](https://instagram.com)\n3. Copy cookies (click `Copy` button in extension popup)\n4. Create file `config/instagram.cookies.txt` and paste there contents from clipboard (it starts with `\"# Netscape HTTP Cookie File\"`)\n5. Check and edit [`tokens.json`](./config/tokens.json) – give a look to property `INSTAGRAM_COOKIE_FILE_LOCATION_FOR_REELS`: set path relative to project's root, default one is in accordance to this example.\n\n#### Osnova\n\nSupports images, videos, GIFs and galleries. Also extracts Twitter and Instagram blocks/links from within and handles them with parsers above.\n\n#### Coub\n\nCreates looped videos with linear audio, limits itself only with audio length and filesize if it ever exceed a reasonable size (20 MB). Uses [`video-audio-merge`](./src/util/video-audio-merge.ts) for merging separated streams (via `ffmpeg`).\n\n#### Joyreactor\n\nSupports images, videos, GIFs and galleries, requires setup of the site's cookies in [`tokens.json`](./config/tokens.json).\n\n#### Tumblr\n\nSupports images and galleries, set OAuth keys and stuff in [`tokens.json`](./config/tokens.json).\n\n#### Kemono\n\nSupports images and galleries, requires setup of the site's cookies in [`tokens.json`](./config/tokens.json).\n\n---\n\n### Read more\n\n- [Social Picker Twitter Scrapper](https://github.com/serguun42/Social-Picker-Twitter-Scrapper) – wrapper [for `twitter-scraper` lib](https://github.com/n0madic/twitter-scraper) in Go\n- [Telegram bot based on this service](https://github.com/serguun42/Anime-Ultra-Bot)\n- [Frontend client built for this service](https://github.com/serguun42/Social-Picker-Vue)\n- [ffmpeg docs](https://ffmpeg.org/ffmpeg.html)\n- [yt-dlp docs](https://github.com/yt-dlp/yt-dlp#readme)\n- [Tumblr API](https://www.tumblr.com/docs/en/api/v2)\n- [Osnova API](https://cmtt-ru.github.io/osnova-api/)\n- [What is Pixiv's Ugoira](https://www.pixiv.help/hc/en-us/articles/235584628-What-are-Ugoira-)\n\n---\n\n### [BSL-1.0 License](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserguun42%2Fsocial-picker-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fserguun42%2Fsocial-picker-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fserguun42%2Fsocial-picker-api/lists"}