{"id":15077200,"url":"https://github.com/riotkit-org/news-feed-provider","last_synced_at":"2025-10-05T11:31:40.013Z","repository":{"id":53458254,"uuid":"100136324","full_name":"riotkit-org/news-feed-provider","owner":"riotkit-org","description":"Provides feeds (articles, publications, posts) from various sources eg. RSS, Facebook, other social media","archived":true,"fork":false,"pushed_at":"2019-05-24T21:18:56.000Z","size":248,"stargazers_count":14,"open_issues_count":6,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-09-30T16:03:09.427Z","etag":null,"topics":["anarchism","facebook","grassroot","grassroot-media","media","microservice","news","news-feed","news-feed-provider","newsapi","php7","rss","rss-aggregator","rss-feed","social","symfony","symfony3","wolnosciowiec"],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/riotkit-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-12T20:30:29.000Z","updated_at":"2024-05-05T11:02:09.000Z","dependencies_parsed_at":"2022-09-12T15:21:50.655Z","dependency_job_id":null,"html_url":"https://github.com/riotkit-org/news-feed-provider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riotkit-org%2Fnews-feed-provider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riotkit-org%2Fnews-feed-provider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riotkit-org%2Fnews-feed-provider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/riotkit-org%2Fnews-feed-provider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/riotkit-org","download_url":"https://codeload.github.com/riotkit-org/news-feed-provider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235393591,"owners_count":18982818,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anarchism","facebook","grassroot","grassroot-media","media","microservice","news","news-feed","news-feed-provider","newsapi","php7","rss","rss-aggregator","rss-feed","social","symfony","symfony3","wolnosciowiec"],"created_at":"2024-09-25T04:04:02.528Z","updated_at":"2025-10-05T11:31:34.680Z","avatar_url":"https://github.com/riotkit-org.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"Wolnościowiec News Feed Provider\n================================\n\nMicroservice.\n\nProvides news from various sources - blogs, portals, facebook pages and exposes via API.\n\nFree software\n-------------\n\nCreated for an anarchist portal, with aim to propagate the freedom and grass-root social movements,\nto automate things for building a better world. Viva la revolución.\n\n- https://wolnosciowiec.net/#/flash\n- http://iwa-ait.org\n- http://zsp.net.pl\n- http://federacja-anarchistyczna.pl/\n\n### Setup\n\nClone the repository, set up a MySQL database, execute `make deploy`\n\n### Getting started\n\nApplication is fully manageable by the HTTP API, so you can run eg. a `postman` to add news boards and attach feed sources.\nThere is an example postman collection exported to file `postman.json`\n\n##### 1. Creating a news board\n\nA news board is a board where your articles will appear, it's like a time line on facebook or twitter.\n\n`{{DOMAIN}}/api/news_board/create`\n\n```\n{\n    \"id\": \"federacja\",\n    \"name\": \"Federacja Anarchistyczna\",\n    \"description\": \"Działająca na terenie Polski federacja grup anarchistycznych. Istniejąca od 1988 roku (początkowo jako Międzymiastówka Anarchistyczna). Dąży do „stworzenia samorządnego społeczeństwa, tworzonego na zasadzie dobrowolności”\"\n} \n```\n\n##### 2. Add sources that will import articles to your news board\n\n`{{DOMAIN}}/api/feed_source/newsBoard-federacja/create`\n\n```\n{\n    \"id\": \"fa-wroclaw\",\n    \"title\": \"FA Wrocław\",\n    \"collectorName\": \"rss\",\n    \"icon\": \"http://wolnywroclaw.pl/wp-content/uploads/2015/05/icon-555f99cdv1_site_icon-256x256.png\",\n    \"sourceData\": {\n        \"url\": \"http://wolnywroclaw.pl/feed/\"\n    },\n    \"defaultLanguage\": \"pl\",\n    \"description\": \"Federacja Anarchistyczna - sekcja Wrocław\",\n    \"enabled\": true,\n    \"scrapingSpecification\": {\n        \"removePaths\": [\n            \"//*[@id='main-content']/article/div[1]/div\"\n        ],\n        \n        \"contentPath\": \"//*[@id='main-content']/article/div[1]\"\n    }\n}\n```\n\nExplanation:\n- collectorName: Must point to a valid name of supported collector, actually only \"rss\" is bundled by default\n  by the plans are to add also a \"facebook_page\"\n  \n- id: You will use it to filter by a source for example\n- defaultLanguage: It's a fallback for articles that do not expose a language via eg. RSS\n- sourceData: Describes parameters that are passed to the collector, in this example to \"rss\" collector which\n  requires only URL\n- scrapingSpecification: Scrapper is a tool that is able to attempt to extract full article content from linked article page, and remove ads\n  everything is operated by xPaths (OPTIONAL)\n  \n##### 3. Browsing feed sources \n\nYou may want to create an option that will allow to select from which sources to show the content.\nThere is an endpoint that shows this list and allows filtering it.\n\n`{{DOMAIN}}/api/feed_source/newsBoard-federacja/search`\n\n```\n{\n    \"title\": \"FA\"\n}\n```\n\n##### 4. Displaying the list of articles\n\n`{{DOMAIN}}/public/api/feed/federacja/browse/1/5`\n\n```\n{\n    \"lanaguage\": [\"pl\"],\n    \"exceptFeedSource\": [\"fa-krakow\"]\n}\n```\n\nExplanation:\n- /browse/1/5 - the first number is a page number, the second is a count of elements to return per page\n- language: Allows to filter by content language (OPTIONAL)\n- exceptFeedSource: Excludes feed sources by id (OPTIONAL)\n- See `{{DOMAIN}}/public/api/feed/search/help` endpoint for more options\n\n### Todo:\n- [ ] Complete test coverage\n- [ ] Implement OAuth2 tokens as currently the service is for internal usage only\n- [ ] Implement Facebook pages posts crawler\n- [ ] Implement a distributed mode (a crawler that collects data from other instance)\n\n#### File Repository integration\n\nCrawled content often contains attached images, and the article itself very often has a main image.\nTo avoid hotlinking entries there is a way to fetch all of those images to a own server\nand replace all usages.\n\n`File Repository` is a [Wolnościowiec's microservice](https://github.com/Wolnosciowiec/file-repository) that exposes an API interface for file storage,\nthe service itself handles de-duplication, caching and of course storing the files in right places.\n\nTo enable the integration simply only all environment variables needs to be filled up properly,\nthe rest is handled by the `ImageRepositoryBundle`.\n\n#### Web-Proxy integration\n\nWhen links to the articles does not have SSL it is possible to use a `Wolnościowiec Web-Proxy` to generate urls secured with SSL.\nThe `web-proxy` service is available here: https://github.com/Wolnosciowiec/web-proxy\n\nExample case:\n1. We have a source url to the article: `http://wolnywroclaw.pl/relacje/protest-lokatorow-z-ul-slicznej-i-zaulka-rogozinskiego-wideo/`\n2. But... our site has SSL, to render the article in iframe we call the `web-proxy`\n3. The webproxy returns: `https://some-proxy.services.someservice.org/?__wp_one_time_token=YT323DEe21FAFAFAZSP1312161F234ewfADSy65Efre4RE23EDW312gtrhtr6`\n4. Embedding the URL from the `web-proxy` allows to render correctly the page through our `web-proxy` server\n\nEnabling:\n\n```\n# web-proxy\nNFP_WEB_PROXY_URL=\"https://some-proxy.services.someservice.org\"\nNFP_WEB_PROXY_PASSPHRASE=\"my-super-secret-passphrase-from-webproxy-service\"\nNFP_PROCESS_INTERNAL_LINKS=1 # process HTML and CSS content, so all static content will be rendered by the proxy\nNFP_EXPIRATION_TIME_MINUTES=1\nNFP_ENABLED_SSL=true\n```\n\n#### Health checking\n\nThere is an endpoint that shows the application health, it's placed under `{{DOMAIN}}/{{NFP_MONITORING_KEY}}/monitor/health/run`\nNFP_MONITORING_KEY is a environment variable you can pass to nginx/apache/docker\n\n#### Setting up using Docker\n\nThere exists an image `wolnosciowiec/news-feed-provider`, it uses environment variables to configure the application.\nYou may want to take a look at the list of variables with example values in the [Dockerfile](./Dockerfile.x86_64).\n\n```\nsudo docker run -e NFP_MAILER_HOST=somehost -e NFP_MAILER_USER=someuser -p 80:80 wolnosciowiec/news-feed-provider\n```\n\nThere are a lot of configuration variables so you might consider using a `docker-compose` for simple usage,\nor `docker-swarm` or `Kubernetes` if scaling in a cluser.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friotkit-org%2Fnews-feed-provider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Friotkit-org%2Fnews-feed-provider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Friotkit-org%2Fnews-feed-provider/lists"}