{"id":17658829,"url":"https://github.com/volfpeter/graphscraper","last_synced_at":"2025-05-07T13:44:32.765Z","repository":{"id":57435905,"uuid":"105275731","full_name":"volfpeter/graphscraper","owner":"volfpeter","description":"Python 3 graph implementation designed to be turned into a web scraper for graph data.","archived":false,"fork":false,"pushed_at":"2023-01-22T10:04:35.000Z","size":32,"stargazers_count":10,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T10:42:26.513Z","etag":null,"topics":["cache-storage","graph","graph-algorithms","graph-database","python","python3","social-network-analysis","web","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/volfpeter.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-29T13:31:53.000Z","updated_at":"2025-03-28T15:54:29.000Z","dependencies_parsed_at":"2023-02-12T15:31:11.389Z","dependency_job_id":null,"html_url":"https://github.com/volfpeter/graphscraper","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volfpeter%2Fgraphscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volfpeter%2Fgraphscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volfpeter%2Fgraphscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volfpeter%2Fgraphscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/volfpeter","download_url":"https://codeload.github.com/volfpeter/graphscraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252889549,"owners_count":21820208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache-storage","graph","graph-algorithms","graph-database","python","python3","social-network-analysis","web","web-scraping"],"created_at":"2024-10-23T15:43:25.049Z","updated_at":"2025-05-07T13:44:32.738Z","avatar_url":"https://github.com/volfpeter.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"|Downloads|\n\nGraphScraper\n=================\n\nGraphScraper is a Python 3 library that contains a base graph implementation designed\nto be turned into a web scraper for graph data. It has two major features:\n\n1) The graph automatically manages a database (using either SQLAlchemy or\nFlask-SQLAlchemy) where it stores all the nodes and edges the graph has seen.\n\n2) The base graph implementation provides hook methods that, if implemented,\nturn the graph into a web scraper.\n\nYet another graph implementation - why\n-------------------------------------------\n\nThere are many excellent graph libraries available for different purposes. I started\nimplementing this one because i haven't found a graph library that is dynamic (i don't\nneed the whole graph in memory - or on disk - before i start working with it), that\ncan be used as a web scraper (to seamlessly load nodes and edges from some remote\ndata source when that piece of data is needed) and that keeps all data (the graph)\nautomatically up-to-date on the disk. GraphScraper aims to satisfy these requirements.\n\nExamples\n----------------------\n\nBesides the base graph implementation, the following working examples are also included\nin the library, that show you how you can implement and use an actual graph scraper:\n\n- `igraphwrapper`: Instead of web-scraping, this example is using an igraph_ graph\n  instance as the \"remote\" source to scrape data from.\n- `spotifyartist`: This example is using the Spotify_ web API to load artists and\n  edges are defined by Artist similarity.\n  \nMore graph implementations\n----------------------------------\n\n- USPTO_ patent citation graph\n- Mastodon_ social graph\n\nRelated projects\n------------------------\n\n- localclustering_: a local graph clustering algorithm\n\nDependencies\n-----------------\n\nIf you wish to use one of the included graph implementations, then please read the\ncorresponding module's description for additional requirements.\n\nContribution\n-----------------\n\nAny form of constructive contribution (feedback, features, bug fixes, tests, additional\ndocumentation, etc.) is welcome.\n\n.. _igraph: http://igraph.org\n.. _localclustering: https://github.com/volfpeter/localclustering\n.. _Spotify: https://developer.spotify.com/web-api/\n.. |Downloads| image:: https://pepy.tech/badge/graphscraper\n.. _USPTO: https://github.com/volfpeter/uspto-patent-citation-graph\n.. _Mastodon: https://github.com/volfpeter/mastodon-social-graph\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvolfpeter%2Fgraphscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvolfpeter%2Fgraphscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvolfpeter%2Fgraphscraper/lists"}