https://github.com/nvalis/MappingYoutube

Generate a visual map of youtube by using channel relations.
https://github.com/nvalis/MappingYoutube

gephi graph scraping youtube

Last synced: 3 months ago
JSON representation

Generate a visual map of youtube by using channel relations.

• Host: GitHub
• URL: https://github.com/nvalis/MappingYoutube
• Owner: nvalis
• Created: 2017-02-07T14:57:40.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2019-02-11T20:06:05.000Z (almost 6 years ago)
• Last Synced: 2024-07-27T05:37:26.524Z (4 months ago)
• Topics: gephi, graph, scraping, youtube
• Language: Python
• Size: 13.2 MB
• Stars: 4
• Watchers: 1
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# MappingYoutube

This python script generates a graph of [Youtube](https://www.youtube.com) by scraping the related channels from the "about" page using `requests` and `lxml` because unfortunately this data is not provided by the Youtube API.

The gathered data is streamed to [Gephi](https://github.com/gephi/gephi) for visualization using the [gephistreamer](https://github.com/totetmatt/GephiStreamer) library.

The scraping is coordinated by a size limited, thread-safe python `set` as the queue to keep track of the channels to explore. It makes use of multithreading and a simple history to accelerate the exploration.

![Screenshot](/MappingYoutube.png "Screenshot")

Example map created in about 14 min of scraping consisting of about 10000 channels. With the ForceAtlas 2 layout in LinLog mode one can observe different youtube communities mostly grouped around one or more central hubs (node color shows its in-degree, red is higher).

## Requirements

For data streaming to Gephi you need the [`Graph Streaming`](https://marketplace.gephi.org/plugin/graph-streaming/) plugin available in Gephis plugin manager. When installed you will need to rightclick -> start the streaming master server.