Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/digitalmethodsinitiative/zeeschuimer

A browser extension to collect social media data with.
https://github.com/digitalmethodsinitiative/zeeschuimer

Last synced: 9 days ago
JSON representation

A browser extension to collect social media data with.

Awesome Lists containing this project

README 

        
# 🏴‍☠️ Zeeschuimer



[![DOI: 10.5281/zenodo.4742622](https://zenodo.org/badge/DOI/10.5281/zenodo.6826877.svg)](https://doi.org/10.5281/zenodo.6826877)

[![License: MPL 2.0](https://img.shields.io/badge/license-MPL--2.0-informational)](https://github.com/digitalmethodsinitiative/4cat/blob/master/LICENSE)



A screenshot of Zeeschuimer's status window



Zeeschuimer is a browser extension that monitors internet traffic while you are browsing a social media site, and 

collects data about the items you see in a platform's web interface for later systematic analysis. Its target audience

is researchers who wish to systematically study content on social media platforms that resist conventional scraping or 

API-based data collection.



You can, for example, browse TikTok and later export a list of all posts you saw in the order you saw them in. Data can 

be exported as a JSON file or exported to a [4CAT](https://github.com/digitalmethodsinitiative/4cat) instance for 

analysis and storage. Zeeschuimer is primarily intended as a companion to 4CAT, but you can also integrate its output

into your own analysis pipeline.



Currently, it supports the following platforms:

* [TikTok](https://www.tiktok.com) (posts and comments)

* [Instagram](https://www.instagram.com) (posts only)

* [X/Twitter](https://www.x.com)

* [LinkedIn](https://www.linkedin.com)

* [9gag](https://9gag.com)

* [Imgur](https://imgur.com)

* [Douyin](https://douyin.com)

* [Gab](https://gab.com)



Platform support requires regular maintenance to keep up with changes to the platforms. If something does not work, we

welcome issues and pull requests. See 'Limitations' below for some known limitations to data capture.



The extension does not interfere with your normal browsing and never uploads data automatically, only when you

explicitly ask it to do so. It uses the

[WebRequest](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest) browser API to

locally collect and parse the data search engines are sending to your browser as you use it.



## Installation

Zeeschuimer is in active development. .xpi files that you can use to install it in your browser are available on the 

[releases](https://github.com/digitalmethodsinitiative/zeeschuimer/releases) page. These are signed and can be installed 

in any Firefox-based browser. If you want to run the latest development version instead, you can [do so from the Firefox

debugging console](https://www.youtube.com/watch?v=J7el77F1ckg) after cloning the repository locally.



## How to use

A [guide to using Zeeschuimer and 4CAT](https://tinyurl.com/nmrw-zeeschuimer-tiktok) is available. Basic instructions 

are as follows: 



Install the browser extension in a Firefox browser. A button with the Zeeschuimer logo (a 'Z') will appear in the 

browser toolbar. Click it to open the Zeeschuimer interface. Enable capturing for the sites you want to capture from.



Next, simply browse a supported platform's site. You will see the amount of items detected per platform increase as you 

browse. When you have the items you need, you can export the data as an [ndjson](https://ndjson.org) file, or upload it

to a 4CAT instance where a 4CAT dataset will be created from the uploaded items. You can then run 4CAT's analytical 

processors on the data.



To upload to 4CAT, copy the URL of the website of the 4CAT instance to the "4CAT instance" field at the top of 

Zeeschuimer's interface. You can then use the "to 4CAT" button to create a new 4CAT dataset from the captured data. 

After uploading, Zeeschuimer will show you a link and the ten most recently uploaded datasets are shown at the bottom of

the interface.



Don't forget to reset the data as needed. For example, if you want to create a dataset for a given TikTok hashtag, first

reset the TikTok data in Zeeschuimer, _then_ go to the hashtag's "Explore" page on TikTok, and then upload the dataset

when you've scrolled down enough to be satisfied with the amount of items.



If you find yourself scrolling a lot to collect data, consider using another browser extension to do it for you, for 

example [FoxScroller](https://addons.mozilla.org/en-US/firefox/addon/foxscroller/).



## Limitations



Due to the technical limitations, it may not be possible to collect all items from all 'views' for each supported 

platform. The following limitations are known:



* *Instagram* items that cannot be captured:

  * Stories

  * Posts from the 'Tagged' and 'Reels' tabs on a profile page

  * Posts from the 'Saved' overview of bookmarked posts

  * Posts from the 'For You' feed on the 'Explore' page (the 'Not personalized' feed _does_ work)

  * 'Suggested for you' and 'Sponsored' posts on the front page feed

* *TikTok* items that cannot be captured:

  * Live streams



Note that these are *known* limitations; data capture may break or change based on platform changes. Always 

cross-reference captured data with what you are seeing in your browser.



## Credits & license

Zeeschuimer was developed by Stijn Peeters for the [Digital Methods Initiative](https://digitalmethods.net) and is 

licensed under the Mozilla Public License, 2.0. Refer to the LICENSE file for more information.



Graphics based on [an image generated by Dall-E](https://labs.openai.com/s/oWvGAHC0pxwWV3bNLfENu7AV), for the prompt 

'detail of a 1914 metaphysical painting by giorgio de chirico depicting a buccaneer pensively looking out over the open 

sea'. Interface icons by [Font Awesome](https://fontawesome.com/license/free). 

[Open Sans](https://fonts.google.com/specimen/Open+Sans) and [Lobster](https://fonts.google.com/specimen/Lobster) fonts 

from Google Fonts.



Development is supported by the Dutch [PDI-SSH](https://pdi-ssh.nl/en/) foundation through the [CAT4SMR 

project](https://cat4smr.humanities.uva.nl/).