https://github.com/techgnosis/loki-twitter

https://github.com/techgnosis/loki-twitter

Last synced: 8 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/techgnosis/loki-twitter
• Owner: techgnosis
• Created: 2021-11-14T23:57:47.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2022-02-07T20:00:12.000Z (almost 3 years ago)
• Last Synced: 2024-08-01T15:32:40.415Z (3 months ago)
• Language: Python
• Size: 30.3 KB
• Stars: 6
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
This repo accompanies the blog post [Going off-label with Grafana Loki](https://grafana.com/blog/2022/02/07/going-off-label-with-grafana-loki-how-to-set-up-a-low-cost-twitter-analysis/)

This repo contains all the code needed to stream tweets into either Loki, GEL, or Grafana Cloud Logs. This repo does not include any code for deploying Loki or GEL. Loki is very easy to run on your laptop but you can also sign up for a [free Grafana Cloud account](https://grafana.com).

GEL stands for Grafana Enterprise Logs. GEL is the paid version of Loki with many features that large companies use, such as out-of-the-box authn/authz and a GUI for speed of initial onboarding.

# Scripts

Use `stream.py` to stream Tweets into either Loki, GEL, or Grafana Cloud Logs. `stream.py` relies on environment variables which are defined at the beginning of the script and explained below.

Use `test-loki.sh` or `test-cloud-or-gel.sh` to insert the data from `test-data.json` into either Loki or GEL. Both scripts use the same environment variables as `stream.py`.

Use `time_in_ns.sh` to get current timestamps for insertion into `test-data.json` so you can find them in Loki or GEL without setting a huge time range. The timestamps that are in `test-data.json` as you read this are dated far back in the past.

# Requirements to run the code

Docker, or you can use your own Python environment if that's in your skillset

# How to run

1. Run `open-docker-env.sh` to create an interactive bash shell in the latest `python:3` container image

1. Run `pip install requests`. The "requests" library is the only dependency that's not in the Python 3 standard library

1. Export the following environment variables:

    * `TWITTER_BEARER_TOKEN`

    * `LOKI_URL`

    * `LOKI_USERNAME` (if required)

    * `LOKI_PASSWORD` (if required)

1. Run `python3 stream.py`

# Paths

The paths for Grafana Cloud Logs, Loki, and GEL are the same: `/loki/api/v1/push`. 

# Other notes

* The code that streams Tweets from the Twitter API filters out tweets that are not in English. The filtering occurs on line 45

* There is a 1.3s wait between each Tweet. The wait occurs on line 47. The author's Twitter account has a limit of 2 million tweets a month and 1.3s gives me exactly 2m tweets if the script ran for a whole month.