https://github.com/pearmaster/distributed_downloader

https://github.com/pearmaster/distributed_downloader

Last synced: about 1 month ago
JSON representation

• Host: GitHub
• URL: https://github.com/pearmaster/distributed_downloader
• Owner: pearmaster
• License: apache-2.0
• Created: 2025-04-30T16:13:53.000Z (about 1 month ago)
• Default Branch: main
• Last Pushed: 2025-04-30T16:33:51.000Z (about 1 month ago)
• Last Synced: 2025-04-30T17:43:13.191Z (about 1 month ago)
• Language: Python
• Size: 8.79 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# distributed_downloader

```py

from distributed_downloader import DistributedDataset, UrlFetcher

from datetime import timedelta

from paho.mqtt.client import Client as MqttClient

from paho.mqtt.enums import CallbackAPIVersion

mqtt_client = MqttClient(callback_api_version=CallbackAPIVersion.VERSION2)

mqtt_client.connect_async("localhost")

mqtt_client.loop_start()

topic = "weather/utah/alerts"

url_fetcher = UrlFetcher("https://api.weather.gov/alerts/active.atom?area=UT", topic)

def print_alerts(topic, payload):

    print(payload)

distrib_data = DistributedDataset(mqtt_client, topic, min_age=timedelta(seconds=10), max_agetimedelta(seconds=300), fetcher=url_fetcher)

distrib_data.add_new_data_callback(print_alerts)

distrib_data.start_up(delay_seconds=3)

'''