https://github.com/ota2000/dlt-community-sources
Community-maintained dlt sources for various APIs
https://github.com/ota2000/dlt-community-sources
app-store-connect community data-engineering data-pipeline dlt etl nextdns python rest-api twilio
Last synced: 3 months ago
JSON representation
Community-maintained dlt sources for various APIs
- Host: GitHub
- URL: https://github.com/ota2000/dlt-community-sources
- Owner: ota2000
- License: mit
- Created: 2026-03-26T14:50:09.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-04-03T10:02:54.000Z (3 months ago)
- Last Synced: 2026-04-03T14:44:03.302Z (3 months ago)
- Topics: app-store-connect, community, data-engineering, data-pipeline, dlt, etl, nextdns, python, rest-api, twilio
- Language: Python
- Homepage: https://pypi.org/project/dlt-community-sources/
- Size: 373 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
# dlt-community-sources
[](https://github.com/ota2000/dlt-community-sources/actions/workflows/ci.yaml)
[](https://pypi.org/project/dlt-community-sources/)
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
Community-maintained [dlt](https://dlthub.com/) sources for various APIs. Install only what you need via extras.
## Available Sources
| Source | Extra | Description |
|---|---|---|
| [App Store Connect](dlt_community_sources/app_store_connect/) | `app-store-connect` | Apple App Store Connect API |
| [Twilio](dlt_community_sources/twilio/) | `twilio` | Twilio SMS, Calls, Usage Records |
| [Meta Ads](dlt_community_sources/meta_ads/) | `meta-ads` | Meta (Facebook) Ads campaigns, ad sets, ads, and insights |
| [NextDNS](dlt_community_sources/nextdns/) | `nextdns` | NextDNS logs and analytics |
| [Microsoft Ads](dlt_community_sources/microsoft_ads/) | `microsoft-ads` | Microsoft Advertising (Bing Ads) with certificate auth support |
| [TikTok Ads](dlt_community_sources/tiktok_ads/) | `tiktok-ads` | TikTok Marketing API (Business API) |
| [Yahoo Ads Search](dlt_community_sources/yahoo_ads_search/) | `yahoo-ads-search` | Yahoo Japan Ads Search API (SS) with MCC support |
## Installation
```bash
pip install dlt-community-sources[app-store-connect]
```
Or with uv:
```bash
uv add dlt-community-sources[app-store-connect]
```
## Usage
```python
import dlt
from dlt_community_sources.twilio import twilio_source
pipeline = dlt.pipeline(
pipeline_name="twilio",
destination="duckdb",
dataset_name="twilio_data",
)
source = twilio_source(
account_sid="your_account_sid",
api_key_sid="your_api_key_sid",
api_key_secret="your_api_key_secret",
resources=["messages", "calls"],
)
load_info = pipeline.run(source)
print(load_info)
```
See each source's README for detailed usage and authentication options.
## Features
All sources share these common features:
- Incremental loading where applicable
- Automatic token/auth refresh
- Rate limit handling with exponential backoff
- Graceful permission handling (skips inaccessible resources)
- Works with any [dlt destination](https://dlthub.com/docs/dlt-ecosystem/destinations/)
## Development
```bash
# Fork the repository first, then:
git clone https://github.com/YOUR_USERNAME/dlt-community-sources.git
cd dlt-community-sources
uv sync --group dev
uv run pytest -v
uv run ruff check .
```
## AI-assisted development
This repository is set up for AI coding assistants with two layers of AI context:
- **Project rules** (`.ai/rules.md`) — coding conventions, testing patterns, and structure specific to this repo
- **[dltHub AI workbench](https://dlthub.com/context/)** — dlt ecosystem knowledge, MCP server, and guided skills for pipeline development
Both are synced to tool-specific locations (Claude Code, Cursor, Codex, etc.) via a single command:
```bash
bash scripts/sync-ai-rules.sh
```
The dltHub workbench provides an MCP server for source search (9,700+ APIs), pipeline inspection, and data exploration — plus skills that guide you through the full pipeline development workflow. See [CONTRIBUTING.md](CONTRIBUTING.md#ai-assisted-development-recommended) for details.
Edit `.ai/` only for project rules, then re-run the sync script. CI checks that files are in sync.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md).
## License
MIT