https://github.com/yoast/singer-tap-builtwith
https://github.com/yoast/singer-tap-builtwith
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/yoast/singer-tap-builtwith
- Owner: Yoast
- License: agpl-3.0
- Created: 2021-05-28T12:54:11.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2022-09-30T08:29:00.000Z (over 3 years ago)
- Last Synced: 2025-01-02T08:31:41.468Z (over 1 year ago)
- Language: Python
- Size: 25.4 KB
- Stars: 1
- Watchers: 4
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# tap-builtwith
This is a [Singer](https://singer.io) tap that produces JSON-formatted data
following the [Singer
spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
This tap:
- Pulls raw data from [BuiltWith](https://api.builtwith.com/)
- Extracts the following resources:
- [Trends](https://api.builtwith.com/trends-api)
- Outputs the schema for each resource
- Incrementally pulls data based on the input state
### Step 1: Create an API key
Create an API key in BuiltWith
### Step 2: Configure
Create a file called `builtwith_config.json` in your working directory, following [builtwith_config.json](builtwith_config.json). The required parameters is the `api_key`.
This requires a `state.json` file to let the tap know from when to retrieve data. For example:
```json
{
"bookmarks": {
"trends": {
"start_date": "2021-05-26"
}
}
}
```
Will replicate data from those dates.
### Step 3: Install and Run
Create a virtual Python environment for this tap. This tap has been tested with Python 3.7, 3.8 and 3.9 and might run on future versions without problems.
```bash
python -m venv singer-builtwith
singer-builtwith/bin/python -m pip install --upgrade pip
singer-builtwith/bin/pip install git+https://github.com/Yoast/singer-tap-builtwith.git
```
This tap can be tested by piping the data to a local JSON target. For example:
Create a virtual Python environment with `singer-json`
```
python -m venv singer-json
singer-json/bin/python -m pip install --upgrade pip
singer-json/bin/pip install target-json
```
Test the tap:
```
singer-builtwith/bin/tap-builtwith --state state.json -c builtwith_config.json | singer-json/bin/target-json >> state_result.json
```
Copyright © 2021 Yoast