https://github.com/unitvectory-labs/http-response-collector
Retrieves HTTP responses and headers from specified endpoints and publishes the collected data to Google Cloud Pub/Sub for further processing.
https://github.com/unitvectory-labs/http-response-collector
Last synced: 6 months ago
JSON representation
Retrieves HTTP responses and headers from specified endpoints and publishes the collected data to Google Cloud Pub/Sub for further processing.
- Host: GitHub
- URL: https://github.com/unitvectory-labs/http-response-collector
- Owner: UnitVectorY-Labs
- License: mit
- Created: 2025-01-30T23:13:11.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-15T21:05:58.000Z (over 1 year ago)
- Last Synced: 2025-02-15T22:19:40.660Z (over 1 year ago)
- Language: Go
- Size: 42 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://opensource.org/licenses/MIT) [](https://guide.unitvectorylabs.com/bestpractices/status/#work-in-progress)
# http-response-collector
Retrieves HTTP responses and headers from specified endpoints and publishes the collected data to Google Cloud Pub/Sub for further processing.
## References
- [http-response-collector](https://github.com/UnitVectorY-Labs/http-response-collector) - Retrieves HTTP responses and headers from specified endpoints and publishes the collected data to Google Cloud Pub/Sub for further processing.
- [http-response-collector-tofu](https://github.com/UnitVectorY-Labs/http-response-collector-tofu) - OpenTofu module for deploying a http-response-collector to GCP
## Features
- Processes requests from Pub/Sub for fetching specified URLs.
- Retrieves HTTP responses from specified URLs.
- Extracts response headers and body content.
- Publishes structured response data to Google Cloud Pub/Sub as JSON.
## Configuration
The application requires the following environment variables:
| Variable | Description |
|------------------------|--------------------------------------------------|
| `GOOGLE_CLOUD_PROJECT` | The GCP project ID where Pub/Sub is hosted. |
| `RESPONSE_PUBSUB` | The Pub/Sub topic name for publishing responses. |
## Request Format
The following JSON format is used to request a URL to be fetched:
```json
{"url":"https://example.com"}
```
## Response Format
The following show examples of the payloads that are published to Pub/Sub.
A successful request whose body is JSON will include the `responseJson` payload:
```json
{
"url": "https://example.com/content.json",
"headers": "{\"Cache-Control\":\"max-age=3600, public, s-maxage=7200, stale-if-error=43200, stale-while-revalidate=3600, immutable\",\"Content-Type\":\"application/json\",\"Date\":\"Tue, 04 Feb 2025 23:37:31 GMT\"}",
"responseJson": "{\"message\":\"Hello, World!\"}",
"responseTime": 366,
"requestTime": "2025-02-04T23:37:31.64365949Z",
"statusCode": 200
}
```
A successful request whose body is not JSON will include the:
```json
{
"url": "https://example.com/text",
"headers": "{\"Content-Length\":\"22\",\"Content-Type\":\"text/plain\",\"Date\":\"Tue, 04 Feb 2025 23:48:27 GMT\"}",
"responseBody": "Body Content Goes Here",
"responseTime": 111,
"requestTime": "2025-02-04T23:48:27.307539426Z",
"statusCode": 200
}
```
A failed request will include the `error` payload:
```json
{
"url": "https://fail.example.com",
"error": "Error fetching URL",
"requestTime": "2025-02-05T01:27:41.915539558Z",
}
```