https://github.com/superglue-ai/superglue
Self-healing open source data connector. Use it as a layer between you and any complex / legacy APIs and always get the data that you want in the format you expect.
https://github.com/superglue-ai/superglue
api api-gateway etl-automation graphql transformations
Last synced: 14 days ago
JSON representation
Self-healing open source data connector. Use it as a layer between you and any complex / legacy APIs and always get the data that you want in the format you expect.
- Host: GitHub
- URL: https://github.com/superglue-ai/superglue
- Owner: superglue-ai
- License: gpl-3.0
- Created: 2025-01-10T18:38:47.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2025-04-02T09:28:05.000Z (22 days ago)
- Last Synced: 2025-04-02T10:33:11.749Z (22 days ago)
- Topics: api, api-gateway, etl-automation, graphql, transformations
- Language: TypeScript
- Homepage: https://superglue.cloud
- Size: 1.05 MB
- Stars: 1,646
- Watchers: 6
- Forks: 73
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
self-healing open source data connector 🍯
superglue is a self-healing open source data connector. You can deploy it as a proxy between you and any complex / legacy APIs and always get the data that you want in the format you expect.
Here's how it works: You define your desired data schema and provide basic instructions about an API endpoint (like "get all issues from jira"). Superglue then does the following:
- Automatically generates the API configuration by analyzing API docs.
- Handles pagination, authentication, and error retries.
- Transforms response data into the exact schema you want using JSONata expressions.
- Validates that all data coming through follows that schema, and fixes transformations when they break.
[](https://github.com/superglue-ai/superglue/blob/main/LICENSE)
[](https://www.ycombinator.com/companies/superglue)
[](https://www.npmjs.com/package/@superglue/client)
[](https://hub.docker.com/r/superglueai/superglue)
[](https://twitter.com/adinagoerres)
[](https://twitter.com/sfaistenauer)
[](https://app.workweave.ai/reports/repository/org_0S2o9PLamHvNsTjHbszc38vC/914997268)
If you’re spending a lot of time writing code connecting to weird APIs, fumbling with custom fields in foreign language ERPs, mapping JSONs, extracting data from compressed CSVs sitting on FTP servers, and making sure your integrations don’t break when something unexpected comes through, superglue might be for you.
```mermaid
flowchart LR
subgraph Input[data sources]
A1[APIs]
A2[files]
A3[legacy systems]
end
subgraph Process[data transformation]
T1[superglue engine]
end
subgraph Output[destination]
D1[your system]
end
Input --> Process
Process --> Output
%% Styling
classDef sourceStyle fill:#f9f,stroke:#333,stroke-width:2px
classDef processStyle fill:#bbf,stroke:#333,stroke-width:2px
classDef outputStyle fill:#bfb,stroke:#333,stroke-width:2px
class Input sourceStyle
class Process processStyle
class Output outputStyle
```
## quick start
### hosted version
1. Sign up for early access to the hosted version of superglue at [superglue.cloud](https://superglue.cloud)
2. Install the superglue js/ts client:
```bash
npm install @superglue/client
```
3. Configure your first api call:
```javascript
import { SuperglueClient } from "@superglue/client";
const superglue = new SuperglueClient({
apiKey: "************"
});
const config = {
urlHost: "https://futuramaapi.com",
urlPath: "/graphql",
instruction: "get all characters from the show",
responseSchema: {
type: "object",
properties: {
characters: {
type: "array",
items: {
type: "object",
properties: {
name: { type: "string" },
species: { type: "string", description: "lowercased" }
}
}
}
}
}
};
const result = await superglue.call({endpoint: config});
console.log(JSON.stringify(result.data, null, 2));
/*
output:
{
"characters": [
{
"name": "Phillip J. Fry",
"species": "human"
},
...
]
}
*/
```
### self-hosted version
Run your own instance of superglue using Docker:
1. Pull the Docker image:
```bash
docker pull superglueai/superglue
```
2. Create a `.env` by copying the `.env.example` file at the root
3. Start the server:
```bash
docker run -d \
--name superglue \
--env-file .env \
-p 3000:3000 \
-p 3001:3001 \
superglueai/superglue
```
4. Verify the installation:
```bash
curl http://localhost:3000/health
> OK
# or open http://localhost:3000/?token=your-auth-token
```
5. Open the dashboard to create your first configuration:
```bash
http://localhost:3001/
```
6. run your first call:
```bash
npm install @superglue/client
```
```javascript
import { SuperglueClient } from "@superglue/client";
const superglue = new SuperglueClient({
endpoint: "http://localhost:3000",
apiKey: "your-auth-token"
});
// either via config object
const config = {
urlHost: "https://futuramaapi.com",
urlPath: "/graphql",
instruction: "get all characters from the show",
};
const result = await superglue.call({endpoint: config});
// or via the api id if you have already created the endpoint
const result2 = await superglue.call({id: "futurama-api"});
console.log(JSON.stringify(result.data, null, 2));
```
## key features
- **LLM-Powered Data Mapping**: Automatically generate data transformations using large language models
- **API Proxy**: Intercept and transform API responses in real-time with minimal added latency
- **File Processing**: Handle various file formats (CSV, JSON, XML) with automatic decompression
- **Schema Validation**: Ensure data compliance with your specified schemas
- **Flexible Authentication**: Support for various auth methods including header auth, api keys, oauth, and more
- **Smart Pagination**: Handle different pagination styles automatically
- **Caching & Retry Logic**: Built-in caching and configurable retry strategies
## 📖 Documentation
For detailed documentation, visit [docs.superglue.cloud](https://docs.superglue.cloud).
## 🤝 contributing
We love contributions! Feel free to open issues for bugs or feature requests.
[//]: # (To contribute to the docs, check out the /docs folder.)
## license
superglue is GPL licensed. The superglue client SDKs are MIT licensed. See [LICENSE](LICENSE) for details.
## 🙋♂️ support
- 💬 Discord: [Join our community](https://discord.gg/vUKnuhHtfW)
- 🐛 Issues: [GitHub Issues](https://github.com/superglue-ai/superglue/issues)
[](https://twitter.com/superglue_d)