https://github.com/banickn/vqlforge
SQL to VQL Migration tool
https://github.com/banickn/vqlforge
denodo migration python sql
Last synced: about 1 month ago
JSON representation
SQL to VQL Migration tool
- Host: GitHub
- URL: https://github.com/banickn/vqlforge
- Owner: banickn
- License: mit
- Created: 2025-04-13T17:22:21.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-26T17:24:44.000Z (about 1 year ago)
- Last Synced: 2025-05-26T17:54:15.126Z (about 1 year ago)
- Topics: denodo, migration, python, sql
- Language: JavaScript
- Homepage:
- Size: 392 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# VQLForge
**Translate SQL to Denodo VQL with ease, powered by AI insights.**
VQLForge translates various SQL dialects into Denodo VQL using a React frontend and a Python/FastAPI backend powered by the `sqlglot` library.
It helps accelerate migrations to Denodo by automating SQL-to-VQL conversion.
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
## Why VQLForge?
Migrating SQL to Denodo's VQL can be a time-consuming and error-prone process. VQLForge aims to:
* **Accelerate Migrations:** Automate the bulk of SQL-to-VQL translation by using the UI or the REST API.
* **Reduce Errors:** Leverage `sqlglot` for robust parsing and AI for error analysis and VQL validation.
* **Improve Developer Productivity:** Provide a user-friendly interface for quick conversions.
## Features
* **Converts 24 SQL dialects** (e.g., DuckDB, Trino, Spark, Snowflake, BigQuery) to Denodo VQL via `sqlglot`.
* *Note:* Conversion capabilities depend on `sqlglot`'s support for specific SQL features per dialect. See [sqlglot documentation](https://github.com/tobymao/sqlglot).
* Web UI for SQL input, dialect selection, VQL output, and AI analysis/validation results.
* Handles common syntax conversions supported by `sqlglot`.
* **AI Assistant** that:
* Analyzes `sqlglot` conversion errors.
* Validates generated VQL against a connected Denodo VDP instance.
* Requires a Gemini API key and a connection to Denodo VDP for validation.
* **Agentic Auto-Forge Mode**: A powerful one-click workflow that automates the entire process:
* **Translate**: Converts the source SQL to VQL.
* **Validate**: Checks the VQL against Denodo.
* **Analyze**: If validation fails, the AI agent diagnoses the error, identifying issues like missing views, incorrect functions, or syntax problems.
* **Correct**: The agent generates a corrected VQL query based on its analysis.
* **Re-Validate**: The agent automatically re-runs validation on the corrected query.
## Technologies Used
* **Frontend:** React
* **Backend:** Python, FastAPI
* **Conversion:** `sqlglot`
* **AI Assistant:** PydanticAI, Google Gemini API
* **Deployment:** Docker, Docker Compose
### Prerequisites
- Docker and Docker Compose
- Access to a Denodo instance
- An API key for an LLM provider (**Google Gemini** or **Azure OpenAI**)
## Installation and Setup
2. **Clone:**
```bash
git clone https://github.com/banickn/VQLForge.git
cd VQLForge
```
3. **Configure Environment:** Create a `.env` file in the project root by copying `template.env` (`cp template.env .env`). Then, modify the following properties:
| Variable | Description | Required for | Example |
|---------------------|-----------------------------------------------------------------------------|------------------------|-----------------------------|
| `DENODO_HOST` | Denodo VDP server URL | AI Validation | `denodo-server.example.com` |
| `DENODO_DB` | Default Denodo Virtual DataBase (VDB) | AI Validation | `my_vdb` |
| `DENODO_USER` | Denodo user with read/execute access to VDBs | AI Validation | `denodo_user` |
| `DENODO_PW` | Password for the Denodo user | AI Validation | `password` |
| `GEMINI_API_KEY` | GEMINI API Key | AI Assistant | `YOUR_AI_API_KEY` |
| `OPENAI_API_KEY` | openAI API Key | AI Assistant | `YOUR_AI_API_KEY` |
| `AZURE_OPENAI_ENDPOINT` | URL to your openAI endpoint deployed in Azure | AI Assistant | `https://endpoint` |
| `AI_MODEL_NAME` | Model name (e.g., gemini-2.5-flash or gpt-5-nano) | AI Assistant | `gemini-2.5-flash` |
| `APP_NETWORK_NAME` | Docker network name for connecting to Denodo (if Denodo is also in Docker) | AI Validation | `denodo-lab-net` |
| `HOST_PROJECT_PATH` | Absolute path to your local VQLForge repository directory. | Translation (VDBs) | `/path/to/your/VQLForge` |
These configurations are required to run the advanced query analysis features.
4. **Configure Denodo VDBs:**
Edit `./backend/vdb_conf.yaml` to list your available Denodo VDB names. This allows them to be selectable in VQLForge when using AI validation features.
```yaml
# ./backend/vdb_conf.yaml
vdbs:
- admin
- test
- placeholder1
```
5. **Docker Network (Required for VDP Validation):** For the AI VQL validation feature, ensure a `denodo-docker-network` exists (`docker network create denodo-docker-network`) and your Denodo VDP container is connected to it.
6. **Run:**
If you want to build the images yourself for local development use:
```bash
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build -d
```
For a prod setup with prebuilt images use:
```bash
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```
Images (`ghcr.io/banickn/vqlforge-backend:main`, `ghcr.io/banickn/vqlforge-frontend:main`) will be pulled from GHCR.
6. **Access:**
* **UI:** `http://localhost:4999`
* **API:** `http://localhost:5000` (Docs: `http://localhost:5000/docs`)
## How to Use
1. Open the UI (`http://localhost:4999`).
2. Select the source SQL dialect.
3. Paste your SQL query.
4. Click "Convert".
5. View the resulting Denodo VQL.
6. If the AI assistant is configured (API key) and VDP connection is available (for validation), view the AI's analysis of conversion errors or VQL validation results.
## Denodo VDP Connection
* The backend connects to Denodo VDP for the AI VQL validation feature.
* **Network:** Expects the Denodo VDP container to run on the `denodo-docker-network`.
* **Credentials:** Uses default Denodo credentials (`admin`/`admin`) for connection. This is **not suitable for production** and should be configured securely if deployed outside local development.
* Basic `sqlglot` conversion (without AI validation) does **not** require a Denodo connection.
## Roadmap
* Improved sqlglot denodo dialect for better translations.
* Advanced mode for SQL performance improvements
* SQL inline comment generation from metadata
* Support for various AI APIs and models
## Contributing
Please open an issue to discuss potential changes or submit a pull request with your improvements. We appreciate contributions for bug fixes, new features, or documentation enhancements.
## License
[MIT](https://choosealicense.com/licenses/mit/)