{"id":29440020,"url":"https://github.com/krokozyab/ofarrow","last_synced_at":"2026-04-05T08:35:37.839Z","repository":{"id":301971381,"uuid":"1010721934","full_name":"krokozyab/ofarrow","owner":"krokozyab","description":"Arrow Flight SQL Server \u0026 SQL query tool (IDE) for Oracle Fusion","archived":false,"fork":false,"pushed_at":"2025-08-17T12:08:35.000Z","size":1828,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-17T12:23:37.613Z","etag":null,"topics":["airflow","analytics","apache-arrow","business-intelligence","dagster","data-extraction","etl","flight-sql","grpc","ide","java","oracle-fusion","oracle-fusion-cloud","oracle-fusion-erp","pandas","polars","python","reporting","stream"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krokozyab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"licenses/LICENSES.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-29T17:06:37.000Z","updated_at":"2025-08-17T12:08:38.000Z","dependencies_parsed_at":"2025-07-27T23:32:55.850Z","dependency_job_id":null,"html_url":"https://github.com/krokozyab/ofarrow","commit_stats":null,"previous_names":["krokozyab/ofarrow"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/krokozyab/ofarrow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krokozyab%2Fofarrow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krokozyab%2Fofarrow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krokozyab%2Fofarrow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krokozyab%2Fofarrow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krokozyab","download_url":"https://codeload.github.com/krokozyab/ofarrow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krokozyab%2Fofarrow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31430009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T08:13:15.228Z","status":"ssl_error","status_checked_at":"2026-04-05T08:13:11.839Z","response_time":75,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","analytics","apache-arrow","business-intelligence","dagster","data-extraction","etl","flight-sql","grpc","ide","java","oracle-fusion","oracle-fusion-cloud","oracle-fusion-erp","pandas","polars","python","reporting","stream"],"created_at":"2025-07-13T10:01:55.406Z","updated_at":"2026-04-05T08:35:37.807Z","avatar_url":"https://github.com/krokozyab.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# ✈️ Oracle Fusion Flight SQL Server\n\n**Zero-Infrastructure Data Access for Oracle Fusion**\n\nTransform your Oracle Fusion into a modern data platform with **no servers to manage, no containers to deploy, no infrastructure to maintain**. Download the Windows or macOS installer and start using the app instantly:\n\n🖥️ **Built-in SQL Editor \u0026 Export** - Execute ad-hoc SQL queries using the integrated editor with database explorer, syntax highlightings, autocompletion and result previews, export results directly to CSV or Excel with one click.   \n✨ **Arrow Flight SQL Protocol** - Connect from Python, R, JavaScript, Go, Rust, and more  \n🌐 **HTTP REST API** - Export data with simple curl/wget commands  \n📊 **Multiple Export Formats** - CSV, JSON, Excel, and Parquet with hive support  \n⚡ **Streaming Data Access** - Handle datasets with automatic pagination   \n🎯 **Minimal Configuration** - Just unarchive OTBI report and provide credentials  \n\nIDE   \n[![Watch the demo](https://img.shields.io/badge/▶️%20Watch%20Demo-red?logo=youtube\u0026logoColor=white)](https://youtu.be/jdHffE1RxuA?si=Fmi1jE9S4vfZ1Z4T)\n\nExcel integration   \n[![Watch the demo](https://img.shields.io/badge/▶️%20Watch%20Demo-red?logo=youtube\u0026logoColor=white)](https://www.youtube.com/watch?v=K63LvHCZZwM)\n\n## Why Choose This Over Traditional ETL?\n\n| Traditional ETL              | Oracle Fusion Flight SQL              |\n|------------------------------|---------------------------------------|\n| Complex infrastructure setup | **Windows \u0026 macOS installers**        |\n| Multiple servers to manage   | **Zero infrastructure**               |\n| Expensive ETL licenses       | **Completely free**                   |\n| Vendor lock-in               | **Open standards (Arrow Flight SQL)** |\n| Limited export formats       | **CSV, JSON, Excel, Parquet, Hive**   |\n| Manual data pipeline setup   | **Instant API access**                |\n\n## 🎯 Well-Suited For\n\n- **Data Scientists** - Direct Python/R access to Oracle Fusion data\n- **DevOps Teams** - Simple shell script automation with curl/wget\n- **Business Analysts** - One-click Excel exports for reporting\n- **Data Engineers** - Parquet exports for data lakes and analytics\n- **Integration Teams** - Standards-based API for any programming language\n- **ETL Orchestration** - Works seamlessly with Airflow, Prefect, Dagster, and other workflow engines\n\n---\n\n## 📄 Table of Contents\n\n- [✨ Features](#-features)\n- [🛠 Prerequisites](#-prerequisites)\n- [📝 Installation](#-installation)\n- [⚙️ Configuration](#-configuration)\n- [❗ Limitations](#-Limitations)\n- [⚠️ Important Disclaimer](#-important-disclaimer)\n- [📝 TODO](#-todo)\n- [📚 Examples](#-examples)\n- [🔗 Other ](#-other)\n- [📫 Contact](#-contact)\n\n---\n\n## ✨ Features\n\n### 📊 Multi-Format Data Export\n- **CSV** - Perfect for spreadsheets and data analysis\n- **JSON** - Ideal for web applications and APIs\n- **Excel** - Ready-to-use business reports with formatting\n- **Parquet** - Optimized columnar format for data lakes and analytics\n- **Hive Partitioned** - Partitioned Parquet files in ZIP archive for data lakes\n\n### ⚡ Optimized Data Access\n- **Streaming processing** - Handle datasets with memory efficiency\n- **Automatic pagination** - Seamless handling of result sets\n- **Connection management** - Efficient resource utilization\n\n### 🌐 Dual Protocol Support\n- **Arrow Flight SQL** - Modern binary protocol for efficient data transfer\n- **HTTP REST API** - Simple endpoints for curl, wget, and web integration\n- **Health monitoring** - Built-in health checks and metrics\n- **Cross-platform** - Works on Windows and macOS\n\n\n\u003c!-- \u003cimg src=\"pics/server_connection.png\" alt=\"Flight SQL Server\" width=\"600\"\u003e --\u003e\n\n\u003c!-- \u003cimg src=\"pics/server_logs.png\" alt=\"Flight SQL Server\" width=\"600\"\u003e --\u003e\n\n\n\n## 🛠 Prerequisites\n\nBefore using this server, ensure you have the following:\n\n- **Oracle Fusion Access:** Valid credentials with access to Oracle Fusion reporting.\n\n---\n\n## 📝 Installation \u0026 Deployment\n\n### 📦 What You Need\n\n1  **Oracle Fusion credentials** with reporting access   \n2  **Download the Windows or macOS installer** from the latest release:\n\n**You're ready to query Oracle Fusion data!** 🎉\n\n   [![Download Installer](https://img.shields.io/static/v1?label=Download\u0026message=Installer\u0026color=blue\u0026style=for-the-badge\u0026logo=github)](https://github.com/krokozyab/ofarrow/releases) \u003c- Download the installer here.\n\n\n### 🔧 Setup Oracle Fusion Report\n\n1. Create report in OTBI. In your Fusion instance, un-archive DM_ARB.xdm.catalog and RP_ARB.xdo.catalog from [OTBI report](https://github.com/krokozyab/ofarrow/tree/master/otbi) into /Shared Folders/Custom/Financials folder.\n2. Install the application by downloading the appropriate installer from the Releases page—available for both Windows and macOS.\n\n## ⚙️ Usage Examples\n\n[### 🐍 Python Data Science](https://gist.github.com/krokozyab/f20b868d4b9c2a1ba12a52e1ada1a07d)\n\n[### 📊 Multi-Format Exports](https://gist.github.com/krokozyab/92e6c977bd3f593f7d39a678a52a57a1)\n\n\n### 🔍 Health Monitoring\n```bash\n# Check server health\ncurl http://localhost:8081/health\n# Returns: {\"status\":\"UP\",\"database\":\"UP\",\"uptime_ms\":12345,\"response_time_ms\":45}\n```\n\n[### 🔄 Apache Airflow Integration](https://gist.github.com/krokozyab/69f880e78ca4654faba021c304d48eb0)\n\n\n### 🗂 Hive Partitioned Export\n```bash\n# Export data partitioned by a specific column (returns ZIP archive)\ncurl -G -o gl_partitioned.zip \\\n  --data-urlencode \"sql=SELECT segment1, segment2, segment3, concatenated_segments FROM gl_code_combinations\" \\\n  --data-urlencode \"format=hive\" \\\n  --data-urlencode \"partition=segment3\" \\\n  \"http://localhost:8081/export\"\n# Extract to see Hive-style directory structure\nunzip gl_partitioned.zip\nls -la\n\n```\n\n** Suited for:**\n- 🏢 **Data Lakes** - Hive-compatible partitioned structure\n- ⚡ **Query Performance** - Partition pruning for faster analytics\n- 📁 **Data Organization** - Logical data separation by column values\n- 🔄 **ETL Pipelines** - Standard format for Spark, Hive, Presto\n\n---\n\n## 🌐 Server Access Points\n\nOnce started, the server provides multiple access methods:\n\n### 📊 Data Export Endpoints\n```bash\n# CSV Export (default)\ncurl \"http://localhost:8081/export?sql=SELECT * FROM fnd_currencies_tl\" -o data.csv\n\n# JSON Export\ncurl \"http://localhost:8081/export?sql=SELECT * FROM fnd_currencies_tl\u0026format=json\" -o data.json\n\n# Excel Export\nwget -O report.xlsx \"http://localhost:8081/export?sql=SELECT * FROM fnd_currencies_tl\u0026format=excel\"\n\n# Parquet Export\ncurl \"http://localhost:8081/export?sql=SELECT * FROM fnd_currencies_tl\u0026format=parquet\" -o data.parquet\n\n# Hive Partitioned Export (ZIP archive)\ncurl -o partitioned.zip \"http://localhost:8081/export?sql=SELECT * FROM fnd_currencies_tl\u0026format=hive\u0026partition=currency_code\"\n```\n\n### ⚡ Arrow Flight SQL (Python)\n```python\nimport pyarrow.flight as fl\nclient = fl.connect(\"grpc://localhost:32010\")\ntable = client.execute(\"SELECT * FROM fnd_currencies_tl\").read_all()\ndf = table.to_pandas()\n```\n\n---\n\n## ❗ Limitations\n\n- Read-only access to Oracle Fusion data\n- Requires Oracle Fusion WSDL reporting setup\n- Limited to SQL SELECT statements\n- Some limitations are inherent to the underlying Oracle Fusion reporting architecture.\n  For further insights on some of these challenges, see this article on using synchronous BIP for data extraction.\n  https://www.ateam-oracle.com/post/using-synchronous-bip-for-extracting-data-dont\n\n\n## ⚠️ Important Disclaimer\n\nConsult with your organization's security team before deployment. Ensure compliance with your security policies and standards.\n\n\n## 📝 Roadmap\n\n- ✅ **Multi-format exports** (CSV, JSON, Excel, Parquet)\n- 🔄 **Query caching** for repeated requests\n- 🔄 **SSL/TLS support** for secure connections\n\n## 📚 Real-World Examples\n\n### 📊 Export to Google sheets\n[View full code on Gist](https://gist.github.com/krokozyab/145eda5f8f53c997a9c0e918f3bc93bc)\n\n### 🐍 Python Data Pipeline\n```python\n# ETL Pipeline with Polars (Efficient Processing)\nimport polars as pl\nimport pyarrow.flight as fl\n\nclient = fl.connect(\"grpc://localhost:32010\")\n\n# Extract large dataset\nsql = \"SELECT * FROM gl_balances WHERE period_name = '2024-01'\"\ntable = client.execute(sql).read_all()\n\n# Transform with Polars\ndf = pl.from_arrow(table)\nresult = df.group_by(\"account_code\").agg(pl.col(\"amount\").sum())\n\n# Load to Parquet\nresult.write_parquet(\"monthly_balances.parquet\")\n```\n\n### 📊 Business Intelligence\n```bash\n#!/bin/bash\n# Daily reporting automation\n\n# Export financial data to Excel\nwget -O \"daily_report_$(date +%Y%m%d).xlsx\" \\\n  \"http://localhost:8081/export?sql=SELECT * FROM daily_summary\u0026format=excel\"\n\n# Upload to cloud storage\naws s3 cp daily_report_*.xlsx s3://reports-bucket/\n```\n\n### 🔄 Data Lake Integration\n```bash\n# Bulk export to data lake with Hive partitioning\nfor table in customers invoices payments; do\n  # Export with partitioning for better query performance\n  curl -o \"${table}_partitioned.zip\" \\\n    \"http://localhost:8081/export?sql=SELECT * FROM ${table}\u0026format=hive\u0026partition=region\"\n  \n  # Extract and upload partitioned structure\n  unzip \"${table}_partitioned.zip\" -d \"${table}_data/\"\n  hdfs dfs -put \"${table}_data/\" /data/oracle_fusion/\ndone\n```\n\n### 🌐 Web Application Integration\n```javascript\n// Fetch data for web dashboard\nfetch('http://localhost:8081/export?sql=SELECT * FROM kpis\u0026format=json')\n  .then(response =\u003e response.json())\n  .then(data =\u003e {\n    // Render charts and dashboards\n    renderDashboard(data);\n  });\n```\n\n### 🔍 Find Outliers in Accounting Entries\n[View full code on Gist](https://gist.github.com/krokozyab/dcd1b3758ebb388ea2327fc67bc51686)\n\n## 🔗 Other\n\n- **Root Project:** [ofjdbc - Oracle Fusion JDBC Driver](https://github.com/krokozyab/ofjdbc)\n\n- **Further reading:** Check out this article on Medium:  \n  [Simplifying Oracle Fusion Data Access with ofarrow](https://medium.com/@rudenko.s/simplifying-oracle-fusion-data-access-with-ofarrow-a78f59a18b12)\n\n\n## 📫 Contact\n\nFor questions or issues, reach out via GitHub Issues or [sergey.rudenko.ba@gmail.com](mailto:sergey.rudenko.ba@gmail.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrokozyab%2Fofarrow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrokozyab%2Fofarrow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrokozyab%2Fofarrow/lists"}