{"id":22929931,"url":"https://github.com/genocs/genocs-form-etl","last_synced_at":"2025-09-02T04:43:25.114Z","repository":{"id":230271776,"uuid":"777329246","full_name":"Genocs/genocs-form-etl","owner":"Genocs","description":"An Extract, Transform, Load (ETL) to upload form, process by means of Cognitive services and store on CosmosDB ","archived":false,"fork":false,"pushed_at":"2024-03-28T18:37:57.000Z","size":381,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-03-28T19:58:15.464Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Genocs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"open_collective":"genocs","custom":["https://www.buymeacoffee.com/genocs"]}},"created_at":"2024-03-25T16:39:28.000Z","updated_at":"2024-03-28T19:58:44.928Z","dependencies_parsed_at":"2024-03-28T19:58:25.829Z","dependency_job_id":"12c18ebb-c8b6-446d-a0c4-87a272a91854","html_url":"https://github.com/Genocs/genocs-form-etl","commit_stats":null,"previous_names":["genocs/genocs-form-etl"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Genocs/genocs-form-etl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fgenocs-form-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fgenocs-form-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fgenocs-form-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fgenocs-form-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Genocs","download_url":"https://codeload.github.com/Genocs/genocs-form-etl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fgenocs-form-etl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273233249,"owners_count":25068725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T10:19:44.719Z","updated_at":"2025-09-02T04:43:25.087Z","avatar_url":"https://github.com/Genocs.png","language":"JavaScript","funding_links":["https://opencollective.com/genocs","https://www.buymeacoffee.com/genocs"],"categories":[],"sub_categories":[],"readme":"# ETL (Extract Transform \u0026 Load) system\n\n**Based on Azure Functions Blob Storage, Cosmos DB, Document Intelligence and OpenAI**\n\nLanguage|Sample code|Tutorial|\n|--|--|--|\n|.NET|[Sample code](dotnet)|[.NET tutorial](https://learn.microsoft.com/en-us/azure/storage/blobs/blob-upload-function-trigger?tabs=azure-portal)|\n|JavaScript|[Sample code](javascript)| [JavaScript tutorial](https://learn.microsoft.com/en-us/azure/storage/blobs/blob-upload-function-trigger-javascript?tabs=storage-resource-visual-studio-code%2Ccomputer-vision-azure-portal%2Ccosmos-db-azure-portal)|\n|NodeJs|[Sample code](nodejs)| [NodeJs tutorial](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-typescript-get-started?tabs=azure-ad)|\n|Python|[Sample code](python)|[Python documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-python-get-started?tabs=azure-ad)|\n\n\nIn this tutorial, you'll learn how to upload an image to Azure Blob Storage and process it using Azure Functions and Computer Vision. You'll also learn how to implement Azure Function triggers and bindings as part of this process. Together, these services will analyze an uploaded image that contains text, extract the text out of it, and then store the text in a database row for later analysis or other purposes.\n\nYou can find the full tutorial on the Microsoft Docs site. Choose the language you prefer from the table above, and then follow the tutorial to create the sample code.\n[Original tutorial](https://learn.microsoft.com/en-us/azure/storage/blobs/blob-upload-function-trigger-javascript?tabs=storage-resource-visual-studio-code%2Ccomputer-vision-azure-portal%2Ccosmos-db-azure-portal)\n  \n\n## Architecture \n![image](genocs-etl.png)\n\n## Introduction\n\nThis sample code is part of a tutorial that shows you how to upload a file to Azure Blob Storage and process it using Azure Functions. The tutorial demonstrates how to create an Azure Function that is triggered by a new blob being uploaded to a container in Azure Blob Storage. The function reads the contents of the blob and logs the content to the console.\n\n## Getting started\n\nTo get started, you must have the following prerequisites:\n\n### Prerequisites    \n\nInstall the following tools:\n- [Visual Studio Code](https://code.visualstudio.com/)\n- [Azure Functions extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurefunctions)\n- [Azure Storage extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurestorage)\n- [Azure Functions Core Tools](https://docs.microsoft.com/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash)\n- [Python](https://www.python.org/downloads/)\n- [NodeJs](https://nodejs.org/)\n- [Azure Storage Emulator](https://docs.microsoft.com/azure/storage/common/storage-use-emulator)\n- [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/)\n- [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli)\n\n\n## Azure resources\n\nYou'll need the following Azure resources to complete the tutorial:\n- Azure Blob Storage account\n- Azure Cognitive Services account\n- Azure Function app\n- Cosmos DB account\n- Azure OpenAI account or OpenAI account\n\n\n\n## nodejs\n\nThe nodejs code is in the `nodejs` folder. The code is an Azure Function that is triggered by a new blob being uploaded to a container in Azure Blob Storage. The function:\n  1. Reads the contents of the blob\n  2. Call the Azure Document Intelligence to extract the file content\n  3. Call The OpenAI CHAT gpt-4 to generate a summary of the content based on the prompt entered.\n\nInside the folder you will find useful scripts to create the resources needed to run the function. Specifically, you will find how to connect the private blob Storage, how to Call the Document Intelligence and how to call the OpenAI API.\n\n\n### Set up\n\n1. Clone the repository\n\n    ```bash\n    git clone\n    ```\n2. Open the repository in Visual Studio Code\n\n    ```bash\n    cd ./genocs-form-etl/nodejs\n    code .\n    ```\n3. Install the required packages npm\n\n    ```bash\n    npm install\n    # use the following command to install the azure functions core tools in case an error is thrown\n    npm i -g azure-functions-core-tools@4.0.5801\n    ```\n4. Run the function locally\n\n    ```bash\n    npm run start\n    ```\n\n5. Open the Azure Storage Explorer and create a new container in the Azure Storage emulator\n\n6. Upload a file to the container\n\n7. Check the console output to see the contents of the uploaded file\n\n8. Check on CosmosDB the content of the analyzed file\n\nYou can debug the function locally by setting breakpoints in the code and using the Azure Functions extension in Visual Studio Code.\nWhen you are ready to deploy the function to Azure, you can use the Azure Functions extension to create a new function app and deploy the function to Azure.\nOfficial documentation [here](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash)\n\n\n## .NET\n\nThe .NET code is in the `dotnet` folder. The code is a simple Azure Function that is triggered by a new blob being uploaded to a container in Azure Blob Storage. The function reads the contents of the blob and logs the content to the console.\n\n### Set up\n\n1. Clone the repository\n\n    ```bash\n    git clone\n    ``` \n2. Open the repository in Visual Studio Code\n\n    ```bash\n    cd ./genocs-form-etl/dotnet\n    code .\n    ```\n3. Create a new Azure Function project\n\n    ```bash\n    func init --worker-runtime dotnet\n    ```\n4. Create a new Azure Function\n\n    ```bash\n    func new --name BlobTrigger --template \"Blob trigger\"\n    ```\n5. Run the function locally\n\n    ```bash\n    func start\n    ``` \n6. Open the Azure Storage Explorer and create a new container in the Azure Storage emulator\n\n7. Upload a file to the container\n\n8. Check the console output to see the contents of the uploaded file\n\n\n\n## Python\n\nThe python code is in the `python` folder. The code is a simple Azure Function that is triggered by a new blob being uploaded to a container in Azure Blob Storage. The function reads the contents of the blob and logs the content to the console.\n\n### Set up\n\n1. Clone the repository\n\n    ```bash\n    git clone\n    ```\n2. Open the repository in Visual Studio Code\n\n    ```bash\n    cd ./genocs-form-etl/python\n    code .\n    ```\n3. Create a virtual environment\n\n    ```bash\n    python -m venv .venv\n    ```\n4. Activate the virtual environment\n\n    ```bash\n    ./.venv/Scripts/activate\n    ```\n5. Install the required packages\n\n    ```bash\n    pip install -r requirements.txt\n    ```\n6. Create a new Azure Function project\n\n    ```bash\n    func init --worker-runtime python\n    ```\n7. Create a new Azure Function\n\n    ```bash\n    func new --name BlobTrigger --template \"Blob trigger\"\n    ```\n8. Install the Azure Storage Blob package\n\n    ```bash \n    pip install azure-storage-blob\n    ```\n9. Run the function locally\n\n    ```bash\n    func start\n    ```\n10. Open the Azure Storage Explorer and create a new container in the Azure Storage emulator\n\n11. Upload a file to the container\n\n12. Check the console output to see the contents of the uploaded file\n\n## Resources\n\n- [Azure Functions Python developer guide](https://docs.microsoft.com/azure/azure-functions/functions-reference-python)\n- [Azure Functions triggers and bindings](https://docs.microsoft.com/azure/azure-functions/functions-triggers-bindings)\n- [Azure Storage Blob package](https://pypi.org/project/azure-storage-blob/)\n- [Azure Functions Core Tools](https://docs.microsoft.com/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash)\n- [Azure Storage Emulator](https://docs.microsoft.com/azure/storage/common/storage-use-emulator)\n- [Azure Storage Explorer](https://azure.microsoft.com/features/storage-explorer/)\n- [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenocs%2Fgenocs-form-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgenocs%2Fgenocs-form-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenocs%2Fgenocs-form-etl/lists"}