{"id":15430535,"url":"https://github.com/lenisha/knowledge-mining","last_synced_at":"2026-07-22T10:32:23.495Z","repository":{"id":116070740,"uuid":"529417697","full_name":"lenisha/knowledge-mining","owner":"lenisha","description":"fork from https://github.com/SenthuranSivananthan/knowledge-mining","archived":false,"fork":false,"pushed_at":"2022-12-08T14:16:49.000Z","size":2688,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-20T04:23:53.773Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lenisha.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-26T22:12:24.000Z","updated_at":"2022-08-26T22:20:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"802e5aeb-1f51-45d6-8f92-eef80511b5da","html_url":"https://github.com/lenisha/knowledge-mining","commit_stats":{"total_commits":1,"total_committers":1,"mean_commits":1.0,"dds":0.0,"last_synced_commit":"a2354a0cb06fd47e6303bf0801c4d66b6ed6bcaa"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lenisha/knowledge-mining","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fknowledge-mining","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fknowledge-mining/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fknowledge-mining/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fknowledge-mining/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lenisha","download_url":"https://codeload.github.com/lenisha/knowledge-mining/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fknowledge-mining/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35759166,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-22T02:00:06.236Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T18:16:50.290Z","updated_at":"2026-07-22T10:32:23.474Z","avatar_url":"https://github.com/lenisha.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThe implementation in this repository is based [Knowledge Mining Solution Accelerator](https://docs.microsoft.com/en-us/samples/azure-samples/azure-search-knowledge-mining/azure-search-knowledge-mining/).\n\nThis repository aims to simplify the onboarding experience through automation.  Therefore, the following changes/additions have been made to the original solution:\n\n1. Converted the ARM templates to [Bicep](https://github.com/Azure/bicep) templates for easier Infrastructure as Code authoring experience.\n\n2. Scripted the Azure Search Index creation through bash to avoid any manual steps through Postman or REST APIs.\n\n3. Scripted deployment using Azure DevOps Pipelines for push-button deployment.\n\n# Azure Services\n\nThe solution uses 5 Azure PaaS Services:\n\n1. **Azure App Service** - Hosting environment for the .NET Core Web Application\n2. **Azure Cognitive Services** - Built-in Machine Learning capabilities to extract insights from documents, images, etc.\n3. **Azure Search** - Search index to explore the insights\n4. **Azure Key Vault** - Secrets \u0026 key management\n5. **Azure Storage Account** - Document repository\n\n# Service Principal Account\n\nAutomated deployment will require one Service Principal Account.  This account is used for:\n\n1. Deploying the Azure Infrastructure\n2. Integrating with Azure Key Vault for key management\n\nThe service principal account will need `Owner` Role assigned (IAM) at the Subscription.  This configuration will allow multiple deployment of the same solution in the same subscription.\n\nOnce the service principal is created, document the following information in a secure place:\n\n1. Enterprise Application Object Id\n2. Application Id\n3. Application Secret\n4. Tenant Id\n\n# Configure Azure DevOps Pipeline\n\nAll Azure DevOps pipeline will use the service principal to interact with Azure.\n\n## Configure Service Connection\n\n1. Navigate to `Project Settings`\n2. Navigate to Pipelines -\u003e `Service Connections`\n3. Click on `New Service Connection`\n4. Click on `Azure Resource Manager`, Click Next\n5. Click on `Service Principal (manual)`, Click Next\n6. Ensure Environment is set to `Azure Cloud`\n7. Ensure Scope Level is set to `Subscription`\n8. Enter the `Subscription Id` for your subscription\n9. Enter a Name for the Subscription\n10.  Enter Application Id of the Service Principal in `Service Principal Id`\n11.  Enter Application Secret of the Service Principal in `Service Principal Key`\n12.  Enter `Tenant Id`\n13.  Click Verify to validate the connection.  This step will ensure that the Service Principal Account has the required permissions to subscriptoin.\n14. Enter `Service connection name`.  This will be used in the Azure Pipelines to reference the connection information.  i.e. `azure-knowledge-mining-arm-connection`\n15. Ensure `Grant access permission to all pipelines` is checked\n16. Click `Verify and Save`\n\n## Configure 3 DevOps Pipelines\n\nThe Service Connection that was setup earlier will be used in each of the pipelines.  The pipeline YAMLs have `azure-knowledge-mining-arm-connection` as the reference to the connection.  This can be changed to your connection name.\n\nLook for references like:\n\n```json\n    azureSubscription: 'azure-knowledge-mining-arm-connection'\n```\n\n1. Click `Pipelines` from the left menu\n2. Click `New Pipeline` button\n3. Select `Azure Repos Git`\n4. Click Azure Repository that has the pipelines\n5. Click `Existing Azure Pipelines YAML file`\n\n### Azure Infrastructure\n\n1. Choose `/azure-pipelines-infra.yml`\n2. Click `Continue`\n3. Click Variables and create the following\n\n    Click on `Let users override this value when running this pipeline` for each variable.\n\n    - LOCATION (value as: CanadaCentral)\n    - RESOURCEGROUP (value as:  knowledge-mining-dev)\n    - SPNOBJECTID (value as: `Enterprise Application Object Id` of the Service Principal from above)\n\n4. Click Save\n5. Click Run\n6. Monitor the job to ensure it is successful.  Check the Azure subscription for the new resource group based on the variable above.\n\n### Azure Search\n\n1. Choose `/azure-pipelines-search.yml`\n2. Click `Continue`\n3. Click Variables and create the following\n\n    Click on `Let users override this value when running this pipeline` for each variable.\n\n    - KEYVAULT (value as: The name of the Azure Key Vault instance.  You will find this name in the Resource Group.  i.e. `akv-j3toevaceiwp4`)\n\n4. Click Save\n5. Click Run\n6. Monitor the job to ensure it is successful.  Navigate to the Azure Search instance deployed in the Resource Group to review the configuration.  You will see `Index`, `Indexer`, `Data sources` and `Skillsets` configured.\n\n\n### Application\n\n1. Choose `/azure-pipelines-app.yml`\n2. Click `Continue`\n3. Click Variables and create the following\n\n    Click on `Let users override this value when running this pipeline` for each variable.\n\n    - APPSERVICENAME (value as: The name of the Azure App Service instance.  You will find this name in the Resource Group.  i.e. `site-j3toevaceiwp4`)\n\n4. Click Save\n5. Click Run\n6. Monitor the job to ensure it is successful.  Navigate to the Azure App Service instance deployed in the Resource Group and click Browse.  This will launch the URL of the application.\n\n\n## Automated Deployments\n\nAll 3 pipelines are configured to automatically trigger when changes are made to their respective folders.\n\nAll you need to do is push/merge changes into `app`, `arm` and `search-index` folders.  Once complete, their respective pipelines will be automatically triggered.\n\n## Manual deployment\n\nCould be done fully in an Azure Cloud Shell (Bash)\n\n- git clone this repo\n\n- build Infra\n\n  -- Verify Subscription\n  ```\n  az account set -s \"\u003csubscription name\u003e\"\n  ```\n  -- [Optional] Create App Registration and note ObjectID, you could also use your current user Identity\n  \n  ```\n  az ad user list --upn \u003cuser email\u003e\n  ``` \n\n  -- Create Resource Group and grant Contributor access to App Registration\n ```\n az group create -l \"Canada Central\" -n \u003cRG NAME\u003e\n  ```\n \n  -- Run Bicep Deployment to provision Infrastructure\n\n```\ncd ~/knowledge-mining/arm\naz deployment group create -g \u003cRG NAME\u003e --template-file env-vnet-integration.bicep --parameters docsContainerName=documents spnObjectId=\u003cobjectID-of-you-or-appregistration\u003e\n```\n\n- Build and Deploy Custom Synonym Mapping Skill\n\nNote: Upload to storage account to container `synonyms` file `thesaurus.json` sample in `skills\\Distinct` dir\n\n```\ncd ~/knowledge-mining/skills\nchmod +x builddeploy.sh\n./builddeploy.sh \u003cRG NAME\u003e \u003cFUNCTION NAME\u003e \"\u003cSTORAGE ACCOUNT with SYNONYMS\u003e\"\n```\n\nExample:\n```\n./builddeploy.sh gackm function-app-e6y6xkw7ancu6 \"DefaultEndpointsProtocol=https;AccountName=stge6y6xkw7an;AccountKey=ddd==;EndpointSuffix=core.windows.net\"\n```\n\n\n- build Search Configuration (index, indexer, skillset)\n```\ncd ~/knowledge-mining/search-index\nchmod +x deploy.sh\n./deploy.sh ~/knowledge-mining/search-index \u003cSTORAGE RESID\u003e  documents \u003cSEARCH ENDPOINT\u003e \u003cSEARCH KEY\u003e \u003cCOG SERVICE KEY\u003e \u003cFUNCTION APPNAME\u003e \u003cFUNCTION KEY\u003e\n```\n\nExample:\n```\n./deploy.sh ~/knowledge-mining/search-index /subscriptions/xxxxx/resourceGroups/gackm/providers/Microsoft.Storage/storageAccounts/storageaccount  documents https://search-xxxxx.search.windows.net DDXXXXX b97a864ccc3a4xxxx  function-app-zzzz Axxxxx==\n```\n\nNote: parameters could be copied from deployment output and keyvault secrets and Funcion itself\n\n\n- Build and Deploy Search Application to App Service\n```\ncd ~/knowledge-mining/app\nchmod +x builddeploy.sh\n./builddeploy.sh \u003cRG NAME\u003e \u003cAPPSVC NAME\u003e\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenisha%2Fknowledge-mining","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flenisha%2Fknowledge-mining","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenisha%2Fknowledge-mining/lists"}