{"id":15430508,"url":"https://github.com/lenisha/survey-data-assistant","last_synced_at":"2026-04-12T07:34:05.328Z","repository":{"id":245470194,"uuid":"818313455","full_name":"lenisha/survey-data-assistant","owner":"lenisha","description":"Demo of OpenAI Assistants API with data in SQL Server and code_interpreter visualizations","archived":false,"fork":false,"pushed_at":"2024-09-06T14:47:46.000Z","size":23069,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-18T06:16:06.940Z","etag":null,"topics":["assistants-api","openai","sql-server","vanna"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lenisha.png","metadata":{"files":{"readme":"README-Azure-Monitor.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-21T15:08:19.000Z","updated_at":"2024-09-06T14:47:50.000Z","dependencies_parsed_at":"2024-09-06T16:59:24.910Z","dependency_job_id":"b7b4e85e-b1f6-450c-b3b9-40dda71b1d66","html_url":"https://github.com/lenisha/survey-data-assistant","commit_stats":{"total_commits":39,"total_committers":1,"mean_commits":39.0,"dds":0.0,"last_synced_commit":"18d9d838f0facb8a02d704dfeb51037d32d7b44f"},"previous_names":["lenisha/survey-data-assistant"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fsurvey-data-assistant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fsurvey-data-assistant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fsurvey-data-assistant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lenisha%2Fsurvey-data-assistant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lenisha","download_url":"https://codeload.github.com/lenisha/survey-data-assistant/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245977108,"owners_count":20703589,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assistants-api","openai","sql-server","vanna"],"created_at":"2024-10-01T18:16:46.592Z","updated_at":"2026-04-12T07:34:05.286Z","avatar_url":"https://github.com/lenisha.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Application Insights / Azure Monitor \n\nThis demo is set up to view your telemetry in Application Insights / Azure Monitor both to capture the Open Telementry traces as well as evaluation of those traces (both by humans and by automated evaluation).\n\n\u003e Note: The way that LLM executions are captured in Open Telemetry, aka the semantic conventions, are still in development by the Open Telemetry community (see [here for the project that is tracking this](https://github.com/open-telemetry/community/blob/main/projects/llm-semconv.md)). The property names and structure used in this demo are based on the current best practices and are guaranteed to change as the OpenAI Semantic Conventions for LLMs are finalized.\n\n### View traces in Application Insights\n\nIn addition to the Promptflow Tracing UI, you can also view the traces in Application Insights. You already set the environment variable `APPLICATIONINSIGHTS_CONNECTION_STRING` in the `.env` file. The value should be the **connection string** of the Application Insights instance you want to use.\n![](images/app-insights-1.png)\n\nTo see the traces here, you can for instance use the End-to-End transaction details view in Application Insights. To get there, follow the clicks as shown in the image below:\n\n![](images/app-insights-2.png)\n\nWhich will give you a view like this:\n\n![](images/app-insights-3.png)\n\n### Building a Dashboard with Promptflow Telemetry\n\n\u003e Note: You can access the telemetry data both through the Application Insights instance and the associated Log Analytics Workspace, albeit using different table and column names (see [here for details](https://learn.microsoft.com/en-us/azure/azure-monitor/app/convert-classic-resource#table-structure)). We are choosing access through the Log Analytics Workspace since it is the recommended way for new projects and it allows Python access to the data which we will be using to create automated evaluations.\n\nTo have the Telemetry from Promptflow and your app show up in and Azure Portal Dashboard or a Grafana Dashboard, you can follow these steps:\n\n1. Navigate to the Log Analytics Workspace: In App Insights, got to Overview page and click on the Workspace link:\n![](images/dashboard-0.png)\n\n2. Under **Maximize your Log Analytics experience**, click on **View logs**:\n![](images/dashboard-1.png)\n\n3. In the Logs view, create a new query. Make sure to edit your query in KQL Mode so you have access to the different tables.\n\n![](images/dashboard-2.png)\n\nHere is an example of a query that shows the tokens used over time: \n\n```kql\nAppDependencies\n| extend\n    total_tokens = toint(Properties[\"llm.usage.total_tokens\"]),\n    prompt_tokens = toint(Properties[\"llm.usage.prompt_tokens\"]),\n    completion_tokens = toint(Properties[\"llm.usage.completion_tokens\"])\n| summarize sum(total_tokens), sum(prompt_tokens), sum(completion_tokens) by bin(TimeGenerated, 5m) \n| render timechart\n```\n\n\n4. Save the query and pin it to a dashboard. You can create a new dashboard or add it to an existing one.\n\n![](images/dashboard-3.png)\n\n5. Once pinned to the Dashboard, you can edit the title and move/resize the chart as you see fit.\n\n![](images/dashboard-4.png)\n\nTo manage your Azure Portal Dashboards, got to the Dashboard hub in the Azure Portal:\n![](images/dashboard-5.png)\n\nHere are a few KQL queries that you can use to get started with building your own dashboards:\n\n\u003e Note: As mentioned above, the property names and structure used in this demo are based on the current best practices and are guaranteed to change as the OpenAI Semantic Conventions for LLMs are finalized.\n\n- Average duration of OpenAI Chat calls model and time:\n```kql\nAppDependencies\n| where Name in (\"openai_chat_async\", \"Iterated(openai_chat)\", \"openai_chat\")\n| extend model = substring(tostring(Properties[\"llm.response.model\"]), 0, 30),\n         duration_sec = DurationMs / 1000\n| summarize avg(duration_sec) by bin(TimeGenerated, 1h), model\n| order by TimeGenerated asc \n| render timechart \n```\n\n- Average duration of OpenAI Chat calls by model:\n```kql\nAppDependencies\n| where Name in (\"openai_chat_async\", \"Iterated(openai_chat)\", \"openai_chat\")\n| extend model = substring(tostring(Properties[\"llm.response.model\"]), 0, 30)\n| project duration_sec = DurationMs / 1000, model \n| summarize avg(duration_sec) by model\n| render columnchart\n```\n\n- Average duration of Assistant Runs by model:\n```kql\nAppDependencies\n| where Name in (\"AssistantAPI.run\")\n| extend model = substring(tostring(Properties[\"llm.response.model\"]), 0, 30)\n| where model != \"\"\n| project duration_sec = DurationMs / 1000, model \n| summarize avg(duration_sec) by model\n| render columnchart\n```\n\n- Tokens used by model over time:\n```kql\nAppDependencies\n| where Name in (\"openai_chat_async\", \"Iterated(openai_chat)\", \"openai_chat\", \"AssistantAPI.run\")\n| extend\n    total_tokens = toint(Properties[\"llm.usage.total_tokens\"]),\n    prompt_tokens = toint(Properties[\"llm.usage.prompt_tokens\"]),\n    completion_tokens = toint(Properties[\"llm.usage.completion_tokens\"]),\n    model = substring(tostring(Properties[\"llm.response.model\"]), 0, 22)\n| where model != \"\"    \n| summarize prompt = sum(prompt_tokens), completion = sum(completion_tokens) by model\n| render columnchart \n```\n\n- Total tokens used by model/deployment\n```kql\nAppDependencies\n| where Name in (\"openai_chat_async\", \"Iterated(openai_chat)\", \"openai_chat\", \"AssistantAPI.run\")\n| extend\n    total_tokens = toint(Properties[\"llm.usage.total_tokens\"]),\n    prompt_tokens = toint(Properties[\"llm.usage.prompt_tokens\"]),\n    completion_tokens = toint(Properties[\"llm.usage.completion_tokens\"]),\n    model = substring(tostring(Properties[\"llm.response.model\"]), 0, 30)\n| summarize prompt = sum(prompt_tokens), completion = sum(completion_tokens) by model\n| render columnchart \n```\n\n\n- User votes over time:\n```kql\nAppTraces\n| where Properties[\"event.name\"] == \"gen_ai.evaluation.user_vote\"\n| extend vote = toint(Properties[\"gen_ai.evaluation.vote\"]) \n| project vote, OperationId, ParentId\n| join kind=innerunique AppDependencies on $left.OperationId == $right.OperationId and $left.ParentId == $right.Id\n| summarize down = countif(vote == 0), up = countif(vote == 1), up_percent = 100*avg(vote) by bin(TimeGenerated, 1d)\n| render timechart  \n```\n\n- Question Quality over time:\n```kql\nAppTraces\n| where Properties[\"event.name\"] == \"gen_ai.evaluation.InDomainQuestion\"\n| extend score = toint(Properties[\"gen_ai.evaluation.score\"]) \n| project score, OperationId, ParentId\n| join kind=innerunique AppDependencies on $left.OperationId == $right.OperationId and $left.ParentId == $right.Id\n| summarize low_score_count = countif(score \u003c= 3), high_score_count = countif(score \u003e 3) by bin(TimeGenerated, 1d)\n| render timechart \n```\n\nHere an example of a Grafana dashboard with the above queries:\n\n![](images/grafana-1.png)\n\n### Query the data in Azure Data Explorer\nIn addition to the Azure UX for the Azure Log Analytics workspace, and you can query the data from Azure Data Explorer (ADE) by follwing these steps:\n\n1. go to https://dataexplorer.azure.com/ and add a connection:\n\n\u003cimg src=\"images/ade-1.png\"  width=\"600\"/\u003e\n\nThen add the URL for you App Insights instance like so:\n\n\u003cimg src=\"images/ade-2.png\"  width=\"400\"/\u003e\n\nFollowing this format:\n`https://ade.loganalytics.io/subscriptions/\u003csubscription-id\u003e/resourcegroups/\u003cresource-group-name\u003e/providers/microsoft.operationalinsights/workspaces/\u003cworkspace-name\u003e`\n\nThis will allow you to execute the same queries as above but in a more developer-friendly fronted. You can then take them back to Azure Monitor or Grafana and pin them to a Dashboard.\n\nIn addition, you can pull data from App Insights to build datasets for validation and fine tuning. For instance, in our example the sub-flow that provides the sales data insights is called `SalesDataInsights`. That means that you will find traces with that name from which you can retrieve the input and output parameters with a query like this:\n\n```kql\nAppDependencies\n| where Name == \"SalesDataInsights\" \n| extend inputs = parse_json(tostring(Properties.inputs)),\n         output = parse_json(tostring(Properties.output))\n| project question = inputs.question, query = output.query, error = output.error\n```\n\nThis will allow you to export the data to a CSV and then use for tasks like human evaluation or fine tuning.\n![](images/ade-3.png)\n\n\n### Programmatic access of telementry data\n\nAzure Monitor / Log Analytics also allows prgrammatic access to you telemetry data by executing KQL queries in Python (and other languages). To access the data you will need the GUID of the Log Analytics Workspace which you can get from the Overview page:\n\n![](images/log-analytics-1.png)\n\n\nHere is an example of how to execute a query on a Log Analytics Workspace using that `workspace_id`:\n\n```python\nimport os\nimport pandas as pd\nfrom datetime import datetime, timezone, timedelta\nfrom azure.monitor.query import LogsQueryClient, LogsQueryStatus\nfrom azure.identity import DefaultAzureCredential\nfrom azure.core.exceptions import HttpResponseError\n\ncredential = DefaultAzureCredential()\nclient = LogsQueryClient(credential)\nworkspace_id=\"********-****-****-****-**********\" \n\nquery = f\"\"\"\nAppDependencies\n| where Name == \"SalesDataInsights\" \n| extend inputs = parse_json(tostring(Properties.inputs)),\n         output = parse_json(tostring(Properties.output))\n| project question = inputs.question, query = output.query, error = output.error\n\"\"\"\n\nend_time=datetime.now(timezone.utc)\nstart_time=end_time - timedelta(days=1)\n\ntry:\n    response = client.query_workspace(\n        workspace_id=workspace_id,\n        query=query,\n        timespan=(start_time, end_time)\n        )\n    if response.status == LogsQueryStatus.PARTIAL:\n        error = response.partial_error\n        data = response.partial_data\n        print(error)\n    elif response.status == LogsQueryStatus.SUCCESS:\n        data = response.tables\n    for table in data:\n        df = pd.DataFrame(data=table.rows, columns=table.columns)\n        \nexcept HttpResponseError as err:\n    print(\"something fatal happened\")\n    print(err)\n\nprint(df)\n```\n\n### Automatic Evaluations\n\nThis demo provides a way to run evaluations against the telemetry collected. The result of the evaluation is then written back to the Application Insights instance as an open telemetry event (i.e. to the AppTraces table) to the relevant span, allowing you to query it through KQL and visualizing it in your dashboards.\n\nThe script requires the following environment variables to be set (e.g. in `.env`):\n- `OPENAI_API_BASE`, `OPENAI_API_KEY`, `OPENAI_API_VERSION`, `OPENAI_EVAL_MODEL`: The model configuration to be used to execute the evaluation prompty.\n- `LOG_ANALYTICS_WORKSPACE_ID`: The GUID of your log analytics workspace (to read the telemetry from) -- see above\n- `APPLICATIONINSIGHTS_CONNECTION_STRING`: The App Insights connection string (to write the evaluation events back) -- see above\n\nTo execute the script with defaults just run:\n```bash\npython src/evaluate/azure_monitor/eval_azure_monitor.py\n```\n\nBy default, the command will run the [in_domain_evaluator.prompty](src/custom_evaluators/in_domain_evaluator.prompty) over the output of this kql query (saved in [sales_data_insights.kql](src/evaluate/azure_monitor/sales_data_insights.kql)):\n\n```kql\nAppDependencies\n| where Name == \"SalesDataInsights\" \n| extend inputs = parse_json(tostring(Properties.inputs)),\n         output = parse_json(tostring(Properties.output)),\n         hash = hash(OperationId, 2)                            // select 1 in 2 traces\n| where hash==0\n| project question = inputs.question, query = output.query, error = output.error, trace_id = OperationId, span_id = Id, time_stamp = TimeGenerated\n| order by time_stamp asc\n```\n\nThe above query will return the input and output of spans of 50% of the traces (1 out of every 2) for the `SalesDataInsights` sub-flow along with the `trace_id`, `span_id` and `time_stamp` fields. The `trace_id` and `span_id` are used to write the evaluation results back to the App Insights instance as events under the respective span. The `time_stamp` is used to keep track of the last timestamp processed by the script, so subsequent exections won't process the same spans again.\n\nTo run with different evaluator, you can pass the path to the promtpy file as an argument to the script. **As you do that, make sure to also change the timestamp file to a new one to start from scratch.** Here is the usage of the script:\n\n```bash\nusage: eval_azure_monitor.py [-h] [--kql-file KQL_FILE] [--timestamp-file TIMESTAMP_FILE] [--evaluator-path EVALUATOR_PATH] [--dry-run]\n\nEvaluate Azure Monitor data\n\noptions:\n  -h, --help            show this help message and exit\n  --kql-file KQL_FILE   KQL query file. Default is sales_data_insights.kql\n  --timestamp-file TIMESTAMP_FILE\n                        Timestamp file. Default is in_domain_evaluator_time_stamp.txt\n  --evaluator-path EVALUATOR_PATH\n                        Evaluator path. Currently only prompty is supported. Default is in_domain_evaluator.prompty\n  --dry-run             When set, the script will not write to App Insights. Default is False.\n```\n\nTo view the evaluation results in a dashboard, you can use the following query:\n\n```kql\nAppTraces\n| where Properties[\"event.name\"] == \"gen_ai.evaluation.InDomainQuestion\"\n| extend score = toint(Properties[\"gen_ai.evaluation.score\"]) \n| project score, OperationId, ParentId\n| join kind=innerunique AppDependencies on $left.OperationId == $right.OperationId and $left.ParentId == $right.Id\n| summarize low_score_count = countif(score \u003c= 3), high_score_count = countif(score \u003e 3) by bin(TimeGenerated, 1d)\n| render columnchart \n```\n\n![](images/log-analytics-2.png)\n(you might need to set the visualization to stacked column chart under \"Chart Formatting\" to get this view)\n\nEnjoy exploring your Promptflow telemetry!\n\n![](images/sad-puppy.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenisha%2Fsurvey-data-assistant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flenisha%2Fsurvey-data-assistant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flenisha%2Fsurvey-data-assistant/lists"}