{"id":23408474,"url":"https://github.com/agithomas/obs-integrations-fields-insights","last_synced_at":"2025-07-20T06:04:17.851Z","repository":{"id":279066865,"uuid":"904817556","full_name":"agithomas/obs-integrations-fields-insights","owner":"agithomas","description":"Agentic workflow to search and find more details of an observability field.","archived":false,"fork":false,"pushed_at":"2025-02-24T05:14:28.000Z","size":82,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-16T10:45:19.852Z","etag":null,"topics":["agent-based","generative-ai","langchain","langgraph","observability"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agithomas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-17T15:56:44.000Z","updated_at":"2025-02-28T06:14:20.000Z","dependencies_parsed_at":"2025-02-23T15:42:02.940Z","dependency_job_id":null,"html_url":"https://github.com/agithomas/obs-integrations-fields-insights","commit_stats":null,"previous_names":["agithomas/obs-integrations-fields-insights"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/agithomas/obs-integrations-fields-insights","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agithomas%2Fobs-integrations-fields-insights","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agithomas%2Fobs-integrations-fields-insights/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agithomas%2Fobs-integrations-fields-insights/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agithomas%2Fobs-integrations-fields-insights/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agithomas","download_url":"https://codeload.github.com/agithomas/obs-integrations-fields-insights/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agithomas%2Fobs-integrations-fields-insights/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266076026,"owners_count":23872729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent-based","generative-ai","langchain","langgraph","observability"],"created_at":"2024-12-22T15:12:20.939Z","updated_at":"2025-07-20T06:04:17.807Z","avatar_url":"https://github.com/agithomas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Observability Integrations Fields Insights (Beta)\n\nThe Observability Integration Fields Insight project bridges gaps in the Observability integration field details. The amount of details added as part of the fields description is often limited due to various reasons including \n- the number of fields added as part of an integration\n- developers or contributors might assume that the users of the integration already familiar with the observed fields, leading to less emphasis on the documentation\n- developers may rely on the documentation of the services or products for which integration is built\n- with a large number of fields, writing detailed descriptions for each one can become a repetitive and low-priority task. \n\nThis project provides extended descriptions for each field, highlighting their relevance and importance in determining service or product availability. Using the name of the field and field description, it pulls insights from a variety of online sources, storing them in databases for reuse. The data is organized into a report schema, supporting both QA and decision-making needs.\n\n\u003e **Warning!!!:** Since the report content generated by running this project ( also used to generate alert threshold or slo settings, alert baselining, etc), is using the contents from random websites, it is important the users validate the sources of these contents (mentioned the report) and the correctness of data before using for Q/A or taking decisions or to be used in other applications.\n\nIt provides advanced insights into observability fields, supporting diverse use cases such as generating asset presets for Alerts and SLOs, extending the knowledge base of Elastic's Observability AI Assistant. With a modular architecture, it operates independently or integrates seamlessly with Elastic Observability to enrich its capabilities. \n\nBy generating recommendations for thresholds, critical metrics, and field relationships, the project simplifies alert creation and validation. It also extends the Elastic's Observability AI Assistant AI assistant’s question-answering capabilities, empowering users to streamline workflows and enhance the observability ecosystem.\n\n## Requirements\n\n- `python 3.11+` and uses poetry for package dependencies\n- Elasticsearch cluster deployed on [Elasticsearch Service](https://www.elastic.co/cloud/) or [download and run](https://www.elastic.co/downloads/) Elasticsearch. Create a key [in Kibana](https://www.elastic.co/guide/en/kibana/8.17/api-keys.html) or [using the Elasticsearch API](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/security-api-create-api-key.html)\n- Visit the [OpenAI API platform](https://platform.openai.com/), sign in, and navigate to the \"API Keys\" section under your account settings to create a new API key.\n- Visit the [Tavily API platform](https://www.tavily.com/), sign in or create an account, then go to the \"API Keys\" section in your account settings to generate a new API key.\n- MySQL 8.1\n- [Langsmith](https://smith.langchain.com/) account (recommended)\n- [Langgraph-studio](https://github.com/langchain-ai/langgraph-studio) (recommended). LangGraph Studio requires docker-compose version 2.22.0+ or higher.\n\n\n## Setup\n\n## Install pip3 and poetry\n```\npython3 -m ensurepip --default-pip\npip install poetry\n```\n### Set up **MySQL 8.1** Docker instance and create a database and table:\n```\n# Pull the MySQL 8.1 Docker image:\ndocker pull mysql:8.1\n\n# Run a MySQL container:\ndocker run --name mysql-container -e MYSQL_ROOT_PASSWORD=rootpassword -d mysql:8.1\n\n# Create a new database:\nCREATE DATABASE your_database_name;\n\n# Execute the SQL script from the install/mysql.sql file to create the required table(s) by running:\nsource /path/to/install/mysql.sql\n```\n\n### Clone repo\n```\n# This is a private repo. Contact agi.thomas@elastic.co for GITHUB TOKEN\ngh repo clone agithomas/obs-integrations-fields-insights\n```\n\n### Move to directory and use poetry to setup venv, install dependencies:\n```\n[ec2-user@ip-172-31-43-26 obs-integrations-fields-insights]$ cd obs-integrations-fields-insights/\n\n[ec2-user@ip-172-31-43-26 obs-integrations-fields-insights]$ poetry shell\n\n(asset-preset-config-generator-py3.11) [ec2-user@ip-172-31-43-26 obs-integrations-fields-insights]$ poetry install\nUpdating dependencies\nResolving dependencies... (10.7s)\n\nPackage operations: 89 installs, 0 updates, 0 removals\n\n  - Installing certifi (2024.12.14)\n  ........\n  - Installing neo4j (5.27.0)\n  - Installing pytest (8.3.4)\n  - Installing tavily-python (0.5.0)\n\nWriting lock file\n\nInstalling the current project: asset-preset-config-generator (0.1.0)\n\n```\n### Configure Environment Variables\n\n```\ncp example.env .env\n```\n\n| Field Name         | Description                                                  |\n|--------------------|--------------------------------------------------------------|\n| `ES_INDEX`         | The vector store index name.                                 |\n| `ES_INDEX_RELEVANCE` | The name of the index for storing the output of the framework run. |\n| `LANGCHAIN_TRACING_V2` | Set to \"true\" to enable langchain tracing. |\n\n\n## Run the service\n```\nlanggraph up\n```\n\n```\nasset-preset-config-generator-py3.11(base) agikthomas@Agis-MBP asset-preset-config-generator % langgraph up                                    \nStarting LangGraph API server...\nFor local dev, requires env var LANGSMITH_API_KEY with access to LangGraph Cloud closed beta.\nFor production use, requires a license key in env var LANGGRAPH_CLOUD_LICENSE_KEY.\nReady!       \n- API: http://localhost:8123\n- Docs: http://localhost:8123/docs\n- LangGraph Studio: https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:8123\n```\n\nUse the `https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:8123` to access the langgraph studio's Web UI for testing.\n\n## Connecting research report data with Elastic's Observability AI Assistant (Optional)\n\nElastic search connectors offers Elastic connectors to sync the data from a variety of sources. The  data stored in the MySQL instance created as part of the framework run can be sync-ed up with Elasticsearch using the [ES Connector for MySQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors-mysql.html). \n\n- Clone the repo : https://github.com/elastic/connectors\n- Login to Kibana, Go to Search -\u003e Content -\u003e Connectors. \n  - Configure a new connector.\n  - Mention the name of an index. This is the index that will have the data of the sync-ed data from MySQL instance. \n      - Example : `obs_assistant_asset_preset_mysql`\n  - Provide the connectivity and authentication details.\n- Update the config.yml as\n  ```\n    connectors:\n    -\n      connector_id: \"\u003cconnector-id\u003e\"\n      service_type: \"mysql\"\n\n    elasticsearch:\n      host: \"\u003celasticsearch endpoint\u003e\"\n      api_key: \"\u003celastisearch api key\u003e\"\n  ```\n- `cd connectors \u0026\u0026 make run`\n- Configure AI Assistant Settings (Optional)\n  - Go to Kibana -\u003e Observability -\u003e Overview\n  - Select AI Assistant Settings\n  - Enter the name of the Search connector index pattern.\n     - Example : `obs_assistant_asset_preset_mysql`\n\n- Follow the steps to [connect AI assistant with search connector](https://www.elastic.co/guide/en/observability/current/obs-ai-assistant.html#obs-ai-search-connectors) and perform the sync operation.\n\n![image](https://github.com/user-attachments/assets/ab9ccb81-9933-48d0-8630-6a3ffe19555a)\n\n\n## Langgraph data flow\n\u003cimg width=\"843\" alt=\"image\" src=\"https://github.com/user-attachments/assets/88669612-6369-4c35-8529-9934d0f9d0ef\" /\u003e\n\n## Troubleshooting\n\n### Connectivity issue\n```\n'NewConnectionError('\u003cpip._vendor.urllib3.connection.HTTPSConnection object at 0x7f2c51c0a610\u003e: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/poetry-core/\n```\nResolution\n```\nsudo nano /etc/docker/daemon.json\n\nAdd the following entries\n\n{\n  \"dns\": [\"8.8.8.8\", \"8.8.4.4\"]\n}\n\nsudo systemctl restart docker\n\n```\n\n### Unable to connect with MySQL\n\nIf you're using Docker Desktop (on macOS or Windows) and the container is running on host networking mode, you can use `host.docker.internal`. However, this doesn't work in Linux or EC2 environments. Make sure that the mysql instance is accessbible from the langgraph docker image created when running `langgraph up`\n\n\n### Other errors\n\nUse the verbose option to debug the issue\n```\nlanggraph up --verbose\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagithomas%2Fobs-integrations-fields-insights","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagithomas%2Fobs-integrations-fields-insights","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagithomas%2Fobs-integrations-fields-insights/lists"}