{"id":20513521,"url":"https://github.com/microwebstacks/markdown-rag-services","last_synced_at":"2025-04-13T23:55:51.702Z","repository":{"id":238540182,"uuid":"796778730","full_name":"MicroWebStacks/markdown-rag-services","owner":"MicroWebStacks","description":"understand the structure of markdown and its elements (tables, diagrams,...) for search and retrieval augmented generation","archived":false,"fork":false,"pushed_at":"2024-08-19T14:25:04.000Z","size":240,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T16:46:42.346Z","etag":null,"topics":["ast","embeddings","instant-search","llm","markdown","rag"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MicroWebStacks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":null,"patreon":null,"open_collective":"micro-web-stacks","ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2024-05-06T15:59:47.000Z","updated_at":"2024-12-30T22:29:38.000Z","dependencies_parsed_at":"2024-05-18T09:31:08.628Z","dependency_job_id":"e4544330-10f8-4f67-a91c-1e0ee7297c59","html_url":"https://github.com/MicroWebStacks/markdown-rag-services","commit_stats":{"total_commits":46,"total_committers":2,"mean_commits":23.0,"dds":0.04347826086956519,"last_synced_commit":"f569ed34064f21f5962dc7ca5fcc928c5bd6dd22"},"previous_names":["microwebstacks/markdown-service"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicroWebStacks%2Fmarkdown-rag-services","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicroWebStacks%2Fmarkdown-rag-services/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicroWebStacks%2Fmarkdown-rag-services/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MicroWebStacks%2Fmarkdown-rag-services/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MicroWebStacks","download_url":"https://codeload.github.com/MicroWebStacks/markdown-rag-services/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248799923,"owners_count":21163403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","embeddings","instant-search","llm","markdown","rag"],"created_at":"2024-11-15T21:11:37.052Z","updated_at":"2025-04-13T23:55:51.682Z","avatar_url":"https://github.com/MicroWebStacks.png","language":"Python","readme":"# Markdown RAG Services\nMarkdown Retrieval Augmented Generation Services.\n\nunderstand the structure of markdown and its elements (tables, diagrams,...) for search and retrieval augmented generation.\n\nMakrdown documents are parsed, the Abstract Syntax Tree is then used for rendering and for creating an intermediate document model that includes elements of types \n* meta data (from the md front matter and others e.g. path, level, order,...)\n* headings\n* tables\n* images\n* code blocks\n* paragraphs\n* links\n* references\n\nThose elements are the basis of a rich search using a schema and a smart LLM embeddings that allows the models to extract proper information from those elements.\n\n# Usage\nstart all services, which will execute the runner sequence and then keeps watching events\n```shell\ndocker compose up -d\n```\n\nor just the runner which will start all other services it depends on (mosquitto, fetcher, markdown, search)\n\n```shell\ndocker compose run --rm runner\n```\n\nstart is events driven mode, without the runner that sequence execution at startup but rather by watching sources only.\n\n```shell\ndocker compose run --profile events\n```\n\n# How does it work\n\nthe root `workflow.yaml` will be executed by the runner service that pulishes each action topics and subscribes to wait for the action to finish\n\nexample\n```yaml\n- action: fetcher/fetch\n  type: github\n  repository: MicroWebStacks/astro-big-doc\n  ref: main\n  filter: content/*\n  resource: test-website\n- action: markdown/build\n  resource: test-website\n  path: /fetch/test-website/content\n- ...\n```\n\n# Services\n* mosquitto : MQTT broker to manage services end points\n* runner : bootstraps execution of actions from a yaml workflow file\n* fetcher : executes fetch actions such as retrieve repos from github\n* markdown : parse, index and render markdown\n  * Markdown Content Parser https://github.com/MicroWebStacks/content-structure\n  * Markdown Renderer https://github.com/MicroWebStacks/astro-big-doc\n* diagrams-viewer : a webapp to test diagrams generation\n* kroki https://kroki.io/ as diagrams generator with REST API. supported formats (plantuml, graphviz, mermaid, drawio,...). It also uses the following services\n  * blockdiag\n  * mermaid\n  * bpmn\n  * diagramsnet\n* typesense : instant search engine https://typesense.org/docs/\n* search : service that collects elements from documents and injects them in the typesense database.\n* llm : embeddings and retrieval augmented generation\n\n# Development\nall services can also run without containers, as the directories mapping respect the same relative path usage outside and inside the containers.\n\nexample\n```shell\ncd fetcher\npython3 -m venv venv\npip install -r requirements.txt\nsource venv/bin/activate\n(venv)# python fetcher.py\n```\n\n# Administration\n\nexample creation of self signed certificates for testing purpose\n\n```bash\nopenssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./certs/localhost.key -out ./certs/localhost.crt -subj \"//CN=localhost\"\n```\n# Survey of related projects\n\n* https://github.com/deepset-ai/haystack\n  * embedding of markdown documents as plain text\n  * optional embedding of meta data for each document\n  * tables as structured element inside documents\n","funding_links":["https://opencollective.com/micro-web-stacks"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrowebstacks%2Fmarkdown-rag-services","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrowebstacks%2Fmarkdown-rag-services","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrowebstacks%2Fmarkdown-rag-services/lists"}