{"id":51502246,"url":"https://github.com/anapaulagomes/virola","last_synced_at":"2026-07-07T20:30:48.048Z","repository":{"id":369010349,"uuid":"1035128502","full_name":"anapaulagomes/virola","owner":"anapaulagomes","description":"[manuscript submitted for peer review] A framework for data-driven syndrome discovery from aggregated health records. 😷","archived":false,"fork":false,"pushed_at":"2026-07-02T11:37:25.000Z","size":2282,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T08:00:11.724Z","etag":null,"topics":["epidemic-intelligence","public-health-surveillance","snf","spectral-clustering","syndrome-discovery","syndromic-surveillance"],"latest_commit_sha":null,"homepage":"https://zenodo.org/records/21131658","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anapaulagomes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-09T18:02:24.000Z","updated_at":"2026-07-03T07:20:01.000Z","dependencies_parsed_at":"2026-07-03T08:01:03.964Z","dependency_job_id":null,"html_url":"https://github.com/anapaulagomes/virola","commit_stats":null,"previous_names":["anapaulagomes/virola"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/anapaulagomes/virola","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anapaulagomes%2Fvirola","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anapaulagomes%2Fvirola/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anapaulagomes%2Fvirola/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anapaulagomes%2Fvirola/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anapaulagomes","download_url":"https://codeload.github.com/anapaulagomes/virola/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anapaulagomes%2Fvirola/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35242497,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["epidemic-intelligence","public-health-surveillance","snf","spectral-clustering","syndrome-discovery","syndromic-surveillance"],"created_at":"2026-07-07T20:30:46.961Z","updated_at":"2026-07-07T20:30:48.040Z","avatar_url":"https://github.com/anapaulagomes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# virola\n\nFind public health threats to be monitored from your data.\n\n`virola` is dataset-agnostic: a framework plus pluggable **adapters**. Two adapters ship today:\n\n- **`datasus`** — Brazilian AESOP/DATASUS data (CID-10 + CIAP-2 + AB terminologies, 10 IBGE cities).\n- **`aktin`** — German AKTIN emergency-department data (ICD-10-GM, country-level stratum).\n\nEach adapter declares its own raw-format reader, terminology source, default embedding model,\nskip patterns, and stratum metadata. Everything downstream of the canonical row\n`(year, week, sex, age_group, code_type, code, quantity, terminology_id, terminology_label, [city])`\nis shared.\n\n## Licensing\n\nThe MIT license (see `LICENSE`) covers the **source code of this project only**.\n\nThe reference and auxiliary files bundled with each adapter — terminology tables,\nclassification files, code lists, and any other third-party data (e.g. CID-10, CIAP-2,\nAB, ICD-10-GM, AESOP code lists) — are **not** covered by this license. They remain\nsubject to the terms and licenses of their respective data sources, and their use and\nredistribution are the responsibility of those sources. Refer to each source for the\napplicable terms before reusing these files.\n\n## Setup (shared)\n\nInstall [uv](https://docs.astral.sh/uv/getting-started/installation/) (the project pins its Python\nversion in `.python-version`), then from the project root:\n\n```bash\nuv sync\nuv run plotly_get_chrome      # required by kaleido for plotly figure export\ncp .env.example .env          # set DATA_DIR / RAW_DATA_DIR / VIROLA_ADAPTER as needed\n```\n\nTo place the virtualenv outside the project: `UV_PROJECT_ENVIRONMENT=/path/to/.venv uv sync`.\n\n### Environment variables\n\nThe project reads `.env` at the project root if present. Recognised variables:\n\n- `DATA_DIR` — overrides the default `./data` location (raw, interim, processed, runs, `virola.db`).\n- `RAW_DATA_DIR` — overrides `data/raw/` only. Point at your adapter's raw export.\n- `VIROLA_ADAPTER` — default adapter for commands that take `--adapter`. CLI flag wins.\n- `LLM_MODEL` — Ollama model used by `vir results llm-descriptions` (default `llama3.2:1b`).\n- `HF_HOME` — cache directory for the SapBERT/XLMR downloads.\n- `HF_FROM_LIMITED_ENV` — set to a non-empty value to force a manual HF snapshot download\n  (SLURM / air-gapped). Empty value = regular environment.\n\n### CLI shape\n\n```text\n# Adapter-level commands — require --adapter (or VIROLA_ADAPTER env var)\nvir setup terminologies   --adapter \u003cname\u003e\nvir setup embeddings      --adapter \u003cname\u003e [--model MODEL]\n\n# Stratum-level commands — adapter is derived from the stratum registry\nvir prepare clean         --stratum \u003cname\u003e [--input PATH] [--output DIR]\nvir pipeline run          \u003cstratum\u003e        [--year-min ...] [--year-max ...] ...\nvir validate ablation     --stratum \u003cname\u003e\nvir results delete        --stratum \u003cname\u003e\n\n# Canonical-row commands — adapter via env or explicit flag\nvir prepare process       --dataset PATH   --adapter \u003cname\u003e\nvir prepare indicative-template --dataset PATH --adapter \u003cname\u003e\nvir view clinical         --dataset NAME   --adapter \u003cname\u003e\nvir view demographic      --dataset NAME\nvir view temporal         --dataset NAME\nvir model snf             --dataset NAME   --adapter \u003cname\u003e\nvir results build-profiles RUN_ID --dataset NAME\nvir results explain        RUN_ID --dataset NAME --adapter \u003cname\u003e\nvir results list\nvir results llm-descriptions RUN_ID\n```\n\n`--city` is accepted as a deprecated alias for `--stratum` (emits a warning).\n\nUse `uv run vir COMMAND --help` for detailed flag descriptions.\n\n---\n\n## DATASUS (Brazil)\n\n### Raw data\n\nThe pipeline expects an AESOP extraction at `data/raw/base_aesop_raw_extracao_01042026.parquet/`\n(multi-file parquet). Schema, period, and the IBGE municipality filter are documented in\n`data/raw/readme-base_aesop_raw_extracao_01042026.md`. Override the location with `RAW_DATA_DIR`\nin `.env`.\n\nThe 10 cities targeted by the pipeline are listed in `vir/adapters/datasus/bin/cities.txt`;\nthe same names are accepted by `vir pipeline run \u003cstratum\u003e`.\n\n### One-time setup\n\nReference files (`vir/adapters/datasus/references/RepositorioTerminologia_202506/tb_cid.csv`,\n`ciap2-cid10.csv`, `abp_ciap2.csv`) are committed. Build the joined terminology table and the\nSapBERT embeddings database once:\n\n```bash\nuv run vir setup terminologies --adapter datasus\nuv run vir setup embeddings    --adapter datasus\n```\n\n`setup embeddings` downloads `cambridgeltl/SapBERT-from-PubMedBERT-fulltext` to `HF_HOME`.\n\n### Run all cities\n\n`vir/adapters/datasus/bin/cities.txt` drives both helpers:\n\n```bash\nvir/adapters/datasus/bin/run_pipeline.sh all                 # clean → process → views → SNF per city\nvir/adapters/datasus/bin/enrich_latest_run.sh all            # build-profiles + explain + llm-descriptions\nuv run vir results list\n```\n\nExtra `vir pipeline run` flags pass through, e.g.\n`vir/adapters/datasus/bin/run_pipeline.sh all --skip-clean --n-clusters 150`.\n\n### Run a single city\n\n```bash\nuv run vir pipeline run \"Belo Horizonte\" --year-min 2022 --year-max 2024 \\\n    --snf-k 10 --n-clusters 200 --notes \"run for paper\"\n\nuv run vir results list\nvir/adapters/datasus/bin/enrich_latest_run.sh\n```\n\n`vir pipeline run` chains `clean → process → views (clinical, demographic, temporal) → model snf`\nand resolves the adapter (`datasus`) from the stratum name via the registry.\n\n### Step-by-step alternative\n\n```bash\nuv run vir prepare clean   --stratum \"Belo Horizonte\"\nuv run vir prepare process --dataset data/interim/cleaned_*belo_horizonte*.parquet \\\n                           --adapter datasus\n\nuv run vir view clinical    --dataset belo_horizonte --adapter datasus \\\n                            --year-min 2022 --year-max 2024\nuv run vir view demographic --dataset belo_horizonte --year-min 2022 --year-max 2024\nuv run vir view temporal    --dataset belo_horizonte --year-min 2022 --year-max 2024 --min-weeks 12\n\nuv run vir model snf --dataset belo_horizonte --adapter datasus \\\n    --year-min 2022 --year-max 2024 --snf-k 10 --n-clusters 200 --notes \"run for paper\"\n\nuv run vir results build-profiles 1 --dataset belo_horizonte\nuv run vir results explain 1 --dataset belo_horizonte --adapter datasus\nuv run vir results llm-descriptions 1                   # requires Ollama (see Optional below)\n```\n\n### SLURM\n\nTwo SLURM scripts mirror the helpers above; edit the `--mail-user` line at the top of each before\nsubmitting and set `DATA_DIR` (in `.env` or via `--export`) when the shared filesystem is not the\nproject root.\n\n```bash\nsbatch vir/adapters/datasus/bin/slurm_pipeline.sh                                  # full pipeline, every city\nsbatch --export=ALL,EXTRA_ARGS=\"--skip-clean\" vir/adapters/datasus/bin/slurm_pipeline.sh\n\nsbatch vir/adapters/datasus/bin/slurm_ablation.sh                                  # ablation array, one task per city\nsbatch --export=ALL,EXTRA_ARGS=\"--permutations 5000\" vir/adapters/datasus/bin/slurm_ablation.sh\n```\n\nThe ablation array index range (`#SBATCH --array=0-9` in `slurm_ablation.sh`) must match the\nnumber of lines in `vir/adapters/datasus/bin/cities.txt`.\n\n---\n\n## AKTIN (Germany)\n\nThe AKTIN adapter treats Germany as a single stratum (`\"germany\"`). Per-encounter raw data lives\nin `*_result/case_data.txt` + `diag_data.txt` pairs (tab-separated). The adapter pools every site\ninto one cleaned interim parquet.\n\n### Raw data\n\nEach AKTIN export ships per-hospital folders/zips. Either:\n\n- a directory of unzipped `\u003cN\u003e_result/` subdirectories under `RAW_DATA_DIR`, **or**\n- a directory containing a single site's `case_data.txt` and `diag_data.txt`.\n\nSet `RAW_DATA_DIR` in `.env` (absolute path, or leave blank to use `data/raw/`).\n\n### One-time setup\n\n```bash\nuv run vir setup terminologies --adapter aktin       # parses BfArM ICD-10-GM 2025 (vendored)\nuv run vir setup embeddings    --adapter aktin       # downloads SapBERT-UMLS-XLMR (~2.2 GB)\n```\n\nThe BfArM ICD-10-GM 2025 classification file is vendored at\n`vir/adapters/aktin/references/icd10gm2025syst_kodes.txt`. AKTIN's default embedding model is\n`cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR` (multilingual; embeds German labels\nreasonably).\n\n### Run the pipeline\n\n```bash\nuv run vir pipeline run germany --year-min 2018 --year-max 2023 \\\n    --snf-k 10 --n-clusters 200 --notes \"first aktin run\"\n\nuv run vir results list\nuv run vir results build-profiles 1 --dataset germany\nuv run vir results explain 1 --dataset germany --adapter aktin\n```\n\n`vir pipeline run` derives the adapter (`aktin`) from the stratum `germany` via the registry and\nthreads it through every step.\n\n### Step-by-step alternative\n\n```bash\nuv run vir prepare clean   --stratum germany                                    # adapter derived\nuv run vir prepare process --dataset data/interim/cleaned_aktin_germany.parquet \\\n                           --adapter aktin\n\nuv run vir view clinical    --dataset germany --adapter aktin \\\n                            --year-min 2018 --year-max 2023\nuv run vir view demographic --dataset germany --year-min 2018 --year-max 2023\nuv run vir view temporal    --dataset germany --year-min 2018 --year-max 2023 --min-weeks 12\n\nuv run vir model snf --dataset germany --adapter aktin \\\n    --year-min 2018 --year-max 2023 --snf-k 10 --n-clusters 200\n\nuv run vir results build-profiles 1 --dataset germany\nuv run vir results explain 1 --dataset germany --adapter aktin\n```\n\n### Indicative-codes template\n\nAKTIN ships **without** an epidemiologist-labeled CSV. To generate the top-volume template for\nlabeling:\n\n```bash\nuv run vir prepare indicative-template \\\n    --dataset data/interim/cleaned_aktin_germany.parquet \\\n    --adapter aktin \\\n    --coverage 0.9\n# writes data/processed/indicative_codes_template_aktin.csv\n```\n\nUntil labels are supplied, AKTIN runs effectively with `--no-indicative` (only Z-chapter codes\nare filtered via `adapter.skip_patterns`).\n\n### Exploratory data analysis\n\nA starter notebook lives at `vir/adapters/aktin/notebooks/eda_aktin.py` (shape, demographics,\nICD-10-GM coverage, monthly/weekly temporal distribution):\n\n```bash\nuv run marimo edit vir/adapters/aktin/notebooks/eda_aktin.py\n```\n\n---\n\n## Adding a new adapter\n\nThe contract is the abstract base class `vir/adapters/base.py::Adapter`. Concrete adapters live\nunder `vir/adapters/\u003cname\u003e/` and own their cleaning, terminology, references, notebooks, bin\nscripts, and tests. Each adapter declares:\n\n- `name`, `default_model`, `terminology_csv_filename` — class attributes.\n- `clean(raw_path, output_path, stratum, **opts)` — read raw → write canonical interim parquet.\n- `build_terminology_labels()` — return `(code, code_type, description)` DataFrame.\n- `stratum_metadata(stratum)` — `{city, region, population_range}`.\n- `cleaned_filename(stratum)` — interim filename convention.\n- `strata`, `skip_patterns`, `code_types` — properties.\n- `indicative_codes_file` — optional `Path`, defaults to `None`.\n\nRegister the instance in `vir/adapters/__init__.py::ADAPTERS`. The registry detects stratum\ncollisions across adapters at import time. See `vir/adapters/datasus/` and `vir/adapters/aktin/`\nfor the two reference implementations.\n\n---\n\n## Optional\n\n### LLM descriptions\n\n`vir results llm-descriptions` is optional. To enable it, install\n[Ollama](https://ollama.com/download/), pull the model named in `LLM_MODEL` (default\n`llama3.2:1b`), and add the `llm-ollama` plugin:\n\n```bash\nollama pull llama3.2:1b\nuv run llm install llm-ollama\n```\n\n### Development\n\n```bash\nmake test            # or: uv run pytest\nmake lint            # ruff format --check \u0026\u0026 ruff check\n```\n\nInteractive marimo edit mode for exploration:\n\n```bash\nuv run marimo edit notebooks/view_clinical.py                              # framework\nuv run marimo edit vir/adapters/datasus/notebooks/view_demographic.py      # DATASUS\nuv run marimo edit vir/adapters/aktin/notebooks/eda_aktin.py               # AKTIN\n```\n\nTwo DATASUS notebook subfolders are organised by purpose:\n\n- `vir/adapters/datasus/notebooks/validation/` — paper-grade validation (ablation, cross-city,\n  reference syndromes, temporal C). The ablation notebook is also exposed as `vir validate\n  ablation`.\n- `vir/adapters/datasus/notebooks/sanity_checks/` — internal exploratory checks per view\n  (`sc_clinical`, `sc_demographic`, `sc_temporal`, feature transformation comparison,\n  temporal distance comparison). Not part of the paper's evidence base.\n\n## Notes\n\n### Terminology (DATASUS)\n\nABP008 is duplicated in the official AB terminology source\n(\u003chttps://integracao.esusab.ufsc.br/ledi/documentacao/estrutura_arquivos/dicionario-fai.html#listaciapcondicaoavaliada\u003e),\nmapping to two distinct conditions: Beribéri (ABP008-1) and Desnutrição (ABP008-2).\nThe suffixes `-1` / `-2` are added in the reference files (`abp_ciap2.csv`,\n`terminology_mapping.csv`) for disambiguation only — the raw data always carries the base code\n`ABP008` without any suffix.\n\n### AESOP validation files (DATASUS)\n\nThe anchor validation used AESOP syndrome indicators, downloaded from the project's\n[repo](https://github.com/cidacslab/AESOP-Data-Documentation/):\n\n- [code_list_arbovirus_apr2024.csv](https://raw.githubusercontent.com/cidacslab/AESOP-Data-Documentation/refs/heads/main/DataPipeline/documentation/code_list_arbovirus_apr2024.csv)\n- [code_list_uri_apr2024.csv](https://raw.githubusercontent.com/cidacslab/AESOP-Data-Documentation/refs/heads/main/DataPipeline/documentation/code_list_uri_apr2024.csv)\n\n### Indicative codes (DATASUS)\n\nAn expert reviewed the codes covering the top 90% of record volume and labeled each as indicative\n(carries an epidemiological signal) or non-indicative. Codes labeled non-indicative are excluded\nfrom the analysis; codes not yet reviewed (outside the top 90%, or reviewed but unlabeled) are\nretained.\n\nTo generate (or regenerate) the labeling template for a different time window:\n\n```bash\nuv run vir prepare indicative-template \\\n    --dataset data/interim/cleaned_*_all_cities.parquet \\\n    --adapter datasus \\\n    --coverage 0.9\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanapaulagomes%2Fvirola","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanapaulagomes%2Fvirola","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanapaulagomes%2Fvirola/lists"}