{"id":51510286,"url":"https://github.com/man4ish/omnibioai-data","last_synced_at":"2026-07-08T05:01:59.707Z","repository":{"id":367872833,"uuid":"1271415373","full_name":"man4ish/omnibioai-data","owner":"man4ish","description":"Runtime data directory for the OmniBioAI platform — directory structure, sample bioinformatics files, and registry configuration for bootstrapping a local instance.","archived":false,"fork":false,"pushed_at":"2026-06-28T03:55:38.000Z","size":5,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-28T05:21:26.735Z","etag":null,"topics":["bed","data-management","fasta","gff3","omnibioai","reference-data","sample-data"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/man4ish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-16T16:32:41.000Z","updated_at":"2026-06-28T03:55:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/man4ish/omnibioai-data","commit_stats":null,"previous_names":["man4ish/omnibioai-data"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/man4ish/omnibioai-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man4ish%2Fomnibioai-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man4ish%2Fomnibioai-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man4ish%2Fomnibioai-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man4ish%2Fomnibioai-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/man4ish","download_url":"https://codeload.github.com/man4ish/omnibioai-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/man4ish%2Fomnibioai-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35252324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bed","data-management","fasta","gff3","omnibioai","reference-data","sample-data"],"created_at":"2026-07-08T05:01:57.377Z","updated_at":"2026-07-08T05:01:59.679Z","avatar_url":"https://github.com/man4ish.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# omnibioai-data\n\nRuntime data directory for the OmniBioAI platform. This repository contains the directory structure, sample files, and registry configuration needed to bootstrap a local OmniBioAI instance.\n\nLarge runtime files (datasets, uploads, results, cached data, tool images) are excluded from version control via `.gitignore`. The directory structure is preserved via `.gitkeep` files.\n\n**v0.4.0-beta** — full reference genome registry (14 organisms), alignment indexes, variant databases, and AI knowledge base (35M+ PubMed abstracts, 108 FAISS indexes).\n\n## Data Scale\n\n| Category | Size |\n|----------|------|\n| Reference genomes \u0026 indexes | ~416 GB |\n| PubMed abstracts \u0026 FAISS indexes | ~195 GB |\n| **Total** | **~611 GB** |\n\n## Directory Structure\n\n```\nomnibioai-data/\n├── datasets/               # User datasets (not tracked)\n├── downloads/              # Downloaded files (not tracked)\n├── uploads/                # User uploads (not tracked)\n├── results/                # Analysis results (not tracked)\n├── reports/                # Generated reports (not tracked)\n├── logs/                   # Application logs (not tracked)\n├── cache/                  # Cached data (not tracked)\n├── igv_snapshots/          # IGV genome browser snapshots (not tracked)\n├── single_cell/            # Single-cell analysis data (not tracked)\n├── omni_object_storage/    # OmniBioAI object storage (not tracked)\n├── omni_objects/           # OmniBioAI objects (not tracked)\n├── objects/                # Runtime objects (not tracked)\n├── reference/              # Reference genomes \u0026 annotations (not tracked)\n│   ├── organisms/          # 14 organism genomes\n│   ├── indexes/            # STAR, BWA, Bowtie2, Salmon, CellRanger\n│   ├── variants/           # ClinVar, dbSNP, gnomAD, COSMIC\n│   ├── databases/          # GO, InterPro, Pfam, UniProt\n│   └── annotation/         # Ensembl, GENCODE, RefSeq, UCSC\n├── PubMed/                 # AI knowledge base (not tracked)\n│   ├── Abstracts/          # 35M+ PubMed abstracts (141 domains)\n│   └── Index/              # FAISS vector indexes (108 domains, 8 GB)\n├── scripts/                # Download and utility scripts\n├── tool-images/            # Tool container images (not tracked)\n├── omnibioai-tool-images/  # OmniBioAI tool images (not tracked)\n├── local_registry/         # Local model/tool registry (not tracked)\n├── object_registry.json    # Object registry index\n├── workflow_registry.json  # Workflow registry index\n└── sample files            # Sample data files for testing\n```\n\n## Reference Genomes\n\n14 organism genomes are stored under `reference/organisms/`, each at the canonical assembly version used by the platform.\n\n| Organism | Assembly |\n|----------|----------|\n| Human | GRCh37, GRCh38, T2T-CHM13 |\n| Mouse | GRCm38, GRCm39 |\n| Rat | GRCr8 |\n| Zebrafish | GRCz11 |\n| Drosophila | BDGP6 |\n| Yeast | R64 |\n| Chimpanzee | Pan_tro_3.0 |\n| Macaque | Mmul_10 |\n| *C. elegans* | WBcel235 |\n| Arabidopsis | TAIR10 |\n| Pig | Sscrofa11.1 |\n| Chicken | GRCg7b |\n\n## Alignment Indexes\n\nPre-built indexes are stored under `reference/indexes/`.\n\n| Index | Organisms |\n|-------|-----------|\n| STAR | All 14 organisms |\n| BWA | All 14 organisms |\n| Bowtie2 | All 14 organisms |\n| Salmon | All 14 organisms |\n| CellRanger (2024-A) | Human only |\n\n\u003e **Note:** The zebrafish (GRCz11) STAR index requires ~141GB RAM\n\u003e to build and cannot be generated on the DGX Spark (128GB).\n\u003e Zebrafish RNA-seq users should use Salmon (available) or\n\u003e Bowtie2 (available) for alignment. The STAR index will be\n\u003e added post-launch when built on a higher-memory instance.\n\n## Variant Databases\n\nVariant files are stored under `reference/variants/`.\n\n| Database | Build | Size |\n|----------|-------|------|\n| ClinVar | GRCh38 | 184 MB |\n| dbSNP (latest) | GRCh38 | 28 GB |\n| gnomAD v4.0 chr1 | GRCh38 | 65 GB |\n| COSMIC v104 | GRCh37 | 1.1 GB |\n| COSMIC v104 | GRCh38 | 1.1 GB |\n| GATK bundle (Mills + 1000G) | GRCh38 | — |\n\n\u003e **Note:** COSMIC download requires a registered account at [cancer.sanger.ac.uk](https://cancer.sanger.ac.uk). See `scripts/download_cosmic.sh` for the authenticated download workflow.\n\n## Functional Databases\n\nProtein and functional annotation databases are stored under `reference/databases/`.\n\n- **GO** — Gene Ontology\n- **InterPro** — protein family and domain signatures\n- **Pfam** — protein family database\n- **UniProt** — reviewed and unreviewed protein sequences\n\nGenome annotations (Ensembl, GENCODE, RefSeq, UCSC) are stored under `reference/annotation/`.\n\n## AI Knowledge Base\n\nThe `PubMed/` subtree provides a local biomedical AI knowledge base for RAG-powered queries within OmniBioAI.\n\n| Component | Details |\n|-----------|---------|\n| Abstracts | 35M+ PubMed abstracts across 141 biomedical domains |\n| FAISS indexes | 108 domain-specific vector indexes, 8.15 GB total |\n| Embedding model | `mxbai-embed-large` |\n\nIndexes are built per domain and queried at inference time by the OmniBioAI retrieval pipeline.\n\n## Scripts\n\nDownload and utility scripts are stored under `scripts/`.\n\n| Script | Purpose |\n|--------|---------|\n| `scripts/download_cosmic.sh` | Authenticated COSMIC v104 download (GRCh37 + GRCh38) |\n| `scripts/download_references.py` | Reference genome downloader for all 14 organisms |\n| `scripts/download_pubmed.sh` | PubMed abstracts bulk downloader |\n\n## Sample Files\n\nSmall sample files are included for testing and development:\n\n| File | Description |\n|------|-------------|\n| `sample1.csv` | Sample expression matrix |\n| `sample2.csv` | Sample expression matrix |\n| `sample1_normalize.csv` | Normalized expression matrix |\n| `sample2_normalize.csv` | Normalized expression matrix |\n| `sample_annotations.gff3` | Sample genome annotations (GFF3) |\n| `sample_annotations.tsv` | Sample annotations (TSV) |\n| `sample_regions.bed` | Sample genomic regions (BED) |\n| `sample_sequences.fasta` | Sample sequences (FASTA) |\n\n## Setup\n\nThis directory is referenced in the OmniBioAI docker-compose stack via the `DATA_DIR` environment variable:\n\n```env\nDATA_DIR=/home/manish/Desktop/machine/omnibioai-data\n```\n\nOn first run, create the required runtime directories:\n\n```bash\nmkdir -p datasets downloads uploads results reports logs cache \\\n         igv_snapshots single_cell omni_object_storage omni_objects \\\n         objects tool-images omnibioai-tool-images local_registry \\\n         reference/organisms reference/indexes reference/variants \\\n         reference/databases reference/annotation \\\n         PubMed/Abstracts PubMed/Index \\\n         scripts\n```\n\n### Reference Genomes\n\nDownload reference genomes for all supported organisms:\n\n```bash\npython scripts/download_references.py --all --outdir reference/organisms\n```\n\nTo download a single organism:\n\n```bash\npython scripts/download_references.py --organism human --assembly GRCh38 \\\n    --outdir reference/organisms\n```\n\n### Building Indexes\n\nAfter downloading genomes, build alignment indexes:\n\n```bash\n# STAR index (example — human GRCh38)\nSTAR --runMode genomeGenerate \\\n     --genomeDir reference/indexes/star/human_GRCh38 \\\n     --genomeFastaFiles reference/organisms/human/GRCh38/genome.fa \\\n     --sjdbGTFfile reference/annotation/human/GRCh38/genes.gtf \\\n     --runThreadN 16\n\n# Salmon index\nsalmon index \\\n    -t reference/organisms/human/GRCh38/transcriptome.fa \\\n    -i reference/indexes/salmon/human_GRCh38\n```\n\n### COSMIC Variant Database\n\nCOSMIC requires a registered account. Set your credentials before running the download script:\n\n```bash\nexport COSMIC_EMAIL=\"your@email.com\"\nexport COSMIC_PASSWORD=\"your-password\"\nbash scripts/download_cosmic.sh\n```\n\n## Related Repositories\n\n- [`omnibioai`](../omnibioai) — main Django workbench application\n- [`omnibioai-tes`](../omnibioai-tes) — task execution service\n- [`omnibioai-tool-images`](../omnibioai-tool-images) — tool container image definitions\n- [`omnibioai-db-init`](../omnibioai-db-init) — database initialization scripts\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fman4ish%2Fomnibioai-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fman4ish%2Fomnibioai-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fman4ish%2Fomnibioai-data/lists"}