{"id":33106901,"url":"https://github.com/bp-kelley/descriptastorus","last_synced_at":"2025-11-16T02:01:32.257Z","repository":{"id":41349740,"uuid":"162162924","full_name":"bp-kelley/descriptastorus","owner":"bp-kelley","description":"Descriptor computation(chemistry) and (optional) storage for machine learning","archived":false,"fork":false,"pushed_at":"2024-10-26T19:43:46.000Z","size":53986,"stargazers_count":229,"open_issues_count":2,"forks_count":66,"subscribers_count":11,"default_branch":"master","last_synced_at":"2024-10-26T21:46:05.593Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bp-kelley.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-17T16:56:09.000Z","updated_at":"2024-10-26T19:43:49.000Z","dependencies_parsed_at":"2024-04-29T12:55:10.282Z","dependency_job_id":null,"html_url":"https://github.com/bp-kelley/descriptastorus","commit_stats":{"total_commits":69,"total_committers":8,"mean_commits":8.625,"dds":0.5507246376811594,"last_synced_commit":"86eedc60546abe6f59cdbcb12025a61157ba178d"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/bp-kelley/descriptastorus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bp-kelley%2Fdescriptastorus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bp-kelley%2Fdescriptastorus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bp-kelley%2Fdescriptastorus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bp-kelley%2Fdescriptastorus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bp-kelley","download_url":"https://codeload.github.com/bp-kelley/descriptastorus/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bp-kelley%2Fdescriptastorus/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":284650210,"owners_count":27041043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-16T02:00:05.974Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-11-15T00:00:27.710Z","updated_at":"2025-11-16T02:01:32.251Z","avatar_url":"https://github.com/bp-kelley.png","language":"Python","funding_links":[],"categories":["Libraries","Machine Learning"],"sub_categories":["Molecular Descriptors"],"readme":"DescriptaStorus\n===============\n\nThe descriptastorus provides \n\n  1. fast random access to rows of properties suitable for\nmachine learning and \n  2. fast random access to indexed molecule files\n  3. A mechanism for generating new descriptors for new molecules\n  4. A mechanism for validating that you can recreate the same storage in different software/hardware environments\n  5. An easy script for making your own descriptor files from raw data.\n\n[n.b.] kyotocabinet is required to read/write the inchiKey and name indices\n  This should be installed in your environment.\n\nThere are three basic ways to use DescriptaStorus:\n  \n  1. Make a DescriptaStore using a script\n  2. Append new data to the store\n  3. Use a DescriptaStore to access properties\n\nInstalling\n==========\n\n```\n1. install rdkit\n2. install scikit-learn\npip install git+https://github.com/bp-kelley/descriptastorus\n```\n\nRequirements are in the setup.py file, but essentially:\n\n 1. python2/3\n 2. rdkit\n 3. [optional but highly recommended] kyotocabinet\n\nUsing RDKit descriptors\n=======================\nGrab a descriptor generator from the registry.\n\nCurrently registered descriptors:\n\n\t* atompaircounts\n\t* morgan3counts\n\t* morganchiral3counts\n\t* morganfeature3counts\n\t* rdkit2d\n\t* rdkit2dnormalized\n        * rdkit2dhistogramnormalized - much faster version of rdkit2dnormalized\n\t* rdkitfpbits\n\nDescriptors are input as a tuple or list to the generator.\n\n```\nfrom descriptastorus.descriptors.DescriptorGenerator import MakeGenerator\ngenerator = MakeGenerator((\"RDKit2D\",))\nfor name, numpy_type in generator.GetColumns():\n  print(\"name: {} data type: {}\".format(name, numpy_type))\n```\n\nThe resulting columns and datatypes look like:\n```\nname: RDKit2D_calculated data type: \u003cclass 'bool'\u003e\nname: BalabanJ data type: \u003cclass 'numpy.float64'\u003e\nname: BertzCT data type: \u003cclass 'numpy.float64'\u003e\nname: Chi0 data type: \u003cclass 'numpy.float64'\u003e\nname: Chi0n data type: \u003cclass 'numpy.float64'\u003e\nname: Chi0v data type: \u003cclass 'numpy.float64'\u003e\nname: Chi1 data type: \u003cclass 'numpy.float64'\u003e\n\n```\n\nNote: RDKit2D_calculated is just a flag for the store to indicate that the\nRDKit2D features were successfully calculated.\n\nTo get combine multiple generators simply add them to the list\nof desired datatypes:\n\n```\nfrom descriptastorus.descriptors.DescriptorGenerator import MakeGenerator\ngenerator = MakeGenerator((\"RDKit2D\", \"Morgan3Counts\"))\nsmiles = \"c1ccccc1\"\ndata = generator.process(smiles)\nassert data[0] is True\n```\n\nThe first element is True if the molecule was successfully processed, this is used\nin the descriptastor to indicate that the row is valid.\n\nIf a molecule is unsuccessfully processed, None is returned\n\n```\ndata = generator.process(\"not a smiles\")\nassert data is None\n```\n\nIndividual descriptor sets can also be used outside of the\ngenerator.\n\n```\nfrom descriptastorus.descriptors import rdNormalizedDescriptors\nfrom rdkit import Chem\nimport logging\n\n# make the normalized descriptor generator\ngenerator = rdNormalizedDescriptors.RDKit2DNormalized()\ngenerator.columns # list of tuples:  (descriptor_name, numpytype) ...\n\n# features = generator.process(smiles)\n# features[0] is True/False if the smiles could be processed correcty\n# features[1:] are the normalized descriptors as described in generator.columns\n\n# example for converting a smiles string into the values\ndef rdkit_2d_normalized_features(smiles: str):\n    # n.b. the first element is true/false if the descriptors were properly computed\n    results = generator.process(smiles)[\n    processed, features = results[0], results[1:]\n    if processed is None:\n       logging.warning(\"Unable to process smiles %s\", smiles)\n    # if processed is None, the features are are default values for the type\n    return features\n```\n\nMaking a DescriptaStore\n=======================\n\nsee scripts/storus.py for more details:\n\n```\nusage: storus.py [-h] [--hasHeader] [--index-inchikey]\n                 [--smilesColumn SMILESCOLUMN] [--nameColumn NAMECOLUMN]\n                 [--seperator SEPERATOR]\n                 smilesfile storage\n\npositional arguments:\n  smilesfile            file containing smiles strings\n  storage               directory in which to store the descriptors\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --hasHeader           Indicate whether the smiles file has a header row\n  --index-inchikey      Optionally index the descriptors with inchi keys\n  --smilesColumn SMILESCOLUMN\n                        Row index (or header name if the file has a header)\n                        for the smiles column\n  --nameColumn NAMECOLUMN\n                        Row index (or header name if the file has a header)\n                        for the name column\n  --seperator SEPERATOR\n                        Row index (or header name if the file has a header)\n                        for the name column\n\n```\n\nExample:\n\nSuppose you have a smiles file like the following:\n\n```\nSMILES STRU_ID\nc1ccccc1 NAME\n```\n\nThis is a whitespace seperated file with a header.  To make the standard\nstorage and also index the inchikey:\n\n```\npython scripts/storus.py --smilesColumn=SMILES --nameColumn=STRU_ID --hasHeader --index-inchikey \\\n  --seperator=\" \" \\\n  smiles.txt mysmiles-store\n```\n\nNote that smiles files are very seperator dependent.  If the smiles or name column\ncan't be found, it is might be because the seperator is misspecified.\n\nThe default properties created are 'Morgan3Counts,RDKit2D'.\n\nUsing a DescriptaStore\n======================\n\nUsing the descriptastore (the descriptastore is a directory of files):\n\n```\nfrom descriptastorus import DescriptaStore\nd = DescriptaStore(\"/db/cix/descriptastorus/store\")\n\n# print out the column names\nprint(d.descriptors().colnames)\n\n# this will take a while!\nfor moldata, descriptors in d:\n    smiles, name = moldata\n    descriptors # is a numpy array of data morgan3 counts + rdkit descriptors\n```\n\nNote that the descriptors may contain status flags named as \"X_Calculated\" where X\nis one of the descriptor sets, such as RDKit2D.\n\nThese are not returned by the iterator, or through the following api points:\n\n```\ncolnames = d.getDescriptorNames()\ndescriptors = d.getDescriptors(index)\nfor moldata, descriptors in d:\n  ...\n```\n\nTo obtain these flags, you can either set the keepCalculatedFlags option\n\n```\ncolnames = d.getDescriptorNames(keepCalculatedFlags=True)\ndescriptors = d.getDescriptors(keepCalculatedFlags=True)\n```\n\nor use the direct descriptor interface:\n\n```\n# to iterate through only the descriptors:\nfor descriptors in d.descriptors():\n    ...\n```\n\n# to lookup by name (requires kyotocabinet)\n\n```\nrows = []\nfor name in names:\n    rows.extend( d.lookupName(name) )\n\n# sorting the rows helps with disk seeking\nrows.sort()\nfor row in rows:\n    descriptors = d.getDescriptors(row)\n    ...\n```\n\n# To lookup by inchikey (requires kyotocabinet)\n\n```\nrows = []\nfor key in inchiKeys:\n    rows.extend( d.lookupInchiKey(key) )\n\nrows.sort()\nfor row in rows:\n    descriptors = d.descriptors().get(row)\n    smiles, name = d.molIndex().get(row)\n    ...\n```\n\nDoing things yourself\n=====================\n    \nCreating a Raw store\n--------------------\n\nThe storage system is quite simple.  It is made by specifying the column names and\nnumpy types to store and also the number of rows to initialize.\n\nExample:\n\n```\n \u003e\u003e\u003e from descriptastorus import raw\n \u003e\u003e\u003e import numpy\n \u003e\u003e\u003e columns = [('exactmw', numpy.float64), ('numRotatableBonds', numpy.int32) ...]\n \u003e\u003e\u003e r = raw.MakeStore( columns, 2, \"store\")\n \u003e\u003e\u003e r.putRow(0, [45.223, 3])\n```\n\nUsing an existing store\n-----------------------\n\nAfter creation, to open the read only storage:\n\n```\n \u003e\u003e\u003e r = raw.RawStore(\"store\")\n```\n\nGet the number or rows:\n\n```\n \u003e\u003e\u003e r.N\n 2\n```\n\nGet the column names:\n\n```\n \u003e\u003e\u003e r.colnames\n ['exactmw', 'numRotatableBonds']\n```\n\nExtract the column:\n\n```\n\u003e\u003e\u003e r.get(0)\n[45.223, 3]\n```\n\nMake a MolFileIndex\n===================\n\nIf the smiles file has a header\n\n```\n\u003e\u003e\u003e from descriptastorus import MolFileIndex\n\u003e\u003e\u003e index = MolFileIndex.MakeSmilesIndex(\"data/test1.smi\", \"test1\", hasHeader=True,\n...                                      smilesColumn=\"smiles\", nameColumn=\"name\")\n\u003e\u003e\u003e index.N\n13\n\u003e\u003e\u003e index.getMol(12)\n'c1ccccc1CCCCCCCCCCCC'\n\u003e\u003e\u003e index.getName(12)\n13\n```\n\nIf the smiles file has no header\n\n```\n\u003e\u003e\u003e from descriptastorus import MolFileIndex\n\u003e\u003e\u003e index = MolFileIndex.MakeSmilesIndex(\"data/test2.smi\", \"test2\", hasHeader=False,\n...                                      smilesColumn=1, nameColumn=0)\n\u003e\u003e\u003e index.N\n13\n\u003e\u003e\u003e index.getMol(12)\n'c1ccccc1CCCCCCCCCCCC'\n\u003e\u003e\u003e index.getName(12)\n13\n```\n\nUse a MolFileIndex\n==================\n\nUsing a molfile index is fairly simple:\n\n```\n\u003e\u003e\u003e from descriptastorus import MolFileIndex\n\u003e\u003e\u003e idx = MolFileIndex(\"/db/cix/descriptastorus/test\")\n\u003e\u003e\u003e idx.get(1000)\n['CC(C)(O)c1ccc(nc1)c4ccc3C=CN(Cc2ccc(F)cc2)c3c4', 'XX-AAAA']\n\u003e\u003e\u003e idx.getName(1000)\n'XX-AAAA'\n\u003e\u003e\u003e idx.getMol(1000)\nCC(C)(O)c1ccc(nc1)c4ccc3C=CN(Cc2ccc(F)cc2)c3c4'\n```\n\nInstallation\n============\n\n```\n  git clone https://bitbucket.org/novartisnibr/rdkit-descriptastorus.git\n  cd rdkit-descriptastorus\n  python setup.py install\n```\n\n\nTODO:\n\n  * fast forwards iteration (fast now, but could be faster)\n  * faster append-only store creation\n  * Fast molecule indexing/lookup (almost done)\n  * Output to bcolz pandas backend\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbp-kelley%2Fdescriptastorus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbp-kelley%2Fdescriptastorus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbp-kelley%2Fdescriptastorus/lists"}