{"id":13572241,"url":"https://github.com/occrp-attic/loom","last_synced_at":"2025-04-04T09:32:09.469Z","repository":{"id":98396227,"uuid":"42299487","full_name":"occrp-attic/loom","owner":"occrp-attic","description":"Weaving SQL databases into graph data. ","archived":false,"fork":false,"pushed_at":"2015-12-09T15:25:12.000Z","size":177,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-11-05T05:34:03.892Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/occrp-attic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-11T09:22:55.000Z","updated_at":"2016-09-07T08:55:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"7562db7a-79c2-45c5-ac05-f72ffb785677","html_url":"https://github.com/occrp-attic/loom","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/occrp-attic%2Floom","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/occrp-attic%2Floom/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/occrp-attic%2Floom/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/occrp-attic%2Floom/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/occrp-attic","download_url":"https://codeload.github.com/occrp-attic/loom/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247153866,"owners_count":20892747,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T14:01:17.465Z","updated_at":"2025-04-04T09:32:04.453Z","avatar_url":"https://github.com/occrp-attic.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Loom\n\n``loom`` is a command-line tool that maps data from its source table structure\nto a common data model, defined through [JSON Schema](http://json-schema.org/).\nOnce data has been modeled into such objects, it is stored as a set of\nstatements in a SQL database and eventually indexed in ElasticSearch.\n\n``loom`` is the backend component for [spindle](https://github.com/occrp/spindle),\na web-based graph data browser/editor tool that uses the same data model.\n\n## Design\n\nThe design goal of ``loom`` is to accept data from many different sources and\nin many different formats, and to integrate it into a single, coherent data\nmodel that can be used for analysis.\n\n### Data integration\n\nImagine, for example, trying to investigate a senior politician. You might find\ndata about that person on WikiData, by scraping a parliament's web site,\nextracting data from expense reports and by looking up company ownership\nrecords in a corporate registry.\n\nThe purpose of ``loom`` is to read data from all of these sources, and to\ntranslate each source record into a (partially-filled) JSON object representing\na person. This object may have nested items, such as information about party\nposts, company directorships etc.\n\n``loom`` will also provide means for de-duplicating entities, so that all the\ndifferent records about the politician coming from various sources will be\nmerged into a single, coherent and complete profile. This is made possible by\nsplitting the information into atomic statements (often called triples or quads,\n[learn more](http://www.w3.org/TR/rdf11-concepts/#section-triples)).\n\n### Indexing\n\nUsing statements, the information can also be re-constructed in different ways\nthan it was originally submitted. For example, after importing a list of people\nwith nested information about what companies they control, you could very\neasily invert that into a list of companies with nested information about the\npeople who control them.\n\nThis aspect is used when indexing the data: ``loom`` will go through all entity\ntypes and try and generate a nested representation of each entity it finds.\nThis means that if you import a record about a person that controls a company,\nyou will end up with two indexed objects: the person (with a reference to the\ncompany) and the company (with a reference to the person).\n\n### Design constraints\n\nThe statement tables generated by ``loom`` are very similar to an RDF triple\nstore. It may be tempting to think about using them to perform recursive graph\nqueries (*Show me all the politicians from Germany that have companies which\nhave subsidiaries in the Cayman Islands*).\n\nThis, however, would quickly lead to an explosion in the join cross-product and\nkill any SQL database. It is not a design goal of ``loom`` to support this. We\nshould, however, provide an export facility to generate RDF data which can be\nimported into a triple store and queried there.\n\n## Configuration\n\nAll of ``loom`` is controlled via two types of configuration files: a core\nconfiguration (which sets the destination database, available data types, and\nthe search index), and a set of source specifications (``spec.yaml``, which\ndefine the source database for data mapping, how to query the source data,\nsource details and, most importantly, a mapping between source data\nstructure and the JSON schema-defined model).\n\nAn example core configuration (passed in via the ``--config`` argument) would\nlook like this:\n\n```yaml\n# A PostgreSQL databases should be provided. It can be empty (statement\n# tables will be created automatically). Since loom uses maintenance functions\n# (COPY) to load data, you may need to provide superuser access to the target\n# database.\ndatabase: postgresql://localhost/statements\n\n# ElasticSearch indexing destination. The index does not need to exist prior to\n# running loom.\nelastic_host: localhost:9200\nelastic_index: graph\n\n# This is the schema registry, which will be used to determine short-hand\n# aliases for specific types. All schemas listed here will be indexed to\n# ElasticSearch.\nschemas:\n    company: https://schema.occrp.org/generic/company.json#\n```\n\nAfter setting these base parameters, a data source can be specified like this:\n\n```yaml\n# The ODS database is expected to contain the source tables referenced below\n# (fo_companies_director, fo_companies_company).\ndatabase: postgresql://localhost/source_database\n\n# Source metadata, limited to three basic fields for the moment.\nsource:\n    slug: foo_companies\n    title: \"Foo Country Company Registry\"\n    url: http://registry.gov.foo/companies\n\n# Source data tables, and how to join them. Tables can also be assigned an\n# alias to allow for recursive joins (i.e. for parent-child relationships).\ntables:\n    - fo_companies_company\n    - fo_companies_director\njoins:\n    - foo_companies_company.id: fo_companies_director.company_id\n\n# Outputs define the set of queries to be run, alongside with a mapping that\n# is used to translate the resulting records into JSON schema objects:\noutputs:\n    # Every output has a name, which is used only for internal purposes:\n    demo:\n        schema:\n            $ref: https://schema.occrp.org/generic/company.json#\n        mapping:\n            # Mapping for a single field in the destination schema:\n            name:\n                column: foo_companies_company.name\n                # Some transformations can be applied:\n                transforms:\n                    - clean\n                    - latinize\n            company_id:\n                columns: foo_companies_company.id\n                format: 'urn:foo:%s'\n            directors:\n                # Nested object, which uses the same mapping method:\n                mapping:\n                    name:\n                        column: fo_companies_director.name\n```\n\nThe content of the ``mapping`` section of this specification file is based on\na separate Python library called [jsonmapping](https://github.com/pudo/jsonmapping).\nRefer to that package's documentation for information about how to perform more\ncomplex mappings and transformations on the source data.\n\n## Installation\n\nBefore installing ``loom``, make sure you have both dependencies - PostgreSQL\nand ElasticSearch - installed.\n\nIf you wish to use the ``loom`` command line tool, you can install the\napplication into a Python [virtual environment](http://docs.python-guide.org/en/latest/dev/virtualenvs/)\nlike this:\n\n```bash\n$ pip install git+https://github.com/occrp/loom\n```\n\nTo do development on the tool, you should instead check out a local copy:\n\n```bash\n$ git clone git@github.com:occrp/loom.git\n$ cd loom\n$ make install\n$ make test\n```\n\nInstead of executing the ``Makefile`` commands (which create a virtual\nenvironment and install the necessary dependencies), you can also run these\nsteps manually.\n\n## Usage\n\nA prerequisite for using ``loom`` is that source data needs to be stored in a\nPostgreSQL database. The specific layout of the source data tables is not\nimportant, since the JSON object mapping will be applied.\n\nAfter installing ``loom``, a command-line tool is available. You can use the\n``--help`` argument to learn more about it's functioning.\n\nA basic sequence of commands might look like this:\n\n```bash\n# translate the source data records into the loom data store as statements:\n$ loom -c config.yaml map spec.yaml\n# index the statements into ElasticSearch:\n$ loom -c config.yaml index --source foo_companies\n# delete statements from the data store:\n$ loom -c config.yaml flush --source foo_companies\n```\n\n## Similar work and references\n\n``loom`` is heavily inspired by Linked Data and RDF. If you're interested in\nsimilar tools in that ecosystem, check out:\n\n* [Grafter](http://grafter.org/)\n* [Silk Framework](http://silk-framework.com/)\n\n## License\n\n``loom`` is free software; it is distributed under the terms of the Affero\nGeneral Public License, version 3.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foccrp-attic%2Floom","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foccrp-attic%2Floom","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foccrp-attic%2Floom/lists"}