{"id":16739769,"url":"https://github.com/ilias-ant/entity-resolution-with-monetdb","last_synced_at":"2026-05-17T15:09:22.209Z","repository":{"id":134760756,"uuid":"338660300","full_name":"ilias-ant/entity-resolution-with-monetdb","owner":"ilias-ant","description":"A proof-of-concept entity resolution approach, with Tensorflow, inside a MonetDB.","archived":false,"fork":false,"pushed_at":"2021-10-03T11:15:31.000Z","size":12055,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-25T19:47:44.493Z","etag":null,"topics":["entity-resolution","monetdb","product-matching","sql","tensorflow","udfs"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ilias-ant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-13T20:10:17.000Z","updated_at":"2023-10-12T14:54:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"87ae611d-b1cf-4f7e-9cb9-de19d13d4778","html_url":"https://github.com/ilias-ant/entity-resolution-with-monetdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ilias-ant/entity-resolution-with-monetdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilias-ant%2Fentity-resolution-with-monetdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilias-ant%2Fentity-resolution-with-monetdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilias-ant%2Fentity-resolution-with-monetdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilias-ant%2Fentity-resolution-with-monetdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ilias-ant","download_url":"https://codeload.github.com/ilias-ant/entity-resolution-with-monetdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ilias-ant%2Fentity-resolution-with-monetdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33143276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["entity-resolution","monetdb","product-matching","sql","tensorflow","udfs"],"created_at":"2024-10-13T00:53:41.250Z","updated_at":"2026-05-17T15:09:22.191Z","avatar_url":"https://github.com/ilias-ant.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Entity Resolution with MonetDB\n**In-Database Machine Learning**\n\nA proof-of-concept approach on an entity resolution task, with [Tensorflow](https://github.com/tensorflow/tensorflow) \nfor the machine learning part and **everything** happening inside a [MonetDB](https://www.monetdb.org/Home) instance. \nThis project was part of the course **Large Scale Data Management** at the MSc in Data Science of AUEB, carried out \nduring the Winter Quarter 2020-21.\n\nIn other words, a MonetDB database instance that performs machine learning and de-duplicates products that are \nessentially the same, in its own right.\n\n## Task Details\n\nThe task consists of identifying which product specifications (in short, specs) from multiple e-commerce websites \nrepresent the same real-world product.\n\n*example product spec*\n```json\n{\n    \"\u003cpage title\u003e\": \"Samsung Smart WB50F Digital Camera White Price in India\",\n    \n    \"brand\": \"Samsung\",\n    \n    \"dimension\": \"101 x 68 x 27.1 mm\",\n    \n    \"display\": \"LCD 3 Inches\",\n    \n    \"pixels\": \"Optical Sensor Resolution (in MegaPixel)\\n16.2 MP\",\n    \n    \"battery\": \"Li-Ion\"\n}\n```\n\nWe are provided with a dataset including ~**30k** specs in JSON format, each spec containing a list of (attribute_name, \nattribute_value) pairs extracted from a different web page, collected across 24 different web sources.\n\nEach spec is stored as a file, and files are organized into directories, each directory corresponding to a different\nweb source (e.g., *www.alibaba.com*).\nAll specs refer to cameras and include information about the camera model (e.g. *Canon EOS 5D Mark II*) and, possibly, \naccessories (e.g. *lens kit, bag, tripod*). Accessories do not contribute to product identification: for instance, a \n*Canon EOS 5D Mark II* that is sold as a bundle with a bag represents the same core product as a *Canon EOS 5D Mark II* \nthat is sold alone.\n\nFor a more detailed view of the task, the datasets, the evaluation process, see the *Other References* section below.\n\n## System Dependencies\n\n- Debian-based distribution (tested against [Ubuntu 18.04.5](https://releases.ubuntu.com/18.04/) and [Pop!_OS 20.04](https://pop.system76.com/))\n- MonetDB with embedded Python3 support (tested against [v11.39.11, v11.41.5](https://www.monetdb.org/Downloads/ReleaseNotes))\n\n## MonetDB setup\n\nThe [official documentation procedure](https://www.monetdb.org/downloads/deb/) should be followed, in order \nfor your deb-based distribution to be able to install the MonetDB via the ``apt`` package manager.\n\nTo be precise, everything up until the ``sudo apt update`` command is sufficient. From there on, we will diverge a bit in order \nto get a MonetDB installation with embedded Python3 support, with the\nfollowing commands:  \n\n```shell\n# root permission may be needed here \napt install monetdb-python3\napt install monetdb-client\n\n# create a db farm, preferably at this project's root dir\nmonetdbd create ecommercedbfarm\nmonetdbd start ecommercedbfarm\n\n# create a database, with python3 support enabled\nmonetdb create ecommercedb\nmonetdb set embedpy3=true ecommercedb\nmonetdb release ecommercedb\n```\n\nNow, to run a simple health check:\n\n```shell\nmclient -u monetdb -d ecommercedb  # password is the default (\u003cmonetdb\u003e)\n```\n\nand run the following:\n\n```sql\nCREATE OR REPLACE FUNCTION python_healthcheck () \nRETURNS STRING \nLANGUAGE python {\n    import sys \n\n\n    return sys.version\n};\n\nSELECT python_healthcheck();\n```\n\nthis should display your system-wide Python version.\n\n## Data Loading\n\nMake sure that the dataset is in extracted form. For example, you can use:\n\n```shell\ntar xvzf data/camera_specs.tar.gz -C ./data\n```\n\nAll the necessary SQL statements (UDFs etc.) are available in the ``entity-resolution-with-monetdb/sql/`` directory, \nwith each independent component hosted in a separate ``.sql`` file.\n\n**NOTE**: Also, for convenience, there is the ``entity-resolution-with-monetdb/main.sql`` available, which contains all the SQL statements together \nand can be used for a quicker build.\n\nSimply cast the SQL statements, preferably in the order below, in the ``mclient`` shell:\n\n1. sql/cameras_loader.sql\n2. sql/specs_loader.sql\n3. sql/cameras_bulk_insert.sql\n4. sql/cameras_constraints.sql\n5. sql/specs_bulk_insert.sql\n6. sql/specs_constraints.sql\n7. sql/labels_create.sql\n8. sql/labels_bulk_insert.sql\n9. sql/labels_constraints.sql\n   \nBy this point, you should have a first, working database schema. \n\n## Blocking\n\nWe would like to block together cameras of the same brand. This will help us restrict the \n\"potential matching\" space, as cameras that belong to different blocks should not, in principle, refer to the same \ncamera! \n\nTo implement the blocking step, we will need the following:\n\n10. sql/text_utils.sql\n11. sql/brands_create.sql\n12. sql/cameras_add_brand_id.sql\n13. sql/brands_bulk_insert.sql\n14. sql/blocking.sql\n\n## Filtering\n\nAfter block formation, we would also like to filter out \"easy\" matches, by extracting-via-heuristics and comparing the \nmodels of the cameras. We would thus end up - for each block - with a subset of cameras that \nmatch both on brand and model (and basically refer to the same camera).\n\nTo implement the filtering step, we will need the following:\n\n15. sql/filtering.sql\n\n## Matching\n\nWhat we have gained from blocking \u0026 filtering is that we now only have to work (and eventually perform pair-wise \ncomparisons) with the camera subsets that remained unmatched within blocks.\n\nBefore we proceed, let's make sure that [Tensorflow](https://github.com/tensorflow/tensorflow) is installed. Running the\nfollowing statement in the ``mclient``:\n\n```sql\nCREATE OR REPLACE FUNCTION tensorflow_healthcheck () \nRETURNS STRING \nLANGUAGE python {\n    import tensorflow as tf \n    \n    \n    return tf.version.VERSION\n};\n\nSELECT tensorflow_healthcheck();\n```\nshould display the Tensorflow version. In case the error ``No module named 'tensorflow'`` emerges, simply install the library:\n\n```shell\npython3 -m pip install tensorflow\n```\n\nand perform again the health check through the ``mclient``. \n\nNow, onto the \"matching\" step.\n\nA first crucial observation is that the labelled dataset is transitively closed (i.e., if A matches with B and B matches\nwith C, then A matches with C).\n\n-- \\# *work in progress* 🚧\n\n## Papers\n\n- [End-to-End Entity Resolution for Big Data: A Survey](https://arxiv.org/pdf/1905.06397.pdf)\n- [Evaluation of entity resolution approaches on real-world match problems](https://dbs.uni-leipzig.de/file/EvaluationOfEntityResolutionApproaches_vldb2010_CameraReady.pdf)\n- [A Survey of Blocking and Filtering Techniques for Entity Resolution](https://arxiv.org/pdf/1905.06167.pdf)\n- [IDEL: In-Database Entity Linking with Neural Embeddings](https://arxiv.org/abs/1803.04884)\n- [Deep Integration of Machine Learning Into Column Stores](https://openproceedings.org/2018/conf/edbt/paper-293.pdf)\n- [Deep Learning for Entity Matching: A Design Space Exploration](http://pages.cs.wisc.edu/~anhai/papers1/deepmatcher-sigmod18.pdf)\n- [Vectorized UDFs in Column-Stores](https://mytherin.github.io/papers/2016-vectorizedudfs.pdf)\n- [devUDF: Increasing UDF development efficiency through IDE](https://openproceedings.org/2019/conf/edbt/EDBT19_paper_242.pdf)\n- [Don’t Keep My UDFs Hostage - Exporting UDFs For\nDebugging Purposes](http://sbbd.org.br/2018/wp-content/uploads/sites/3/2018/02/p246-251.pdf)\n\n## Other References\n\n- [SIGMOD 2020 Contest: Task Details](http://www.inf.uniroma3.it/db/sigmod2020contest/task.html)\n- [MonetDB/Python Loader Functions](https://www.monetdb.org/blog/monetdbpython-loader-functions)  \n- [Embedded Python/NumPy in MonetDB](https://www.monetdb.org/blog/embedded-pythonnumpy-monetdb)\n- [FaBIAM Architecture Overview](https://fashionbrain-project.eu/showcase/MonetDB/output1.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Filias-ant%2Fentity-resolution-with-monetdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Filias-ant%2Fentity-resolution-with-monetdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Filias-ant%2Fentity-resolution-with-monetdb/lists"}