{"id":20063163,"url":"https://github.com/alan-turing-institute/csv_wrangling","last_synced_at":"2025-11-26T18:02:30.184Z","repository":{"id":33578645,"uuid":"158363564","full_name":"alan-turing-institute/CSV_Wrangling","owner":"alan-turing-institute","description":"Repository for reproducibility of the CSV file project","archived":false,"fork":false,"pushed_at":"2022-01-20T22:13:47.000Z","size":6506,"stargazers_count":28,"open_issues_count":0,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-12T22:41:56.993Z","etag":null,"topics":["csv","csv-files","csv-parsing","reproducibility","reproducible-paper","reproducible-research","reproducible-science"],"latest_commit_sha":null,"homepage":null,"language":"TeX","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alan-turing-institute.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-20T09:28:05.000Z","updated_at":"2024-12-10T16:50:53.000Z","dependencies_parsed_at":"2022-08-07T22:15:23.591Z","dependency_job_id":null,"html_url":"https://github.com/alan-turing-institute/CSV_Wrangling","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alan-turing-institute%2FCSV_Wrangling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alan-turing-institute%2FCSV_Wrangling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alan-turing-institute%2FCSV_Wrangling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alan-turing-institute%2FCSV_Wrangling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alan-turing-institute","download_url":"https://codeload.github.com/alan-turing-institute/CSV_Wrangling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241490624,"owners_count":19971294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","csv-files","csv-parsing","reproducibility","reproducible-paper","reproducible-research","reproducible-science"],"created_at":"2024-11-13T13:40:47.609Z","updated_at":"2025-11-26T18:02:30.105Z","avatar_url":"https://github.com/alan-turing-institute.png","language":"TeX","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV Wrangling\n\n[![Build Status](https://travis-ci.org/alan-turing-institute/CSV_Wrangling.svg?branch=master)](https://travis-ci.org/alan-turing-institute/CSV_Wrangling)\n[![DOI](https://zenodo.org/badge/158363564.svg)](https://zenodo.org/badge/latestdoi/158363564)\n\nThis is the repository for reproducing the experiments in the paper:\n\n[**Wrangling Messy CSV files by Detecting Row and Type \nPatterns**](https://rdcu.be/bLVur) \n[(PDF)](https://gertjanvandenburg.com/papers/VandenBurg_Nazabal_Sutton_-_Wrangling_Messy_CSV_Files_by_Detecting_Row_and_Type_Patterns_2019.pdf)\n\nby [G.J.J. van den Burg](https://gertjanvandenburg.com), [A. \nNazabal](https://scholar.google.co.uk/citations?user=IanHvT4AAAAJ\u0026hl=en\u0026oi=ao) \nand [C. Sutton](https://homepages.inf.ed.ac.uk/csutton/).\n\nFor an implementation of the method developed in the paper, see the \n[CleverCSV](https://github.com/alan-turing-institute/CleverCSV) repository.\n\nIf you use this paper or this code in your own work, please ***cite the \npaper*** using for instance the following BibTeX citation:\n\n```bibtex\n@article{van2019wrangling,\n  title = {Wrangling Messy {CSV} Files by Detecting Row and Type Patterns},\n  author = {{van den Burg}, G. J. J. and Naz{\\'a}bal, A. and Sutton, C.},\n  journal = {Data Mining and Knowledge Discovery},\n  year = {2019},\n  volume = {33},\n  number = {6},\n  pages = {1799--1820},\n  issn = {1573-756X},\n  doi = {10.1007/s10618-019-00646-y},\n}\n```\n\n## Introduction\n\nOur experiments are made reproducible through the use of [GNU \nMake](https://www.gnu.org/software/make/). You can either set up your local \nenvironment with the necessary dependencies as described under \n[Requirements](#requirements), or use the Dockerfile included in the \nrepository.\n\nThere are two ways to reproduce our results. The first only reproduces the \nfigures, tables, and constants in the paper from the raw detection results, \nwhile the second runs the detection methods as well.\n\n1. You can reproduce the figures, tables, and constants from the raw \n   experimental results included in this repository. This will not re-run all \n   the experiments but will regenerate the output used in the paper. The \n   command for this is:\n\n   ```bash\n   $ make output\n   ```\n\n2. You can fully reproduce our experiments by downloading the data and \n   rerunning the detection methods on all the files. This might take a while \n   depending on the speed of your machine and the number of cores available. \n   Total wall-clock computation time for a single core is estimated at 11 \n   days. The following commands will do all of this.\n\n   ```bash\n   $ make clean       # remove existing output files, except human annotated\n   $ make data        # download the data\n   $ make results     # run all the detectors and generate the result files\n   ```\n\n   If you'd like to use multiple cores, you can replace the last command with:\n\n   ```bash\n   $ make -j X results\n   ```\n\n   where ``X`` is the desired number of cores.\n\n\n## Data\n\nThere are two datasets that are used in the experiments. Because we don't own \nthe rights to all these files, we can't package these files and make them \navailable in a single download. We can however provide URLs to the files and \nadd a download script, which is what we do here. The data can be downloaded \nwith:\n\n```bash\n$ make data\n```\n\nIf you wish to change the download location of the data, please edit the \n``DATA_DIR`` variable in the Makefile.\n\n**Note:** We are aware of the fact that some of the files may change or become \nunavailable in the future. This is an unfortunate side-effect of using \npublically available data in this way. The data downloader skips files that \nare unavailable or that have changed. Note that this may affect the exact \nreproducibility of the results.\n\nThe above downloads the \"test\" set that was used for the evaluation in the \npaper. For the \"working set\" that was used to develop our algorithm, run \n``make dev-data``.\n\nIf the above datasets are insufficient, the complete original data sets are \navailable on request for research purposes. Contact ``gertjanvandenburg at \ngmail dot com``.\n\n## Requirements\n\nBelow are the requirements for reproducing the experiments if you're not using \nDocker. Note that at the moment only Linux-based systems are supported.  MacOS \nwill probably work, but hasn't been tested.\n\n- Python 3.x with the packages in the ``requirements.txt`` file. These can be \n  installed with: ``pip install --user -r requirements.txt``.\n\n- R with the external packages installed through: \n  ``install.packages(c('devtools', 'rjson', 'data.tree', 'RecordLinkage', \n  'readr', 'tibble'))``.\n\n- A working [LaTeX](https://www.latex-project.org/) installation is needed for \n  creating the figures (at least ``texlive-latex-extra`` and \n  ``texlive-pictures``), as well as a working \n  [LaTeXMK](https://mg.readthedocs.io/latexmk.html) installation.\n\n\n## Instructions\n\nTo clone this repository and all its submodules do:\n\n```bash\n$ git clone --recurse-submodules https://github.com/alan-turing-institute/CSV_Wrangling\n```\n\nThen install the requirements as listed above and run the ``make`` command of \nyour choice.\n\n## License\n\nWith the exception of the submodule in ``scripts/detection/lib/hypoparsr`` \nthis code is licensed under the [MIT \nlicense](https://en.wikipedia.org/wiki/MIT_License). See the LICENSE file for \nmore details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falan-turing-institute%2Fcsv_wrangling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falan-turing-institute%2Fcsv_wrangling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falan-turing-institute%2Fcsv_wrangling/lists"}