{"id":18564411,"url":"https://github.com/sdam-au/li_etl","last_synced_at":"2026-02-10T09:32:48.606Z","repository":{"id":90089549,"uuid":"370980222","full_name":"sdam-au/LI_ETL","owner":"sdam-au","description":"Deduplicated and enriched merge of the EDH and EDCS dataset","archived":false,"fork":false,"pushed_at":"2024-01-09T10:05:25.000Z","size":106910,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-26T11:43:01.944Z","etag":null,"topics":["data-science","dataset","machine-learning","pandas","python"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-sa-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sdam-au.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-05-26T09:37:02.000Z","updated_at":"2023-04-27T08:08:47.000Z","dependencies_parsed_at":"2023-10-11T18:18:04.540Z","dependency_job_id":null,"html_url":"https://github.com/sdam-au/LI_ETL","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdam-au%2FLI_ETL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdam-au%2FLI_ETL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdam-au%2FLI_ETL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdam-au%2FLI_ETL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sdam-au","download_url":"https://codeload.github.com/sdam-au/LI_ETL/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239293948,"owners_count":19615043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","dataset","machine-learning","pandas","python"],"created_at":"2024-11-06T22:15:06.994Z","updated_at":"2025-11-01T13:30:33.533Z","avatar_url":"https://github.com/sdam-au.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LI ETL (Latin Inscriptions - Extract, Transform, Load)\n\n---\n## Authors\n* Vojtěch Kaše [![](https://orcid.org/sites/default/files/images/orcid_16x16.png)](https://orcid.org/0000-0002-6601-1605), SDAM project, vojtech.kase@gmail.com\n* Petra Hermankova [![](https://orcid.org/sites/default/files/images/orcid_16x16.png)](https://orcid.org/0000-0002-6349-0540), SDAM project, petra@ancientsocialcomplexity.org\n* Adela Sobotkova [![](https://orcid.org/sites/default/files/images/orcid_16x16.png)](https://orcid.org/0000-0002-4541-3963), SDAM project, admin@ancientsocialcomplexity.org\n\n\n## Description\n\nThis repository serves for generation of two datasets: LIST (Latin Inscriptions in Space and Time, https://zenodo.org/record/7587556#.ZEor6i9BxhF) and LIRE (Latin Inscriptions of the Roman Empire, https://zenodo.org/record/5776109#.ZEosBC9BxpQ), where the second is a filtered, spatio-temporally more restricted, version of the first one. Both were created by aggregation of EDH and EDCS epigraphic datasets enriched by additional metadata. The repository does not contain the datasets as such, but the scripts used to generating them (see the scripts subdirectory). \n\nFor inscriptions which are covered by both EDCS and EDH source datasets, it contains attributes from both of them. In cases in which an inscription is available only in one dataset, it contains attributes only from that one dataset. Some crucial attributes shared by both datasets:\n* `clean_text_interpretive_word`: text of the inscription\n* `not_before`: start of the dating interval\n* `not_after` : end of the dating interval\n* `geography` : latitude/longitude defining geospatial position in form of a point\nIn the case of other metadata attributes, the information cannot be easily transferred between the two sources. For instance, EDCS has the attribute `inscr_type` which should bear approximately the same information as `type_of_inscription_clean` in EDH. However, the `inscr_type` attribute from EDCS uses a different classification system than EDH, relies on latin labels of inscription types etc. This project overcomes this issue by developing and applying a machine learning classification model (see `scripts/CLASSIFIER_TRAINING\u0026TESTING.ipynb` and `scripts/CLASSIFIER-APPLICATION.ipynb`). This way the dataset is enriched by two additional attributes: `type_of_inscription_auto` and `type_of_inscription_prob`.\n\nFor an overview of all metadata, see `LIST_v0.4_metadata.csv`. For an overview of the data, see the jupyter notebook `5_DATA-OVERVIEW.ipynb` in the scripts subdirectory. \n\nThe final datasets are available via Zenodo:\n* LIST dataset: https://zenodo.org/record/7870085#.ZEoyjy9BxhE (using geopandas library, you can load the data directly into your Python environment using the following command: `LIST = gpd.read_parquet(\"https://zenodo.org/record/7870085/files/LIST_v0-4.parquet?download=1\")`)\n* LIRE dataset: https://zenodo.org/record/7577788#.ZEo3rS9BxhE (using geopandas library, you can load the data directly into your Python environment using the following command: `LIRE = gpd.read_parquet(\"https://zenodo.org/record/7577788/files/LIRE_v2-1.parquet?download=1\")`)\n\n\n### Source Data\n[EDCS](http://www.manfredclauss.de/) dataset is accessed and transformed by the series of Python and R scripts in [EDCS ETL repository](https://github.com/sdam-au/EDCS_ETL), created by the SDAM project. The latest version of the dataset (as JSON file) can be accessed via Sciencedata.dk  using the following url: https://sciencedata.dk/shared/1f5f56d09903fe259c0906add8b3a55e. \n\n[EDH]() dataset is accessed and transformed by the series of Python and R scripts in [EDH ETL repository](https://github.com/sdam-au/EDH_ETL) and in [EDH exploration repository](https://github.com/sdam-au/EDH_exploration), created by the SDAM Project. The latest version of the dataset (as JSON file) can be accessed via Sciencedata.dk using the following url: https://sciencedata.dk/shared/b6b6afdb969d378b70929e86e58ad975.\n\n### Software\n* Python 3\n* Jupyter notebooks app/JupyterLab/JupyterHub\n* Python 3 additional libraries listed `requirements.txt`\n\n### Getting Started\n\nAfter you clone the repository, we recommend you to create a  virtual environment **lire_venv** using the `virtualenv` library and to run the notebooks with it as their kernel:\n```bash\nvirtualenv li_venv\nsource li_venv/bin/activate\npip install --upgrade pip\npip install -r requirements.txt # install anything in requirements.txt\npython -m ipykernel install --user --name=li_venv # add to kernels\n\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdam-au%2Fli_etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsdam-au%2Fli_etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdam-au%2Fli_etl/lists"}