{"id":21229306,"url":"https://github.com/ndrplz/eurlex-toolbox","last_synced_at":"2025-07-10T15:31:58.438Z","repository":{"id":40980195,"uuid":"203158818","full_name":"ndrplz/eurlex-toolbox","owner":"ndrplz","description":"Python toolbox to load, parse and process Official Journals of the European Union (EU).","archived":false,"fork":false,"pushed_at":"2024-05-03T19:47:28.000Z","size":6279,"stargazers_count":12,"open_issues_count":3,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-03T21:00:52.765Z","etag":null,"topics":["dataset","eur-lex","eurlex","european-union","legal-acts","natural-language-processing","nlp","official-journals","python-toolbox"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ndrplz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-19T11:25:33.000Z","updated_at":"2024-03-23T14:46:38.000Z","dependencies_parsed_at":"2022-09-01T10:21:00.739Z","dependency_job_id":null,"html_url":"https://github.com/ndrplz/eurlex-toolbox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Feurlex-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Feurlex-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Feurlex-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ndrplz%2Feurlex-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ndrplz","download_url":"https://codeload.github.com/ndrplz/eurlex-toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225645507,"owners_count":17501694,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","eur-lex","eurlex","european-union","legal-acts","natural-language-processing","nlp","official-journals","python-toolbox"],"created_at":"2024-11-20T23:26:38.015Z","updated_at":"2024-11-20T23:26:38.605Z","avatar_url":"https://github.com/ndrplz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Eurlex-Toolbox\n\nThis repository contains a python toolbox to load, parse and process Official Journals of the European Union (EU).\n\n## EU Law text corpus\n\n:arrow_double_down: Download the dataset [here](https://drive.google.com/open?id=15XKFh69Fsy907D9tcs9qcniGrsTRQPxk).\n\nText corpus containing all the legal acts (the L series of the Official Journal) of the European Union adopted since the entry into force of the Lisbon Treaty, 1st December 2009, until 30th June 2019.\n\n**The resulting corpus contains 24134 documents totalling approximately 43 million words.**\n\nWe present it divided in three files. \n\n* **Legislation** contains the legal acts the Union can adopt to exercise its competences (those listed under Article 288 Treaty on the Functioning of the European Union): regulations, directives, decisions, recommendations and opinions. It also contains the EU acts implementing them.\n* **International Agreements and EEA acts** contains, in addition to Treaties concluded by the European Union, texts with EEA relevance.\n* **Other** contains guidelines, interinstitutional agreements, notices, procedural rules, etc.\n\n## Software Overview\n\nEuropean Union law documents are publicly available in human-readable format in the [EUR-Lex portal](https://eur-lex.europa.eu/homepage.html). To enable automatic analysis, the same documents are released as structured text in the [EU Open Data Portal](https://data.europa.eu/euodp/en/home). Options for bulk download of entire blocks of documents are available there.\n\nThis software allows to handle the Official Journals of the EU as they are released in XML-Formex format [here](https://data.europa.eu/euodp/en/data/dataset/official-journals-of-the-european-union-in-english).\n\n## Data preparation\nTo maximize reproducibility and ease of use, the download and decompression of XML data from [here](https://data.europa.eu/euodp/en/data/dataset/official-journals-of-the-european-union-in-english) is automatized thanks to the `download_and_unzip.py` script. Usage:\n```bash\npython download_and_unzip.py \u003cdataset_root\u003e \u003clanguage\u003e\n```\n![download_en](./img/download_en.gif)\n\nOnce the `download_and_unzip` script has finished, the expected directory structure should look like the following:\n```bash\n\u003ceurlex_root\u003e/\n  ├── JOx_FMX_EN_2004/\n    ├── file0.doc.xml\n    ├── file0.xml\n    ├── ...\n    ├── fileN.doc.xml\n    └── fileN.xml\n  ├── JOx_FMX_EN_2005/\n      ├── ...\n  ├── ...\n  └── JOx_FMX_EN_2019/\n      ├── ...\n```\n\n## Hello World!\nOnce the raw XML data are in place, the toolbox (entry point `main.py`) can be used to manipulate the data and create huamn readable text corpora. \n\nDocuments can be dumped in human readable format in few lines of code.\n```python\nfrom eurlex_ds import EurLexDataset\n\ndataset = EurLexDataset(data_root=\u003ceurlex_root\u003e)\nprint(f'Number of documents: {len(dataset)}')\n\n# Dump all concatenated docs in human readable text\ndataset.dump_to_txt('all_txt.txt', mode='text')\n\n# Dump all document headers\ndataset.dump_to_txt('stats.csv', mode='headers')\n```\nNB: To save time, the `EurLexDataset` object can also be initialized form a text file containing a list of paths pointing to the docs to be loaded (see [here](https://github.com/ndrplz/eurlex-toolbox/blob/9a7526f63c15800114016b4854fa636eac4e46c4/eurlex_ds.py#L164-L174)). This avoids listing all file on disk every time.\n\n## Advanced features\nThe `EurLexDataset` object encapsulates the official journals dataset and it can be used to perform more sophisticated analyses and queries. Few examples:\n\n* Create a separate text file for all the documents of each year from 2009 to 2019:\n```python\nfrom eurlex_ds import EurLexDataset\n\ndataset = EurLexDataset(data_root=args.data_root)\n\nfor year in range(2009, 2020):\n  items = [it for it in dataset if it.date.startswith(str(year))]\n  year_all_txt = ('\\n' * 5).join([it.to_txt() for it in items])\n  Path(f'all_txt_{year}.txt').write_text(year_all_txt, encoding='utf-8')\n```\n\n* Filter the dataset keeping only decisions:\n```python\nfrom eurlex_ds import EurLexDataset\n\ndataset = EurLexDataset(data_root=args.data_root)\ndataset.items[:] = [it for it in dataset if it.meta.is_dec()]\n```\n\n* Print legal value and title of all documents to console:\n```python\nfrom eurlex_ds import EurLexDataset\n\ndataset = EurLexDataset(data_root=args.data_root)\nfor item in dataset:\n  print(f'{item.meta.legval}: {item.meta.title}\\n')\n```\netc.\n\nMany other features are already available, but still undocumented since we are still working on them and the API might change a lot. These includes tokenization, extraction of geographical entities etc. However, the code is open source, so feel free to explore it and take what you may find useful :)\n\n## Citation\n\nThe code contained this repository accompanies the following publication:\n\n\u003e Palazzi, Andrea and Luigi Lonardo. \"A Dataset, Software Toolbox, and Interdisciplinary Research Agenda for the Common Foreign and Security Policy\", to appear in the European Foreign Affairs Review Vol. 25, Issue 2 (July 2020).\n\nIn case you find this dataset and toolbox helpful for your research (we hope so!), please mention the paper above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrplz%2Feurlex-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fndrplz%2Feurlex-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fndrplz%2Feurlex-toolbox/lists"}