{"id":16284676,"url":"https://github.com/pachadotdev/uncomtrademisc","last_synced_at":"2025-04-09T00:36:19.609Z","repository":{"id":81294115,"uuid":"483446750","full_name":"pachadotdev/uncomtrademisc","owner":"pachadotdev","description":"Ease some data munging when using UN COMTRADE data.","archived":false,"fork":false,"pushed_at":"2023-08-22T03:42:05.000Z","size":587,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-14T18:52:31.132Z","etag":null,"topics":["gravity","r","trade"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pachadotdev.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-19T23:59:55.000Z","updated_at":"2023-08-08T14:43:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"775b11d4-8962-4f67-8821-deb24378e2b3","html_url":"https://github.com/pachadotdev/uncomtrademisc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pachadotdev%2Funcomtrademisc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pachadotdev%2Funcomtrademisc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pachadotdev%2Funcomtrademisc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pachadotdev%2Funcomtrademisc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pachadotdev","download_url":"https://codeload.github.com/pachadotdev/uncomtrademisc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247952048,"owners_count":21023854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gravity","r","trade"],"created_at":"2024-10-10T19:20:24.926Z","updated_at":"2025-04-09T00:36:19.590Z","avatar_url":"https://github.com/pachadotdev.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# UN COMTRADE Misc Functions\n\nThe goal of uncomtrademisc is to ease some data munging when using UN COMTRADE data.\n\n## 2023-07-17 update\n\nThe package will not be maintained anymore after 2023-07-30. A new package will be\ncreated to work with the new UN COMTRADE Plus.\n\n## 2023-07-15 update\n\nTo get the downloads to work, we need to alter `/usr/lib/ssl/openssl.cnf` and add this under `[system_default_sect]`\n\n```\nOptions = UnsafeLegacyRenegotiation\n```\n\nThen it is required to restart the computer and try again, this is because of the error `OpenSSL: error:0A000152:SSL routines::unsafe legacy renegotiation disabled` which is created by UN COMTRADE using an old SSL protocol.\n\n## Installation\n\nYou can install the development version of uncomtrademisc like so:\n\n``` r\nremotes::install_github(\"pachadotdev/uncomtrademisc\")\n```\n\nThis one-liner will install the next packages if needed: *dplyr, stringr,\npurrr, janitor, RPostgres.*\n\n## Usage\n\nThis is a summary of the [tradestatistics-database-postgresql repository](https://github.com/tradestatistics/tradestatistics-database-postgresql).\n\n### Declare environment variables\n\nRun `usethis::edit_r_environ()` and add the following variables\n\n```\nCOMTRADE_TOKEN=\"THE_TOKEN\"\nLOCAL_SQL_USR=\"THE_USER\"\nLOCAL_SQL_PWD=\"THE_PWD\"\n```\n\nRestart the R session after changing your environment.\n\n### Clone the database project from GitHub\n\nClone https://github.com/tradestatistics/database-postgresql and open the \nproject in RStudio.\n\nFrom a command line\n```bash\ngit clone git@github.com:tradestatistics/database-postgresql.git\n```\n\nOr:\n```bash\ngit clone https://github.com/tradestatistics/database-postgresql.git\n```\n\nYou can also use GitHub Desktop, but I don't know how to use it.\n\n### Download data from UN COMTRADE\n\nRun this code:\n```r\nuncomtrademisc::data_downloading()\n```\n\nTo avoid the prompts, and assuming you followed the previous instructions, you can pass these arguments to `data_downloading()`, which will write to the `umcomtrade` PostgreSQL database in `localhost` after tidying the tables, unless you pass `postgres = F` to just download the data as zipped CSV files.\n\n```r\nlibrary(uncomtrademisc)\n\n# hs92\ndata_downloading(postgres = T, token = T, dataset = 1, remove_old_files = 1, subset_years = 1988:2021, parallel = 2, skip_updates = F)\n\n# hs96\ndata_downloading(postgres = T, token = T, dataset = 2, remove_old_files = 1, subset_years = 1996:2021, parallel = 2, skip_updates = F)\n\n# hs02\ndata_downloading(postgres = T, token = T, dataset = 3, remove_old_files = 1, subset_years = 2002:2021, parallel = 2, skip_updates = F)\n\n# hs07\ndata_downloading(postgres = T, token = T, dataset = 4, remove_old_files = 1, subset_years = 2007:2021, parallel = 2, skip_updates = F)\n\n# hs12\ndata_downloading(postgres = T, token = T, dataset = 5, remove_old_files = 1, subset_years = 2012:2021, parallel = 2, skip_updates = F)\n\n# sitc1\ndata_downloading(postgres = T, token = T, dataset = 6, remove_old_files = 1, subset_years = 1962:2021, parallel = 2, skip_updates = F)\n\n# sitc2\ndata_downloading(postgres = T, token = T, dataset = 7, remove_old_files = 1, subset_years = 1976:2021, parallel = 2, skip_updates = F)\n```\n\n\n### Upload the data to the server\n\nThese functions by default write to the public schema and will create new\ntables or delete the existing tables and overwrite the content table by table.\nFor example, if you defined `TRADESTATISTICS_SQL_DB=\"mydb_main\"` instead of\n`TRADESTATISTICS_SQL_DB=\"mydb_testing\"`, you'll be writing irreversible changes\nto your main database, so please check twice before running the code.\n\n# Downsides\n\nThe code to download the data from UN COMTRADE relies on specific UNIX\nmulticore tools, in order to speed up the downloading process. On Windows the\ndownload is sequential, file by file, and therefore much slower (around 10x slower).\n\n`RPostgres` uses the system package libpq. On Windows, it's not a problem. On\nUbuntu/Mac, you'll need to install `libpq-dev` or similar (i.e., the name depends\non the specific platform).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpachadotdev%2Funcomtrademisc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpachadotdev%2Funcomtrademisc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpachadotdev%2Funcomtrademisc/lists"}