{"id":19772495,"url":"https://github.com/mit-spatial-action/update-from-remote","last_synced_at":"2025-09-16T20:14:00.114Z","repository":{"id":231565880,"uuid":"782085073","full_name":"mit-spatial-action/update-from-remote","owner":"mit-spatial-action","description":"Lil' utility that pulls the last modified date of a remote file and checks it against previously logged downloads in order to determine whether there has been an update on the remote. If the remote has been updated, it downloads the updated file, pushes it to Dropbox, and moves the outdated local copy to an archive folder. ","archived":false,"fork":false,"pushed_at":"2024-04-08T17:19:14.000Z","size":64,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-28T11:44:18.111Z","etag":null,"topics":["automation","bash","dropbox-api"],"latest_commit_sha":null,"homepage":"https://mit-spatial-action.github.io/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mit-spatial-action.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-04T15:55:42.000Z","updated_at":"2024-04-04T21:49:09.000Z","dependencies_parsed_at":"2025-01-11T01:10:48.761Z","dependency_job_id":"57f5ad5e-1e32-4dcc-852a-6336528c2929","html_url":"https://github.com/mit-spatial-action/update-from-remote","commit_stats":null,"previous_names":["mit-spatial-action/update-from-remote"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mit-spatial-action/update-from-remote","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-spatial-action%2Fupdate-from-remote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-spatial-action%2Fupdate-from-remote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-spatial-action%2Fupdate-from-remote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-spatial-action%2Fupdate-from-remote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mit-spatial-action","download_url":"https://codeload.github.com/mit-spatial-action/update-from-remote/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mit-spatial-action%2Fupdate-from-remote/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273377153,"owners_count":25094528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","bash","dropbox-api"],"created_at":"2024-11-12T05:06:42.940Z","updated_at":"2025-09-16T20:13:55.028Z","avatar_url":"https://github.com/mit-spatial-action.png","language":"Shell","readme":"# update-from-remote\nLil' utilities for automatically managing recurrent downloads of remote files. Writen to minimize dependencies (which means clunky but functional `JSON` parsing---sorry, [JQ](https://jqlang.github.io/jq/) stans). Tested on macOS and Ubuntu.\n\n1. `update-from-remote.sh` checks the last modified date of a remote file and compares it to the previous download in order to determine whether there has been an update on the remote. (This is handled by simpty writing a timestamp into the filename.) If the remote has been updated, it downloads the updated file and moves the outdated local copy to an archive folder.\n2. `push-to-dropbox.sh` pushes a local file to a folder associated with a Dropbox application, which also handles the 150MB payload limit for the base [`upload/`](https://www.dropbox.com/developers/documentation/http/documentation#files-upload) endpont by splitting large files and using the [`upload_session/`](https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-append) endpoint. Handles all the stuff you'd expect (checking whether files exists on Dropbox before pushing file, etc.).\n\nCurrently used to create an archive of the [MassGIS Property Tax Parcels database](https://www.mass.gov/info-details/massgis-data-property-tax-parcels). Archive is available in [this Dropbox folder](https://www.dropbox.com/scl/fo/8tb0boh3ejckizdx3w9q8/h?rlkey=ye2s8zgs16dif81usc2jhx8fm\u0026dl=0). We began using this script in April 2024, so updates will be ongoing (approximately monthly).\n\n## Configuration\n`update-from-remote.sh` has one required option (`-u`) which expects a remote file URL. An optional argument (`-a`) allows you to specify a local archive directory. (By default, this is set to `'./archive/'`). To use `push-to-dropbox.sh`, you'll have to create a `.env` file, which includes the following:\n\n```\ndbox_refresh_token=\"\"\ndbox_app_key=\"\"\ndbox_app_secret=\"\"\ndbox_out_path=\"\"\n```\n\nYou must first [create a Dropbox application](https://www.dropbox.com/developers/apps/create?_tk=pilot_lp\u0026_ad=ctabtn1\u0026_camp=create). Because this script is built to run on a server without intervention, you'll also need to set up a Dropbox refresh token, which is necessary because the API has moved to an `OAuth 2.0` authentication system for its API. See the [Dropbox OAuth documentation here](https://developers.dropbox.com/oauth-guide). Application key and secret are available through the Application dashboard.\n\nIf you want to schedule updates on on a `crontab`, you should be able to simply use the below, obviously modifying the schedule for your needs.\n\n```\n# m h dom mon dow command\n59 23 * * cd \u003crepo_path\u003e \u0026\u0026 bash ./update-from-remote.sh -u \"\u003cremote_file_of_interest\u003e\" \u003e\u003e update-from-remote.log 2\u003e\u00261 \u0026\u0026 bash ./push-to-dropbox.sh \u003e\u003e update-from-remote.log 2\u003e\u00261\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-spatial-action%2Fupdate-from-remote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmit-spatial-action%2Fupdate-from-remote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmit-spatial-action%2Fupdate-from-remote/lists"}