{"id":20766451,"url":"https://github.com/agrc/palletjack","last_synced_at":"2026-04-02T20:29:48.118Z","repository":{"id":37502316,"uuid":"422629038","full_name":"agrc/palletjack","owner":"agrc","description":"A library for updating AGOL data from various external sources","archived":false,"fork":false,"pushed_at":"2026-02-17T20:46:26.000Z","size":693,"stargazers_count":12,"open_issues_count":18,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-02-17T21:48:23.827Z","etag":null,"topics":["government-app","python-package","scheduled-tool","spatial-data-life-cycle","terraform-managed"],"latest_commit_sha":null,"homepage":"https://agrc.github.io/palletjack/palletjack/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agrc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-10-29T15:39:35.000Z","updated_at":"2026-02-17T20:46:02.000Z","dependencies_parsed_at":"2026-01-09T03:07:57.876Z","dependency_job_id":null,"html_url":"https://github.com/agrc/palletjack","commit_stats":{"total_commits":246,"total_committers":4,"mean_commits":61.5,"dds":"0.028455284552845517","last_synced_commit":"e815d4ba23575fcaf4d41b07f280e975995f907d"},"previous_names":[],"tags_count":44,"template":false,"template_full_name":"agrc/python","purl":"pkg:github/agrc/palletjack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrc%2Fpalletjack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrc%2Fpalletjack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrc%2Fpalletjack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrc%2Fpalletjack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agrc","download_url":"https://codeload.github.com/agrc/palletjack/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agrc%2Fpalletjack/sbom","scorecard":{"id":171136,"data":{"date":"2025-08-11","repo":{"name":"github.com/agrc/palletjack","commit":"49f3716197f1bb368be7fda3016c11e35646c1af"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":6,"checks":[{"name":"Maintained","score":10,"reason":"16 commit(s) and 3 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":5,"reason":"Found 7/12 approved changesets -- score normalized to 5","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pull_request.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/pull_request.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pull_request.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/pull_request.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/pull_request.yml:44: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/pull_request.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/push.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/push.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/release.yml:48: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:54: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/release.yml:66: update your workflow using https://app.stepsecurity.io/secureworkflow/agrc/palletjack/release.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/pull_request.yml:36","Info:   0 out of   6 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   4 third-party GitHubAction dependencies pinned","Info:   0 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: jobLevel 'contents' permission set to 'write': .github/workflows/push.yml:18","Info: jobLevel 'contents' permission set to 'read': .github/workflows/release.yml:61","Info: found token with 'none' permissions: .github/workflows/pull_request.yml:1","Warn: no topLevel permission defined: .github/workflows/push.yml:1","Warn: no topLevel permission defined: .github/workflows/release.yml:1"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":5,"reason":"branch protection is not maximal on development and all release branches","details":["Info: 'allow deletion' disabled on branch 'main'","Info: 'force pushes' disabled on branch 'main'","Warn: 'branch protection settings apply to administrators' is disabled on branch 'main'","Info: 'stale review dismissal' is required to merge on branch 'main'","Warn: required approving review count is 1 on branch 'main'","Warn: codeowners review is not required on branch 'main'","Info: 'last push approval' is required to merge on branch 'main'","Warn: 'up-to-date branches' is disabled on branch 'main'","Info: status check found to merge onto on branch 'main'","Info: PRs are required in order to make changes on branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/release.yml:36"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":7,"reason":"SAST tool is not run on all commits -- score normalized to 7","details":["Warn: 20 commits out of 28 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-16T16:29:48.864Z","repository_id":37502316,"created_at":"2025-08-16T16:29:48.864Z","updated_at":"2025-08-16T16:29:48.864Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29559966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T21:50:49.831Z","status":"ssl_error","status_checked_at":"2026-02-17T21:46:15.313Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["government-app","python-package","scheduled-tool","spatial-data-life-cycle","terraform-managed"],"created_at":"2024-11-17T11:23:52.829Z","updated_at":"2026-02-17T22:06:03.053Z","avatar_url":"https://github.com/agrc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# palletjack\n\n[![Release Events](https://github.com/agrc/palletjack/actions/workflows/release.yml/badge.svg)](https://github.com/agrc/palletjack/actions/workflows/release.yml)\n[![codecov](https://codecov.io/gh/agrc/palletjack/branch/main/graph/badge.svg)](https://codecov.io/gh/agrc/palletjack)\n\nA library of classes and methods for automatically updating AGOL feature services with data from several different types of external sources. Client apps (sometimes called 'skids') can reuse these classes for common use cases. The code modules are oriented around each step in the extract, transform, and load process.\n\n`palletjack` works with pandas DataFrames (either regular for tabular data or Esri's spatially-enabled dataframes for spatial data). The extract and transform methods return dataframes and the load methods consume dataframes as their source data.\n\nThe [documentation](https://agrc.github.io/palletjack/palletjack) includes a user guide along with an API description of the available classes and methods.\n\nPallet jack: [forklift's](https://www.github.com/agrc/forklift) little brother.\n\n## Dependencies\n\n`palletjack` relies on the dependencies listed in `setup.py`. These are all available on PyPI and can be installed in most environments, including Google Cloud Functions.\n\nThe `arcgis` library does all the heavy lifting for spatial data. If the `arcpy` library is not available (such as in a cloud function), it relies on `shapely` for its geometry engine.\n\n## Installation\n\n1. Activate your application's environment\n1. `pip install ugrc-palletjack`\n\n## Quick start\n\n1. Import the desired modules\n1. Use a class in `extract` to load a dataframe from an external source\n1. Transform your dataframe as desired with helper methods from `transform`\n1. Use the dataframe to update a hosted feature service using the methods in `load`\n\n   ```python\n   from palletjack import extract, transform, load\n\n   #: Load the data from a Google Sheet\n   gsheet_extractor = extract.GSheetLoader(path_to_service_account_json)\n   sheet_df = gsheet_extractor.load_specific_worksheet_into_dataframe(sheet_id, 'title of desired sheet', by_title=True)\n\n   #: Convert the data to points using lat/long fields, clean for uploading\n   spatial_df = pd.DataFrame.spatial.from_xy(input_df, x_column='longitude', y_column='latitude')\n   renamed_df = transform.DataCleaning.rename_dataframe_columns_for_agol(spatial_df)\n   cleaned_df = transform.DataCleaning.switch_to_nullable_int(renamed_df, ['an_int_field_with_null_values'])\n\n   #: Truncate the existing feature service data and load the new data\n   gis = arcgis.gis.GIS('my_agol_org_url', 'username', 'super-duper-secure-password')\n   updater = load.ServiceUpdater(gis, 'feature_service_item_id')\n   updates = updater.truncate_and_load(cleaned_df)\n\n   #: It even works with stand-alone tables!\n   table_updater = load.TableUpdater(gis, 'table_service_item_id', service_type='table')\n   table_updates = table_updater.truncate_and_load(cleaned_df)\n   ```\n\n## Development\n\n1. Create a conda environment with Python 3.13 (use defaults channel if you won't be deploying to an environment with ArcGIS Pro installed)\n   - `conda create -n palletjack -c defaults python=3.13`\n   - `activate palletjack`\n1. Clone the repo\n1. Install in dev mode with development dependencies\n   - `pip install -e .[tests]`\n\n### Troubleshooting Weird Append Errors\n\nIf a `FeatureLayer.append()` call (within a load.ServiceUpdater method) fails with an \"Unknown Error: 500\" error or something like that, you can query the results to get more info. The `urllib3` debug-level logs will include the HTTP GET or POST call, something like the following:\n`https://services1.arcgis.com:443 POST /\u003cunique string\u003e/arcgis/rest/services/\u003cfeature layer name\u003e/FeatureServer/\u003clayer id\u003e/append/jobs/\u003cjob guid\u003e?f=json token=\u003ccrazy long token string\u003e`. The defualt `basicConfig` logger includes the `urllib3` logs (`logging.basicConfig(level=logging.DEBUG)`) and is great for development debugging, or you can add a specific logger for `urllib3` in your code and set it's level to debug.\n\nYou can use this and a token from an AGOL tab to build a new job status url. To get the token, log into AGOL in a browser and open a private hosted feature layer item. Click the layer, and then open the developer console. With the Network tab of the console open, click on the \"View\" link for the service URL. You should see a document in the list whose name includes \"?token=\u003creally long token string\u003e\". Copy the name and then copy out the token string.\n\nNow that you've got the token string, you can build the status query:\n`https://services1.arcgis.com/\u003cunique string\u003e/arcgis/rest/services/\u003cfeature layer name\u003e/FeatureServer/\u003clayer id\u003e/append/jobs/\u003cjob guid\u003e?f=json\u0026\u003ctoken from agol\u003e`\n\nCalling this URL in a browser should return a message that will hopefully give you more info as to why it failed.\n\n### Updating Docs\n\npalletjack uses `pdoc3` to generate HTML docs in `docs/palletjack` from the docstrings within the code itself. These are then served up via github pages.\n\nThe github pages are served from the `gh-pages` branch. After you make edits to the code and update the docstrings, rebase this branch onto the updated `main` branch. To prevent github pages from trying to generate a site from the contents of `docs/palletjack` with jekyll, add a `.nojekyll` file to `docs/palletjack`.\n\nTo generate the docs, run `pdoc --html -o docs\\ c:\\palletjack\\repo\\src\\palletjack --force`. The code's docstrings should be Google-style docstrings with proper indentation to ensure the argument lists, etc are parsed and displayed correctly.\n\n`docs/README.md` is included at the top package level by adding the line `.. include:: ../../docs/README.md` in `__init__.py`'s docstring. This tells pdoc to insert that markdown into the HTML generated for that docstring, and the include directive can be used for more in-depth documentation anywhere else as well. Note that `pdoc` tries to create links for anything surrounded by backticks, which are also used for code blocks. You may need to manually edit the HTML to remove the links if they change the content of your code blocks (such as in the example import statement).\n\nOnce the contents of `docs/palletjack` look correct, force push the `gh-pages` branch to github. This will trigger the action to republish the site. The docs are then accessible at [agrc.github.io/palletjack/palletjack/index.html].\n\n## Attribution\n\nThis project was developed with the assistance of [GitHub Copilot](https://github.com/features/copilot).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagrc%2Fpalletjack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagrc%2Fpalletjack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagrc%2Fpalletjack/lists"}