{"id":19056739,"url":"https://github.com/geonode/geonode-importer","last_synced_at":"2025-09-21T18:25:20.377Z","repository":{"id":41409317,"uuid":"496950246","full_name":"GeoNode/geonode-importer","owner":"GeoNode","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-08T09:59:25.000Z","size":1659,"stargazers_count":4,"open_issues_count":6,"forks_count":16,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-24T04:48:55.899Z","etag":null,"topics":["geojson","geonode","gpkg","importer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoNode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-27T10:13:21.000Z","updated_at":"2025-04-08T09:57:41.000Z","dependencies_parsed_at":"2024-03-27T11:44:42.563Z","dependency_job_id":"a158e418-7bb1-49e3-bef8-244456c02caf","html_url":"https://github.com/GeoNode/geonode-importer","commit_stats":{"total_commits":333,"total_committers":8,"mean_commits":41.625,"dds":"0.21021021021021025","last_synced_commit":"1c0306284b8f6f7869e85b12b89952cc2ba41dc1"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoNode%2Fgeonode-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoNode%2Fgeonode-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoNode%2Fgeonode-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoNode%2Fgeonode-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoNode","download_url":"https://codeload.github.com/GeoNode/geonode-importer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250566443,"owners_count":21451230,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geojson","geonode","gpkg","importer"],"created_at":"2024-11-08T23:51:15.436Z","updated_at":"2025-09-21T18:25:15.321Z","avatar_url":"https://github.com/GeoNode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![PyPI - Downloads](https://img.shields.io/pypi/dm/geonode-importer) ![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/GeoNode/geonode-importer/runtests.yml) ![GitHub top language](https://img.shields.io/github/languages/top/GeoNode/geonode-importer)\n\n# NOTE:\nGeoNode 5 (master branch) includes the importer in its core. This repository and the `geonode-importer` package will be maintained for GeoNode \u003c= 4.4.x\n\nFor more information https://github.com/GeoNode/geonode/issues/12368.\n\n# geonode-importer\n\nA GeoNode app that implements a flow to upload/import data files.  \nThe modular logic adapts to different file types, and can be extended by implementing custom handlers.  \n\n## Some history\n`geonode-importer` has been created as a GeoNode 4.0 app to handle some formats that at the time were unsupported by GeoNode.\nIt has then been extended to include all previously handled formats.  \nIn GeoNode 4.1 `geonode-importer` replaced the previous importer logic.\n\n## Supported file formats\n- **ESRI Shapefile** - Vector\n- **GeoPackage** - Vector\n- **GeoJSON** - Vector\n- **KML** - Vector\n- **CSV** - Vector\n- **GeoTiff** - Raster\n- **XML** - Update XML file for a given resource\n- **SLD** - Update SLD file for a given resource\n\n**IMPORTANT**: At the moment the importer doesn't support overwriting/skipping existing layers from the UI. Every upload will create a new dataset.\nOverwriting a layer (`overwrite_existing_layer`) and skipping an already existing layer (`skip_existing_layers`) is supported through the API. \nRefer to the [API documentation](http://localhost:5500/_build/html/en/devel/api/usage/index.html#resource-upload) for more details and exmplaes.\n\n### GeoPackage\n- Features in the same table must have the same geometry type. Mixed geometry tpyes are not supported, therefore `GEOMETRY` columns are not accepted\n- The XML file and the SLD file uploaded along with the GPKG are ignored\n- The number of layers in a GPKG must be lower than the `max_parallel_upload` configuration value\n### GeoJSON\n- The filename should not contain dots, for example \"invalid.file.name.geojson\" -\u003e \"valid_file_name.geojson\"\n\n### CSV\n- The CSV colum accepted for lat/long CSVs (`POINTS`) are the followings:\n  - `lat`, `latitude`, `y`\n  - `long`, `longitude`, `x`\n- For any other geometry type the following columns are accepted:\n  - `geom`, `geometry`, `the_geom`, `wkt_geom`\n\n\n## Installation\n**Starting from GeoNode 4.1.0 the new importer is installed and configured by default**. \n\nThe following documentation is only meant to report what is automatically done under the hood.\n### System dependencies\n\nThe importer relies on the gdal utilities to perform format conversions and manipulations. \n\nYou need to install the `gdal-bin` package in your system, be it a base system or a docker environment; in the latter case, make sure it is installed in the `celery` and in the `django` services.  \nIn a ubuntu/debian system you can install `gdal-bin` with the command:\n\n    apt install gdal-bin\n\nYou need at least version `3.2.2` (this is the version that has been tested).  \nTo check your version please run either:\n\n    $ gdalinfo --version\n    GDAL 3.3.2, released 2021/09/01   \n\nor\n\n    $ ogrinfo --version\n    GDAL 3.3.2, released 2021/09/01\n\nTo install `gdal-bin` on other platforms please refer to https://gdal.org/download.html.\n\n### Install the package\n\nMake sure you activated the virtualenv if you are using one.\n```bash\npip install -e git+https://github.com/geosolutions-it/geonode-importer.git@master#egg=geonode_importer\n```\n\n### Configuration\n\nThe following settings in GeoNode's `settings.py` drive the importer functionality:\n\n```python\nINSTALLED_APPS += ('dynamic_models', 'importer', 'importer.handlers',)\n\nCELERY_TASK_QUEUES += (\n    Queue('importer.import_orchestrator', GEONODE_EXCHANGE, routing_key='importer.import_orchestrator'),\n    Queue('importer.import_resource', GEONODE_EXCHANGE, routing_key='importer.import_resource', max_priority=8),\n    Queue('importer.publish_resource', GEONODE_EXCHANGE, routing_key='importer.publish_resource', max_priority=8),\n    Queue('importer.create_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.create_geonode_resource', max_priority=8),\n    Queue('importer.import_with_ogr2ogr', GEONODE_EXCHANGE, routing_key='importer.import_with_ogr2ogr', max_priority=10),\n    Queue('importer.import_next_step', GEONODE_EXCHANGE, routing_key='importer.import_next_step', max_priority=3),\n    Queue('importer.create_dynamic_structure', GEONODE_EXCHANGE, routing_key='importer.create_dynamic_structure', max_priority=10),\n    Queue('importer.copy_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_resource', max_priority=0),\n    Queue('importer.copy_dynamic_model', GEONODE_EXCHANGE, routing_key='importer.copy_dynamic_model'),\n    Queue('importer.copy_geonode_data_table', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_data_table'),\n    Queue('importer.copy_raster_file', GEONODE_EXCHANGE, routing_key='importer.copy_raster_file'),\n    Queue('importer.rollback', GEONODE_EXCHANGE, routing_key='importer.rollback'),\n\n)\n\nDATABASE_ROUTERS = [\"importer.db_router.DatastoreRouter\"]\n\nSIZE_RESTRICTED_FILE_UPLOAD_ELEGIBLE_URL_NAMES += ('importer_upload',)\n\nIMPORTER_HANDLERS = os.getenv('IMPORTER_HANDLERS', [\n    'importer.handlers.gpkg.handler.GPKGFileHandler',\n    'importer.handlers.geojson.handler.GeoJsonFileHandler',\n    'importer.handlers.shapefile.handler.ShapeFileHandler',\n    'importer.handlers.kml.handler.KMLFileHandler',\n    'importer.handlers.csv.handler.CSVFileHandler',\n    'importer.handlers.geotiff.handler.GeoTiffFileHandler',\n    'importer.handlers.xml.handler.XMLFileHandler',\n    'importer.handlers.sld.handler.SLDFileHandler'\n])\n\n```\n\n## NOTE:\nIn case of a local environment, Geoserver and Geonode should be able to reach the default `MEDIA_ROOT`.\n\nIf some permission is missing, please change the `FILE_UPLOAD_DIRECTORY_PERMISSIONS` to make the folder accessible to both\n\n### DB migration\n\nRun migrations:\n\n```bash\npython manage.py migrate\npython manage.py migrate --database datastore\n```\n\n## Available environment variables\n\nTo change the task rate limit, please update the following env variables:\n\n```\nIMPORTER_GLOBAL_RATE_LIMIT= # default 5\nIMPORTER_PUBLISHING_RATE_LIMIT= # default 5\nIMPORTER_RESOURCE_CREATION_RATE_LIMIT= # default 10\nIMPORTER_RESOURCE_COPY_RATE_LIMIT = # default 10\n\n# https://github.com/OSGeo/gdal/issues/8674\nOGR2OGR_COPY_WITH_DUMP = If true, will pipe the PG dump to psql.\n```\n\n## Troubleshooting\n\n### GeoPackage\n\nThe importer will return different error codes according to the encountered error.  \nHere a description of the various codes:\n\n| Code    |   Error                | Description |\n|---------|------------------------|:------------|\n| `RQ1`   | Invalid layer name     | Layer names must start with a letter, and valid characters are lowercase a-z, numbers, or underscores.|\n| `RQ2`   | Empty layer            | Layers must have at least one feature.|\n| `RQ13`  | SRS mismatch           | It is required to give all GEOMETRY features the same default spatial reference system|\n| `RQ14`  | Unknown geometry type  | The geometry_type_name from the gpkg_geometry_columns table must be one of POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, or MULTIPOLYGON|\n| `RQ15`  | Geometry type mismatch | All table geometries must match the geometry_type_name from the gpkg_geometry_columns table|\n| `RC18`  | Geometry attr mismatch | It is recommended to give all GEOMETRY type columns the same name.|\n\n## Character conversion\n\nBy default, we use the `LAUNDER` option in ogr2ogr to convert a list of special chars to make them compatible with PostgreSQL.\n\nAlong with this, also the layer name during the dataset import is converted to make it aligned with ogr2ogr.\n\nIn detail the list is the following:\n\n| From | TO  |\n|---|---|\n| - (space) # \\  | _  |\n| .)(,\u0026  | empty_string  |\n\n\n## Run tests\n\nA simple docker compose is provided along with the project. To run the geonode-importer test suite please follow this steps:\n\nThe compose project, provides what is needed to run the tests. This includes:\n\n1) Database\n1) Geoserver\n1) Django with all the dependecies needed\n\n\n#### Step 1: build the project\n\nBuild and run the docker compose project\n```\ndocker compose -f docker-compose-test.yaml up --build -d\n```\n\n#### Step 2: Run Scripts\n\nTwo scripts are provided along with the compose file:\n\n1) entrypoint_test.sh\n1) runtest.sh\n\n#### 1) entrypoint_test.sh\nIs a script which create the needed extension and database to run the tests. Is a setup script\n\n#### 2) runtest.sh\nIt actually runs the tests.\n\n\nNote: the tests inside the docker environment can be run manually too\n\n## Env File\nThe `.env_test` file contains all the environment variable needed to successfully run the tests\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeonode%2Fgeonode-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeonode%2Fgeonode-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeonode%2Fgeonode-importer/lists"}