{"id":13502521,"url":"https://github.com/augustomen/simple-data-importer","last_synced_at":"2025-03-29T12:32:03.581Z","repository":{"id":9652749,"uuid":"11589270","full_name":"augustomen/simple-data-importer","owner":"augustomen","description":null,"archived":false,"fork":false,"pushed_at":"2014-10-08T14:40:07.000Z","size":209,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2024-10-31T22:32:09.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/augustomen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-22T18:49:16.000Z","updated_at":"2020-05-27T01:59:32.000Z","dependencies_parsed_at":"2022-09-01T13:40:32.992Z","dependency_job_id":null,"html_url":"https://github.com/augustomen/simple-data-importer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustomen%2Fsimple-data-importer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustomen%2Fsimple-data-importer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustomen%2Fsimple-data-importer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/augustomen%2Fsimple-data-importer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/augustomen","download_url":"https://codeload.github.com/augustomen/simple-data-importer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246186816,"owners_count":20737453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T22:02:16.680Z","updated_at":"2025-03-29T12:32:03.311Z","avatar_url":"https://github.com/augustomen.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Simple Data Importer\n====================\n\nA simple, yet useful framework for data reading, processing and saving.\n\nGiven a data source (which may be a CSV file, Excel worksheet, DB query or any\niterable), the importer will iterate through the rows or objects, validating\nall data before saving the record.\n\nGetting started\n---------------\n\nCreate your own:\n\n```python\nclass MyImporter(BaseImporter):\n\n    # fields is a list of dicts with the setup of each field read.\n    # The fields are read and cleaned in this order.\n    fields = [\n        {\n            'field_names': (),\n            # One or more attribute or key names\n            # The first will determine clean_* method name and row_values key\n\n            'get_value': None,\n            # A function (row_number, row, field_names) to return\n            # the field value from a row or object.\n\n            'required': False,\n            # default: False. If True and the value is not provided, will\n            # raise an RequiredFieldMissing exception\n\n            'coerce': int,\n            # A single-parameter callable applied to the value\n            # BEFORE calling clean_field_name\n        },\n    ]\n\n    def before_import(self):\n        # Called before the import process\n        pass\n\n    def clean_field_name(self, row_number, row_values, value):\n        # Validate and coerce value to desired type.\n        # If an exception is raised, it will be passed to\n        # exception_handler.\n\n        # - row_values is the dict of values retrieved so far (in the same\n        # order fields were provided).\n        # - row_number is a 0-based index of the row being imported\n        return value\n\n    def clean(self, row_number, row_values):\n        # Validate the entire row, after all clean_* have been called.\n        # Must return the same row_values or a completely new object\n        # that will be passed to save().\n        # - row_values is the complete dict of values retrieved and cleaned\n        return row_values\n\n    def save(self, row_number, row_values):\n        # Saves the record.\n        # - row_values is the complete dict of values retrieved and cleaned\n        pass\n\n    def after_import(self):\n        # Called after the import process.\n        # This is always called, even when an exception is raised, so be\n        # sure to check self.aborted for untreated errors.\n        pass\n\n    def exception_handler(self, row_number, exception):\n        # Called when any exception is raised.\n        # Return True to continue processing the next row.\n        return False\n```\n\nCreate an instance of `MyImporter` with any iterable (such as the ones in\n`simpledataimporter.readers`):\n\n```python\nmycsv = SlugCSVReader('/path/to/file.csv')\nmyimporter = MyImporter(mycsv)\nmyimporter.run()  # run!\n```\n\nFor each valid row, the function `save()` will be called.\n\n\nTips\n----\nAt any time during the proccess, the following attributes are accessible:\n\n- self.rows_saved: The number of rows that have been successfully saved.\n- self.rows_processed: The number of rows that have been read and processed,\n    even if it has not been saved.\n- self.current_row: contains the object retrieved from source. Not available\n    before and after import.\n- self.current_row_number: the current row number (starting at 0).\n- self.exception_count: The number of exceptions raised during the process.\n- self.aborted: True if the process has been aborted by exception_handler.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faugustomen%2Fsimple-data-importer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faugustomen%2Fsimple-data-importer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faugustomen%2Fsimple-data-importer/lists"}