{"id":18446513,"url":"https://github.com/ovh/osarchiver","last_synced_at":"2025-04-08T00:31:52.433Z","repository":{"id":37669823,"uuid":"190537965","full_name":"ovh/osarchiver","owner":"ovh","description":"OpenStack databases archiver","archived":false,"fork":false,"pushed_at":"2024-07-09T17:39:09.000Z","size":110,"stargazers_count":14,"open_issues_count":3,"forks_count":8,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-23T03:11:16.878Z","etag":null,"topics":["archiver","database","openstack"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-06-06T07:45:05.000Z","updated_at":"2024-11-27T14:29:04.000Z","dependencies_parsed_at":"2022-07-08T07:41:56.460Z","dependency_job_id":null,"html_url":"https://github.com/ovh/osarchiver","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fosarchiver","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fosarchiver/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fosarchiver/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fosarchiver/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovh","download_url":"https://codeload.github.com/ovh/osarchiver/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247755388,"owners_count":20990616,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archiver","database","openstack"],"created_at":"2024-11-06T07:09:34.958Z","updated_at":"2025-04-08T00:31:52.094Z","avatar_url":"https://github.com/ovh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# OSArchiver: OpenStack databases archiver\n\nOSArchiver is a python package that aims to archive and remove soft deleted data from OpenStack databases.\nThe package is shiped with a main script called osarchiver that reads a configuration file and run the archivers.\n\n# Philosophy\n\n* OSArchiver doesn't have any knowledge of Openstack business objects\n* OSArchiver purely relies on the common way of how OpenStack marks data as deleted by setting the column 'deleted_at' to a datetime.\nIt means that a row is archivable/removable if the 'deleted_at' column is not NULL\n\n# Limitations\n\n* Support Mysql/MariaDB as db backend.\n* python \u003e= 3.5\n\n# Design\n\nOSArchiver reads an INI configuration file in which you can define:\n\n* archivers: a section that hold one source and a non mandatory list of destinations\n* sources: a section that define a source of where the data should be read (basically the OS DB)\n* destinations: a section that define where the data should be archived\n\n# How does it works:\n\n                                       .----------.\n            .--------------------------| Archiver |-----------------------------.\n            |                          '----------'                             |\n            |                                                                   |\n            |                                                                   |\n            |                                                                   |\n            v                        _______________                            v\n       .--------.                    \\              \\                    .-------------.\n       | Source |--------------------\u003e) ARCHIVE DATA )------------------\u003e| Desinations |\n       '--------'                    /______________/                    '-------------'\n            |                                |                                  |\n            |                                |                                  |\n            |                                |                                  |\n            |                                |                                  |\n            |                                |                                  |\n            |                                v                                  |\n            |                  .--------------------------.                     |\n            v                 ( No error and delete_data=1 )                    |\n                               '--------------------------'                     |\n        _.-----._                            |                    _.-----._     |\n      .-         -.                          |                  .-         -.   |   ___   \n      |-_       _-|                          |                  |-_       _-|   |  |   |\\ \n      |  ~-----~  |                          |                  |  ~-----~  |\u003c--'-\u003e|   ' ___   \n      |           |                          |                  |           |      | SQL|   |\\ \n      `._       _.'                          |                  `._       _.'      |____|   '-|---.\n         \"-----\"                             |                     \"-----\"              | CSV |   |\n      OpenStack DB                           v                  Archiving DB            |_____|   |\n            ^                        _______________                                              v\n            |                        \\              \\                                 .-----------------------.\n            '-------------------------) DELETE DATA  )                               ( remote_store configured )\n                                     /______________/                                 '-----------------------'\n                                                                                                  |\n                                                                                                  v\n                                                                                             __________ \n                                                                                            [_|||||||_°]\n                                                                                            [_|||||||_°]\n                                                                                            [_|||||||_°]\n\n                                                                                     Remote Storage (Swift, ...)\n\n# Installation\n\n```\ngit clone https://github.com/ovh/osarchiver.git\ncd osarchiver\npip install -r requirements.txt\npip setup.py install\n```\n\n# osarchiver script\n\n```\n# osarchiver --help\nusage: osarchiver [-h] --config CONFIG [--log-file LOG_FILE]\n                  [--log-level {info,warn,error,debug}] [--debug] [--dry-run]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --config CONFIG       Configuration file to read\n  --log-file LOG_FILE   Append log to the specified file\n  --log-level {info,warn,error,debug}\n                        Set log level\n  --debug               Enable debug mode\n  --dry-run             Display what would be done without really deleting or\n                        writing data\n```\n\n# Configuration\nThe configuation is an INI file containing several sections. You configure your\ndifferents archivers in this configuration file. An example is available at the\nroot of the repository.\n\n## DEFAULT section:\n* Drescription: default section that define default/fallback value for options\n* Format **[DEFAULT]**\n* configuration parameters: all the parameters of archiver, source, destination\n  and backend section can be added in this section, those will be the fallback\n  value if the value is not set in a section.\n\n## Archiver section:\n\n* Description: defines where to read data and where to archive them and/or delete.\n* Format **[archiver:*name*]**\n* configuration parameters:\n    * **src**: name of the src section\n    * **dst**: comma separated list of destination section names\n    * **enable**: 1 or 0, if set to 0 the archiver is ignored and not run\n\nExample:\n```properties\n[archiver:My_Archiver]\nsrc: os_prod\ndst: file, db\n\n[src:os_prod]\n...\n\n[dst:file]\n...\n\n[dst:db]\n....\n```\n\n## Source section:\n\n* Description: defines where the OpenStack database are. It supports for now\n  one backend (db) but it may be easily extended\n* Format **[src:*name*]**\n* configuration parameters:\n    * **backend**: the name of backend to use, only `db` is supported\n    * **retention**: 12 MONTH\n    * **archive_data**: 0 or 1 if set to 1 expect a dest to archive the data else\n      won't run the archiving step just the delete step.\n    * **delete_data**: 0 or 1 if set to 1 will run the delete step. If the\n      archive step fails the delete step is not run to prevent loose of data.\n    * *backend specific options*\n\n\n## Destination section:\n\n* Description: defines where the data should be written. It supports for now\n  two backends (db for datatabase and file [csv, sql]) and may be extended\n* Format **[dst:*name*]**\n* configuration parameters:\n    * **backend**: the name of backend to use, `db` or `file`\n    * *backend specific options*\n\n\n## Backends options:\n\n### db\n* Description: is the database (mysql/mariadb) backend\n* options:\n    * **host**: DB host to connect to\n    * **port**: port of MariaDB server is running on\n    * **user**: login of MariaDB server to connect with\n    * **password**: password of user\n    * **delete_limit**: apply a LIMIT to DELETE statement\n    * **select_limit**: apply a LIMIT to SELECT statement\n    * **bulk_insert**: data are inserted in DB every builk_insert rows\n    * **deleted_column**: name of column that holds the date of soft delete, is\n      also used to filter table to archive, it means that the table must have\n      the deleted_column to be archived\n    * **where**: the literal SQL where applied to the select statement\n    Ex: where=${deleted_column} \u003c= SUBDATE(NOW(), INTERVAL ${retention})\n    * **foreign_key_check**: true or false if set to false disable foreign key\n      check (default true)\n    * **retention**: how long time of data to keep in database (SQL format: 12\n      MONTH, 1 DAY, etc..)\n    * **excluded_databases**: comma, cariage return or semicolon separated\n      regexp of DB to exclude when specfiying '*' as database. The following DB\n      are akways ignored:  'mysql', 'performance_schema', 'information_schema'\n    * **excluded_tables**: comma, cariage return or semicolon separated regexp\n      of DB to exclude when specifying '*' as table. Ex: shadow_.*,.*_archived\n    * **db_suffix**: a non mendatory suffix to apply to the archiving DB. The\n    default suffix '_archive' is applied if you archive on same host than\n    source without setting a db_suffix or table_suffix (avoid reading and\n    writing on the same db.table)\n    * **table_suffix**: apply a suffix to the archiving table if specified\n\n### file\n* Description: is the file archiving destination type, it writes SQL data in a\n  file using one or several formats (supported: SQL, CSV)\n    * **directory**: the directory path where to archive data. You may use the\n      {date} keyword to append automaticaly the date to the directory path.\n      (/backup/archive_{date})\n    * **formats**: a comma, semicolon or cariage return separated list that\n      define the format in witch archive the data (csv, sql)\n\nYou've developed a new cool feature ? Fixed an annoying bug ? We'd be happy\n\nto hear from you !\n\nHave a look in [CONTRIBUTING.md](https://github.com/ovh/osarchiver/blob/master/CONTRIBUTING.md)\n\n# Related links\n\n * Contribute: https://github.com/ovh/osarchiver/blob/master/CONTRIBUTING.md\n * Report bugs: https://github.com/ovh/osarchiver/issues\n\n# License\n\nSee https://github.com/ovh/osarchiver/blob/master/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fosarchiver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovh%2Fosarchiver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fosarchiver/lists"}