{"id":16388244,"url":"https://github.com/tlesort/continual_learning_data_former","last_synced_at":"2025-03-21T02:31:28.959Z","repository":{"id":78327666,"uuid":"198824802","full_name":"TLESORT/Continual_Learning_Data_Former","owner":"TLESORT","description":"A pytorch compatible data loader to create sequence of tasks for Continual Learning","archived":false,"fork":false,"pushed_at":"2020-04-14T09:11:25.000Z","size":1366,"stargazers_count":33,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-17T19:52:20.204Z","etag":null,"topics":["continual-learning","continualai","deep-learning","incremental-learning","lifelong-learning","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TLESORT.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-25T12:12:08.000Z","updated_at":"2024-01-04T16:36:04.000Z","dependencies_parsed_at":"2023-03-12T04:08:46.210Z","dependency_job_id":null,"html_url":"https://github.com/TLESORT/Continual_Learning_Data_Former","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TLESORT%2FContinual_Learning_Data_Former","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TLESORT%2FContinual_Learning_Data_Former/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TLESORT%2FContinual_Learning_Data_Former/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TLESORT%2FContinual_Learning_Data_Former/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TLESORT","download_url":"https://codeload.github.com/TLESORT/Continual_Learning_Data_Former/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244725552,"owners_count":20499628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["continual-learning","continualai","deep-learning","incremental-learning","lifelong-learning","machine-learning"],"created_at":"2024-10-11T04:28:40.812Z","updated_at":"2025-03-21T02:31:28.944Z","avatar_url":"https://github.com/TLESORT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Continuum: A dataloader for continual learning\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/9273eb0f97b946308248b0007e054e54)](https://app.codacy.com/app/TLESORT/Continual_Learning_Data_Former?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=TLESORT/Continual_Learning_Data_Former\u0026utm_campaign=Badge_Grade_Dashboard)\n[![DOI](https://zenodo.org/badge/198824802.svg)](https://zenodo.org/badge/latestdoi/198824802)\n\n\n### Intro\n\nThis repositery proprose several script to create sequence of tasks for continual learning. The spirit is the following : \nInstead of managing the sequence of tasks while learning, we create the sequence of tasks first and then we load tasks \none by one while learning.\n\nIt makes programming easier and code cleaner.\n\n### Installation\n\n```bash\ngit clone https://github.com/TLESORT/Continual_Learning_Data_Former\ncd Continual_Learning_Data_Former\npip install .\n```\n\n### Few possible invocations\n\n-   Disjoint tasks\n\n```python\nfrom continuum.disjoint import Disjoint\n\n#MNIST with 10 tasks of one class\ncontinuum = Disjoint(path=\"./Data\", dataset=\"MNIST\", task_number=10, download=True, train=True)\n```\n-   Rotations tasks\n\n```python\nfrom continuum.rotations import Rotations\n\n#MNIST with 5 tasks with various rotations\ncontinuum = Rotations(path=\"./Data\", dataset=\"MNIST\", tasks_number=5, download=True, train=True, min_rot=0.0,\n                 max_rot=90.0)\n```\n\n-   Permutations tasks\n\n```python\nfrom continuum.permutations import Permutations\n\n#MNIST with 5 tasks with different permutations\ncontinuum = Permutations(path=\"./Data\", dataset=\"MNIST\", tasks_number=1, download=False, train=True)\n```\n\n### Use example\n\n```python\nfrom continuum.disjoint import Disjoint\nfrom torch.utils import data\n\n# create continuum dataset\ncontinuum = Disjoint(path=\".\", dataset=\"MNIST\", task_number=10, download=True, train=True)\n\n# create pytorch dataloader\ntrain_loader = data.DataLoader(data_set, batch_size=64, shuffle=True, num_workers=6)\n\n#set the task on 0 for example with the data_set\ncontinuum.set_task(0)\n\n# iterate on task 0\nfor t, (data, target) in enumerate(train_loader):\n    print(target)\n    \n#change the task to 2 for example\ncontinuum.set_task(2)\n\n# iterate on task 2\nfor t, (data, target) in enumerate(train_loader):\n    print(target)\n\n# We can visualize samples from the sequence of tasks\nfor i in range(10):\n    continuum.set_task(i)\n    \n    folder = \"./Samples/disjoint_10_tasks/\"\n    \n    if not os.path.exists(folder):\n        os.makedirs(folder)\n    \n    path_samples = os.path.join(folder, \"MNIST_task_{}.png\".format(i))\n    continuum.visualize_sample(path_samples , number=100, shape=[28,28,1])\n    \n```\n\n\n### Task sequences possibilities\n\n-   **Disjoint tasks** : each task propose new classes\n-   **Rotations tasks** : each tasks propose same data but with different rotations of datata point\n-   **Permutations tasks** : each tasks propose same data but with different permutations of pixels\n-   **Mnist Fellowship task** : each task is a new mnist like dataset (this sequence of task is an original contribution of this repository)\n\n### An example with MNIST 5 dijoint tasks\n\n|\u003cimg src=\"/Samples/disjoint_5_tasks/MNIST_task_0.png\" width=\"150\"\u003e|\u003cimg src=\"/Samples/disjoint_5_tasks/MNIST_task_1.png\" width=\"150\"\u003e|\u003cimg src=\"/Samples/disjoint_5_tasks/MNIST_task_2.png\" width=\"150\"\u003e|\u003cimg src=\"/Samples/disjoint_5_tasks/MNIST_task_3.png\" width=\"150\"\u003e|\u003cimg src=\"/Samples/disjoint_5_tasks/MNIST_task_4.png\" width=\"150\"\u003e|    \n|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|\n|Task 0 | Task 1 | Task 2 | Task 3 | Task 4|\n\nMore examples at [Samples](/Samples)\n\n### Datasets\n\n-   Mnist\n-   fashion-Mnist\n-   kmnist\n-   cifar10\n-   Core50/Core10\n\n### Some supplementary option are possible\n-   The number of tasks can be choosed (1, 3, 5 and 10 have been tested normally)\n-   Classes order can be shuffled for disjoint tasks\n-   We can choose the magnitude of rotation for rotations mnist\n\n\n\n\n\n### Citing the Project\n\n```Array.\u003cstring\u003e\n@software{timothee_lesort_2020_3605202,\n  author       = {Timothée LESORT},\n  title        = {Continual Learning Data Former},\n  month        = jan,\n  year         = 2020,\n  publisher    = {Zenodo},\n  version      = {v1.0},\n  doi          = {10.5281/zenodo.3605202},\n  url          = {https://doi.org/10.5281/zenodo.3605202}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlesort%2Fcontinual_learning_data_former","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftlesort%2Fcontinual_learning_data_former","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlesort%2Fcontinual_learning_data_former/lists"}