{"id":24549969,"url":"https://github.com/jonasfrey96/asl","last_synced_at":"2025-09-05T03:33:37.207Z","repository":{"id":38352064,"uuid":"322561819","full_name":"JonasFrey96/ASL","owner":"JonasFrey96","description":"My Master Thesis at the ASL supervised by Hermann Blum, Francesco Milano and Dr. Cadena Cesar","archived":false,"fork":false,"pushed_at":"2022-12-20T13:50:43.000Z","size":290321,"stargazers_count":11,"open_issues_count":2,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-15T20:15:23.358Z","etag":null,"topics":["continual-learning","incremental-learning","semantic-segmentation-network"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JonasFrey96.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-18T10:25:21.000Z","updated_at":"2024-03-17T15:10:32.000Z","dependencies_parsed_at":"2023-01-30T00:46:10.203Z","dependency_job_id":null,"html_url":"https://github.com/JonasFrey96/ASL","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JonasFrey96/ASL","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasFrey96%2FASL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasFrey96%2FASL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasFrey96%2FASL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasFrey96%2FASL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JonasFrey96","download_url":"https://codeload.github.com/JonasFrey96/ASL/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JonasFrey96%2FASL/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273707346,"owners_count":25153724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["continual-learning","incremental-learning","semantic-segmentation-network"],"created_at":"2025-01-23T00:17:45.957Z","updated_at":"2025-09-05T03:33:32.192Z","avatar_url":"https://github.com/JonasFrey96.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 1. ASL Master Thesis\n\nhttps://markdownlivepreview.com/\nThis is the repository accompanying my master's thesis.\n \nAdditionally, we created the accompanying repositories:  \n 1.) Pose estimation using [ORBSLAM2 Fork](https://github.com/JonasFrey96/orb_slam_2_ros)  \n 2.) Optical Flow estimation [RAFT Fork](https://github.com/JonasFrey96/RAFT)   \n 3.) [Kimera Semantics Fork](https://github.com/JonasFrey96/Kimera-Semantics) (Allows fast protobuf export of semantics)  \n 4.) [Kimera Interfacer](https://github.com/JonasFrey96/Kimera-Interfacer) (Generate maps and pseudo labels for ScanNet and Lab data)  \n \nWe only elaborate on how to run all the experiments and reproduce the results.  \nFor the technical background, I refer to my [Master's Thesis](https://github.com/JonasFrey96/ASL/tree/main/docs/Jonas_Frey_Master_Thesis.pdf). \n\n## Repository Overview:\nAll code related to continual learning experiments is contained in the **src** folder.  \n\n* **src** Contains all code for training the neural network\n  * **src/supervisor.py** Starts all the training tasks.  \n  * **src/train_task.py** Can be called directly or is executed by the supervisor.  \nSet's up logger, trainer, and model and executes a single task.  \n    * **src/lightning/lightning.py** Contains the full neural network model and logging.  \nIt is the best starting point to get familiar with the implementation. \n    * **src/task/task_generator.py** Depending on the configuration returns a list of parameters to create multiple training and associated test datasets for each task. \n\n* **docker** Contains the docker files to generate the pseudo labels using 3D mapping  \n  * **docker/orbslam2_ros** ORBSLAM2 Implementation  \n  * **docker/semantics** Kimera Semantics + Kimera Interfacer  \n  * **docker/vio** Not used in the final report (Kimera VIO)  \n  * **docker/maplab** Not used in the final report (Maplab)  \n\n* **notebooks** \n  * **notebooks/report** includes all the jupyter notebooks used to generate all figures in the report  \n \n* **cfg** includes configuration files for:  \n  * **cfg/env** Setting up your environment file\n  * **cfg/exp** Experiment files to run all experiments\n  * **cfg/conda** Python conda enviornment\n\n\n## Continual Learning Experiments\n\n### Getting started:\nClone the repository:  \n```bash \ncd ~ \u0026\u0026 git clone https://github.com/JonasFrey96/ASL\n```\n \nCreate and activate conda environment:\n```bash \ncd ~/ASL \u0026\u0026 conda env create -f cfg/conda/cl.yml\nconda activate cl \n```\nAssumes CUDA 11.0 is installed and uses ```pytorch=1.7.1``` and ```pytorch-lightning=1.2.0```.  \n\nYou may have to adapt the conda environment depending on your system. \n\n#### Environment Configuration Explained\nSetting up the enviornment file depending on your system.  \nYou can either create a new enviornment file or edit an existin one in the ```env``` folder.  \nFor example you can create ```ASL/cfg/env/your_env_name.yml```, with the following content:\n\n```yaml\n# Configuration\nworkstation: True                 \n# If False uses Proxy and copies Dataset to $TMPDIR folder\n# -\u003e allows to train on Leonhard and Euler Cluster\n\nbase: path_to_your_logging_directory # Here all your experiments are logged\n \n# Datasets\ncityscapes: ../datasets/cityscapes\nnyuv2: ../datasets/nyuv2\ncoco2014: ../datasets/coco\nmlhypersim: ../datasets/mlhypersim\nscannet: ../datasets/scannet\nscannet_frames_25k: ../datasets/scannet_frames_25k\ncocostuff164k: ../datasets/cocostuff164k\n \n# Pseudo Labels Path\nlabels_generic: ../datasets/\n```\n \nThe correct env file is chosen based on the global variable ```ENV_WORKSTATION_NAME```.\n```bash\nexport ENV_WORKSTATION_NAME=\"\"\"your_env_name\"\"\"\n```\nIf available set your NeptuneAPI Token to log all the experiments to NeptuneAI:\n```bash\nexport NEPTUNE_API_TOKEN=\"\"\"YOUR_NEPTUNE_KEY\"\"\"\n```\nWith this you have setup the configuration. \nHow to download and install the datasets is explained in [Dataset Preperation](Dataset-Preperation).\n \n### Running Experiments\n \nRunning a full CL-scenario:\n```bash\ncd ~/ASL \u0026\u0026 python supervisor.py --exp=cfg/exp/MA/scannet/25k_pretrain/pretrain.py\n```\n```exp``` arguments provides the correct experiment configuration path.\n \n#### Experiment Configuration Explained\n \n```yaml\nname: test/run1 #Name of the experiment folders\nneptne_project_name: YourName/YourProject #NeptuneAI project name  \noffline_mode: False # False uses tensorboard, True uses NeptuneAI\nTODO\n```\n#### Overview Provided Experiments\n**Hypersim**\n1. basic: Standard Memory Buffer\n2. memory_size: Evaluate different Memory Sizes\n3. sgd_setting: Training Setting Search\n \n \n\n**ScanNet**\n1. basic: Standard Memory Buffer\n2. buffer_filling: Different Buffer Filling Strategies\n3. buffer_size: Buffer Size\n4. latent_replay: Latent Replay Experiment\n5. no_replay: Catastrophic Forgetting\n6. replay_ratios: Replay Ratios\n7. single_task: Train on all datasets simultaneously\n8. student_teacher: Experiments with soft and hard teacher for replay\n \n## Supervision Signal Generation\n\nHow to run\n\n### CRF \u0026 SLIC\nTODO\n \n### Optical Flow\nTo create the optical flow we used [RAFT](https://github.com/princeton-vl/RAFT)  \nDownload the pre-trained models.  \nYou can create the optical flow for the ScanNet dataset using the jupyter Notebook **notebook/optical_flow/**  \nYou need to make sure to add the correct paths to import the RAFT model and the dataset.  \nThe notebook **Report/pseudo_labels_flow.ipynb** creates all reported plots and figures.  \n\n### 3D Mapping\nResult:  \n![](https://github.com/JonasFrey96/ASL/tree/main/docs/create_map_gt_rviz.gif)\n\n## Dataset Preparation\n### Hypersim\nDownload the Hypersim Dataset and extract the files following the [author's instructions](https://github.com/apple/ml-hypersim)  \nWe additionally provide a .tar file with the first 10 scenes.\n \nFolder structure:\n```\nmlhypersim\n  ai_XXX_XXX\n    images\n      scene\n        scene_cam_00_final_hdf5\n          frame.0000.color.hdf5\n          ...\n          frame.XXXX.color.hdf5\n        scene_cam_00_geometry_hdf5\n          frame.0000.semantic.hdf5\n          ...\n          frame.XXX.semantic.hdf5\n  ...\n  ...\n```\t\t\n### ScanNet\nDownload the ScanNet dataset and follow the [author's instructions](http://www.scan-net.org/) to extract the .sens files into individual files.  \nAlso, make sure to download the 25k files if you want to perform the pre-training.  \nHere it's important to delete samples from the first 10 scenes to avoid overlapping continual learning and pre-training tasks.  \nWe additionally provide a .tar file with the extracted first 10 scenes.  \n \nFolder structure:\n```\nscannet\n  scannet_frames_25k\n    scene0010_00\n      color\n        000000.jpg\n        ...\n        XXXXXX.jpg\n      depth\n        000000.png\n        ...\n        XXXXXX.png\n      label\n        000000.png\n        ...\n        XXXXXX.png\n      pose\n        000000.txt\n        ...\n        XXXXXX.txt\n      intriniscs_color.txt\n      intrinsics_depth.txt\n    ...\n    ...\n    scene0706_00\n      ...\n  scans\n    scene0000_00\n      color\n        000000.jpg\n        ...\n        XXXXXX.jpg\n      depth\n        000000.png\n        ...\n        XXXXXX.png\n      label-filt\n        000000.png\n        ...\n        XXXXXX.png\n      pose\n        000000.txt\n        ...\n        XXXXXX.txt\n      intrinsics\n        intriniscs_color.txt\n        intrinsics_depth.txt\n    ...\n    scene0009_02\n    ...\n\n```\n### COCO2014\nThis dataset is only used for pre-training.\nDownload the MS COCO 2014 dataset https://cocodataset.org/#home.\n \nFolder structure:\n```\ncoco\n  train2014\n    COCO_train2014_000000000009.jpg\n    ...\n    COCO_train2014_000000581921.jpg\n  val2014\n    COCO_val2014_000000000042.jpg\n    ...\n    COCO_val2014_000000581929.jpg\n  annotations\n    instances_train2014.json\n    instances_val2014.json\n```\n### COCO164k\nThis dataset is only used for pre-training.\nDownload the MS COCO 2017 dataset https://cocodataset.org/#home.\n \nFolder structure:\n```\ncocostuff164k\n  images\n    train2017\n      000000000009.jpg\n      ...\n      000000581929.jpg\n    val2017\n      000000000139.jpg\n      ...\n      000000581781.jpg\n  annotations\n    instances_train2014.json\n    instances_val2014.json\n```\n\n# 2. Acknowledgment \n\n- The authors of [Fast-SCNN](https://arxiv.org/pdf/1902.04502.pdf)  \n- TRAMAC implementing [Fast-SCNN in PyTorch](https://github.com/Tramac/Fast-SCNN-pytorch)  \n- The authors of [RAFT](https://github.com/princeton-vl/RAFT)  \n-  The authors of [ORBSLAM2](https://github.com/appliedAI-Initiative/orb_slam_2_ros)  \n- People at \u003chttp://continualai.org\u003e for the inspiration\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonasfrey96%2Fasl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonasfrey96%2Fasl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonasfrey96%2Fasl/lists"}