{"id":19066265,"url":"https://github.com/epfml/gradientalignment","last_synced_at":"2025-10-17T05:51:58.453Z","repository":{"id":109046750,"uuid":"371413021","full_name":"epfml/GradientAlignment","owner":"epfml","description":null,"archived":false,"fork":false,"pushed_at":"2022-03-31T06:58:59.000Z","size":6150,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-02T14:28:37.071Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epfml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-27T15:01:40.000Z","updated_at":"2022-01-03T08:20:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"29c74c56-2b58-462c-97ad-fa72d361a5cf","html_url":"https://github.com/epfml/GradientAlignment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2FGradientAlignment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2FGradientAlignment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2FGradientAlignment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epfml%2FGradientAlignment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epfml","download_url":"https://codeload.github.com/epfml/GradientAlignment/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240118419,"owners_count":19750491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T00:55:41.385Z","updated_at":"2025-10-17T05:51:53.404Z","avatar_url":"https://github.com/epfml.png","language":"Python","readme":"# Implicit Gradient Alignment in Distributed and Federated Learning\n\n\n**Requirements:**\nIt requires Python 3 with the installation of torch, torchvision and wandb is recommended.\nWe recommend to use pip to install them\n\npip install torch\n\npip install torchvision \n\n**Weights and Biases:**\nTo log the experiments using wandb, a free account of Weights and biases is needed. Please go to https://www.wandb.com to create a free account.\nOnce your account is created you need to instal wandb.\nWith pip3 it requires to run:\n\npip install wandb\n\nThen login in your account by running in the terminal the line:\n\nwandb login WANDB_API_KEY\n\nwhere WANDB_API_KEY can be found in https://app.wandb.ai/settings after loging in. \n\nYou are done and ready to run experiments.\n\n**Code samples:**\n\nFor FedAvg run it like:\n\npython interface.py --batch_size=100 --gpu_ids=[0] --learning_rate=0.2 --model=cnn_cifar --nsubsets=10 --nprocesses=10 --nrounds=2001 --nlogging_steps=10 --port=28 --nsteps=20 --task=CIFAR10 --wandb_exp_name=test_cifar --wandb --weight_decay=0.001 --percentage_iid=1.0\n\nFor MiniBatch SGD (largeBatchSGD) run it like:\n\npython interface.py --batch_size=2400 --gpu_ids=[0,1,3,4] --learning_rate=0.2 --model=cnn_mnist --nsubsets=47 --nprocesses=10 --nrounds=2001 --nlogging_steps=5 --port=83--task=EMNIST --wandb_exp_name=test_emnist --wandb --weight_decay=0.001 --percentage_iid=0.0 --largeBatchSGD\n\nFor SCAFFOLD run like:\n\npython interface.py --batch_size=2400 --gpu_ids=[0] --learning_rate=0.05 --model=cnn_mnist --nsubsets=47 --nprocesses=10 --nrounds=2001 --nlogging_steps=5 --port=49 --server_momentum=0.0 --nsteps=10 --task=EMNIST --wandb_exp_name=test_emnist --wandb --weight_decay=0.001 --percentage_iid=0.0 --model=cnn_mnist --scaffold\n\nFor FedGA run like:\n\npython interface.py --batch_size=2400 --gpu_ids=[0] --learning_rate=0.05 --model=cnn_mnist --nprocesses=10 --nrounds=2001 --nsubsets=47 --nlogging_steps=5 --port=59 --server_momentum=0.0 --nsteps=10 --task=EMNIST --wandb_exp_name=test_emnist --wandb --weight_decay=0.001 --percentage_iid=0.0 --grad_align --beta=0.025\n\n**Possible variables:**\n\n--**nsubsets**, default=100, type=int, choices=range(1, 1001), help=\"Total number of clients to be used\"\n\n--**nprocesses**, default=1, type=int, choices=range(1, 48), help=\"number of processes, i.e., number of clients sampled in each round\"\n\n--**batch_size**, default=50, type=int, choices=range(1, 5001), help=\"batch size of every process\"\n\n--**nsteps**, default=5, type=int, choices=range(0, 2001), help=\"number of local steps per round\"\n\n--**nrounds**, default=1001, type=int, choices=range(1, 400002), help=\"number of rounds\"\n\n--**nlogging_steps**, default=100, type=int, choices=range(1, 1001), help=\"number of rounds for logging, i.e., there is logging whenever round % nlogging_steps == 0\"\n\n--**learning_rate**, default=0.01, type=float, help=\"learning rate for each of the local steps\"\n\n--**task**, default=\"MNIST, type=str, choices={\"MNIST\", \"EMNIST\", \"CIFAR10\", \"CIFAR100\"}, help=\"Dataset to be used\"\n\n--**model**, default=\"cnn_mnist\", choices={\"cnn_mnist\", \"cnn_cifar\", \"resnet18\"}, type=str, help=\"model to be used\"\n\n--**gradient_clipping**, boolean, help=\"Clip gradients of workers\"\n\n--**clipping_norm**, default=1.0, type=float,\n                    help=\"max magnitud of L_2 norm when using gradient clipping with full gradient\"\n--**target_acc**, default=100.0, type=float, help=\"Target Accuracy of this run, it stops when reaching this accuracy if the parameter is set\"\n\n--**wandb**, boolean, help=\"Use weights and biases logging system. You need an account and to be logged in for this to work.\"\n\n--**wandb_exp_name**, default='random_experiments', type=str, help=\"Experiment name in wandb\"\n\n--**load_seed, type=int**, default=-1, help=\"Used to set the random seed of the algorithm.\"\n\n--**scheduler**, default=\"[1000000000], type=str, help=\"scheduler decreases the learning rates when the the round is in the given list\"\n\n--**lr_decay**, default=0.1, type=float, help=\"Learning rate decay\"\n\n--**port**, default=21, type=int, choices=range(0, 100), help=\"final digit of port number for the distributed process. No two processes with the same port can run simultaneously. \"\n\n--**gpu_ids**, default=\"[0,1], type=str, help=\"a list with the indices of the gpus to run the experiment. Processes will be distributed evenly among them.\"\n\n--**percentage_iid**, default=1.0, type=float, help=\"percentage of iid data in workers' partition. 1.0 for fully IID data and 0.0 for completely heterogeneous data.\"\n\n--**weight_decay**, default=0.001, type=float, help=\"Optimizer's weight decay\"\n\n--**server_momentum**, default=0.0, type=float, help=\"server momentum\"\n\n--**momentum**, default=0.0, type=float, help=\"local momentum\"\n\n--**server_learning_rate**, default=1.0, type=float, help=\"server step learning rate**, default is 1.0\"\n\n--**server_nesterov**, boolean, help=\"Use Nesterov's server momentum\"\n\n--**batchnorm**, boolean, help=\"Use batch normalization (only available in Resnet Models)\"\n\n\n--**beta**, default=0.1, type=float, help=\"Corresponds to the beta constant in the FedGA algorithm\"\n\n--**fg_batch_size**, default=-1, type=int, help=\"Full gradient Batch size\"\n\n--**plot_grad_alignment**, boolean, help=\"Set to true to plot the gradient variance after each round.\"\n\n--**fedavg**, boolean, help=\"Use FedAvg algorithm, it is also used by default if no other is set\"\n\n--**grad_align**, boolean, help=\"Use the FedGA algorithm\"\n\n--**scaffold**, boolean, help=\"Use SCAFFOLD algorithm.\"\n\n--**largeBatchSGD**, boolean, help=\"Use largeBatch SGD\"","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fgradientalignment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepfml%2Fgradientalignment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepfml%2Fgradientalignment/lists"}