{"id":23870081,"url":"https://github.com/yuyatinnefeld/parallel-programming","last_synced_at":"2025-02-22T18:27:48.328Z","repository":{"id":87113009,"uuid":"381662113","full_name":"yuyatinnefeld/parallel-programming","owner":"yuyatinnefeld","description":"📕 Learning Guide 📕 | Parallel Programming (Python \u0026 Java)","archived":false,"fork":false,"pushed_at":"2022-04-24T16:09:08.000Z","size":2826,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-03T13:52:31.379Z","etag":null,"topics":["parallel-computing","parallel-programming"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuyatinnefeld.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-30T10:23:43.000Z","updated_at":"2023-06-01T06:33:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"88aac6d5-d351-4812-a0a9-8b285548874f","html_url":"https://github.com/yuyatinnefeld/parallel-programming","commit_stats":null,"previous_names":["yuyatinnefeld/parallel-programming"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fparallel-programming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fparallel-programming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fparallel-programming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuyatinnefeld%2Fparallel-programming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuyatinnefeld","download_url":"https://codeload.github.com/yuyatinnefeld/parallel-programming/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240217308,"owners_count":19766745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel-computing","parallel-programming"],"created_at":"2025-01-03T13:52:11.977Z","updated_at":"2025-02-22T18:27:48.322Z","avatar_url":"https://github.com/yuyatinnefeld.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e Parallel Programing (Python / Java)\u003c/h1\u003e \u003cbr\u003e\n\u003ch2\u003e🚀 Table of Contents 🚀 \u003c/h2\u003e\n\n- [About](#about)\n- [Info](#info)\n- [Concept](#concept)\n- [Setup](#setup)\n\n\n## ⚡ About ⚡ \nIn this project you can learn parallel programming (Python \u0026 Java) to execute multiple instructions simultaneously, increases the overall processing throughput\nThis can help to writing faster and more efficient applications. \n\n## Info\n- https://en.wikipedia.org/wiki/Scheduling_(computing)\n\n## Concept\n\n### Flynn's Taxonomy\n\n![GitHub Logo](/images/flynns_taxonomy.png)\n- SISD (Single Instruction, Single Data)\n- SIMD (Single Instruction, Multiple Data)\n- MISD (Multiple Instruction, Single Data)\n- MIMD (Multiple Instruction, Multiple Data)\n  \n![GitHub Logo](/images/flynns_taxonomy2.png)\n\n\n### Parallel Programming Architecture\n- SPMD (Simple Program, Multiple Data)\n- MPMD (Multiple Program, Multiple Data) =\u003e Manager \u0026 Worker Node\n\n## Memory\n### Shared Memory (all CPUs access the same Memory with the global address space)\n    - UMA (Uniform Memory Access)\n    - NUMA (Non-Uniform Memory Access)\n\n\n![GitHub Logo](/images/uma.png)\n\n![GitHub Logo](/images/numa.png)\n\n### Distributed Memory (all CPUs)\n    - Scalable\n    - Linear\n    - Fixed\n\n![GitHub Logo](/images/dsm.png)\n\n## Threads and Process\nPython\n\u003e example code: project/examples/multiple_threads.py\n\n\u003e example code: project/examples/multiple_processes.py\n\n\n### Threads\n- Thread is an execution unit = part of a process.\n- Threads are \"lightweight\" - require less overhead to create and terminate\n- OS can switch between threads faster than processes\n\n### Inter-process Communication (IPC)\n- Socket and pipes\n- Shared Memory\n- RPC\n\n### Concurrency\n\n![GitHub Logo](/images/concurrency.png)\n- Application only has ONE CPU and makes progress on multiple tasks, but not same time\n\n### Parallel Execution\n\n![GitHub Logo](/images/parallel_execution.png)\n- Application has MANY CPUs and makes progress on multiple task at once\n\n### Parallelism\n\n![GitHub Logo](/images/parallelism.png)\n- Application splits its tasks up into smaller subtasks which can be processed in parallel, for instance on multiple CPUs at the exact same time. \n- Thus, parallelism does not refer to the same execution model as parallel concurrent execution - even if they may look similar on the surface.\n\n### Parallel Hardware\n- Multi-Core Processors\n- GPU\n- Computer Cluster\n\n\n### GIL (Global interpreter lock)\n- GIL is a mutex (mutual exclusion) that allows ONLY ONE thread to hold the control of the Python interpreter\n- only one thread can be in a state of execution at any point in time.\n- This mutex is necessary mainly because CPython's memory management is not thread-safe.\n- By I/O Bounced Application GIL is NOT a bottleneck\n- BY CPU Bounced Application GIL can negatively impact performance\n\n### Scheduling\n\nPython\n\u003e example code: project/examples/execution_scheduling.py\n\nJava\n\u003e example code: com.parallel_programming.ExecutionSchedulingDemo.java\n\n#### Scheduling Algorithms\n- First come, first served\n- Shortest job next\n- Priority\n- Shortest remaining time\n- Round-robin\n- Multiple-level queues\n\n#### Scheduling Goals\n- Maximize throughput\n- Maximize fairness\n- Minimize wait time\n- Minimize latency\n\n### Lifecycle of Thread\n\n![GitHub Logo](/images/lifecyle_of_thread.png)\n\nThread Status\nPython: OBJECT.is_alive() =\u003e false, true\nJava: OBJECT.getState() =\u003e NEW, RUNNABLE, BLOCK, WAITING, TIMED_WAITING, TERMINATED\n\nThread Priority\n1 = low, 10 = high\n\nCreate Thread in JAVA\n- create thread class which extends Thread =\u003e Thread olivia = new ChefOlivia();\n- implement the runnable interface =\u003e Thread olivia = new Thread(new ChefOlivia1());\n\nPython\n\u003e example code: project/examples/thread_lifecycle.py\n\nJava\n\u003e example code: com.parallel_programming.ThreadLifecycleDemo.java\n\u003e example code: com.parallel_programming.RunnableDemo.java\n\nThread vs. Runnable (Java)\nExtending Thread class\n- Cannot extend additional classes\n- each instance is a separate object\n\nImplementing Runnable interface (more flexible)\n- can implement other interfaces and extend another class\n- multiple threads can share a single runnable object\n\n### Daemon (Background) Thread\n\n- Daemon thread is a low priority thread (in context of JVM) that runs in background to perform tasks such as garbage collection (gc) etc.\n- When a new thread is created it inherits the daemon status of its parent.\n- they do not prevent the JVM from exiting (even if the daemon thread itself is running) when all the user threads (non-daemon threads) finish their execution.\n- We do not care whether Daemon thread is running or not\n- If the daemon thread is running, it does shutdown itself later\n- Python =\u003e by default non-daemon\n- Python =\u003e set the daemon property to change status before starting thread\n\nPython:\n\u003e example code: project/examples/daemon_threads.py\n\nJava:\n\u003e example code: com.parallel_programming.DaemonThreadDemo.java\n\n## Mutex\n### Data Race\nPython:\n\u003e example code: project/examples/data_race.py\n\nJava:\n\u003e example code: com.parallel_programming.DataRaceDemo.java\n\n\n- Data Race Problem =\u003e two or more concurrent threads access the same memory location and reading, changing \u0026 overwriting wrong information\n- Detecting Data Race is very hard\n  \n### Mutex\nPython:\n\u003e example code: project/examples/mutex0.py\n\n\n\u003e example code: project/examples/mutex1.py\n\nJava:\n\u003e example code: com.parallel_programming.MutexDemo0.java\n\nSolution\n\u003e example code: com.parallel_programming.MutexDemo1.java\n\nSolution with AtomicInteger\n\u003e example code: com.parallel_programming.MutexDemo2.java\n\nSolution with SynchronizedMethodDemo\n\u003e example code: com.parallel_programming.SynchronizedMethodDemo.java\n\nSynchronized vs. Locks by Java:\n- Synchronized =\u003e easier to implement and prevents many pitfalls of locks\n- Locks =\u003e provide more flexibility to implement certain algorithms\n\n- Mutex (Mutual Exclution) is used for the concurrency control to avoid Data Race\n- By Mutex can only one thread or process can possess at a time\n- Limits access to critical section (ex. data storing)\n- atomic operations = execute as a single action, relative to other threads\n- cannot be interrupted by other concurrent threads\n- acquiring a lock = if lock is already taken, block/wait for it to be available\n- keep protected sections of code AS SHORT AS POSSIBLE\n\n## Locks\nPython:\n\u003e example code: project/examples/reentrant_lock.py\n\nJava:\n\u003e example code: com.parallel_programming.ReentrantLock.java\n\n- Deadlock =\u003e all processes and threads are unable to continue executing because they are waiting for another members\n- Reentrant Mutex\n    - can be locked multiple time by the same thread\n    - must be unlocked as many times as it was locked\n- Common Terms\n    - Reentrant mutex\n    - Reentrant lock\n    - Recursive mutex\n    - Recursive lock\n    \n### Lock vs. RLock / ReentrantLock\n- Lock can be released by a different thread than was used to acquire it\n- RLock must be released by the same thread that acquired it\n\n### Try lock\n- Non-blocking lock/acquire method for mutex\n- if the mutex is available, lock it and return TRUE\n- if the mutex is not available, immediately return FALSE\n- used mostly for the multiple threads which have multiple tasks\n\nPython:\n\u003e example code: project/examples/nonblocking_acquire0.py\n\n\u003e example code: project/examples/nonblocking_acquire1.py\n \nJava:\n\u003e example code: com.parallel_programming.TryLockDemo0.java\n\n\u003e example code: com.parallel_programming.TryLockDemo1.java\n\n\n### Reader-Writer Lock / shared_mutex\n- SHARED READ =\u003e multiple threads at once\n- EXCLUSIVE WRITE =\u003e only one thread at a time\n\nPython:\n\u003e example code: project/examples/readwrite_lock0.py\n\n\u003e example code: project/examples/readwrite_lock1.py\n\nJava:\n\u003e example code: com.parallel_programming.TryLockDemo0.java\n\n\u003e example code: com.parallel_programming.TryLockDemo1.java\n\n\n```bash\npip install readerwriterlock\n```\n\n#### RWLock\nmarker = rwlock.RWLockFair()\n- fair priority for reader/writers\n- gen_rlock() generates a reader lock object\n- gen_wlock() generates a writer lock object\n\nJava:\nReentrantReadWriteLock.ReadLock\nReentrantReadWriteLock.WriteLock\n\n\n## Liveness (how to avoid Deadlock)\n- Liveness guarantees are important properties in operating systems and distributed systems\n- Liveness requires a system to make progress despite the fact that its concurrently executing components (\"processes\") may have to \"take turns\" in critical sections, parts of the program that cannot be simultaneously run by multiple processes\n- Lock Ordering\n  - ensure locks are always taken in the same order by any thread\n  - example code: project/examples/deadlock1.py\n- Lock Timeout\n  - Put a timeout on lock attempts\n  \nPython:\n\u003e example code: project/examples/deadlock0.py\n\nJava:\n\u003e example code: com.parallel_programming.DeadlockDemo.java\n\n\n  \n### Abandoned Lock\nPython:\n\u003e example code: project/examples/abandoned_lock0.py\n\n\u003e example code: project/examples/abandoned_lock1.py\n\n\u003e example code: project/examples/abandoned_lock2.py\n\nJava:\n\u003e example code: com.parallel_programming.AbandonedLockDemo.java\n\n\u003e example code: com.parallel_programming.AbandonedLockDemo.java\n\n\n### Starvation Lock\nPython:\n\u003e example code: project/examples/starvation.py\n\nJava:\n\u003e example code: com.parallel_programming.Starvation.java\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyatinnefeld%2Fparallel-programming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuyatinnefeld%2Fparallel-programming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuyatinnefeld%2Fparallel-programming/lists"}