{"id":13585892,"url":"https://github.com/abhishek-ch/around-dataengineering","last_synced_at":"2025-04-08T16:09:48.272Z","repository":{"id":38798380,"uuid":"296615591","full_name":"abhishek-ch/around-dataengineering","owner":"abhishek-ch","description":"A Data Engineering \u0026 Machine Learning Knowledge Hub","archived":false,"fork":false,"pushed_at":"2024-02-02T13:09:13.000Z","size":12774,"stargazers_count":1123,"open_issues_count":2,"forks_count":225,"subscribers_count":83,"default_branch":"master","last_synced_at":"2025-04-01T15:12:59.095Z","etag":null,"topics":["airflow","data-engineering","datascience","devops","infrastructure","machine-learning","mlops","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhishek-ch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-18T12:31:08.000Z","updated_at":"2025-03-25T07:28:41.000Z","dependencies_parsed_at":"2024-10-12T00:00:46.193Z","dependency_job_id":"e211e1b5-1ba5-409b-8323-b2458751f456","html_url":"https://github.com/abhishek-ch/around-dataengineering","commit_stats":{"total_commits":308,"total_committers":4,"mean_commits":77.0,"dds":"0.11688311688311692","last_synced_commit":"ac7dc54ed259691a297779c12e2f48cabbed70b8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhishek-ch%2Faround-dataengineering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhishek-ch%2Faround-dataengineering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhishek-ch%2Faround-dataengineering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhishek-ch%2Faround-dataengineering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhishek-ch","download_url":"https://codeload.github.com/abhishek-ch/around-dataengineering/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247878022,"owners_count":21011158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","data-engineering","datascience","devops","infrastructure","machine-learning","mlops","spark"],"created_at":"2024-08-01T15:05:12.490Z","updated_at":"2025-04-08T16:09:48.253Z","avatar_url":"https://github.com/abhishek-ch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# A very Long never ending Learning around Data Engineering \u0026 Machine Learning\n\n![](sketchnotes/background.jpeg)\n\n## New Tech\n* [Dragonfly is a faster Redis or Memcached alternative, that I recently tried.](https://www.linkedin.com/posts/iamabhishekchoudhary_github-dragonflydbdragonfly-a-modern-activity-6987806780424617984-sfOk?utm_source=share\u0026utm_medium=member_desktop)\n\n## Interesting Reads\n\n* [How to choose a Distributed Database](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/how_to_choose_db.md)\n* [Cockroach DB Architecture](https://www.linkedin.com/posts/iamabhishekchoudhary_cockroachdbarchitecture-activity-6834117247510855680-VeCF)\n* [Amundsen Review](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/amundsen_review.md)\n* [Deep Dive - Foundation DB](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/foundationdb.md)\n* [The What, Why, and When of Single-Table Design with DynamoDB](https://www.alexdebrie.com/posts/dynamodb-single-table/)\n* [How To Manage And Monitor Apache Spark On Kubernetes](https://www.lightbend.com/blog/how-to-manage-monitor-spark-on-kubernetes-deep-dive-kubernetes-operator-for-spark)\n* [Git is hard: screwing up is easy, and figuring out how to fix your mistakes is fucking impossible](https://ohshitgit.com/)\n* [8 Practical Use Cases of Change Data Capture](https://medium.com/event-driven-utopia/8-practical-use-cases-of-change-data-capture-8f059da4c3b7)\n* [Apache Iceberg- Links](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/apache_iceberg_read.md)\n* [Kubernetes Port Forwarding Manager](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-kuberforwarder-dataengineering-activity-6821345311655555072-aEqx)\n* [Querying Parquet with Precision using DuckDB - Much faster compared to Pandas](https://www.linkedin.com/posts/iamabhishekchoudhary_querying-parquet-with-precision-using-duckdb-activity-6821024627578466304-TgRS)\n* [What is Apache Pinot - Usecases \u0026 Architecture](https://www.linkedin.com/posts/iamabhishekchoudhary_building-latency-sensitive-user-facing-analytics-activity-6818810335286370304-gZAQ)\n* [Change Data Streaming Patterns in Distributedsystems](https://www.linkedin.com/posts/iamabhishekchoudhary_cdcpatterns-activity-6828573352563654656-CDS1)\n* [Cuckoo Hashing - An alternative to chaining and linear probing for collision handling ](https://www.linkedin.com/posts/iamabhishekchoudhary_cuckoo-hashing-activity-6832911622126784512-3rZB)\n* [Riak Database](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/riakdb.md)\n* [Database Indexing](https://www.linkedin.com/posts/iamabhishekchoudhary_databaseindexes-activity-6844509295476924416-Jzto)\n* [Parallel Databases using Map Reduce](https://www.linkedin.com/posts/iamabhishekchoudhary_parallel-databases-using-map-reduce-activity-6858620336083136512-NtTz)\n* [REST vs GraphQL](https://www.linkedin.com/posts/iamabhishekchoudhary_restapi-vs-graphql-activity-6879712973284364288-O8ad)\n* [Linux Namespace \u0026 Control Group(cgroup)](https://www.linkedin.com/posts/iamabhishekchoudhary_namespaces-and-cgroups-the-basis-of-linux-activity-6897116562046672897-B5va)\n* [SQL Lexical Structure](https://www.linkedin.com/posts/iamabhishekchoudhary_database-golang-dataengineering-activity-6911963969658220544-LCxB)\n* [Everything about the Linux kernel](https://www.linkedin.com/posts/iamabhishekchoudhary_github-0xaxlinux-insides-a-little-bit-activity-6932325041955061760-QlwG?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n\n\n## Weekly Digest\n\n* [How #dataengineering get complicated over time](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-realtime-streamprocessing-activity-6808383006567383040-xY-g)\n* [What is eBPF - Sandboxing Programs inside #linux Kernel](https://www.linkedin.com/posts/iamabhishekchoudhary_facebook-google-isovalent-microsoft-and-activity-6831817260462608384-p6QN0)\n* [Absolute Basic Explanation of SSTable \u0026 Log Structured Merge Trees - Sorted String Table \u0026 Faster Random Writes](https://www.linkedin.com/posts/iamabhishekchoudhary_datastructures-cassandra-dataengineering-activity-6833629108484784128-u-bN)\n\n\n## The Data Engineering\n\n#### Level 0\n* [Getting started with #dataengineering Volume 6 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-datapipelines-dag-activity-6770259833363980288-7biE)\n* [Getting started with Dataengineering Volume 5 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-infrastructure-docker-activity-6763429524764975104-lVvy)\n* [Getting started with Data Engineering, volume 4 🎉💡](sketchnotes/Getting_started_with_de_vol4.PNG)\n* [Getting started with Data Engineering, volume 3 🎉💡](sketchnotes/Getting_started_with_de_vol3.png)\n* [Getting started with Data Engineering, volume 2 🎉💡](sketchnotes/Getting_started_with_de_vol2.png)\n* [Getting started with Data Engineering, volume 1 🎉💡](sketchnotes/gettingstart_dataengg.png)\n* [Getting started with #dataengineering from basics](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-mapreduce-docker-activity-6803350978432180224-GnWW)\n* [Apache Airflow 2.0](sketchnotes/Airflow_2_0.jpg)\n* [Some Interesting essentials while learning Apache Airflow](sketchnotes/airflow_checklist.png)\n* [Dagster Release 0.10.0 - Everything about Exactly-once, Fault-Tolerant Scheduling - Extremely Important Release 🎉🎉🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_dagster-machinelearning-kubernetes-activity-6757931107041255424-KAG1)\n* [#getdbt or Data Build Tools interface across all major Data Workflow Management Platform 💯✨🔥](https://www.linkedin.com/posts/iamabhishekchoudhary_getdbt-apacheairflow-dagster-activity-6750389785275240448-N8t_)\n* [Apache Superset - An #opensource Fully Featured Business Intelligence Application 🎊🎊🎊](https://www.linkedin.com/posts/iamabhishekchoudhary_apache-superset-10-is-out-activity-6759044550578229248-rerO)\n* [The Hop Orchestration Platform, or Apache #Hop (Incubating), aims to facilitate all aspects of data and metadata orchestration 💯💡⭐](https://www.linkedin.com/posts/iamabhishekchoudhary_hop-apachespark-apacheflink-activity-6761641585437396993-ydHg)\n* [Apache Iceberg Partitioning is way better than Hive ! Hidden Partitioning makes everything easier! 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheiceberg-dataengineering-bigdata-activity-6764569890650238976-0skJ)\n* [Trino aka #prestosql is different from Apache Spark SQL - Exclusively designed for Distributed SQL 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_prestosql-bigdata-apachespark-activity-6764906382325010432-qg00)\n* [Apache Spark is NOT a Map \nbut an MPP/MPI Engine](https://www.linkedin.com/posts/iamabhishekchoudhary_apachespark-mapreduce-dataengineering-activity-6765550560524476416-2c7f)\n* [Apache Hudi - Design Principles](https://www.linkedin.com/posts/iamabhishekchoudhary_apachehudi-hadoop-streamprocessing-activity-6767445258792947712-m9c2)\n* [OpenTelemetry specification V1.0](https://www.linkedin.com/posts/iamabhishekchoudhary_opentelemetry-specification-v100-tracing-activity-6767783538310873089-N9q5)\n* [Everything Around PySpark Pandas UDF 📖](https://www.linkedin.com/posts/iamabhishekchoudhary_revisitingpandasudf-activity-6775378803897221120-rS3z)\n* [Important skill-set of a Dataengineer - Reduce Cost](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineer-cloud-dataengineering-activity-6783023471485091840-2QfG)\n* [Everything on PyFlink - Python with Apache Flink](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheflink-dataengineering-python-activity-6785834728302948353-0I0R)\n* [Delta Lake Cheat Sheet](https://www.linkedin.com/posts/iamabhishekchoudhary_deltalakecheatsheet-activity-6787337208899678208-LSG3)\n* [Dataengineering schedule breakdown, a very flexible estimate](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-distributedsystems-mlops-activity-6791336683653648384-RY3_)\n* [Parquet - Introduction \u0026 Design, An OpenSource File Format](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheparquet-protocolbuffers-dataengineering-activity-6823244946166857729-0CZf)\n* [SQL - Avoiding Antipatterns](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlantipattern-activity-6826491257410396160-xX5P)\n* [Explaining Apache Kafka - In children's book format](https://www.linkedin.com/posts/iamabhishekchoudhary_apachekafka-streamprocessing-dataengineering-activity-6811538408763121664-5qpL)\n* [The Perfect #dataengineering: Top INVALID Reasons behind #datapipelines failures](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-datapipelines-data-activity-6839946812015603712-FnZq)\n* [What is ETL](https://www.linkedin.com/posts/iamabhishekchoudhary_etl-introduction-ugcPost-6864433492600635392-sbBl)\n* [What is Proxy \u0026 Reverse Proxy](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-activity-6907655591100354560-5DDk)\n\n\n#### Level 1\n\n* [DataEngg Skills to work with DataScience](sketchnotes/DE_skills_work_with_DS.jpg)\n* [Data Quality, A necessity for Data Driven Projects](sketchnotes/arond_dq.png)\n* [Essential Cloud Skills for Data Engineering](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-machinelearnig-aws-activity-6758329553036345346-b-40)\n* [Open Source Technologies in Data Engineering](sketchnotes/DE_OS.jpg)\n* [Kubernetes Fundamentals Required as a Data Engineer](sketchnotes/kubernetes_fundamentals.png)\n* [Apache Superset, OSS Business Intelligence for 2021](https://www.linkedin.com/posts/iamabhishekchoudhary_apache-superset-10-is-out-activity-6759044550578229248-rerO)\n* [#apachekafka as a Database - Summary on both the sides , Arguments, Trade-offs \u0026 exceptional 💬 quotes ⏳💡⏳](https://www.linkedin.com/posts/iamabhishekchoudhary_kafka-as-a-database-yes-or-no-a-summary-activity-6757228852923158528-qVBc)\n* [Processing Guarantees in #apachekafka 💯🔆🎉 - The best resource](https://www.linkedin.com/posts/iamabhishekchoudhary_processing-guarantees-in-kafka-activity-6756149463359791104-sJcD)\n* [Change Data Analysis with Debezium and Apache Pinot 🎉💡🚿](https://www.linkedin.com/posts/iamabhishekchoudhary_apachepinot-debezium-eventsourcing-activity-6754311102718382080-gWL1)\n* [Optimizing Apache Kafka Producers \u0026 Consumers 🎊📈🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_apachekafka-strimzi-streamprocessing-activity-6753977207800037376-kNWi)\n* [Redpanda -A NON-JVM Streaming Platform for mission critical workloads 💡🎉🔆](https://www.linkedin.com/posts/iamabhishekchoudhary_redpanda-activity-6749640173656559616-Ol6C)\n* [Apache Hudi - Turn Batch Jobs to Incremental Model | Complete file management on a Data Lake](https://www.linkedin.com/posts/iamabhishekchoudhary_building-a-large-scale-transactional-data-activity-6760855681277980672-kiZF)\n* [Apache Iceberg - an open table format for huge analytic datasets](https://www.linkedin.com/posts/iamabhishekchoudhary_apache-iceberg-activity-6760465145954131968-v0tp)\n* [Ballista - Distributed computing platform built primarily on Rust and powered by Apache Arrow](https://www.linkedin.com/posts/iamabhishekchoudhary_ballista-a-distributed-compute-platform-activity-6763021778580246528-Zwj8)\n* [ZooKeeper, a distributed, open-source coordination service for distributed applications](https://www.linkedin.com/posts/iamabhishekchoudhary_zookeeperpaper-activity-6764214157366620160-84rM)\n* [Apache Iceberg - Partition Evolution, its simple but its so amazing](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheiceberg-dataengineering-datascience-activity-6766299661784403968-T_-i)\n* [ApacheKafka without ZooKeeper Sneak Peak🔎](https://www.linkedin.com/posts/iamabhishekchoudhary_apachekafka-distributedsystems-zookeeper-activity-6782957154392432640-3s5r)\n* [Why Data Discovery is important for Data Engineering](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-bigdata-datagovernance-activity-6797526472681771008-C-XM)\n* [Queue vs Log - Event driven Architecture](https://www.linkedin.com/posts/iamabhishekchoudhary_eventsourcing-distributedsystems-bigdata-activity-6798554566800523264-xa88)\n* [Database Indexing](https://www.linkedin.com/posts/iamabhishekchoudhary_postgres-indexes-for-newbies-activity-6889845127477563392-04Kp)\n\n\n#### Level 1.1\n* [Multiple criteria search at scale with Apache Pinot \u0026 Theta Sketches](https://www.linkedin.com/posts/iamabhishekchoudhary_solving-for-the-cardinality-of-set-intersection-activity-6789846326893969408-4exe)\n* [VM vs Containers - Similar but Different](https://www.linkedin.com/posts/iamabhishekchoudhary_docker-dataengineering-bigdata-activity-6790586642504724480-Zq0e)\n* [State of Trino aka PrestoSQL](https://www.linkedin.com/posts/iamabhishekchoudhary_trino-aka-prestosql-state-activity-6790920739471069184-Ipcq)\n* [ETL is an extremely important component for any modern business](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-datawarehouse-bigdata-activity-6789114504010625024-V-lz)\n* [Top 5 ways to complicate a #dataengineering pipeline/application 💥](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-functionalprogramming-bigdata-activity-6802551801179631616-u6a0)\n* [Leader election is commonly used aka Master/Namenode/Leader/Driver](https://www.linkedin.com/posts/iamabhishekchoudhary_leaderelection-activity-6802979035270938624-VYYF)\n* [Dagster vs Airflow - A comparison](https://www.linkedin.com/posts/iamabhishekchoudhary_moving-past-airflow-why-dagster-is-the-next-generation-activity-6800808471768969216-__wj)\n* [About Single Source of Truth in DataEngineering](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-machinelearning-dataanalytics-activity-6801541986638733312-81gM)\n* [Change Data Capture for Distributed Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_change-data-capture-for-distributed-databases-activity-6821429243944206336-HiJL)\n* [Deep Dive on Why Apache Iceberg for Change Data Capture, using Apache Flink 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheiceberg-deltalake-realtime-activity-6813803694627336192-W7M7)\n* [OpenMetadata is an Open Standard for Metadata. A Single place to Discover, Collaborate, and Get your data right](https://www.linkedin.com/posts/iamabhishekchoudhary_metadata-datamesh-amundsen-activity-6833223767699845120-8U5h)\n* [About Lakehouse](https://www.linkedin.com/posts/iamabhishekchoudhary_datawarehouse-lakehouse-machinelearning-activity-6839506317648941056-d23r)\n* [etcd - A distributed, reliable key-value store for the most critical data of a distributed system](https://www.linkedin.com/posts/iamabhishekchoudhary_etcd-distributed-reliable-key-value-store-activity-6854393688827711488-1mts)\n* [What is Redis](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-redis-data-store-activity-6859358023492648960-YxoZ)\n* [What is Hive](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-apache-hive-activity-6862273737388032000-nRSq)\n* [What is Data Warehouse - An Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-data-warehouse-activity-6862625346521583616-x6Yl)\n* [Fundamentals of Designing Data Warehouse](https://www.linkedin.com/posts/iamabhishekchoudhary_fundamentals-of-designing-data-warehouse-activity-6865875658413793280-3Cln)\n* [Database Relational Model - A way of looking at Data](https://www.linkedin.com/posts/iamabhishekchoudhary_database-relational-model-activity-6888768611230511104-Eyu4)\n* [Data Engineering Infrastructure Notes](https://www.linkedin.com/posts/iamabhishekchoudhary_data-engineering-infrastructure-notes-activity-6901541636677910528-jjxL)\n\n#### Dataengineering Core \n\n* [A Data Engineering Story - The Beginning](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/blog1/index.md)\n* [Data Engineering - More towards Data Science or Data Analytics or ...](https://github.com/abhishek-ch/around-dataengineering/tree/blog2) \n* [Data Engineering Interview Patterns](sketchnotes/DE_Interview.jpg)\n* [Basic Checklists while learning Apache Spark](sketchnotes/spark_checklist.png)\n* [#apachespark for Distributed Analytics or #businessinteligence Platform - Worth or not ?](https://www.linkedin.com/posts/iamabhishekchoudhary_apachespark-businessinteligence-analytics-activity-6754429988021387264-kA7V)\n* [Apache Beam for Search: An Introduction \u0026 Addressing the challenge of the Time Problem 🔐💡🔒](https://www.linkedin.com/posts/iamabhishekchoudhary_apachebeam-apachekafka-dataengineering-activity-6753591516708573184-1-p6)\n* [Nextflow is a Workflow Manager exclusively for #bioinformatics 🩹💊🩹](https://www.linkedin.com/posts/iamabhishekchoudhary_bioinformatics-distributedsystems-kubernetes-activity-6749644503558238209-ufTg)\n* [#apachespark Project Zen Update - Making PySpark Better 💡🔗💡](https://www.linkedin.com/posts/iamabhishekchoudhary_project-zen-improving-apache-spark-for-python-activity-6748904711069474816-3iQN)\n* [Design - Exactly Once Delivery \u0026 Transactional Messaging in #apachekafka 🎊📋🎊](https://www.linkedin.com/posts/iamabhishekchoudhary_exactlyoncekafka-activity-6762359693802299392-Gch3)\n* [underrated but important skill of a Data Engineer](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineer-datascience-dataengineering-activity-6775777891335647232-6573)\n* [Fallacies of Distributed Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-distributedsystems-cloud-activity-6776439768591998976-aE6h)\n* [As a Data Engineer, some Essentials I did which really helped Data Scientists and the Team](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineer-datascientists-kubernetes-activity-6779784194651504640-zIUP)\n* [A very normal Data Engineering work 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-python-kubernetes-activity-6772461203500384256-yrAH)\n* [What can go wrong in Distributed Data Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineers-dataengineering-machinelearning-activity-6771387632715935744-wZCk)\n* [Architect and build an #machinelearning use case end to end using Amazon SageMaker 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_architect-and-build-the-full-machine-learning-activity-6775727947325214721-iYAW)\n* [Around Data Discovery or Metadata Management Platforms](https://www.linkedin.com/posts/iamabhishekchoudhary_datadiscovery-metadatamanagement-datascientists-activity-6777238261258682368-fkGa)\n* [Amazon S3 Object Lambda - Provide Different Views of Data to Multiple Applications](https://www.linkedin.com/posts/iamabhishekchoudhary_introducing-amazon-s3-object-lambda-use-activity-6778582018067378176-fzvy)\n* [Full Stack Data Engineer](https://www.linkedin.com/posts/iamabhishekchoudhary_hadoop-apachespark-kubernetes-activity-6781152322971074560-VWvK)\n* [Data cleaning is Hard but why](https://www.linkedin.com/posts/iamabhishekchoudhary_whys-it-hard-to-teach-data-cleaning-activity-6782684736637693953-JiLg)\n* [Most exciting things about #dataengineering](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-distributedsystems-machinelearning-activity-6783666378680401920-E23H)\n* [The real impact of Disks on #rocksdb State Backend in Apache Flink](https://www.linkedin.com/posts/iamabhishekchoudhary_the-impact-of-disks-on-rocksdb-state-backend-activity-6784115938338906112-FTVY)\n* [Tips for Distributed System High Availability](https://www.linkedin.com/posts/iamabhishekchoudhary_tips-for-high-availability-activity-6785942817693855744-8KFF)\n* [interesting way of collaboration between a Dataengineer \u0026 Datascientis](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineer-datascientist-datascience-activity-6786211019640369152-QrFF)\n* [Building DistributedLog: High-performance replicated log service 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystems-dataengineering-bigdata-activity-6786283090441506816-kCob)\n* [Whiz: Data Analytics Execution Framework based on Intermediate Data](https://www.linkedin.com/posts/iamabhishekchoudhary_whiz-data-driven-analytics-execution-activity-6788003925300707328-2Yuy)\n* [Adding unlimited Nodes in a #dataengineering platform will eventually drop](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-distributedsystems-bigdata-activity-6788418825047080960-iTiv)\n* [A typical Data Engineering Pipeline](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-data-bigdata-activity-6800075858917773312-W_Lc)\n* ['Log' is a fundamental component of a Data Engineering Ecosystem](https://www.linkedin.com/posts/iamabhishekchoudhary_apacheiceberg-deltalake-dataengineering-activity-6804701361796636672-MOFm)\n* [Flink CDC](https://github.com/abhishek-ch/around-dataengineering/blob/master/flink_cdc.md)\n* [Readings Around Databases](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/Reading_ardound_database.md)\n* [Code Review Best Practice, bcz Developers, hate code reviews](https://www.linkedin.com/posts/iamabhishekchoudhary_codereviewbestpractice-activity-6812358358906019840-Qfw5)\n* [Important Performance Criteria to measure DataEngineering Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-distributedsystems-datastructures-activity-6848612433121157120-wrNi)\n* [Database Internals - Storage](https://www.linkedin.com/posts/iamabhishekchoudhary_database-storage-internals-activity-6863372646696972288-Hsia)\n* [Data Integration for Databases \u0026 Data Warehousing - An Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_data-integration-introduction-activity-6872779193617309696-HU54)\n* [What is Protocol Buffer - An excellent important data interchange format for serialization, \"Zero Copy\" format](https://www.linkedin.com/posts/iamabhishekchoudhary_protocol-buffer-activity-6886960430330232832-ONrn)\n* [Memcached, Redis \u0026 Elasticache - To accelerate your data or databases](https://www.linkedin.com/posts/iamabhishekchoudhary_memory-memcached-redis-elasticache-activity-6891708019474788352-8ur_)\n* [What is LSM-Tree](https://www.linkedin.com/posts/iamabhishekchoudhary_lsm-tree-activity-6901823852150624256-cnbq)\n* [Tor aka Onion Router - How does it work?](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-activity-6909070959350059009-QfFq?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n\n\n#### Infrastructure\n\n* [SQL Database on Kubernetes - Best Practices](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-database-on-kubernetes-considerations-activity-6755432993130643457-Zts2)\n* [Devtron - An Open Source DevOps on Kubernetes, written in Go 🥇🎁🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_devtron-multicloud-infrastructure-activity-6749978071970975744-PcPs)\n* [Most Popular #opensource BI \u0026 Data Analytics Platforms 🎊💡🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_opensource-dataengineering-datascience-activity-6749258103369322496-hc5C)\n* [datapipelines Dataframe APi is now available with #apachebeam 💯🔥💯](https://www.linkedin.com/posts/iamabhishekchoudhary_datapipelines-apachebeam-python-activity-6748167921433948160-oWfT)\n* [Disaster Recovery for Multi-Region Apache Kafka \u0026 Data Consumption using #apacheflink 🔅🎉🔅](https://www.linkedin.com/posts/iamabhishekchoudhary_disaster-recovery-for-multi-region-kafka-activity-6747422409218940928-70Rf)\n* [Kubernetes Api Structure 💯✔️💯](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-dataengineering-machinelearning-activity-6759074145817944064-dHIl)\n* [Architecting a Kubernetes Infrastructure 💯](https://www.linkedin.com/posts/iamabhishekchoudhary_a-deep-dive-into-architecting-a-kubernetes-activity-6761257898438918144-N74g)\n* [Exploring Kubernetes Operator Pattern 💡](https://www.linkedin.com/posts/iamabhishekchoudhary_exploring-kubernetes-operator-pattern-activity-6762000800911814656-Jre7)\n* [Docker is an interal part of Data Engineering ML pipeline \u0026 that makes security 🔐 extremely essential](https://www.linkedin.com/posts/iamabhishekchoudhary_docker-security-activity-6763766659426660353-_MIa)\n* [Rack awareness for #apachekafka Streams Proposal 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_apachekafka-kafkastreams-dataengineering-activity-6779000842264592384-NA4J)\n* [Dolt is Git for Data 🎊](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-bigdata-datascience-activity-6774623291329060864-QKYb)\n* [Toward Better Data Culture From First Principles by Ube](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-dataengineer-bigdata-activity-6780824810315403264-e0hC)\n* [Fast and Reliable Schema-Agnostic Log Analytics Platform by Uber](https://www.linkedin.com/posts/iamabhishekchoudhary_elasticsearch-elk-dataengineering-activity-6783364868515934208-jhj6)\n* [Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems📋](https://www.linkedin.com/posts/iamabhishekchoudhary_simple-testing-can-prevent-most-critical-failures-activity-6784411707830779904-ds3_)\n* [Diving Deep on S3 Consistency - Insightful](https://www.linkedin.com/posts/iamabhishekchoudhary_diving-deep-on-s3-consistency-activity-6793185152844513280-k48V)\n* [Ray- General Purpose ML Infrastructure](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/ray_distributed_ml.md)\n* [Kubernetes Hardening Guide by National Security Agency](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-hardening-guide-activity-6831118268779020288-KQfU)\n* [Everything around Load Balancer](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystems-datainfrastructure-loadbalancing-activity-6840255837957607424-oOmk)\n* [Data Lakehouse - Is it really the end of Datawarehouse](https://www.linkedin.com/posts/iamabhishekchoudhary_datawarehouse-lakehouse-machinelearning-activity-6839506317648941056-d23r)\n* [Real-Time Exactly-Once Processing with Apache Flink, Kafka, and Pinot @uber](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystems-apacheflink-apachekafka-activity-6850759002620522496-zq8w)\n* [WTH is Kubernetes Operator? - An Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_k8s-operator-introduction-activity-6850331132089679872-_sVt)\n* [Lessons Learned from Sharding Postgres](https://www.linkedin.com/posts/iamabhishekchoudhary_herding-elephants-lessons-learned-from-sharding-activity-6856096160822697984-x4es)\n* [What is Kubernetes- An introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-introduction-activity-6861168802697220096-wCI1)\n* [ELK Stack - Introduction of Scalable Monitoring](https://www.linkedin.com/posts/iamabhishekchoudhary_elk-stack-introduction-activity-6864796949678223360-j-oo)\n* [What is NGINX - An Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_nginx-introduction-activity-6867698654820519937-Z_cX)\n* [What is Load Balancer - An Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-load-balancer-an-introduction-activity-6866977826793435136-SmCk)\n* [What is OAuth 2 - Introduction|API Based Authorization](https://www.linkedin.com/posts/iamabhishekchoudhary_oauth-2-introduction-activity-6877542164289728512-m1WG)\n* [Kubernetes \u0026 Networks - It's hard because multiple options are available](https://www.linkedin.com/posts/iamabhishekchoudhary_k8s-networks-activity-6880088004594151424-FhbK)\n* [Kubernetes Reconciliation](https://www.linkedin.com/posts/iamabhishekchoudhary_k8s-what-is-reconciliation-activity-6881167406342004736-FbuO)\n* [Troubleshooting Kubernetes App](https://www.linkedin.com/posts/iamabhishekchoudhary_troubleshooting-kubernetes-app-activity-6885146702148435968-n1oo)\n* [Kubernetes Best Practices - Classics](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-best-practices-activity-6886580619527155712--h7D)\n* [Paper: Serverless Computing: A Survey of Opportunities, Challenges, and Applications](https://www.linkedin.com/posts/iamabhishekchoudhary_serverless-computing-activity-6890329219063189504-7P-5)\n* [Choosing the Kubernetes Local Cluster](https://www.linkedin.com/posts/iamabhishekchoudhary_local-kubernetes-cluster-which-why-activity-6897194508409278464-CV61)\n* [Monitoring Kubernetes - Fundamentals of #kubernetes Infrastructure Monitoring](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-monitoring-fundamental-activity-6897830067133571072-hmnv)\n* [Kubernetes Controller Manager](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-controller-manager-concept-activity-6900449709870104577-4T0d)\n* [Kubernetes: Why the Pod is still in the Pending State?](https://www.linkedin.com/posts/iamabhishekchoudhary_understanding-kubernetes-pod-pending-problems-activity-6931949104704962560-OOOl?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [Kubernetes Liveness \u0026 Readiness Probe](https://www.linkedin.com/posts/iamabhishekchoudhary_introduction-to-kubernetes-probes-activity-6936950535275048961-8wYL?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [Kubernetes Pod/Node Affinity](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-dataengineering-activity-6976177528545669121-QgjV?utm_source=share\u0026utm_medium=member_desktop)\n\n\n## SQL\n* [Advanced SQL - Reference CS 564 Database Management Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_advanced-sql-2-activity-6845598540790599680-gTgH)\n* [SQL and Advanced SQL - An asset](https://www.linkedin.com/posts/iamabhishekchoudhary_advancedsql-activity-6844892160798584832-gBTl)\n* [Database Indexing - Almost Everything](https://www.linkedin.com/posts/iamabhishekchoudhary_databaseindexes-activity-6844509295476924416-Jzto)\n* [Tuning SQL queries - Tips for writing efficient \u0026 faster Queries](https://www.linkedin.com/posts/iamabhishekchoudhary_query-optimization-techniques-sql-activity-6844123194836742144-aptY)\n* [Database Schema Design - Schema Design is a Complicated Necessity](https://www.linkedin.com/posts/iamabhishekchoudhary_database-schema-design-activity-6843509027025022976-qbRF)\n* [SQL Query Processing Plan - Basics](https://www.linkedin.com/posts/iamabhishekchoudhary_queryprocessingplan-activity-6843139272497774592-10sT)\n* [Revisiting SQL Basics - The beginning of Data Science \u0026 Data Engineering](https://www.linkedin.com/posts/iamabhishekchoudhary_basic-sql-activity-6842758586142142464-kJ_U)\n* [Distributed Advanced Queries - Presto/Trino](https://www.linkedin.com/posts/iamabhishekchoudhary_advanced-sql-distributed-query-engine-prestotrino-activity-6845952140645593088-r6BJ)\n* [SQL Notes For Professionals, 100+ pages](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlnotes4professional-activity-6846303034914471936-q1T9)\n* [Table Partitioning](https://www.linkedin.com/posts/iamabhishekchoudhary_database-dataengineering-sql-activity-6846689189979844609-yXQS)\n* [SQL complex Queries - Nested Queries \u0026 Aggregation](https://www.linkedin.com/posts/iamabhishekchoudhary_nested-queries-and-aggregation-activity-6847059870416474112-tSkY)\n* [Gossip Protocols - Designed for Data Consistency \u0026 Fault-Tolerance](https://www.linkedin.com/posts/iamabhishekchoudhary_p2p-gossip-protocol-activity-6847021932785786880-A2Oj)\n* [Table Partitioning - An Important Concept](https://www.linkedin.com/posts/iamabhishekchoudhary_database-dataengineering-sql-activity-6846689189979844609-yXQS)\n* [Database Concurrency Control : 2 Phase Locking](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-database-dataengineering-activity-6847472978335195136-qqaC)\n* [Database Entity Relationship Model](https://www.linkedin.com/posts/iamabhishekchoudhary_the-entity-relationship-er-model-activity-6848113792358907904-bwRc)\n* [SQL Join Fundamentals](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-joins-activity-6848474520647458816-3ACA)\n* [Database Indexing](https://www.linkedin.com/posts/iamabhishekchoudhary_database-performanceandindexes-activity-6848848899260653569-l1gz)\n* [Database Indexing Notes](https://www.inf.unibz.it/~artale/DB2/handout2.pdf)\n* [SQL Injection Introduction](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-injection-introduction-activity-6849573020533608448-tsWX)\n* [SQL Constraints Fundamentals](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-constraints-activity-6849207436176363520-hiDU)\n* [The fundamental of writing SQL queries is different from](https://www.linkedin.com/posts/iamabhishekchoudhary_programming-dataengineering-datanalytics-activity-6797939314098520064-hMfd)\n* [Building a NoSQL Database using Git](https://www.linkedin.com/posts/iamabhishekchoudhary_gitasnosqldatabase-activity-6820624465861390336-qwpo)\n* [Against SQL - An article on What is not good with #sql ](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-database-data-activity-6819964297423179776-OPr5)\n* [Using `EXPLAIN` for Data Problems - Things beyond SQL](https://www.linkedin.com/posts/iamabhishekchoudhary_databaseexplain-activity-6819573919389941760-euD5)\n* [10 SQL Queries to Blow Your Mind 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlmightiestqueries-activity-6829031645870374912-HW88)\n* [Views, Stored Procedures, Functions \u0026 Triggers - SQL ](https://www.linkedin.com/posts/iamabhishekchoudhary_viewstoredproceduretriggersql-activity-6850663245687115776-1V75)\n* [SQL Transaction \u0026 ACID Property](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-transaction-activity-6851020805803958272-w69d)\n* [How to Solve complex SQL queries](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlcomplexqueriesapproach-activity-6851755823228604416-uoVK)\n* [Apache Spark SQL - The Introduction from RDMBS till SparkSQL](https://www.linkedin.com/posts/iamabhishekchoudhary_sparksqlintroduction-activity-6852476659761782785-MOLf)\n* [Advanced SQL \u0026 Functions](https://www.linkedin.com/posts/iamabhishekchoudhary_advanced-sql-functions-activity-6853198639423123456-0g0q)\n* [Basic \u0026 Intermediate on Database Sharding](https://www.linkedin.com/posts/iamabhishekchoudhary_database-sharding-basic-intermediate-activity-6855026526132477952-UGya)\n* [Complex Database Queries with PostgreSQL](https://www.linkedin.com/posts/iamabhishekchoudhary_complex-database-queries-activity-6854642670451650560-P7xv)\n* [Query Evaluation - Technical Details \"when you execute SQL Query\"](https://www.linkedin.com/posts/iamabhishekchoudhary_queryevaluation-activity-6853910123237183488-kdw7)\n* [What is Materialized View \u0026 how does work in Distributed Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_materialized-view-activity-6853558113245773824-6gBQ)\n* [Breaking Down NoSQL Sharding, Replication \u0026 Consistency](https://www.linkedin.com/posts/iamabhishekchoudhary_whynosqlshardingiscomplicated-activity-6855744363792781312-KxQz)\n* [Database Query Optimization Technique](https://www.linkedin.com/posts/iamabhishekchoudhary_queryoptimizationtechniques-activity-6856460121778790400-PWrb)\n* [Intermediate SQL](https://www.linkedin.com/posts/iamabhishekchoudhary_intermediate-sql-activity-6856827085349040128-IxvM)\n* [SQL Stored Procedures](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlproceduresdeep-dive-activity-6859022836573507584-3Khi)\n* [OLAP \u0026 OLTP - Datawarehouse Data Mining](https://www.linkedin.com/posts/iamabhishekchoudhary_olap-vs-oltp-activity-6860811031833296896-fapF)\n* [Database Fundamentals](https://www.linkedin.com/posts/iamabhishekchoudhary_database-fundamentals-activity-6863695598084775936-ZgKM)\n* [SQL Subqueries](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-subqueries-activity-6868420033782059008-wfr8)\n* [NewSQL Introduction - Basic to Intermediate](https://www.linkedin.com/posts/iamabhishekchoudhary_newsql-introduction-activity-6867339297767387136-O-sG)\n* [SQL Intermediate \u0026 basics Deep Dive](https://www.linkedin.com/posts/iamabhishekchoudhary_introduction-to-sql-intermediate-deep-dive-activity-6869861428271091712-Pbmm)\n* [SQL Basics - The Starting point](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-basics-ugcPost-6869502880605360128-78iw)\n* [Data Warehousing \u0026 OLAP Technology](https://www.linkedin.com/posts/iamabhishekchoudhary_dwholap-activity-6871337092929138689-QMcd)\n* [Snowflake Datawarehouse](https://www.linkedin.com/posts/iamabhishekchoudhary_snowflake-activity-6871720966620282880-23OI)\n* [RelationalAlgebra \u0026 SQL](https://www.linkedin.com/posts/iamabhishekchoudhary_relationalalgebrasql-activity-6874631115001393152-nMFW)\n* [Logical Schema Design: SQL Database](https://www.linkedin.com/posts/iamabhishekchoudhary_logical-schema-design-sql-activity-6876437920027439104-uncn)\n* [Kubernetes Pod Internals - Deep Dive](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-pod-internals-activity-6878625725881171968-MfQn)\n* [The Illustrated Children's Guide to Kubernetes](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-children-book-activity-6884036918624034816-DtTK)\n* [SQL Subqueries by Example](https://www.linkedin.com/posts/iamabhishekchoudhary_understanding-subqueries-by-examples-activity-6890591994150760448-mdsN)\n* [What is Write-Ahead-Logging (WAL)](https://www.linkedin.com/posts/iamabhishekchoudhary_wal-write-ahead-log-activity-6894184345448321024-dW5p)\n* [SQL Transactions](SQL Transactions - a sequence of database operations)\n* [Linux Productivity Tools - This is a Data Infrastructure necessity](https://www.linkedin.com/posts/iamabhishekchoudhary_linux-productivity-activity-6905439443684007937-A5PF)\n\n\n### NoSQL\n* [NoSQL \u0026 MongoDB]https://www.linkedin.com/posts/iamabhishekchoudhary_nosql-mongodb-activity-6874231633654935553-Z66u)\n* [CouchDB Introduction - • Document Storage Database](https://www.linkedin.com/posts/iamabhishekchoudhary_couchdb-document-db-activity-6892768901944401921-zxkA)\n\n\n## Machine Learning\n\n#### MLOPS\n* [Machine Learning Workflow :100: ](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dataengineering-distributedsystems-activity-6754711934441984000-WZ1G)\n* [Dummy Notes On Machine Learning Infrastructire](sketchnotes/ML_Infra_RoughNotes.png)\n* [Machine Learning Feature Store :100:](sketchnotes/feature_store.jpg)\n* [Deploying #machinelearning model in Production is really HARD but #MLOps can fix that.](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-mlops-datascience-activity-6752596084746436609-r99U)\n* [List of #machinelearning \u0026 #dataengineering Technologies will be following in 2021 🎉💡🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dataengineering-getdbt-activity-6751786250769903616-sEUb)\n* [MLOps - ZenML #machinelearning with reproducible pipelines ✅💯✅](https://www.linkedin.com/posts/iamabhishekchoudhary_maiot-iozenml-activity-6748236907542577152-nBOj)\n* [Why? Data Versioning is a complicated problem for Dataengineers](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineers-apacheiceberg-deltalake-activity-6793880325475901440-Lsxx)\n* [Explainable AI Cheat Sheet](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-mlops-dataengineering-activity-6794228285145452545-YEwQ)\n* [Designing Machine Learning infrastructure](https://www.linkedin.com/posts/iamabhishekchoudhary_lifescience-dataengineering-activity-6883778465296990208-8RUn)\n* [What is Log - Foundation behind Databases \u0026 Distributed Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystem-database-lifescience-activity-6905525481894948864-L2nG)\n* [How does the GIT version control work?](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dataengineering-activity-6906610679105912832-K7wL)\n\n\n#### Project\n* [Streamlit Healthcare Machine Learning Data App](https://github.com/abhishek-ch/streamlit-healthcare-ML-App)\n* [Dstack AI - An open-source tool to develop data applications with Python 🎉💭🎊](https://www.linkedin.com/posts/iamabhishekchoudhary_dstack-machinelearning-businessinteligence-activity-6753247298588852225-CbcA)\n* [Adversarial Robustness Toolbox - a Python library for #machinelearning Security 💡🔎🔓](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dataengineering-datascience-activity-6752863649414680576-bSK8)\n* [Biopython is a set of freely available tools for biological computation written in #Python 💊⌛️💊](https://www.linkedin.com/posts/iamabhishekchoudhary_python-bioinformatics-apachespark-activity-6749263102992269312-SCut)\n\n\n#### Insightful\n* [Time to Know More about DASK](sketchnotes/TimeToLearnDask.png)\n* [DataEngineering vs Machine Learning](sketchnotes/DataEngineeringVsMachineLearningengineering.jpg)\n* [A good #machinelearning Model is only possible with a good quality of #data. ⌛️](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-data-datascience-activity-6757609262110826496-CmGj)\n* [Statistics for #softwareengineer 🔥💯🔥](https://www.linkedin.com/posts/iamabhishekchoudhary_statsforhackers-activity-6752265241486331904-moUz)\n* [Monitoring #machinelearning Applications 🎉🎁 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dataengineering-prometheus-activity-6751436872456687616-ZTvW)\n* [Dagster is a data orchestrator for machine learning, analytics, and ETL - Officially #machinelearning driven 🥇🥇🥇](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-dagster-apacheairflow-activity-6749618916609314816-YPnW)\n* [Short Notes on -Open source #machinelearning Tracking System](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-datascience-mlflow-activity-6759484697518096384-Szi9)\n* [The best example of Randomness is - #machinelearning model in Production. 🔐💭🔎](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-infrastructure-datascientists-activity-6759792666403246081-Jq3Q)\n* [Flyte is declarative, structured, and highly scalable cloud-native workflow orchestration platform for Distributed Machine Learning](https://www.linkedin.com/posts/iamabhishekchoudhary_mlops-golang-datalineage-activity-6778218137268625408-WWdm)\n* [Tips for Distributed System High Availability 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_tips-for-high-availability-activity-6785942817693855744-8KFF)\n* [Building DistributedLog: High-performance replicated log service 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystems-dataengineering-bigdata-activity-6786283090441506816-kCob)\n* [How to scale Kubernetes with Assurance](https://www.linkedin.com/posts/iamabhishekchoudhary_kubernetes-infrastructure-dataengineering-activity-6786619511165501440-pcNS)\n* [Apache Calcite - Building Sql Query Processor from Scratch over Lucene](https://www.linkedin.com/posts/iamabhishekchoudhary_calcite-tutorial-lucene-activity-6833330741460447233-CCfP)\n* [Database Storage](https://www.linkedin.com/posts/iamabhishekchoudhary_database-storage-basics-activity-6858266378051350528-0F1-)\n* [ACID is the foundation of Database, BASE is for NoSQL Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_acid-base-activity-6861531231998275584-q8ih)\n* [Some common elements behind many Distributed Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_databases-dataengineering-datainfrastructure-activity-6870277667103526912-d0d6)\n* [Failure Recovery in TrinoDB](https://www.linkedin.com/posts/iamabhishekchoudhary_failure-recovery-in-trino-activity-6893925269191610368-7g98)\n* [What is LLVM](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-activity-6909794301438799872-yjJt)\n* [What is Garbage Collection](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-activity-6910230886785445888-p1qS)\n* [What is Canary Deployment](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-activity-6913058649288409088-He1R?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n\n## Paper\n\n### Distributed System\n\n#### Crazy\n* [The Snowflake Paper - Core idea is to build an enterprise-ready #datawarehouse solution for the #cloud 🎉📰📕](https://www.linkedin.com/posts/iamabhishekchoudhary_snowflakepaper-activity-6752909762079326208-o1cC)\n* [Most important points around Distributed #dataengineering Platform](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-programminglanguages-distributedsystem-activity-6742777508380360704-qqvV)\n* [Fundamental of #distributedsystems Scaling - Avoiding Co-ordination 🎊♨️🔆](https://www.linkedin.com/posts/iamabhishekchoudhary_marcs-blog-activity-6758678779893231616-OX_h)\n* [Technical Debt in #dataengineering #softwareengineering 🔕💡🔕](https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-softwareengineering-programmers-activity-6756946484488400896-2uQx)\n* [Paper on Wander Join: Online Aggregation via Random Walks 📃💭📑 Join problem](https://www.linkedin.com/posts/iamabhishekchoudhary_wanderjoinonline-aggregation-problem-activity-6756537519698956288-jTdS)\n* [The Delta Lake Paper - High-Performance ACID Table Storage 📋💡📋](https://www.linkedin.com/posts/iamabhishekchoudhary_thedeltalakepaper-activity-6755858637895335936-kwaT)\n* [Dynamo - AWS Highly Available Key-value Store #distributedsystem 💬💡🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_dynamodbkv-activity-6755499276400431104-DofI)\n* [An Efficient and Syntactically Idiomatic Approach to Management of Streams and Tables, A Single SQL for all 💡📩📩](https://www.linkedin.com/posts/iamabhishekchoudhary_sqlforall-activity-6755119918598955008-WKY5)\n* [Secure \u0026 Robust Machine Learning in #healthcare 💊🧪🥳](https://www.linkedin.com/posts/iamabhishekchoudhary_robustmlinhealthcare-activity-6754772406499377152-giZp)\n* [Progress in Medical Science using #deeplearning 💊💡💉](https://www.linkedin.com/posts/iamabhishekchoudhary_deeplearningcnnmedical-activity-6754374045988380672-exQZ)\n* [The Amazon Redshift Paper - A fast, fully managed, petabyte-scale data warehouse solution that makes it simple and cost-effective to efficiently analyze large volumes of data using existing #businessintelligence tools 📂📰💭](https://www.linkedin.com/posts/iamabhishekchoudhary_redhistinitialpaper-activity-6753300440881926145-qZY6)\n* [Advancing #drugdiscovery via Artificial Intelligence 💊🏥🏥](https://www.linkedin.com/posts/iamabhishekchoudhary_drugdiscoveryai-activity-6751143165618651136-6EX8)\n* [Apache Calcite is a dynamic data management framework 🎉📚🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_apachecalcite-activity-6751114208450015232-67Wn)\n* [Lakehouse - A Paper on new Generation of #datawarehouse technology 💡🔎💡](https://www.linkedin.com/posts/iamabhishekchoudhary_datawarehouse-apacheparquet-machinelearning-activity-6747813594395668480-hepU)\n* [Calvin: Fast Distributed Transactions for Partitioned Database Systems 📝📝](https://www.linkedin.com/posts/iamabhishekchoudhary_calvin-fast-distributed-transactions-activity-6760230219019354113-b-Bi)\n* [Presto or Trino - #SQL on Everything ( The Design, Motivation \u0026 Performance) #presto 💭🎊💡](https://www.linkedin.com/posts/iamabhishekchoudhary_trinoprestosqloneverything-activity-6759416018025680896-DPfu)\n* [Design - Exactly Once Delivery \u0026 Transactional Messaging in Apache Kafka](https://www.linkedin.com/posts/iamabhishekchoudhary_exactlyoncekafka-activity-6762359693802299392-Gch3)\n* [Apache Kafka Paper : Distributed Messaging System for Log Processing](https://www.linkedin.com/posts/iamabhishekchoudhary_apachekafkapaper-activity-6765285530247684097-ufXy)\n* [Paper: Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size](https://www.linkedin.com/posts/iamabhishekchoudhary_bigdatagoogle-activity-6765973408174923776-jrJz)\n* [Paper: Ground is an open-source data context service, a system to manage all the information that informs the use of data](https://www.linkedin.com/posts/iamabhishekchoudhary_metadatadatacontextservice-activity-6767048256766697472-0A8Q)\n* [Azure Data Lake Store(ADLS) is a fully-managed, elastic, scalable, and secure file system that supports #hadoop distributed file system (HDFS) and Cosmos semantics](https://www.linkedin.com/posts/iamabhishekchoudhary_azure-data-lake-store-activity-6776108273075589120-hqWr)\n* [An LFU (Least Frequently Used) Cache eviction algorithm of O(1) Runtime complexity](https://www.linkedin.com/posts/iamabhishekchoudhary_lfu-cache-eviction-scheme-activity-6780125496920346624-BfKP)\n* [The Berkeley View on Cloud Computing - Paper](https://www.linkedin.com/posts/iamabhishekchoudhary_serverlesscomputing-activity-6773574741639917570-uEB5)\n* [The Google File System - The Paper 🎉](https://www.linkedin.com/posts/iamabhishekchoudhary_the-google-file-system-paper-activity-6778681429632049152-EF03)\n* [Paper: Report on Distributed Deep Learning on Data Systems 📂](https://www.linkedin.com/posts/iamabhishekchoudhary_tech-report-of-distributed-deep-learning-activity-6787729738124341249-fFG8)\n* [Crystal: A Unified Cache Storage System for Analytical Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_crystal-semantic-caching-activity-6834495561652076544-UGVO)\n* [VoltDB](https://www.linkedin.com/feed/update/urn:li:activity:6846055501105553408/)\n* [Magnet - Apache Spark Shuffle mechanism to handle petabytes of daily shuffled data and clusters with thousands of nodes](https://www.linkedin.com/posts/iamabhishekchoudhary_magnet-sharkshuffleservice-activity-6851903731227799552-EmLz)\n\n#### Set 2\n\n* [Paper: Real-time Data Infrastructure @ Uber](https://www.linkedin.com/posts/iamabhishekchoudhary_real-time-data-infrastructure-activity-6792384716063469568-UKc-)\n* [Paper: DBLog, A Watermark Based Change-Data-Capture Framework by Netflix](https://www.linkedin.com/posts/iamabhishekchoudhary_change-data-capture-dblog-activity-6793492331967365120-517C)\n* [Paper: Large Scale Distributed Systems Tracing Infrastructure](https://www.linkedin.com/posts/iamabhishekchoudhary_dapper-a-large-scale-distributed-tracing-infrastructure-activity-6796076018366070784-tHMZ)\n* [Paper:Paxos vs Raft: Distributed Consensus 📋](https://www.linkedin.com/posts/iamabhishekchoudhary_paxosvsraft-activity-6798953621909250048-4kgw)\n* [Paper: Sorting in a #distributedsystem 📋](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystemsorting-activity-6800374901174792192-6C5O)\n* [Paper: A large scale analysis of hundreds of in-memory cache clusters](https://www.linkedin.com/posts/iamabhishekchoudhary_a-large-scale-analysis-in-memory-cache-clusters-activity-6802207087511298048-piMh)\n* [Design \u0026 Architecture of Amazon Timestream - Streaming at Scale](https://www.linkedin.com/posts/iamabhishekchoudhary_sql-data-dataengineering-activity-6805811689687281664-4edZ)\n* [Distributed System Synchronization](https://www.linkedin.com/posts/iamabhishekchoudhary_distributedsystemclocksynchronization-activity-6808035719366373376-T_GK)\n* [Paper: Consistent hashing - Resizing cluster or Load in a #distributedsystems with a simple concept](https://www.linkedin.com/posts/iamabhishekchoudhary_consistenthashing-activity-6807248364808282112-2ef2)\n* [Deep Dive - Foundation DB (unbundled database, OLTP, strict serializability, multi-version concurrency control, optimistic concurrency control, simulation testing)](https://www.linkedin.com/posts/iamabhishekchoudhary_foundationdb-opensource-database-activity-6828300478351237120-ENDF)\n* [Distributed Database - ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook Database](https://www.linkedin.com/posts/iamabhishekchoudhary_database-rocksdb-zookeeper-activity-6830432000072048640-y8nw)\n* [BigData Metadata Management System](https://github.com/abhishek-ch/around-dataengineering/blob/master/docs/Metadata_management.md)\n* [Machine Learning for Database Optimizations](https://www.linkedin.com/posts/iamabhishekchoudhary_databases-machinelearning-dataengineering-activity-6841287258293846016-QRVZ)\n* [SingleStore - A Distributed Database Management System. It's really more than a Database](https://www.linkedin.com/posts/iamabhishekchoudhary_singlestore-database-activity-6842396205964681216-3SLY)\n* [ArrowSAM, in-memory genomics SAM format based on Apache Arrow](https://www.linkedin.com/posts/iamabhishekchoudhary_arrowsam-apache-arrow-based-genomics-sam-activity-6843895456158814208-K86l)\n* [Realtime Data Processing FB - Deep Dive on #streamprocessing](https://www.linkedin.com/posts/iamabhishekchoudhary_realtimedataprocessingatfacebook-activity-6849698562062786560-UhTQ)\n* [ArangoDB - Native multi-model NoSQL Distributed #database, From #sql to NoSQL](https://www.linkedin.com/posts/iamabhishekchoudhary_arangodb-from-relational-to-nosql-activity-6853329534280888325-8GSX)\n* [To BLOB or Not To BLOB: Large Object Storage in a Database or a Filesystem](https://www.linkedin.com/posts/iamabhishekchoudhary_large-object-storage-in-a-database-or-a-filesystem-activity-6859838164094255104-1P4M)\n* [How to bring robustness while Designing Large Scale Complex Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_robustness-in-complex-system-activity-6861671058144010240-18bY)\n* [Facebook Datawarehouse](https://www.linkedin.com/posts/iamabhishekchoudhary_data-warehouse-analytics-infrastructure-activity-6872096948296454145-KrwG)\n* [Building a performant OLTP system on an open-source columnar format, and supporting near-zero overhead data export to external tools](https://www.linkedin.com/posts/iamabhishekchoudhary_columnar-format-for-both-oltp-olap-using-activity-6876921762639036416-hYV1)\n* [Towards Demystifying Serverless Machine Learning Training](https://www.linkedin.com/posts/iamabhishekchoudhary_towards-demystifying-serverless-machine-learning-activity-6880184328421294080-WVi7)\n* [Paper: Scalable Linear Algebra on top of Distributed Databases, this will simplify Machine Learning on Databases](https://www.linkedin.com/posts/iamabhishekchoudhary_scalable-linear-algebra-on-a-relational-database-activity-6883378575798611968-PZg1)\n* [Paper: Are You Sure You Want to Use MMAP in Your Database Management System](https://www.linkedin.com/posts/iamabhishekchoudhary_paper-dont-use-mmap-in-your-database-management-activity-6888862226975924224-J4Tn)\n* [What is RBAC or Role-Based Access Control](https://www.linkedin.com/posts/iamabhishekchoudhary_rbac-activity-6939859154085814272-v_nB?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [Vectorization vs. Compilation in Query Execution](https://www.linkedin.com/posts/iamabhishekchoudhary_vectorization-vs-compilation-in-query-execution-activity-6967759062994194432-HX9e?utm_source=share\u0026utm_medium=member_desktop)\n* [SQLite vs DuckDB](https://www.vldb.org/pvldb/vol15/p3535-gaffney.pdf)\n\n\n#### Advanced\n* [Glow is an open-source toolkit for working with genomic data at biobank-scale and beyond using #apachespark \u0026 #deltalake 💊🎉💉](https://www.linkedin.com/posts/iamabhishekchoudhary_apachespark-deltalake-bioinformatics-activity-6758414696254119936-RCz5)\n* [ExPASy - Databases and software tools in proteomics, #genomics, phylogeny, systems biology, evolution, population genetics, and transcriptomics 💡💊💊](https://www.linkedin.com/posts/iamabhishekchoudhary_genomics-bioinformatics-expasy-activity-6750752435607793664-4M1Q)\n* [What is Metadata - A Data Engineering necessity](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-metadata-activity-6860433758181474304-qLcY)\n* [What is Distributed Database](https://www.linkedin.com/posts/iamabhishekchoudhary_distributeddatabaseintroduction-activity-6864058792556982272-ulZZ)\n* [To Partition, or Not to Partition, That is the Join Question in a Real System](https://www.linkedin.com/posts/iamabhishekchoudhary_when-does-radix-partitioning-pay-of-in-database-activity-6879826992397742080-Fqzn)\n* [Paper: Solana- A new architecture for a high performance blockchain-inspired by Distributed Systems](https://www.linkedin.com/posts/iamabhishekchoudhary_solana-whitepaper-distributed-blockchain-activity-6881606001796235264-jDBU)\n* [Scaling Large Production Clusters with Partitioned Synchronization](https://www.linkedin.com/posts/iamabhishekchoudhary_scaling-large-production-clusters-activity-6881968002917638144-5CiQ)\n* [Paper: Volcano Operator Model is based on relational algebra](https://www.linkedin.com/posts/iamabhishekchoudhary_volcano-query-processing-engine-activity-6884139657123241984-ZwEY)\n* [Paper: Faster and Cheaper Serverless Computing on Harvested Resources](https://www.linkedin.com/posts/iamabhishekchoudhary_faster-and-cheaper-serverless-computing-on-activity-6892139456795598850-X-Qs)\n* [DBOS: A Paper on DBMS-oriented Operating System](https://www.linkedin.com/posts/iamabhishekchoudhary_dbos-database-operating-system-activity-6895067128035127296-V7Hh)\n* [SSD Storage - Scale Caching without increasing too much cost \u0026 Smart Indexing for faster data query](https://www.linkedin.com/posts/iamabhishekchoudhary_lifescience-dataengineering-datainfrastructure-activity-6902532964022849537-EBYE)\n* [Paper: Lineage Tracing for General Data Warehouse Transformations](https://www.linkedin.com/posts/iamabhishekchoudhary_paper-lineage-tracing-activity-6909529244867960832-z-kS?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [What Every Programmer Should Know About Memory](https://www.linkedin.com/posts/iamabhishekchoudhary_what-every-programmer-should-know-about-memory-activity-6929358484312342528-cqtq)\n* [Deployment Archetypes for Cloud Applications](https://www.linkedin.com/posts/iamabhishekchoudhary_deployment-archetypes-for-cloud-applications-activity-6934872860444299264-_2Dm?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [PolarDB Serverless: A Cloud Native Database for Disaggregated Data Centers](https://www.linkedin.com/posts/iamabhishekchoudhary_polardb-serverless-activity-6948647647712837633-_gRs?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask](https://www.linkedin.com/posts/iamabhishekchoudhary_vectorizationvscompilation-activity-6951179494887981056-DC3b?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n* [Dual use of artifcial-intelligence-powered drug\ndiscovery](https://www.nature.com/articles/s42256-022-00465-9.pdf)\n\n## Discussions\n* [Should you pick Managed Service or build self Managed Open Source Infrastructure](https://www.linkedin.com/posts/iamabhishekchoudhary_data-engineering-managed-service-vs-self-activity-6901439190475579392-BxxY)\n* [What is Sigstore](https://www.linkedin.com/posts/iamabhishekchoudhary_github-sigstoresigstore-common-go-library-activity-6927946358867824640-qbpK)\n* [Security Threat Model](https://owasp.org/www-community/Threat_Modeling_Process)\n* [Kubernetes Security \u0026 Secrets](https://www.macchaffee.com/blog/2022/k8s-secrets/)\n* [2 ways of Data/ML Product Development](https://www.linkedin.com/posts/iamabhishekchoudhary_machinelearning-activity-6931979116527177729-EQzG?utm_source=linkedin_share\u0026utm_medium=member_desktop_web)\n\n\n## Basics\n* [What is Compiler - Programming language Processor](https://www.linkedin.com/posts/iamabhishekchoudhary_what-is-compiler-activity-6897471758522212353-w9rs)\n* [Understanding Raft Consensus](https://www.linkedin.com/posts/iamabhishekchoudhary_raft-consensus-activity-6898202083418279936-Bl-J)\n* [How does SSH (Secure Shell) Work](https://www.linkedin.com/posts/iamabhishekchoudhary_lifescience-dataengineering-datainfrastructure-activity-6903984507943227392-Rqcw)\n* [Operation System Memory Management - Why Do you even need Virtual Memory](https://www.linkedin.com/posts/iamabhishekchoudhary_lifescience-dataengineering-datainfrastructure-activity-6904349820576690176-1pgR)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhishek-ch%2Faround-dataengineering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhishek-ch%2Faround-dataengineering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhishek-ch%2Faround-dataengineering/lists"}