Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Jupyter-Guide

Jupyter Guide
https://github.com/mikeroyal/Jupyter-Guide

Last synced: 4 days ago
JSON representation

Apache Spark Learning Resources

Apache Spark Tools, Libraries, and Frameworks

  • Apache MXNet
  • AutoGluon - accuracy deep learning models on tabular, image, and text data.
  • Spark SQL
  • Spark Streaming - tolerant stream processing engine built on the Spark SQL engine. It can express your streaming computation the same way you would express a batch computation on static data from various sources including Apache Kafka, Apache Flume, and Amazon Kinesis.
  • MLib - level optimization primitives and higher-level pipeline APIs.
  • Graphx - parallel computation. At a high-level, GraphX extends the Spark RDD by introducing the Resilient Distributed Property Graph: a directed multigraph with properties attached to each vertex and edge.
  • PySpark
  • MLflow
  • Tracking component
  • Projects component
  • Models component
  • Model Registry
  • Apache Flume
  • Apache HBase™ - source, NoSQL, distributed big data store. It enables random, strictly consistent, real-time access to petabytes of data. HBase is very effective for handling large, sparse datasets. HBase serves as a direct input and output to the Apache MapReduce framework for Hadoop, and works with Apache Phoenix to enable SQL-like queries over HBase tables.
  • Apache Beam - specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs).
  • Apache Arrow - independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs.
  • Neo4j - strength graph database that combines native graph storage, advanced security, scalable speed-optimized architecture, and ACID compliance to ensure predictability and integrity of relationship-based queries.
  • ElasticSearch - capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java.
  • Trino - us/azure/architecture/data-guide/relational-data/etl), allow them all to use standard SQL statement, and work with numerous data sources and targets all in the same system.
  • Extract, transform, and load (ETL)
  • Redis(REmote DIctionary Server) - memory data structure store, used as a database, cache, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.
  • Koalas - docs/stable/reference/api/pandas.DataFrame.html) on top of Apache Spark.
  • Hadoop Distributed File System (HDFS) - yarn/hadoop-yarn-site/YARN.html).
  • Logstash
  • Kibana

C/C++ Learning Resources

C/C++ Tools and Frameworks

  • AWS SDK for C++
  • Visual Studio - rich application that can be used for many aspects of software development. Visual Studio makes it easy to edit, debug, build, and publish your app. By using Microsoft software development platforms such as Windows API, Windows Forms, Windows Presentation Foundation, and Windows Store.
  • ReSharper C++
  • AppCode - fixes to resolve them automatically. AppCode provides lots of code inspections for Objective-C, Swift, C/C++, and a number of code inspections for other supported languages. All code inspections are run on the fly.
  • CLion - platform IDE for C and C++ developers developed by JetBrains.
  • Code::Blocks
  • Conan
  • High Performance Computing (HPC) SDK
  • Boost - edge C++. Boost has been a participant in the annual Google Summer of Code since 2007, in which students develop their skills by working on Boost Library development.
  • Automake
  • Cmake - source, cross-platform family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files, and generate native makefiles and workspaces that can be used in the compiler environment of your choice.
  • GDB
  • GCC - C, Fortran, Ada, Go, and D, as well as libraries for these languages.
  • GSL - squares fitting. There are over 1000 functions in total with an extensive test suite.
  • OpenGL Extension Wrangler Library (GLEW) - platform open-source C/C++ extension loading library. GLEW provides efficient run-time mechanisms for determining which OpenGL extensions are supported on the target platform.
  • Libtool
  • Maven
  • TAU (Tuning And Analysis Utilities) - based sampling. All C++ language features are supported including templates and namespaces.
  • Clang - C, C++ and Objective-C++ compiler when targeting X86-32, X86-64, and ARM (other targets may have caveats, but are usually easy to fix). Clang is used in production to build performance-critical software like Google Chrome or Firefox.
  • OpenCV - time applications. Cross-Platform C++, Python and Java interfaces support Linux, MacOS, Windows, iOS, and Android.
  • ANTLR (ANother Tool for Language Recognition)
  • Oat++ - efficient web application. It's zero-dependency and easy-portable.
  • Cython
  • Infer - C, and C. Infer is written in OCaml.
  • Azure SDK for C++
  • Azure SDK for C
  • C++ Client Libraries for Google Cloud Services
  • Vcpkg
  • CppSharp
  • Thrust - level interface greatly enhances programmer productivity while enabling performance portability between GPUs and multicore CPUs. Interoperability with established technologies such as CUDA, TBB, and OpenMP integrates with existing software.
  • JavaCPP
  • Spdlog - only/compiled, C++ logging library.
  • Scala for TensorFlow

Deep Learning Learning Resources

Deep Learning Tools, Libraries, and Frameworks

  • Open Neural Network Exchange(ONNX) - in operators and standard data types.
  • Anaconda
  • Scikit-Learn
  • NVIDIA cuDNN - accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN accelerates widely used deep learning frameworks, including Caffe2, Chainer, Keras, MATLAB, MxNet, PyTorch, and TensorFlow.
  • Jupyter Notebook - source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Jupyter is used widely in industries that do data cleaning and transformation, numerical simulation, statistical modeling, data visualization, data science, and machine learning.
  • Apache Spark - scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.
  • Apache PredictionIO
  • BigDL
  • Eclipse Deeplearning4J (DL4J) - based(Scala, Kotlin, Clojure, and Groovy) deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.
  • XGBoost
  • NVIDIA DLSS (Deep Learning Super Sampling)
  • AMD FidelityFX Super Resolution (FSR) - quality solution for producing high resolution frames from lower resolution inputs. It uses a collection of cutting-edge Deep Learning algorithms with a particular emphasis on creating high-quality edges, giving large performance improvements compared to rendering at native resolution directly. FSR enables “practical performance” for costly render operations, such as hardware ray tracing for the AMD RDNA™ and AMD RDNA™ 2 architectures.
  • Intel Xe Super Sampling (XeSS) - cores to run XeSS. The GPUs will have Xe Matrix eXtenstions matrix (XMX) engines for hardware-accelerated AI processing. XeSS will be able to run on devices without XMX, including integrated graphics, though, the performance of XeSS will be lower on non-Intel graphics cards because it will be powered by DP4a instruction.
  • Deep Learning Toolbox™ - term memory (LSTM) networks to perform classification and regression on image, time-series, and text data. You can build network architectures such as generative adversarial networks (GANs) and Siamese networks using automatic differentiation, custom training loops, and shared weights. With the Deep Network Designer app, you can design, analyze, and train networks graphically. It can exchange models with TensorFlow™ and PyTorch through the ONNX format and import models from TensorFlow-Keras and Caffe. The toolbox supports transfer learning with DarkNet-53, ResNet-50, NASNet, SqueezeNet and many other pretrained models.
  • Reinforcement Learning Toolbox™ - making algorithms for complex applications such as resource allocation, robotics, and autonomous systems.
  • Deep Learning HDL Toolbox™ - built bitstreams for running a variety of deep learning networks on supported Xilinx® and Intel® FPGA and SoC devices. Profiling and estimation tools let you customize a deep learning network by exploring design, performance, and resource utilization tradeoffs.
  • Parallel Computing Toolbox™ - intensive problems using multicore processors, GPUs, and computer clusters. High-level constructs such as parallel for-loops, special array types, and parallelized numerical algorithms enable you to parallelize MATLAB® applications without CUDA or MPI programming. The toolbox lets you use parallel-enabled functions in MATLAB and other toolboxes. You can use the toolbox with Simulink® to run multiple simulations of a model in parallel. Programs and models can run in both interactive and batch modes.
  • LIBSVM - SVC, nu-SVC), regression (epsilon-SVR, nu-SVR) and distribution estimation (one-class SVM). It supports multi-class classification.
  • Microsoft Project Bonsai - code AI platform that speeds AI-powered automation development and part of the Autonomous Systems suite from Microsoft. Bonsai is used to build AI components that can provide operator guidance or make independent decisions to optimize process variables, improve production efficiency, and reduce downtime.
  • Microsoft AirSim - source, cross platform, and supports software-in-the-loop simulation with popular flight controllers such as PX4 & ArduPilot and hardware-in-loop with PX4 for physically and visually realistic simulations. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. AirSim is being developed as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles.
  • ROS Toolbox
  • Robotics Toolbox™ - holonomic vehicle. The Toolbox also including a detailed Simulink model for a quadrotor flying robot.
  • Computer Vision Toolbox™
  • Model Predictive Control Toolbox™ - loop simulations, you can evaluate controller performance.
  • Predictive Maintenance Toolbox™ - based and model-based techniques, including statistical, spectral, and time-series analysis.
  • Vision HDL Toolbox™ - streaming algorithms for the design and implementation of vision systems on FPGAs and ASICs. It provides a design framework that supports a diverse set of interface types, frame sizes, and frame rates. The image processing, video, and computer vision algorithms in the toolbox use an architecture appropriate for HDL implementations.
  • Automated Driving Toolbox™ - eye-view plot and scope for sensor coverage, detections and tracks, and displays for video, lidar, and maps. The toolbox lets you import and work with HERE HD Live Map data and OpenDRIVE® road networks. It also provides reference application examples for common ADAS and automated driving features, including FCW, AEB, ACC, LKA, and parking valet. The toolbox supports C/C++ code generation for rapid prototyping and HIL testing, with support for sensor fusion, tracking, path planning, and vehicle controller algorithms.
  • UAV Toolbox
  • Navigation Toolbox™ - based path planners, as well as metrics for validating and comparing paths. You can create 2D and 3D map representations, generate maps using SLAM algorithms, and interactively visualize and debug map generation with the SLAM map builder app.
  • Lidar Toolbox™ - camera cross calibration for workflows that combine computer vision and lidar processing.
  • Mapping Toolbox™
  • PlaidML
  • Caffe
  • Theano - dimensional arrays efficiently including tight integration with NumPy.
  • Apache Spark Connector for SQL Server and Azure SQL - performance connector that enables you to use transactional data in big data analytics and persists results for ad-hoc queries or reporting. The connector allows you to use any SQL database, on-premises or in the cloud, as an input data source or output data sink for Spark jobs.
  • Cluster Manager for Apache Kafka(CMAK)
  • CARLA - source simulator for autonomous driving research. CARLA has been developed from the ground up to support development, training, and validation of autonomous driving systems. In addition to open-source code and protocols, CARLA provides open digital assets (urban layouts, buildings, vehicles) that were created for this purpose and can be used freely.
  • ROS/ROS2 bridge for CARLA(package) - way communication between ROS and CARLA. The information from the CARLA server is translated to ROS topics. In the same way, the messages sent between nodes in ROS get translated to commands to be applied in CARLA.
  • Image Processing Toolbox™ - standard algorithms and workflow apps for image processing, analysis, visualization, and algorithm development. You can perform image segmentation, image enhancement, noise reduction, geometric transformations, image registration, and 3D image processing.
  • OpenCV - time computer vision applications. The C++, Python, and Java interfaces support Linux, MacOS, Windows, iOS, and Android.
  • Azure Databricks - based big data analytics service designed for data science and data engineering. Azure Databricks, sets up your Apache Spark environment in minutes, autoscale, and collaborate on shared projects in an interactive workspace. Azure Databricks supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch, and scikit-learn.

Julia Learning Resources

Julia Tools, Libraries and Frameworks