https://github.com/taabishhh/llm_training

This project implements a distributed pipeline for NLP model training using Apache Spark and DeepLearning4J (DL4J). The methodology utilizes a sliding window approach for data preparation, positional embeddings for token encoding, and Word2Vec model training with parallel processing. The model and training process is designed for scalability and op

apache-spark deeplearning4j dl4j llm llm-training logback-classic mapreduce-scala scalatest sliding-window tensorflow word2vec

Last synced: 06 Mar 2025