{"id":31686990,"url":"https://github.com/straightchlorine/quantum-pipeline","last_synced_at":"2025-10-08T09:58:17.812Z","repository":{"id":273704435,"uuid":"867991652","full_name":"straightchlorine/quantum-pipeline","owner":"straightchlorine","description":"A Python module for executing and monitoring quantum algorithms across local simulators and IBM Quantum platforms. Seamlessly handles data collection, organization, and streaming to Apache Kafka","archived":false,"fork":false,"pushed_at":"2025-09-14T16:49:50.000Z","size":512,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-14T18:34:34.112Z","etag":null,"topics":["apache-kafka","apache-spark","aws-s3","cuda","docker","gpu-acceleration","ibm-cloud","ibm-quantum","minio","qiskit","qiskit-aer","qiskit-nature","quantum-computing","visualizations","vqe"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/straightchlorine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-05T07:24:18.000Z","updated_at":"2025-09-14T16:49:48.000Z","dependencies_parsed_at":"2025-01-22T14:20:05.523Z","dependency_job_id":"121aa7de-5486-4b17-ab9e-4368fff5892d","html_url":"https://github.com/straightchlorine/quantum-pipeline","commit_stats":null,"previous_names":["straightchlorine/quantum-pipeline"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/straightchlorine/quantum-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/straightchlorine%2Fquantum-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/straightchlorine%2Fquantum-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/straightchlorine%2Fquantum-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/straightchlorine%2Fquantum-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/straightchlorine","download_url":"https://codeload.github.com/straightchlorine/quantum-pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/straightchlorine%2Fquantum-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278924141,"owners_count":26069400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","apache-spark","aws-s3","cuda","docker","gpu-acceleration","ibm-cloud","ibm-quantum","minio","qiskit","qiskit-aer","qiskit-nature","quantum-computing","visualizations","vqe"],"created_at":"2025-10-08T09:58:13.476Z","updated_at":"2025-10-08T09:58:17.799Z","avatar_url":"https://github.com/straightchlorine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Quantum Pipeline\n\n## Overview\n\nThe Quantum Pipeline project is an extensible framework designed for exploring Variational Quantum Eigensolver (VQE) algorithms. It combines quantum and classical computing to estimate the ground-state energy of molecular systems with a comprehensive data engineering pipeline.\n\nThe framework provides modules to handle algorithm orchestration, parametrising it as well as monitoring and data visualization. Data is organised in extensible dataclasses, which can be streamed via Kafka for real-time processing, transformed into ML features using Apache Spark, and stored in Apache Iceberg tables for scalable analytics.\n\nCurrently, it offers VQE as its primary algorithm with production-grade data processing capabilities, including automated workflow orchestration via Apache Airflow, and aims to evolve into a convenient platform for running various quantum algorithms at scale.\n\n---\n\n## Features\n\n### Core Quantum Computing\n- **Molecule Loading:** Load and validate molecular data from files.\n- **Hamiltonian Preparation:** Generate second-quantized Hamiltonians for molecular systems.\n- **Quantum Circuit Construction:** Create parameterized ansatz circuits with customizable repetitions.\n- **VQE Execution:** Solve Hamiltonians using the VQE algorithm with support for various optimizers.\n- **Advanced Backend Options:** Customize simulation parameters such as qubit count, shot count, and optimization levels.\n\n### Data Engineering Pipeline\n- **Real-time Streaming:** Stream simulation results to Apache Kafka with Avro serialization for real-time data processing.\n- **ML Feature Engineering:** Transform quantum experiment data into ML features using Apache Spark with incremental processing.\n- **Data Lake Storage:** Store processed data in Apache Iceberg tables with versioning and time-travel capabilities.\n- **Object Storage:** Persist data using MinIO S3-compatible storage with automated backup and retention.\n- **Workflow Orchestration:** Automate data processing workflows using Apache Airflow with monitoring and alerting.\n\n### Analytics and Visualization\n- **Visualization Tools:** Plot molecular structures, energy convergence, and operator coefficients.\n- **Report Generation:** Automatically generate detailed reports for each processed molecule.\n- **Feature Tables:** Access structured data through 9 specialized ML feature tables (molecules, iterations, parameters, etc.).\n- **Processing Metadata:** Track data lineage and processing history with comprehensive metadata management.\n\n### Production Deployment\n- **Containerized Execution:** Deploy as multi-service Docker containers with GPU support.\n- **CI/CD Pipeline:** Automated testing, building, and publishing of Docker images via GitHub Actions.\n- **Scalable Architecture:** Distributed processing with Spark clusters and horizontal scaling capabilities.\n- **Security:** Comprehensive secrets management and secure communication between services.\n\n---\n\n## Directory Structure\n\n```\nquantum_pipeline/\n├── configs/              # Configuration settings and argument parsers\n├── drivers/              # Molecule loading and basis set validation\n├── features/             # Quantum circuit and Hamiltonian features\n├── mappers/              # Fermionic-to-qubit mapping implementations\n├── report/               # Report generation utilities\n├── runners/              # VQE execution logic\n├── solvers/              # VQE solver implementations\n├── stream/               # Kafka streaming and messaging utilities\n├── structures/           # Quantum and classical data structures\n├── utils/                # Utility functions (logging, visualization, etc.)\n├── visual/               # Visualization tools for molecules and operators\n├── docker/               # Docker configurations and deployment files\n│   ├── airflow/          # Airflow DAGs and Spark processing scripts\n│   ├── connectors/       # Kafka Connect configurations\n│   ├── Dockerfile.cpu    # CPU-optimized container\n│   ├── Dockerfile.gpu    # GPU-accelerated container\n│   ├── Dockerfile.spark  # Spark cluster container\n│   └── Dockerfile.airflow # Airflow services container\n├── notebooks/            # Jupyter notebooks for data analysis and exploration\n├── .github/              # CI/CD workflows and automation\n└── quantum_pipeline.py   # Main entry point\n```\n\n---\n\n## Installation\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/your-repo/quantum_pipeline.git\n   cd quantum_pipeline\n   ```\n\n2. **Set Up a Virtual Environment** (optional but recommended):\n   ```bash\n   python3 -m venv env\n   source env/bin/activate\n   ```\n\n3. **Install Dependencies**:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. **(Optional) Deploy Full Platform with Docker Compose**:\n   ```bash\n   docker-compose up --build\n   ```\n\n   This launches the complete data platform including:\n   - Quantum Pipeline (CPU/GPU)\n   - Apache Kafka with Schema Registry\n   - Apache Spark cluster (master + workers)\n   - Apache Airflow (webserver, scheduler, triggerer)\n   - MinIO object storage\n   - PostgreSQL database\n\n5. **(Alternative) Build Individual Containers**:\n   ```bash\n   # CPU-optimized container\n   docker build -f docker/Dockerfile.cpu .\n\n   # GPU-accelerated container (requires NVIDIA Docker)\n   docker build -f docker/Dockerfile.gpu .\n\n   # Spark processing container\n   docker build -f docker/Dockerfile.spark .\n   ```\n\n6. **(Production) Use Pre-built Images**:\n   Docker images are automatically built and published via GitHub Actions:\n   ```bash\n   # Latest stable release\n   docker pull straightchlorine/quantum-pipeline:latest\n\n   # GPU-enabled version\n   docker pull straightchlorine/quantum-pipeline:latest-gpu\n   ```\n\n---\n\n## Usage\n\n### 1. Prepare Input Data\n\nMolecules should be defined like this:\n\n```json\n[\n    {\n        \"symbols\": [\"H\", \"H\"],\n        \"coords\": [[0.0, 0.0, 0.0], [0.0, 0.0, 0.74]],\n        \"multiplicity\": 1,\n        \"charge\": 0,\n        \"units\": \"angstrom\",\n        \"masses\": [1.008, 1.008]\n    },\n    {\n        \"symbols\": [\"O\", \"H\", \"H\"],\n        \"coords\": [[0.0, 0.0, 0.0], [0.0, 0.757, 0.586], [0.0, -0.757, 0.586]],\n        \"multiplicity\": 1,\n        \"charge\": 0,\n        \"units\": \"angstrom\",\n        \"masses\": [15.999, 1.008, 1.008]\n    }\n]\n```\n\n### 2. Run the Pipeline\n\nRun the main script to process molecules:\n\n```bash\npython quantum_pipeline.py -f data/molecule.json -b sto-3g --max-iterations 100 --optimizer COBYLA --report\n```\n\nDefaults for each option can be found in `configs/defaults.py` and the help message (`python quantum_pipeline.py -h`). Other available parameters include:\n\n- `-f FILE, --file FILE`: Path to the molecule data file (required).\n- `-b BASIS, --basis BASIS`: Specify the basis set for the simulation.\n- `--local`: Use a local quantum simulator instead of IBM Quantum.\n- `--min-qubits MIN_QUBITS`: Specify the minimum number of qubits required.\n- `--max-iterations MAX_ITERATIONS`: Set the maximum number of VQE iterations.\n- `--optimizer OPTIMIZER`: Choose from a variety of optimization algorithms.\n- `--output-dir OUTPUT_DIR`: Specify the directory for storing output files.\n- `--log-level {DEBUG,INFO,WARNING,ERROR}`: Set the logging level.\n- `--shots SHOTS`: Number of shots for quantum circuit execution.\n- `--optimization-level {0,1,2,3}`: Circuit optimization level.\n- `--report`: Generate a PDF report after simulation.\n- `--kafka`: Stream data to Apache Kafka for real-time processing.\n\n### Example Configurations\n\nBasic configuration (utilizes the `defaults.py` config) emphasizes performance over accuracy:\n```bash\npython quantum_pipeline.py -f data/molecules.json\n```\n\nConfiguration with custom parameters:\n```bash\npython quantum_pipeline.py -f data/molecule.json -b cc-pvdz --max-iterations 200 --optimizer L-BFGS-B --shots 2048 --report\n```\n\n### 3. Data Platform Integration\n\n**Kafka Streaming**: Enable real-time streaming to Apache Kafka:\n```bash\npython quantum_pipeline.py -f data/molecule.json --kafka\n```\n\n**Full Platform Deployment**: Launch with complete data processing pipeline:\n```bash\n# Start all services\ndocker-compose up -d\n\n# Run quantum pipeline with data streaming\ndocker-compose exec quantum-pipeline python quantum_pipeline.py -f data/molecules.json --kafka --gpu\n```\n\n**Airflow Orchestration**: Access the Airflow web interface at `http://localhost:8084` to:\n- Monitor automated daily processing workflows\n- View data processing logs and metrics\n- Manage DAG schedules and configurations\n\n**Spark Analytics**: Process and analyze quantum experiment data:\n```bash\n# Access Spark master UI at http://localhost:8080\n# MinIO console at http://localhost:9001\n# Kafka UI available through connect APIs\n```\n\n---\n\n## Examples\n\n### Python API\n\nThe framework can be used programmatically:\n\n```python\nfrom quantum_pipeline.runners.vqe_runner import VQERunner\n\nbackend = VQERunner.default_backend()\nrunner = VQERunner(\n    filepath='data/molecules.json',\n    basis_set='sto3g',\n    max_iterations=1,\n    convergence_threshold=1e-6,\n    optimizer='COBYLA',\n    ansatz_reps=3\n)\nrunner.run(backend)\n```\n\n### Docker Examples\n\n**Single Container Execution**:\n```bash\n# CPU version\ndocker run --rm straightchlorine/quantum-pipeline:latest --file /app/data/molecule.json --basis sto-3g --max-iterations 10\n\n# GPU version (requires NVIDIA Docker)\ndocker run --rm --gpus all straightchlorine/quantum-pipeline:latest-gpu --file /app/data/molecule.json --basis sto-3g --gpu\n```\n\n**Platform Deployment**:\n```bash\n# Deploy complete data platform\ndocker-compose up -d\n\n# Execute quantum simulation with full data processing\ndocker-compose exec quantum-pipeline python quantum_pipeline.py \\\n  -f data/molecules.json \\\n  --kafka \\\n  --gpu \\\n  --max-iterations 150 \\\n  --report\n```\n\n### Example KafkaConsumer\n\nYou can test the Kafka integration with a simple consumer like this:\n\n```python\nfrom kafka import KafkaConsumer\nfrom quantum_pipeline.stream.serialization.interfaces.vqe import VQEDecoratedResultInterface\n\nclass KafkaMessageConsumer:\n    def __init__(self, topic='vqe_results', bootstrap_servers='localhost:9092'):\n        self.deserializer = VQEDecoratedResultInterface()\n        self.consumer = KafkaConsumer(\n            topic,\n            bootstrap_servers=bootstrap_servers,\n            value_deserializer=self.deserializer.from_avro_bytes,\n            auto_offset_reset='earliest',\n            enable_auto_commit=True,\n            group_id='vqe_consumer_group'\n        )\n\n    def consume_messages(self):\n        try:\n            for message in self.consumer:\n                try:\n                    # Process the message\n                    decoded_message = message.value\n                    yield decoded_message\n                except Exception as e:\n                    print(f\"Error processing message: {str(e)}\")\n                    continue\n        except Exception as e:\n            print(f\"Error in consumer: {str(e)}\")\n        finally:\n            self.consumer.close()\n```\n\nThen you can use the consumer like this:\n```python\nconsumer = KafkaMessageConsumer()\nfor msg in consumer.consume_messages():\n    print(f\"Received message: {msg}\")\n```\n\n### Data Analytics with Spark\n\nAccess processed quantum data through Iceberg tables:\n```python\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder \\\n    .appName(\"Quantum Data Analytics\") \\\n    .config(\"spark.sql.catalog.quantum_catalog\", \"org.apache.iceberg.spark.SparkCatalog\") \\\n    .getOrCreate()\n\n# Query VQE results\nvqe_results = spark.sql(\"\"\"\n    SELECT molecule_id, basis_set, minimum_energy, total_iterations\n    FROM quantum_catalog.quantum_features.vqe_results\n    WHERE processing_date \u003e= '2025-01-01'\n\"\"\")\n\n# Analyze convergence patterns\nconvergence = spark.sql(\"\"\"\n    SELECT experiment_id, iteration_step, iteration_energy\n    FROM quantum_catalog.quantum_features.vqe_iterations\n    ORDER BY experiment_id, iteration_step\n\"\"\")\n```\n---\n\n## Architecture Overview\n\nThe platform follows a modern data architecture with the following components:\n\n```\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│  Quantum        │───▶│  Apache Kafka    │───▶│  Apache Spark   │\n│  Pipeline       │    │  (Streaming)     │    │  (Processing)   │\n│  (VQE Runner)   │    │                  │    │                 │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n                                │                        │\n                                ▼                        ▼\n┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐\n│  Apache Airflow │    │  Schema Registry │    │  Apache Iceberg │\n│  (Orchestration)│    │  (Avro Schemas)  │    │  (Data Lake)    │\n└─────────────────┘    └──────────────────┘    └─────────────────┘\n         │                       │                       │\n         └───────────────────────┼───────────────────────┘\n                                 ▼\n                    ┌──────────────────┐\n                    │  MinIO Storage   │\n                    │  (Object Store)  │\n                    └──────────────────┘\n```\n\n## CI/CD and Deployment\n\nThe project includes comprehensive CI/CD pipelines via GitHub Actions (`.github/` folder):\n\n- **Automated Testing**: Python tests with flake8 linting on every PR\n- **Docker Image Building**: Automatic builds for CPU and GPU variants\n- **Security Scanning**: Trivy vulnerability scans for all container images\n- **DockerHub Publishing**: Automated daily and tag-based releases\n- **Image Signing**: Cosign-based container signing for security\n\nAvailable Docker images:\n- `straightchlorine/quantum-pipeline:latest` (CPU optimized)\n- `straightchlorine/quantum-pipeline:latest-gpu` (GPU accelerated)\n- `straightchlorine/quantum-pipeline:nightly-cpu` (Development builds)\n- `straightchlorine/quantum-pipeline:nightly-gpu` (Development builds)\n\n## Contributing\n\nFor now, this project is not open for contributions since it is a university project, but feel free to fork it and make your own version.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the `LICENSE` file for more details.\n\n---\n\n## Contact\n\nFor questions or support, please reach out to:\n- **Email:** piotrlis555@gmail.com\n- **GitHub:** [straightchlorine](https://github.com/straightchlorine)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstraightchlorine%2Fquantum-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstraightchlorine%2Fquantum-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstraightchlorine%2Fquantum-pipeline/lists"}