{"id":46566855,"url":"https://github.com/amfelso/curiosity-pipeline","last_synced_at":"2026-03-07T07:31:04.743Z","repository":{"id":266396404,"uuid":"898139250","full_name":"amfelso/curiosity-pipeline","owner":"amfelso","description":"A creative project simulating a memory system for NASA’s Curiosity Rover, blending AI, cloud computing, and storytelling. Uses AWS Step Functions, PineconeDB, and serverless architecture to automate image processing, generate reflective diaries, and embed memories for RAG applications.","archived":false,"fork":false,"pushed_at":"2025-12-19T19:12:16.000Z","size":148,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-12-22T08:26:04.968Z","etag":null,"topics":["aws","chatbot","dynamodb","eventbridge","gpt","lambda","mars-rover","nasa","serverless","step-functions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amfelso.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-03T21:24:50.000Z","updated_at":"2025-12-19T19:12:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"26390ead-2ce5-4bc0-96d9-545b2ebd9d25","html_url":"https://github.com/amfelso/curiosity-pipeline","commit_stats":null,"previous_names":["amfelso/curiosity-pipeline"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/amfelso/curiosity-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amfelso%2Fcuriosity-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amfelso%2Fcuriosity-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amfelso%2Fcuriosity-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amfelso%2Fcuriosity-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amfelso","download_url":"https://codeload.github.com/amfelso/curiosity-pipeline/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amfelso%2Fcuriosity-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30209719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T05:23:27.321Z","status":"ssl_error","status_checked_at":"2026-03-07T05:00:17.256Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","chatbot","dynamodb","eventbridge","gpt","lambda","mars-rover","nasa","serverless","step-functions"],"created_at":"2026-03-07T07:31:00.407Z","updated_at":"2026-03-07T07:31:04.738Z","avatar_url":"https://github.com/amfelso.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Curiosity Rover Memory System\n\n[![.github/workflows/Develop.yml](https://github.com/amfelso/curiosity-pipeline/actions/workflows/Develop.yml/badge.svg)](https://github.com/amfelso/curiosity-pipeline/actions/workflows/Develop.yml)\n[![.github/workflows/Release.yml](https://github.com/amfelso/curiosity-pipeline/actions/workflows/Release.yml/badge.svg?branch=release)](https://github.com/amfelso/curiosity-pipeline/actions/workflows/Release.yml)\n\nThis application automates the retrieval, processing, and embedding of Mars rover images into a memory system designed for Retrieval-Augmented Generation (RAG). The pipeline is built on AWS Step Functions and serverless architecture to orchestrate data processing with scalability and efficiency.\n\n## Overview\n\nThe pipeline runs on a nightly schedule (disabled by default to save costs) and performs the following steps:\n\n0. **Daily Scheduler**:\n\n   - Triggers the pipeline on a nightly schedule to automate the retrieval and processing of Mars rover images.\n\n1. **Fetch Images and Metadata**:\n\n   - Retrieves 1-5 random images from NASA's Mars Rover API for a specific date (Earth date or sol).\n   - Outputs a list of image URLs and associated metadata.\n\n2. **Generate Memories and Diary**:\n\n   - Writes daily memory entries for each image, describing key features, speculation, and reflection.\n   - Writes a daily diary entry summarizing all image memories for the date.\n   - Stores these entries in an **S3 bucket** structured by date.\n\n3. **Embed Memories into PineconeDB**:\n   - Embeds memories and diary entries into Pinecone for use in RAG workflows and chatbot conversations.\n\nThe pipeline is designed to enable a chatbot with contextual memory, simulating the ability to \"remember\" and reference Mars Rover data in conversations.\n\n---\n\n## Project Structure\n\n- **`functions`**: Code for Lambda functions handling each pipeline step:\n  - `daily_scheduler`: Triggers the pipeline on a nightly schedule.\n  - `fetch_images_with_metadata`: Retrieves images and metadata.\n  - `generate_memories_and_diary`: Creates structured memory and diary entries in S3.\n  - `embed_memories_to_pinecone`: Embeds memories and diary entries for RAG use.\n- **`statemachines`**: Step Function definition orchestrating the pipeline's tasks.\n- **`tests`**: Unit and integration tests for pipeline components.\n- **`template.yaml`**: AWS SAM template defining serverless resources.\n\n---\n\n## Setup and Deployment\n\n### Local Development Setup\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/amfelso/curiosity-pipeline.git\n   cd curiosity-pipeline\n   ```\n\n2. **Set up your environment**\n   ```bash\n   make setup\n   ```\n   This will:\n   - Install Python dependencies from `layers/curiosity_pipeline/requirements.txt`\n   - Create a `.env` file from `.env.example`\n\n3. **Configure your API keys**\n   Edit the `.env` file with your actual API keys:\n   ```bash\n   PINECONE_API_KEY=your-pinecone-api-key-here\n   OPENAI_API_KEY=your-openai-api-key-here\n   NASA_API_KEY=your-nasa-api-key-here\n   ```\n\n### Available Make Commands\n\n- `make check-tools` - Verify required tools installed\n- `make setup` - Create venv, install dependencies, and create `.env`\n- `make install` - Install Python dependencies only\n- `make login` - Configure AWS credentials from `.env`\n- `make test` - Run all tests (automatically loads `.env`)\n- `make test-unit` - Run unit tests only\n- `make test-integration` - Run integration tests only (requires deployed stack)\n- `make lint` - Run flake8 linter and SAM template validation\n- `make build` - Build SAM application\n- `make deploy` - Lint, test, build, and deploy to AWS\n- `make clean` - Clean build artifacts, venv, and Python cache files\n\n### Deploying the Pipeline\n\n**Automatic Deployment:**\nPipeline will automatically deploy via Github Actions when code updates are merged to the release branch.\n\n**Manual Deployment:**\n```bash\nmake deploy\n```\n\nThis will lint, test, build, and deploy the application to AWS using SAM.\n\n## **Simulated Dates Table and EventBridge**\n\nThe **Simulated Dates Table** and **EventBridge** work together to manage and trigger the Mars Rover simulation.\n\n### **Simulated Dates Table**\n\nThis DynamoDB table stores the current Earth date for each active simulation. It allows the simulation to track and increment the Earth date daily or maintain a static date for testing purposes.\n\n#### **Table Structure**\n\n| Attribute       | Type   | Description                                                              |\n| --------------- | ------ | ------------------------------------------------------------------------ |\n| `simulation_id` | String | Primary key that uniquely identifies a simulation (e.g., `mvp`, `test`). |\n| `earth_date`    | String | Current Earth date for the simulation in `YYYY-MM-DD` format.            |\n\n#### **Example Table Entry**\n\n```json\n{\n  \"simulation_id\": \"mvp\",\n  \"earth_date\": \"2012-08-06\"\n}\n```\n\n### **EventBridge and Daily Scheduler**\n\nEventBridge is used to schedule the simulation’s daily updates. It triggers the **DailySchedulerLambda**, which handles the following tasks:\n\n1. **Fetch the Simulation Date**:\n\n   - Reads the current `earth_date` for the specified `simulation_id` from the **Simulated Dates Table**.\n\n2. **Increment the Date**:\n\n   - Increments the `earth_date` for simulations like `mvp`. For `test`, the date remains static.\n\n3. **Trigger the Pipeline**:\n   - Starts the Step Function for the pipeline with the current `earth_date`.\n\n#### **EventBridge Rule**\n\n- **Frequency**: `\"rate(1 day)\"` ensures the simulation progresses daily.\n- **Target**: The rule invokes the **DailySchedulerLambda** with a payload specifying the `simulation_id`.\n\n#### **Example EventBridge Payload**\n\n```json\n{\n  \"simulation_id\": \"mvp\"\n}\n```\n\n### Enabling the Schedule\n\nThe pipeline's nightly schedule is disabled by default. To enable it:\n\n1. Open the `template.yaml` file in the project directory.\n2. Locate the `MVPEventBridgeRule` resource under the `Resources` section.\n3. Update the `State` property to `ENABLED`:\n\n```yaml\nMVPEventBridgeRule:\n  Type: AWS::Events::Rule\n  Properties:\n    ScheduleExpression: \"rate(1 day)\"\n    Targets:\n      - Arn: !GetAtt DailySchedulerLambda.Arn\n        Id: \"DailySchedulerLambdaTarget\"\n        Input: '{\"simulation_id\": \"mvp\"}'\n    State: DISABLED\n```\n\n---\n\n## **DynamoDB Pipeline Log**\n\nThe DynamoDB pipeline log is used to track the execution status and outputs of each stage in the pipeline. It ensures a complete record of the pipeline’s progress and aids in debugging or auditing.\n\n### **Table Name**\n\n- `PipelineTransactionLogTable`\n\n### **Primary Key**\n\n- `EarthDate` (String): Represents the Earth date corresponding to the pipeline run.\n\n### **Attributes**\n\nThe table structure includes the following attributes:\n\n| Attribute                   | Type   | Description                                                                         |\n| --------------------------- | ------ | ----------------------------------------------------------------------------------- |\n| `EarthDate`                 | String | Primary key indicating the Earth date of the pipeline execution.                    |\n| `sol`                       | Number | Corresponding Mars Sol (Martian day) for the Earth date.                            |\n| `Lambda1__FetchImages`      | Map    | Contains the status, output, and update timestamp for the Fetch Images Lambda.      |\n| `Lambda2__GenerateMemories` | Map    | Contains the status, output, and update timestamp for the Generate Memories Lambda. |\n| `Lambda3__EmbedToPinecone`  | Map    | Contains the status, output, and update timestamp for the Embed to Pinecone Lambda. |\n| `updated_at`                | String | Timestamp of the most recent update to the log entry.                               |\n\n### **Lambda Logs Structure**\n\nEach Lambda log entry is stored as a map with the following keys:\n\n| Key          | Type     | Description                                                      |\n| ------------ | -------- | ---------------------------------------------------------------- |\n| `output`     | List/Map | The output of the Lambda, such as URLs, metadata, or embeddings. |\n| `status`     | String   | Execution status of the Lambda (`Success`, `Error`, etc.).       |\n| `updated_at` | String   | Timestamp of the last update for this Lambda entry.              |\n\n### **Example Log Entry**\n\n```json\n{\n  \"EarthDate\": \"2012-08-07\",\n  \"sol\": 1,\n  \"Lambda1__FetchImages\": {\n    \"output\": [\n      {\n        \"earth_date\": \"2012-08-07\",\n        \"id\": 2674,\n        \"img_src\": \"http://mars.jpl.nasa.gov/msl-raw-images/proj/msl/redops/ods/surface/sol/00001/opgs/edr/ncam/NRA_397586928EDR_F0010008AUT_04096M_.JPG\",\n        \"sol\": 1\n      }\n    ],\n    \"status\": \"Success\",\n    \"updated_at\": \"2024-12-07T20:42:50.015223\"\n  },\n  \"Lambda2__GenerateMemories\": {\n    \"output\": [\n      \"https://curiosity-data.s3.amazonaws.com/memories/2012-08-07/image2674_memory.txt\"\n    ],\n    \"status\": \"Success\",\n    \"updated_at\": \"2024-12-07T21:09:47.448240\"\n  },\n  \"Lambda3__EmbedToPinecone\": {\n    \"output\": [\n      {\n        \"date\": \"2012-08-07\",\n        \"id\": \"4a8f85ba-bd21-404c-8de8-a85ee5801396\",\n        \"s3_url\": \"https://curiosity-data.s3.amazonaws.com/memories/2012-08-07/image2674_memory.txt\",\n        \"type\": \"memory\"\n      }\n    ],\n    \"status\": \"Success\",\n    \"updated_at\": \"2024-12-07T21:03:25.350127\"\n  },\n  \"updated_at\": \"2024-12-07T21:09:47.448240\"\n}\n```\n\n### **Usage**\n\n1. **Log Updates:**\n   Each Lambda function updates its corresponding entry in the DynamoDB log upon completion or failure.\n2. **Tracking Progress:**\n   Use the `EarthDate` key to retrieve pipeline logs for a specific date and check the progress or status of each Lambda.\n3. **Error Handling:**\n   The `status` field in each Lambda log helps identify and debug pipeline failures.\n\n## Folder Structure for Memories\n\nMemories are stored in an S3 bucket with the following structure for simplicity and cost savings:\n\n```plaintext\nmemories/\n├── YYYY-MM-DD/\n│   ├── image1_memory.txt\n│   ├── image2_memory.txt\n│   └── diary.txt\n```\n\n- **`imageX_memory.txt`**: Contains memory details for each image (data, description, speculation, and reflection).\n- **`diary.txt`**: Summarizes the day’s memories into a single diary entry.\n\n---\n\n## Tests\n\nTests ensure the functionality of individual Lambda functions and the pipeline as a whole.\n\n```bash\n# Run all tests\nmake test\n\n# Run only unit tests\nmake test-unit\n\n# Run only integration tests (requires the stack to be deployed)\nmake test-integration\n```\n\n**Note:** Integration tests require the SAM stack to be deployed to AWS first. Unit tests can run locally without any AWS resources.\n\n---\n\n## Resources\n\n- [AWS SAM Developer Guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html): Introduction to SAM specification, the SAM CLI, and serverless application concepts.\n- [NASA Mars Rover API](https://api.nasa.gov/): Official NASA API documentation for accessing Mars rover data.\n- [Pinecone Documentation](https://www.pinecone.io/docs/): Guide to setting up and managing vector embeddings for efficient RAG workflows.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famfelso%2Fcuriosity-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famfelso%2Fcuriosity-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famfelso%2Fcuriosity-pipeline/lists"}