{"id":28047903,"url":"https://github.com/camilajaviera91/mock-data-factory","last_synced_at":"2026-05-04T10:38:45.396Z","repository":{"id":290646678,"uuid":"975151303","full_name":"CamilaJaviera91/mock-data-factory","owner":"CamilaJaviera91","description":"Generate large-scale synthetic datasets using SQL and BigQuery. ","archived":false,"fork":false,"pushed_at":"2025-05-10T02:41:44.000Z","size":50,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-11T21:55:32.722Z","etag":null,"topics":["bigquery","dbt","dotenv","exceute-batch","faker","load-dotenv","os","postgresql","psycopg2","psycopg2-extras","random","sql","yml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CamilaJaviera91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-29T21:38:33.000Z","updated_at":"2025-05-10T02:41:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"5816c3fd-fa4b-4991-b989-b3f721e010e6","html_url":"https://github.com/CamilaJaviera91/mock-data-factory","commit_stats":null,"previous_names":["camilajaviera91/mock-data-factory"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CamilaJaviera91%2Fmock-data-factory","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CamilaJaviera91%2Fmock-data-factory/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CamilaJaviera91%2Fmock-data-factory/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CamilaJaviera91%2Fmock-data-factory/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CamilaJaviera91","download_url":"https://codeload.github.com/CamilaJaviera91/mock-data-factory/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253639575,"owners_count":21940446,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","dbt","dotenv","exceute-batch","faker","load-dotenv","os","postgresql","psycopg2","psycopg2-extras","random","sql","yml"],"created_at":"2025-05-11T21:55:37.170Z","updated_at":"2026-05-04T10:38:45.354Z","avatar_url":"https://github.com/CamilaJaviera91.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🏭 Mock Data Factory\n\n## 🧰 What Is Mock Data Factory?\n\nThis project is designed to generate large-scale synthetic datasets using SQL and BigQuery. It's particularly useful for:\n\n- Testing and prototyping data pipelines\n\n- Demonstrating dashboards or analytical models with realistic (but fake) data\n\n- Simulating business scenarios with high data volumes\n\n---\n\n## 🔧 Key Features\n\n- **SQL-Based Data Generation:** Utilizes SQL for creating synthetic data tailored for BigQuery environments.\n\n- **Modular Models:** Includes separate models for customers, products, and orders, allowing for flexible data generation.\n\n- **Parameterized Row Volume:** Supports generating datasets with varying sizes (e.g., 10K, 1M rows) to suit different testing needs.\n\n- **DBT-Ready Structure:** Structured to integrate seamlessly with DBT (Data Build Tool) for scalable development.\n\n- **Pipeline Integration:** Designed for easy integration with data pipeline tools like Airflow and Cloud Composer.\n\n---\n\n## 🚀 How to Get Started\n\n1. Clone the repository:\n\n```\ngit clone https://github.com/CamilaJaviera91/mock-data-factory.git\ncd mock-data-factory/mockdata\n```\n\n2. Install dependencies:\n\nMake sure Python and pip are installed, then run:\n\n```\npip install -r requirements.txt\n```\n\n3. Configure DBT:\n\nEnsure DBT is installed and set up to connect to your BigQuery project.\n\n4. Generate synthetic data:\n\nRun the Python script:\n\n```\npython scripts/generate_fake_data.py\n```\n\n5. Run DBT models:\n\nCompile and execute the models to populate data in BigQuery:\n\n```\ndbt run\n```\n\n---\n\n## 🗂️ Project Structure\n\nThe repository follows a structure compatible with DBT (Data Build Tool), which makes it easy to integrate with orchestration tools like Airflow or Cloud Composer.\n\n```\n├── .gitignore\n├── logs\n│   └── dbt.log\n├── mockdata\n│   ├── analyses\n│   │   └── .gitkeep\n│   ├── dbt_project.yml\n│   ├── .gitignore\n│   ├── macros\n│   │   └── .gitkeep\n│   ├── models\n│   │   ├── client.sql\n|   |   ├── orders.sql\n|   |   ├── product.sql\n│   │   └── schema.yml\n│   ├── scripts\n│   │   └── generate_fake_data.py\n│   ├── README.md\n│   ├── seeds\n│   │   └── .gitkeep\n│   ├── snapshots\n│   │   └── .gitkeep\n│   └── tests\n│       └── .gitkeep\n└── README.md\n```\n\n---\n\n## 🧪 DBT Integration\n\nTry running the following commands.\n\n- Builds or materializes the tables/views based on your transformations.\n\n```\ndbt run\n```\n\n- Runs checks on the data to ensure quality and correctness.\n\n```\ndbt test\n```\n\n---\n\n## 🐍 Python Script: generate_fake_data.py\n\n- Located in the **mockdata/scripts/** directory, this Python script uses the Faker library to generate synthetic data for a PostgreSQL database. It creates and populates three tables: client, product, and orders. \n\n- The script is customizable and can be extended to fit specific testing scenarios.\n\n---\n\n### 🔧 Key Features\n\n- SQL-based data generation tailored for BigQuery.\n\n- Modular models for entities like customers, products, and orders.\n\n- Parameterized row volume, allowing generation of datasets with varying sizes (e.g., 10K, 1M rows).\n\n- DBT-ready structure for scalable development.\n\n- Easy integration with data pipeline tools such as Airflow and Cloud Composer.\n\n---\n\n### 🚀 Setup\n\n#### ⚙️ Step 1: Prerequisites\n\nMake sure you have the following installed:\n\n- Python 3.7+\n\n- PostgreSQL (running locally or accessible remotely)\n\n#### 🔗 Step 2: Configure Database Connection\n\nUpdate the PostgreSQL credentials in the script:\n\n```\nconn = psycopg2.connect(\n    host=\"localhost\",\n    database=\"db_name\",\n    user=\"user_name\",\n    password=\"*****\"\n)\n```\n\n- Adjust the values to match your environment.\n\n#### ▶️ Step 3: Run the Script\n\nOnce configured, run the script:\n\n```\npython generate_fake_data.py\n```\n\nYou'll see log messages indicating progress:\n\n```\n✅ Successfully connected to PostgreSQL.\n✅ Tables created or verified.\n✅ Inserted 200 clients.\n✅ Inserted 30 products.\n✅ Inserted 500 orders.\n✅ PostgreSQL connection closed. Data generation complete.\n```\n---\n\n## 🗃️ Table Schemas\n\n```client```\n\n| Column    | Type               |\n|-----------|--------------------|\n| client_id | SERIAL PRIMARY KEY |\n| name      | TEXT               |\n| email     | TEXT               |\n| address   | TEXT               |\n| city      | TEXT               |\n\n```product```\n| Column      | Type                |\n|-------------|---------------------|\n| product_id  | SERIAL PRIMARY KEY  |\n| name        | TEXT                |\n| price       | NUMERIC(10,2)       |\n| category    | TEXT                |\n\n```salesman```\n\n| Column      | Type               |\n|-------------|--------------------|\n| salesman_id | SERIAL PRIMARY KEY |\n| name        | TEXT               |\n| city        | TEXT               |\n\n```orders```\n| Column      | Type                    |\n|-------------|-------------------------|\n| order_id    | SERIAL PRIMARY KEY      |\n| client_id   | INTEGER (FK to client)  |\n| product_id  | INTEGER (FK to product) |\n| order_date  | DATE                    |\n| quantity    | INTEGER                 |\n| total       | NUMERIC(10,2)           |\n\n---\n\n## 🗃️ Model\n\n```resume```\n| Column | Type          |\n|--------|---------------|\n| year   | NUMERIC(10,2) |\n| store  | TEXT          |\n| total  | NUMERIC(10,2) |\n\n---\n\n## 🛠️ Customization\n\nTo generate more or fewer rows, change the arguments in generate_clients(), generate_products(), generate_salesman() or generate_orders(). You can modify the list of product names and categories to suit your business domain.\n\n---\n\n## 📎 Related Projects\n\nI have other related repositories that might be of interest:​\n\n- ```sql-mock-data:``` Earlier version or a related project focusing on SQL-based mock data generation.\n    - [repository](https://github.com/CamilaJaviera91/sql-mock-data)\n\n- ```dbt-transformations-sql-mock-data:``` This project contains transformations and documentation for the data model generated in ```sql-mock-data```.​\n    - [repository](https://github.com/CamilaJaviera91/dbt-transformations-sql-mock-data)\n\n---\n\n## 💭 Final Thoughts\n\n- The **mock-data-factory** repository offers a robust framework for generating synthetic datasets, making it a valuable tool for data engineers and analysts working with BigQuery and DBT. Its modular design and integration capabilities make it adaptable to various testing and development scenarios.\n\n- If you need assistance setting up the project or customizing it for your specific use case, feel free to ask!\n\n---\n\n## 📚 Learn More About dbt\n\n- 📖 **Read the [official dbt documentation](https://docs.getdbt.com/docs/introduction)** — A great starting point to understand how dbt works and how to get started.\n- 💬 **Visit [Discourse](https://discourse.getdbt.com/)** — Explore frequently asked questions and community discussions.\n- 💻 **Join the [dbt Slack community](https://community.getdbt.com/)** — Get live support, ask questions, and connect with other data practitioners.\n- 📅 **Browse upcoming [dbt events](https://events.getdbt.com)** — Find webinars, meetups, and conferences near you.\n- 📰 **Read the [dbt blog](https://blog.getdbt.com/)** — Stay up to date with product updates, best practices, and community highlights.\n\n---\n\n## 📄 License\n\nThis project is released under the **MIT License**.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcamilajaviera91%2Fmock-data-factory","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcamilajaviera91%2Fmock-data-factory","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcamilajaviera91%2Fmock-data-factory/lists"}