{"id":28932339,"url":"https://github.com/juspay/code-as-data","last_synced_at":"2026-03-02T06:03:31.646Z","repository":{"id":293233393,"uuid":"954440559","full_name":"juspay/code-as-data","owner":"juspay","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-17T03:56:23.000Z","size":4593,"stargazers_count":1,"open_issues_count":5,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-12-20T17:42:01.907Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/juspay.png","metadata":{"files":{"readme":"readme-dev.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-25T04:54:09.000Z","updated_at":"2025-12-17T03:54:55.000Z","dependencies_parsed_at":"2025-07-07T08:20:50.184Z","dependency_job_id":"7e9180df-0684-4fbf-b4af-a1a292d37b52","html_url":"https://github.com/juspay/code-as-data","commit_stats":null,"previous_names":["juspay/code-as-data"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/juspay/code-as-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juspay%2Fcode-as-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juspay%2Fcode-as-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juspay%2Fcode-as-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juspay%2Fcode-as-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/juspay","download_url":"https://codeload.github.com/juspay/code-as-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/juspay%2Fcode-as-data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29993549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T01:47:34.672Z","status":"online","status_checked_at":"2026-03-02T02:00:07.342Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-22T16:41:39.580Z","updated_at":"2026-03-02T06:03:26.636Z","avatar_url":"https://github.com/juspay.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Code Analysis Tool\n\nA powerful tool for analyzing code structure and relationships in large Haskell codebases.\n\n## Overview\n\nThis tool parses data generated by the GHC Plugin Fdep , extracts structural information (functions, types, classes, imports, etc.), and stores it in a PostgreSQL database for efficient querying and analysis. It's designed to help developers understand complex codebases by enabling exploration of function call graphs, dependency relationships, and code patterns.\n\n![Image](./images/mermaid-diagram-2025-03-25-100823.png)\n\n## Features\n\n- **Parse and analyze code structure**: Extract functions, types, classes, imports, and instances from source code\n- **Database storage**: Store code structure in a PostgreSQL database for efficient querying\n- **Function call graphs**: Analyze function dependencies and call hierarchies\n- **Module relationships**: Understand dependencies between modules\n- **Advanced queries**: Find complex patterns and relationships in your code\n- **High performance**: Uses optimized bulk loading and parallel processing for handling large codebases\n\n## Setup\n\n### Prerequisites\n\n- Haskell code base\n- Data generated by this plugin \u003chttps://github.com/juspay/spider/tree/ghc-9.2.8/fdep\u003e\n- Python 3.10+\n- PostgreSQL database\n\n### Installation\n\n1. Clone the repository:\n\n   ```bash\n   git clone https://github.com/juspay/code-as-data.git\n   cd code-as-data\n   ```\n\nNOTE: if you have nix just run `nix develop` and jump to step 6\n\n2. Install dependencies:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n3. Install the package in development mode:\n\n   ```bash\n   pip install -e .\n   ```\n\n4. Create a PostgreSQL database:\n\n   ```bash\n   createdb code_as_data\n   ```\n\n5. Set environment variables in a `.env` file:\n\n   ```\n   DB_USER=postgres\n   DB_PASSWORD=postgres\n   DB_HOST=localhost\n   DB_PORT=5432\n   DB_NAME=code_as_data\n   DB_POOL_SIZE=10\n   DB_MAX_OVERFLOW=20\n   DB_POOL_TIMEOUT=30\n   DB_POOL_RECYCLE=1800\n   ```\n\n6. Set up the database schema:\n\n   ```bash\n   python scripts/setup_db.py\n   ```\n\n## Usage\n\n### Importing Code Data\n\nTo import code analysis data into the database:\n\n```bash\npython scripts/import_dumps.py /path/to/fdep_output\n```\n\nThe `--clear` flag will clear existing data before importing.\n\n### Querying the Database\n\nUse the query script to explore code structure:\n\n```bash\n# List all modules\npython examples/query.py list-modules\n\n# Show module details\npython examples/query.py module \"module.name\"\n\n# Show function details\npython examples/query.py function \"functionName\" --module \"module.name\"\n\n# Show type details\npython examples/query.py type \"TypeName\" --module \"module.name\"\n\n# Find most called functions\npython examples/query.py most-called --limit 20\n\n# Generate function call graph\npython examples/query.py call-graph \"functionName\" \"module.name\" --depth 3\n\n# Search for functions containing specific patterns\npython examples/query.py search \"pattern\"\n```\n\n## Using as a Library\n\n### Class Reference\n\n#### Database Models\n\n**Module Class (`src.db.models.Module`)**\n\n- Represents a code module in the database\n- Relationships:\n  - `functions`: Associated functions\n  - `imports`: Import statements\n  - `types`: Type definitions\n  - `classes`: Class definitions\n  - `instances`: Type instances\n\n**Function Class (`src.db.models.Function`)**\n\n- Represents a function definition\n- Key attributes:\n  - `name`: Function name\n  - `function_signature`: Type signature\n  - `raw_string`: Original code\n  - `src_loc`: Source location\n- Relationships:\n  - `module`: Parent module\n  - `where_functions`: Nested functions\n  - `called_functions`: Functions this function calls\n  - `called_by`: Functions that call this function\n\n**Type Class (`src.db.models.Type`)**\n\n- Represents a type definition\n- Key attributes:\n  - `type_name`: Name of the type\n  - `raw_code`: Original type definition\n  - `type_of_type`: Category (DATA, SUMTYPE, TYPE, NEWTYPE, CLASS, INSTANCE)\n- Relationships:\n  - `module`: Parent module\n  - `constructors`: Type constructors\n  - `dependent_types`: Types that depend on this type\n\n**Constructor Class (`src.db.models.Constructor`)**\n\n- Represents a data constructor for a type\n- Relationships:\n  - `type`: Parent type\n  - `fields`: Constructor fields\n\n**Class Class (`src.db.models.Class`)**\n\n- Represents a class definition\n- Key attributes:\n  - `class_name`: Name of the class\n  - `class_definition`: Original code\n\n**Import Class (`src.db.models.Import`)**\n\n- Represents an import statement\n- Key attributes:\n  - `module_name`: Module being imported\n  - `package_name`: Package name\n  - `qualified_style`: Import style\n\n**Instance Class (`src.db.models.Instance`)**\n\n- Represents a type instance\n- Relationships:\n  - `module`: Parent module\n  - `instance_functions`: Functions in this instance\n\n#### Parsers\n\n**FunctionParser (`src.parsers.function_parser.FunctionParser`)**\n\n- Methods:\n  - `load_all_files()`: Load all function definitions\n  - `get_functions()`: Get all parsed functions as domain objects\n  - `_process_module_data()`: Process function data for a module\n  - `_deduplicate_functions_called()`: Remove duplicate function calls\n\n**TypeParser (`src.parsers.type_parser.TypeParser`)**\n\n- Methods:\n  - `load()`: Load all type definitions\n  - `process_single_module()`: Process types for a single module\n  - `_load_raw_code_file()`: Load raw type code\n  - `_load_type_file()`: Parse type information\n\n**ClassParser (`src.parsers.class_parser.ClassParser`)**\n\n- Methods:\n  - `load()`: Load all class definitions\n  - `process_single_class_module()`: Process a single class module\n  - `_process_class_data()`: Process individual class data\n\n**ImportParser (`src.parsers.import_parser.ImportParser`)**\n\n- Methods:\n  - `load()`: Load all import statements\n  - `process_single_module()`: Process imports for a module\n  - `get_imports_for_module()`: Get imports for a specific module\n\n**InstanceParser (`src.parsers.instance_parser.InstanceParser`)**\n\n- Methods:\n  - `load_all_files()`: Load all instance definitions\n  - `process_single_module()`: Process a single module's instances\n  - `find_functions_by_line()`: Find functions within a line range\n\n#### Services\n\n**QueryService (`src.services.query_service.QueryService`)**\n\n- Core query methods:\n  - `get_all_modules()`: Get all modules\n  - `get_module_by_name()`: Find module by name\n  - `get_functions_by_module()`: Get all functions in a module\n  - `get_function_by_name()`: Find functions by name\n  - `get_function_details()`: Get detailed information about a function\n  - `get_most_called_functions()`: Find most frequently called functions\n\n- Advanced query methods:\n  - `execute_advanced_query()`: Run complex queries with joins and conditions\n  - `pattern_match()`: Find patterns in code\n  - `find_similar_functions()`: Find functions similar to a reference\n  - `find_code_patterns()`: Find recurring code patterns\n  - `group_similar_functions()`: Group functions by similarity\n  - `find_cross_module_dependencies()`: Analyze module dependencies\n  - `analyze_module_coupling()`: Measure coupling between modules\n  - `find_complex_functions()`: Identify complex functions\n  - `execute_custom_query()`: Run raw SQL queries\n\n**DumpService (`src.services.dump_service.DumpService`)**\n\n- Methods:\n  - `insert_data()`: Main method to process and insert all data\n  - `process_functions()`: Process function data\n  - `process_classes()`: Process class data\n  - `process_imports()`: Process import data\n  - `process_types()`: Process type data\n  - `process_instances()`: Process instance data\n  - `extreme_parallel_processing()`: Process data in parallel\n  - `prepare_database()`: Prepare database for high-speed inserts\n  - `execute_bulk_load()`: Perform bulk loading of data\n\n### Basic Usage\n\n```python\nfrom src.db.connection import SessionLocal\nfrom src.services.query_service import QueryService\n\n# Create a database session\ndb = SessionLocal()\n\ntry:\n    # Initialize the query service\n    query_service = QueryService(db)\n    \n    # Get all modules\n    modules = query_service.get_all_modules()\n    \n    # Get functions for a module\n    module = query_service.get_module_by_name(\"your.module.name\")\n    if module:\n        functions = query_service.get_functions_by_module(module.id)\n        \n    # Get function details\n    function = query_service.get_function_by_name(\"functionName\", module.id)[0]\n    details = query_service.get_function_details(function.id)\n    \n    # Find the most called functions\n    top_functions = query_service.get_most_called_functions(limit=10)\n    \nfinally:\n    db.close()\n```\n\n### Example Queries\n\nHere are some common query patterns for analyzing code structure:\n\n### Finding Functions by Name Pattern\n\n```python\n# Find all functions with names containing \"Handler\"\nfunctions = db.query(Function).filter(Function.name.like(\"%Handler%\")).all()\n```\n\n### Finding Functions that Call a Specific Function\n\n```python\n# Find functions that call a function named \"processData\"\nfrom sqlalchemy.orm import aliased\n\nCaller = aliased(Function)\nCallee = aliased(Function)\n\nfunctions = (\n    db.query(Caller)\n    .join(function_dependency, Caller.id == function_dependency.c.caller_id)\n    .join(Callee, function_dependency.c.callee_id == Callee.id)\n    .filter(Callee.name == \"processData\")\n    .all()\n)\n```\n\n### Finding Functions by Module\n\n```python\n# Find all functions in a specific module\nmodule = db.query(Module).filter(Module.name == \"your.module.name\").first()\nfunctions = db.query(Function).filter(Function.module_id == module.id).all()\n```\n\n### Finding Types Used in a Function\n\n```python\n# Find types used in a function's signature\nfunction = db.query(Function).filter(Function.name == \"someFunction\").first()\nif function and function.function_signature:\n    # This is a simplified approach - a real implementation would need to\n    # parse the signature more carefully\n    types = db.query(Type).filter(\n        Type.type_name.in_([\n            t.strip() for t in function.function_signature.split(\"-\u003e\")\n        ])\n    ).all()\n```\n\n### Finding Modules that Import a Specific Module\n\n```python\n# Find modules that import a specific module\nmodules = (\n    db.query(Module)\n    .join(Import)\n    .filter(Import.module_name == \"target.module.name\")\n    .all()\n)\n```\n\n## Advanced Queries\n\nThe `QueryService` provides powerful advanced query capabilities through a flexible query language that lets you express complex relationships and patterns in your code.\n\n### Query Language Overview\n\nThe advanced query system is based on a structured JSON format that can express:\n\n- Conditions on entity attributes\n- Joins between related entities\n- Nested relationships\n- Complex patterns\n\n### Supported Operators\n\nThe query language supports the following operators:\n\n| Operator     | Description           | Example                                                                        |\n| ------------ | --------------------- | ------------------------------------------------------------------------------ |\n| `eq`         | Equal to              | `{\"field\": \"name\", \"operator\": \"eq\", \"value\": \"processData\"}`                  |\n| `ne`         | Not equal to          | `{\"field\": \"type_enum\", \"operator\": \"ne\", \"value\": \"where_function\"}`          |\n| `gt`         | Greater than          | `{\"field\": \"line_number_start\", \"operator\": \"gt\", \"value\": 100}`               |\n| `lt`         | Less than             | `{\"field\": \"line_number_end\", \"operator\": \"lt\", \"value\": 200}`                 |\n| `ge`         | Greater than or equal | `{\"field\": \"line_number_start\", \"operator\": \"ge\", \"value\": 100}`               |\n| `le`         | Less than or equal    | `{\"field\": \"line_number_end\", \"operator\": \"le\", \"value\": 200}`                 |\n| `like`       | SQL LIKE              | `{\"field\": \"name\", \"operator\": \"like\", \"value\": \"%Handler%\"}`                  |\n| `ilike`      | Case-insensitive LIKE | `{\"field\": \"name\", \"operator\": \"ilike\", \"value\": \"%handler%\"}`                 |\n| `in`         | In a list             | `{\"field\": \"name\", \"operator\": \"in\", \"value\": [\"init\", \"setup\", \"configure\"]}` |\n| `not_in`     | Not in a list         | `{\"field\": \"name\", \"operator\": \"not_in\", \"value\": [\"private\", \"internal\"]}`    |\n| `contains`   | Contains substring    | `{\"field\": \"raw_string\", \"operator\": \"contains\", \"value\": \"error handling\"}`   |\n| `startswith` | Starts with           | `{\"field\": \"name\", \"operator\": \"startswith\", \"value\": \"get\"}`                  |\n| `endswith`   | Ends with             | `{\"field\": \"name\", \"operator\": \"endswith\", \"value\": \"Handler\"}`                |\n| `between`    | Between values        | `{\"field\": \"line_number_start\", \"operator\": \"between\", \"value\": [100, 200]}`   |\n| `is_null`    | Is null               | `{\"field\": \"function_signature\", \"operator\": \"is_null\", \"value\": true}`        |\n\n### Advanced Query Examples\n\n```python\nfrom src.db.connection import SessionLocal\nfrom src.services.query_service import QueryService\n\ndb = SessionLocal()\n\ntry:\n    query_service = QueryService(db)\n    \n    # Example 1: Find functions with names containing \"Handler\" that call a function named \"processData\"\n    query = {\n        \"type\": \"function\",\n        \"conditions\": [\n            {\"field\": \"name\", \"operator\": \"like\", \"value\": \"%Handler%\"}\n        ],\n        \"joins\": [\n            {\n                \"type\": \"called_function\", \n                \"conditions\": [\n                    {\"field\": \"name\", \"operator\": \"eq\", \"value\": \"processData\"}\n                ]\n            }\n        ]\n    }\n    \n    results = query_service.execute_advanced_query(query)\n    \n    # Example 2: Find functions in a specific module that are called by functions in other modules\n    query = {\n        \"type\": \"function\",\n        \"conditions\": [\n            {\"field\": \"module_id\", \"operator\": \"eq\", \"value\": 42}\n        ],\n        \"joins\": [\n            {\n                \"type\": \"calling_function\",  # Functions that call this function\n                \"conditions\": [\n                    {\"field\": \"module_id\", \"operator\": \"ne\", \"value\": 42}  # Different module\n                ]\n            }\n        ]\n    }\n    \n    results = query_service.execute_advanced_query(query)\n    \n    # Example 3: Find type definitions used by functions with high complexity\n    query = {\n        \"type\": \"type\",\n        \"joins\": [\n            {\n                \"type\": \"function\",\n                \"conditions\": [\n                    # Join with functions that use this type and have complex implementation\n                    {\"field\": \"raw_string\", \"operator\": \"contains\", \"value\": \"case\"},\n                    {\"field\": \"raw_string\", \"operator\": \"contains\", \"value\": \"where\"}\n                ]\n            }\n        ]\n    }\n    \n    results = query_service.execute_advanced_query(query)\n    \n    # Example 4: Find large functions (by line count) that call many different functions\n    # This example combines multiple conditions and complex relationship criteria\n    query = {\n        \"type\": \"function\",\n        \"conditions\": [\n            # Functions with more than 50 lines\n            {\"field\": \"line_number_end\", \"operator\": \"gt\", \"value\": 0},  # Must have valid line numbers\n            {\n                \"field\": \"line_number_end\", \n                \"operator\": \"gt\", \n                \"value\": \"function.line_number_start + 50\"  # Pseudo-code, actual implementation would use raw SQL\n            }\n        ],\n        \"joins\": [\n            {\n                \"type\": \"called_function\",\n                \"conditions\": [],  # No specific conditions on called functions\n                # The query service counts these relationships automatically\n            }\n        ]\n    }\n    \n    # For this specific example, you might need a custom method:\n    complex_functions = query_service.find_complex_functions(\n        complexity_threshold=15,  # Combined score from metrics\n        min_lines=50,            # Minimum lines of code\n        min_dependencies=10      # Minimum number of function calls\n    )\n    \nfinally:\n    db.close()\n```\n\n### Pattern Matching\n\nThe pattern matching system allows you to search for specific code structures and relationships:\n\n```python\n# Example 1: Find call relationships (function A calls function B)\npattern = {\n    \"type\": \"function_call\",\n    \"caller\": \"process\",  # Function names containing \"process\"\n    \"callee\": \"validate\", # Function names containing \"validate\"\n    \"mode\": \"calls\"       # Direction of the relationship\n}\n\nmatches = query_service.pattern_match(pattern)\n\n# Example 2: Find functions called by specific functions (function B is called by function A)\npattern = {\n    \"type\": \"function_call\",\n    \"caller\": \"Controller\",  # Functions with \"Controller\" in the name\n    \"callee\": \"Model\",       # Functions with \"Model\" in the name\n    \"mode\": \"called_by\"      # Reverse direction\n}\n\nmatches = query_service.pattern_match(pattern)\n\n# Example 3: Find types used in specific functions\npattern = {\n    \"type\": \"type_usage\",\n    \"type_name\": \"State\",    # Types containing \"State\" \n    \"usage_in\": \"function\"   # Used in functions\n}\n\nmatches = query_service.pattern_match(pattern)\n\n# Example 4: Find code structure patterns (functions with nested functions)\npattern = {\n    \"type\": \"code_structure\",\n    \"structure_type\": \"nested_function\"  # Look for functions with where functions\n}\n\nmatches = query_service.pattern_match(pattern)\n```\n\n### Similarity Analysis\n\nYou can find functions similar to each other based on code structure and signature:\n\n```python\n# Find functions similar to a reference function\nfunction = query_service.get_function_by_name(\"processData\")[0]\nsimilar_functions = query_service.find_similar_functions(\n    function.id,\n    threshold=0.7  # Similarity threshold (0.0 to 1.0)\n)\n\n# Group all similar functions in the codebase\nfunction_groups = query_service.group_similar_functions(similarity_threshold=0.7)\n```\n\n### Complexity and Coupling Analysis\n\nAnalyze code complexity and coupling between modules:\n\n```python\n# Find complex functions\ncomplex_functions = query_service.find_complex_functions(\n    complexity_threshold=15  # Combined complexity score threshold\n)\n\n# Analyze dependencies between modules\nmodule_dependencies = query_service.find_cross_module_dependencies()\n\n# Get comprehensive coupling metrics\ncoupling_metrics = query_service.analyze_module_coupling()\n```\n\n### Code Pattern Identification\n\nFind recurring patterns across your codebase:\n\n```python\n# Find code patterns that repeat across multiple functions\npattern_code = \"\"\"\n  Right val -\u003e val\n  Left (err :: SomeException) -\u003e\n\"\"\"\n\npattern_matches = query_service.find_code_patterns(\n    pattern_code,\n    min_matches=2  # Minimum number of matching lines\n)\n```\n\n### Custom SQL Queries\n\nFor ultimate flexibility, you can execute custom SQL:\n\n```python\n# Execute custom SQL query\nresults = query_service.execute_custom_query(\n    \"\"\"\n    SELECT f.name AS function_name, m.name AS module_name, COUNT(fd.callee_id) AS call_count\n    FROM function f\n    JOIN module m ON f.module_id = m.id\n    LEFT JOIN function_dependency fd ON f.id = fd.caller_id\n    GROUP BY f.id, m.name\n    HAVING COUNT(fd.callee_id) \u003e 5\n    ORDER BY call_count DESC\n    LIMIT 20\n    \"\"\"\n)\n```\n\n### Custom Parsing\n\nYou can use the parsers directly to process your own code dumps:\n\n```python\nfrom src.parsers.function_parser import FunctionParser\nfrom src.parsers.type_parser import TypeParser\nfrom src.parsers.class_parser import ClassParser\nfrom src.parsers.import_parser import ImportParser\n\n# Parse functions\nfunction_parser = FunctionParser(\"/path/to/dumps\")\nfunction_parser.load_all_files()\nfunctions = function_parser.get_functions()\n\n# Parse types\ntype_parser = TypeParser(\"/path/to/dumps\")\ntypes_by_module = type_parser.load()\n\n# Parse classes\nclass_parser = ClassParser(\"/path/to/dumps\")\nclasses_by_module = class_parser.load()\n\n# Parse imports\nimport_parser = ImportParser(\"/path/to/dumps\")\nimports_by_module = import_parser.load()\n```\n\n## Database Schema\n\n![Image](./images/mermaid-flow-transparent.svg)\n\n### Main Tables\n\n- **module**: Code modules\n  - `id`: Primary key\n  - `name`: Module name\n  - `path`: File path\n\n- **function**: Functions in the codebase\n  - `id`: Primary key\n  - `name`: Function name\n  - `function_signature`: Type signature\n  - `raw_string`: Function code\n  - `src_loc`: Source location\n  - `line_number_start`, `line_number_end`: Line numbers\n  - `type_enum`: Type of function\n  - `module_id`: Foreign key to module\n  - `function_input`, `function_output`: Input/output metadata\n\n- **where_function**: Nested functions defined within other functions\n  - `id`: Primary key\n  - `name`: Function name\n  - `function_signature`: Type signature\n  - `raw_string`: Function code\n  - `src_loc`: Source location\n  - `parent_function_id`: Foreign key to parent function\n\n- **function_called**: References to function calls\n  - `id`: Primary key\n  - `module_name`: Module containing the called function\n  - `name`, `function_name`: Function name\n  - `package_name`: Package name\n  - `src_loc`: Source location\n  - `_type`, `type_enum`: Type information\n  - `function_id`: Foreign key to calling function\n  - `where_function_id`: Foreign key to calling where function\n\n- **class**: Class definitions\n  - `id`: Primary key\n  - `class_name`: Class name\n  - `class_definition`: Class code\n  - `src_location`: Source location\n  - `line_number_start`, `line_number_end`: Line numbers\n  - `module_id`: Foreign key to module\n\n- **import**: Import statements\n  - `id`: Primary key\n  - `module_name`: Imported module name\n  - `package_name`: Package name\n  - `src_loc`: Source location\n  - `is_boot_source`, `is_safe`, `is_implicit`: Import flags\n  - `as_module_name`: Alias name\n  - `qualified_style`: Import style\n  - `is_hiding`: Whether imports are hidden\n  - `hiding_specs`: What is hidden\n  - `line_number_start`, `line_number_end`: Line numbers\n  - `module_id`: Foreign key to module\n\n- **type**: Type definitions\n  - `id`: Primary key\n  - `type_name`: Type name\n  - `raw_code`: Type definition code\n  - `src_loc`: Source location\n  - `type_of_type`: Category (DATA, SUMTYPE, TYPE, etc.)\n  - `line_number_start`, `line_number_end`: Line numbers\n  - `module_id`: Foreign key to module\n\n- **constructor**: Constructors for types\n  - `id`: Primary key\n  - `name`: Constructor name\n  - `type_id`: Foreign key to type\n\n- **field**: Fields in type constructors\n  - `id`: Primary key\n  - `field_name`: Field name\n  - `field_type_raw`: Raw field type\n  - `field_type_structure`: Structured type information\n  - `constructor_id`: Foreign key to constructor\n\n- **instance**: Type instances\n  - `id`: Primary key\n  - `instance_definition`: Instance definition\n  - `instance_signature`: Instance signature\n  - `src_loc`: Source location\n  - `line_number_start`, `line_number_end`: Line numbers\n  - `module_id`: Foreign key to module\n\n- **instance_function**: Linking instances to their functions\n  - `id`: Primary key\n  - `instance_id`: Foreign key to instance\n  - `function_id`: Foreign key to function\n\n- **function_dependency**: Function call relationships\n  - `caller_id`: ID of the calling function\n  - `callee_id`: ID of the called function\n\n- **type_dependency**: Type dependency relationships\n  - `dependent_id`: ID of the dependent type\n  - `dependency_id`: ID of the dependency type\n\n## Development\n\n### Domain Models\n\nThe project also includes Pydantic models for domain objects:\n\n**Function Model (`src.models.function_model.Function`)**\n\n- Represents a parsed function with its relationships\n- Key methods:\n  - `get_function_prompt()`: Generate a formatted representation for prompts\n\n**Type Model (`src.models.type_model.Type`)**\n\n- Represents a type with constructors and fields\n- Key methods:\n  - `get_prompt()`: Return formatted type definition\n  - Properties:\n    - `id`: Generate a unique identifier\n\n**Class Model (`src.models.class_model.Class`)**\n\n- Represents a class definition\n- Key methods:\n  - `get_prompt()`: Return class definition for prompting\n  - Properties:\n    - `id`: Generate a unique identifier\n\n**Import Model (`src.models.import_model.Import`)**\n\n- Represents an import statement\n- Key methods:\n  - `get_prompt()`: Generate a JSON representation\n\n**Instance Model (`src.models.instance_model.Instance`)**\n\n- Represents an instance\n- Properties:\n  - `id`: Generate a unique identifier\n\n### Tests\n\nTo run tests run the following command\n\n```\n\npytest tests\n\n```\n\n### Performance Tuning\n\nThe tool includes performance optimizations for handling large codebases:\n\n1. **Parallel Processing**: Uses multi-core processing for faster data parsing\n2. **Bulk Loading**: Uses PostgreSQL COPY for high-speed data insertion\n3. **Connection Pooling**: Optimizes database connection usage\n4. **Memory Management**: Carefully manages memory for large datasets\n\nTo tune performance:\n\n1. Adjust database connection pool settings in the `.env` file:\n\n   ```\n   DB_POOL_SIZE=10         # Concurrent connections\n   DB_MAX_OVERFLOW=20      # Additional connections when needed\n   DB_POOL_TIMEOUT=30      # Connection timeout in seconds\n   DB_POOL_RECYCLE=1800    # Connection recycle time (30 minutes)\n   ```\n\n2. Optimize PostgreSQL for bulk loading during import:\n\n   ```python\n   # In src/services/dump_service.py\n   # Adjust these settings for your hardware\n   db.execute(text(\"SET work_mem = '256MB'\"))\n   db.execute(text(\"SET maintenance_work_mem = '1GB'\"))\n   ```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature-name`\n3. Commit your changes: `git commit -m 'Add some feature'`\n4. Push to the branch: `git push origin feature-name`\n5. Submit a pull request\n\n## TODO\n\n- [ ] Multiple repo support\n- [ ] Multiple commit support\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuspay%2Fcode-as-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjuspay%2Fcode-as-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjuspay%2Fcode-as-data/lists"}