{"id":28474297,"url":"https://github.com/snowflakedb/snowpark-checkpoints","last_synced_at":"2025-08-31T14:03:52.483Z","repository":{"id":274987056,"uuid":"850042652","full_name":"snowflakedb/snowpark-checkpoints","owner":"snowflakedb","description":"Snowpark Python / Spark Migration Testing Tools","archived":false,"fork":false,"pushed_at":"2025-07-30T14:41:50.000Z","size":15980,"stargazers_count":5,"open_issues_count":2,"forks_count":0,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-07-30T18:39:32.813Z","etag":null,"topics":["data-analytics","data-engineering","data-science","python","snowflake","sql"],"latest_commit_sha":null,"homepage":"https://docs.snowflake.com/en/developer-guide/snowpark/python/snowpark-checkpoints-library","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snowflakedb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-30T18:57:42.000Z","updated_at":"2025-07-30T14:41:53.000Z","dependencies_parsed_at":"2025-05-20T16:31:54.801Z","dependency_job_id":"daaeec01-7d53-4c13-90b3-dfc587162aef","html_url":"https://github.com/snowflakedb/snowpark-checkpoints","commit_stats":null,"previous_names":["snowflakedb/snowpark-checkpoints"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/snowflakedb/snowpark-checkpoints","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowflakedb%2Fsnowpark-checkpoints","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowflakedb%2Fsnowpark-checkpoints/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowflakedb%2Fsnowpark-checkpoints/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowflakedb%2Fsnowpark-checkpoints/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snowflakedb","download_url":"https://codeload.github.com/snowflakedb/snowpark-checkpoints/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowflakedb%2Fsnowpark-checkpoints/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272900157,"owners_count":25012033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analytics","data-engineering","data-science","python","snowflake","sql"],"created_at":"2025-06-07T13:06:41.620Z","updated_at":"2025-08-31T14:03:52.463Z","avatar_url":"https://github.com/snowflakedb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# snowpark-checkpoints\n\nSnowpark Python / Spark Migration Testing Tools\n\n\n[![Build and Test](https://github.com/snowflakedb/snowpark-checkpoints/actions/workflows/snowpark-checkpoints-all-tests.yml/badge.svg?branch=main)](https://github.com/snowflakedb/snowpark-checkpoints/actions/workflows/snowpark-checkpoints-all-tests.yml)\n[![codecov](https://codecov.io/gh/snowflakedb/snowpark-checkpoints/branch/main/graph/badge.svg)](https://codecov.io/gh/snowflakedb/snowpark-checkpoints)\n[![PyPi](https://img.shields.io/pypi/v/snowpark-checkpoints.svg)](https://pypi.org/project/snowpark-checkpoints)\n[![License Apache-2.0](https://img.shields.io/:license-Apache%202-brightgreen.svg)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n[![Codestyle Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\nThe **snowpark-checkpoints**  package is a testing library that will help you validate your migrated Snowpark code and discover any behavioral differences with the original Apache PySpark code.\n\n[Source code][source code] | [Snowpark Checkpoints Developer guide][Snowpark Checkpoints Developer guide] | [Snowpark Checkpoints API references][Snowpark Checkpoints API references] \n\n---\n##### This package is on Public Preview.\n---\n---\n## Install the library \n```bash\npip install snowpark-checkpoints\n```\nThis package requires PySpark to be installed in the same environment. If you do not have it, you can install PySpark alongside Snowpark Checkpoints by running the following command:\n```bash\npip install \"snowpark-checkpoints[pyspark]\"\n```\n---\n\n## Getting started\n\nThis bundle includes:\n- **snowpark-checkpoints-collectors**: Extracts information from PySpark dataframes for validation against Snowpark dataframes.\n- **snowpark-checkpoints-validators**: Validates Snowpark dataframes against predefined schemas and checkpoints.\n- **snowpark-checkpoints-hypothesis**: Generates Snowpark dataframes using the Hypothesis library for testing and data generation.\n- **snowpark-checkpoints-configuration**: Loads `checkpoint.json` and provides a model, working automatically with collectors and validators.\n---\n## snowpark-checkpoints-collectors\n\n\n**snowpark-checkpoints-collector** package offers a function for extracting information from PySpark dataframes. We can then use that data to validate against the converted Snowpark dataframes to ensure that behavioral equivalence has been achieved.\n## Features\n\n- Schema inference collected data mode (Schema): This is the default mode, which leverages Pandera schema inference to obtain the metadata and checks that will be evaluated for the specified dataframe. This mode also collects custom data from columns of the DataFrame based on the PySpark type.\n- DataFrame collected data mode (DataFrame): This mode collects the data of the PySpark dataframe. In this case, the mechanism saves all data of the given dataframe in parquet format. Using the default user Snowflake connection, it tries to upload the parquet files into the Snowflake temporal stage and create a table based on the information in the stage. The name of the file and the table is the same as the checkpoint.\n\n\n\n## Functionalities\n\n### Collect DataFrame Checkpoint\n\n\n\n```python\nfrom pyspark.sql import DataFrame as SparkDataFrame\nfrom snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode\nfrom typing import Optional\n\n# Signature of the function\ndef collect_dataframe_checkpoint(\n    df: SparkDataFrame,\n    checkpoint_name: str,\n    sample: Optional[float] = None,\n    mode: Optional[CheckpointMode] = None,\n    output_path: Optional[str] = None,\n) -\u003e None:\n    ...\n```\n\n- `df`: The input Spark dataframe to collect.\n- `checkpoint_name`: Name of the checkpoint schema file or dataframe.\n- `sample`: Fraction of DataFrame to sample for schema inference, defaults to 1.0.\n- `mode`: The mode to execution the collection (Schema or Dataframe), defaults to CheckpointMode.Schema.\n- `output_path`: The output path to save the checkpoint, defaults to current working directory.\n\n\n### Skip DataFrame Checkpoint Collection\n\n\n\n```python\nfrom pyspark.sql import DataFrame as SparkDataFrame\nfrom snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode\nfrom typing import Optional\n\n# Signature of the function\ndef xcollect_dataframe_checkpoint(\n    df: SparkDataFrame,\n    checkpoint_name: str,\n    sample: Optional[float] = None,\n    mode: Optional[CheckpointMode] = None,\n    output_path: Optional[str] = None,\n) -\u003e None:\n    ...\n```\n\nThe signature of the method is the same of `collect_dataframe_checkpoint`.\n\n## Usage Example\n\n### Schema mode\n\n```python\nfrom pyspark.sql import SparkSession\nfrom snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint\nfrom snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode\n\nspark_session = SparkSession.builder.getOrCreate()\nsample_size = 1.0\n\npyspark_df = spark_session.createDataFrame(\n    [(\"apple\", 21), (\"lemon\", 34), (\"banana\", 50)], schema=\"fruit string, age integer\"\n)\n\ncollect_dataframe_checkpoint(\n    pyspark_df,\n    checkpoint_name=\"collect_checkpoint_mode_1\",\n    sample=sample_size,\n    mode=CheckpointMode.SCHEMA,\n)\n```\n\n\n### Dataframe mode\n\n```python\nfrom pyspark.sql import SparkSession\nfrom snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint\nfrom snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode\nfrom pyspark.sql.types import StructType, StructField, ByteType, StringType, IntegerType \n\nspark_schema = StructType(\n    [\n        StructField(\"BYTE\", ByteType(), True),\n        StructField(\"STRING\", StringType(), True),\n        StructField(\"INTEGER\", IntegerType(), True)\n    ]\n)\n\ndata = [(1, \"apple\", 21), (2, \"lemon\", 34), (3, \"banana\", 50)]\n\nspark_session = SparkSession.builder.getOrCreate()\npyspark_df = spark_session.createDataFrame(data, schema=spark_schema).orderBy(\n    \"INTEGER\"\n)\n\ncollect_dataframe_checkpoint(\n    pyspark_df,\n    checkpoint_name=\"collect_checkpoint_mode_2\",\n    mode=CheckpointMode.DATAFRAME,\n)\n```\n\n---\n\n# snowpark-checkpoints-validators\n\n**snowpark-checkpoints-validators** is a package designed to validate Snowpark DataFrames against predefined schemas and checkpoints. This package ensures data integrity and consistency by performing schema and data validation checks at various stages of a Snowpark pipeline.\n\n## Features\n\n- Validate Snowpark DataFrames against predefined Pandera schemas.\n- Perform custom checks and skip specific checks as needed.\n- Generate validation results and log them for further analysis.\n- Support for sampling strategies to validate large datasets efficiently.\n- Integration with PySpark for cross-validation between Snowpark and PySpark DataFrames.\n\n## Functionalities\n\n### Validate DataFrame Schema from File\n\nThe `validate_dataframe_checkpoint` function validates a Snowpark DataFrame against a checkpoint schema file or dataframe.\n\n```python\nfrom snowflake.snowpark import DataFrame as SnowparkDataFrame\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom snowflake.snowpark_checkpoints.utils.constant import (\n    CheckpointMode,\n)\nfrom snowflake.snowpark_checkpoints.spark_migration import SamplingStrategy\nfrom typing import Any, Optional\n\n# Signature of the function\ndef validate_dataframe_checkpoint(\n    df: SnowparkDataFrame,\n    checkpoint_name: str,\n    job_context: Optional[SnowparkJobContext] = None,\n    mode: Optional[CheckpointMode] = CheckpointMode.SCHEMA,\n    custom_checks: Optional[dict[Any, Any]] = None,\n    skip_checks: Optional[dict[Any, Any]] = None,\n    sample_frac: Optional[float] = 1.0,\n    sample_number: Optional[int] = None,\n    sampling_strategy: Optional[SamplingStrategy] = SamplingStrategy.RANDOM_SAMPLE,\n    output_path: Optional[str] = None,\n):\n    ...\n```\n\n- `df`: Snowpark dataframe to validate.\n- `checkpoint_name`: Name of the checkpoint schema file or dataframe.\n- `job_context`: Snowpark job context.\n- `mode`: Checkpoint mode (schema or data).\n- `custom_checks`: Custom checks to perform.\n- `skip_checks`: Checks to skip.\n- `sample_frac`: Fraction of the dataframe to sample.\n- `sample_number`: Number of rows to sample.\n- `sampling_strategy`: Sampling strategy to use.\n- `output_path`: Output path for the checkpoint report.\n\n### Usage Example\n\n```python\nfrom snowflake.snowpark import Session\nfrom snowflake.snowpark_checkpoints.utils.constant import (\n    CheckpointMode,\n)\nfrom snowflake.snowpark_checkpoints.checkpoint import validate_dataframe_checkpoint\nfrom snowflake.snowpark_checkpoints.spark_migration import SamplingStrategy\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom pyspark.sql import SparkSession\n\nsession = Session.builder.getOrCreate()\njob_context = SnowparkJobContext(\n    session, SparkSession.builder.getOrCreate(), \"job_context\", True\n)\ndf = session.read.format(\"csv\").load(\"data.csv\")\n\nvalidate_dataframe_checkpoint(\n    df,\n    \"schema_checkpoint\",\n    job_context=job_context,\n    mode=CheckpointMode.SCHEMA,\n    sample_frac=0.1,\n    sampling_strategy=SamplingStrategy.RANDOM_SAMPLE\n)\n```\n\n### Check with Spark Decorator\n\nThe `check_with_spark` decorator converts any Snowpark dataframe arguments to a function, samples them, and converts them to PySpark dataframe. It then executes a provided Spark function and compares the outputs between the two implementations.\n\n```python\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom snowflake.snowpark_checkpoints.spark_migration import SamplingStrategy\nfrom typing import Callable, Optional, TypeVar\n\nfn = TypeVar(\"F\", bound=Callable)\n\n# Signature of the decorator\ndef check_with_spark(\n    job_context: Optional[SnowparkJobContext],\n    spark_function: fn,\n    checkpoint_name: str,\n    sample_number: Optional[int] = 100,\n    sampling_strategy: Optional[SamplingStrategy] = SamplingStrategy.RANDOM_SAMPLE,\n    output_path: Optional[str] = None,\n) -\u003e Callable[[fn], fn]:\n    ...\n```\n\n- `job_context`: Snowpark job context.\n- `spark_function`: PySpark function to execute.\n- `checkpoint_name`: Name of the check.\n- `sample_number`: Number of rows to sample.\n- `sampling_strategy`: Sampling strategy to use.\n- `output_path`: Output path for the checkpoint report.\n\n### Usage Example\n\n```python\nfrom snowflake.snowpark import Session\nfrom snowflake.snowpark import DataFrame as SnowparkDataFrame\nfrom snowflake.snowpark_checkpoints.spark_migration import check_with_spark\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom pyspark.sql import DataFrame as SparkDataFrame, SparkSession\n\nsession = Session.builder.getOrCreate()\njob_context = SnowparkJobContext(\n    session, SparkSession.builder.getOrCreate(), \"job_context\", True\n)\n\ndef my_spark_scalar_fn(df: SparkDataFrame):\n    return df.count()\n\n@check_with_spark(\n    job_context=job_context,\n    spark_function=my_spark_scalar_fn,\n    checkpoint_name=\"count_checkpoint\",\n)\ndef my_snowpark_scalar_fn(df: SnowparkDataFrame):\n    return df.count()\n\ndf = job_context.snowpark_session.create_dataframe(\n    [[1, 2], [3, 4]], schema=[\"a\", \"b\"]\n)\ncount = my_snowpark_scalar_fn(df)\n```\n\n### Pandera Snowpark Decorators\n\nThe decorators `@check_input_schema` and `@check_output_schema` allow for sampled schema validation of Snowpark dataframes in the input arguments or in the return value.\n\n```python\nfrom snowflake.snowpark_checkpoints.spark_migration import SamplingStrategy\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom pandera import DataFrameSchema\nfrom typing import Optional\n\n# Signature of the decorator\ndef check_input_schema(\n    pandera_schema: DataFrameSchema,\n    checkpoint_name: str,\n    sample_frac: Optional[float] = 1.0,\n    sample_number: Optional[int] = None,\n    sampling_strategy: Optional[SamplingStrategy] = SamplingStrategy.RANDOM_SAMPLE,\n    job_context: Optional[SnowparkJobContext] = None,\n    output_path: Optional[str] = None,\n):\n    ...\n\n# Signature of the decorator\ndef check_output_schema(\n    pandera_schema: DataFrameSchema,\n    checkpoint_name: str,\n    sample_frac: Optional[float] = 1.0,\n    sample_number: Optional[int] = None,\n    sampling_strategy: Optional[SamplingStrategy] = SamplingStrategy.RANDOM_SAMPLE,\n    job_context: Optional[SnowparkJobContext] = None,\n    output_path: Optional[str] = None,\n):\n    ...\n```\n\n- `pandera_schema`: Pandera schema to validate.\n- `checkpoint_name`: Name of the checkpoint schema file or DataFrame.\n- `sample_frac`: Fraction of the DataFrame to sample.\n- `sample_number`: Number of rows to sample.\n- `sampling_strategy`: Sampling strategy to use.\n- `job_context`: Snowpark job context.\n- `output_path`: Output path for the checkpoint report.\n\n### Usage Example\n\n#### Check Input Schema Example\n```python\nfrom pandas import DataFrame as PandasDataFrame\nfrom pandera import DataFrameSchema, Column, Check\nfrom snowflake.snowpark import Session\nfrom snowflake.snowpark import DataFrame as SnowparkDataFrame\nfrom snowflake.snowpark_checkpoints.checkpoint import check_input_schema\nfrom numpy import int8\n\ndf = PandasDataFrame(\n    {\n        \"COLUMN1\": [1, 4, 0, 10, 9],\n        \"COLUMN2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n    }\n)\n\nin_schema = DataFrameSchema(\n    {\n        \"COLUMN1\": Column(int8, Check(lambda x: 0 \u003c= x \u003c= 10, element_wise=True)),\n        \"COLUMN2\": Column(float, Check(lambda x: x \u003c -1.2, element_wise=True)),\n    }\n)\n\n@check_input_schema(in_schema, \"input_schema_checkpoint\")\ndef preprocessor(dataframe: SnowparkDataFrame):\n    dataframe = dataframe.withColumn(\n        \"COLUMN3\", dataframe[\"COLUMN1\"] + dataframe[\"COLUMN2\"]\n    )\n    return dataframe\n\nsession = Session.builder.getOrCreate()\nsp_dataframe = session.create_dataframe(df)\n\npreprocessed_dataframe = preprocessor(sp_dataframe)\n```\n\n#### Check Input Schema Example\n```python\nfrom pandas import DataFrame as PandasDataFrame\nfrom pandera import DataFrameSchema, Column, Check\nfrom snowflake.snowpark import Session\nfrom snowflake.snowpark import DataFrame as SnowparkDataFrame\nfrom snowflake.snowpark_checkpoints.checkpoint import check_output_schema\nfrom numpy import int8\n\ndf = PandasDataFrame(\n    {\n        \"COLUMN1\": [1, 4, 0, 10, 9],\n        \"COLUMN2\": [-1.3, -1.4, -2.9, -10.1, -20.4],\n    }\n)\n\nout_schema = DataFrameSchema(\n    {\n        \"COLUMN1\": Column(int8, Check.between(0, 10, include_max=True, include_min=True)),\n        \"COLUMN2\": Column(float, Check.less_than_or_equal_to(-1.2)),\n        \"COLUMN3\": Column(float, Check.less_than(10)),\n    }\n)\n\n@check_output_schema(out_schema, \"output_schema_checkpoint\")\ndef preprocessor(dataframe: SnowparkDataFrame):\n    return dataframe.with_column(\n        \"COLUMN3\", dataframe[\"COLUMN1\"] + dataframe[\"COLUMN2\"]\n    )\n\nsession = Session.builder.getOrCreate()\nsp_dataframe = session.create_dataframe(df)\n\npreprocessed_dataframe = preprocessor(sp_dataframe)\n```\n\n### Skip validation\nThe method `xvalidate_dataframe_checkpoint` can be used to avoid the validation of the checkpoint defined.\nThe result of the checkpoint will be `SKIP` and a message will be logged in the terminal. The method contains the same signature of `validate_dataframe_checkpoint`.\n\n#### Usage Example\n\n```python\nfrom snowflake.snowpark import Session\nfrom snowflake.snowpark_checkpoints.utils.constant import (\n    CheckpointMode,\n)\nfrom snowflake.snowpark_checkpoints.checkpoint import validate_dataframe_checkpoint\nfrom snowflake.snowpark_checkpoints.spark_migration import SamplingStrategy\nfrom snowflake.snowpark_checkpoints.job_context import SnowparkJobContext\nfrom pyspark.sql import SparkSession\n\nsession = Session.builder.getOrCreate()\njob_context = SnowparkJobContext(\n    session, SparkSession.builder.getOrCreate(), \"job_context\", True\n)\ndf = session.read.format(\"csv\").load(\"data.csv\")\n\nxvalidate_dataframe_checkpoint(\n    df,\n    \"schema_checkpoint\",\n    job_context=job_context,\n    mode=CheckpointMode.SCHEMA,\n    sample_frac=0.1,\n    sampling_strategy=SamplingStrategy.RANDOM_SAMPLE\n)\n```\n\n\n---\n\n# snowpark-checkpoints-hypothesis\n\n**snowpark-checkpoints-hypothesis** is a [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) extension for generating Snowpark DataFrames. This project provides strategies to facilitate testing and data generation for Snowpark DataFrames using the Hypothesis library.\n\n## Installation\n\nYou can install this package using either **pip** or **conda**:\n\n```shell\npip install snowpark-checkpoints-hypothesis\n--or--\nconda install snowpark-checkpoints-hypothesis\n```\n\n## Usage\n\nThe typical workflow for using the Hypothesis library to generate Snowpark dataframes is as follows:\n\n1. Create a standard Python test function with the different assertions or conditions your code should satisfy for all inputs.\n2. Add the Hypothesis `@given` decorator to your test function and pass the `dataframe_strategy` function as an argument.\n3. Run the test. When the test is executed, Hypothesis will automatically provide the generated inputs as arguments to the test.\n\n### Example 1: Generate Snowpark DataFrames from a JSON schema file\n\nYou can use the `dataframe_strategy` function to create Snowpark DataFrames from a JSON schema file generated by the `collect_dataframe_checkpoint` function of the [snowpark-checkpoints-collectors](https://pypi.org/project/snowpark-checkpoints-collectors/) package:\n\n```python\nfrom hypothesis import given\nfrom snowflake.hypothesis_snowpark import dataframe_strategy\nfrom snowflake.snowpark import DataFrame, Session\n\n\n@given(\n    df=dataframe_strategy(\n        schema=\"path/to/schema.json\",\n        session=Session.builder.getOrCreate(),\n        size=10,\n    )\n)\ndef test_my_function(df: DataFrame):\n    # Test your function here\n    ...\n```\n\n### Example 2: Generate Snowpark DataFrames from a Pandera DataFrameSchema object\n\nYou can also use the `dataframe_strategy` function to create Snowpark DataFrames from a Pandera DataFrameSchema object:\n\n```python\nimport pandera as pa\nfrom hypothesis import given\nfrom snowflake.hypothesis_snowpark import dataframe_strategy\nfrom snowflake.snowpark import DataFrame, Session\n\n@given(\n    df=dataframe_strategy(\n        schema=pa.DataFrameSchema(\n            {\n                \"A\": pa.Column(pa.Int, checks=pa.Check.in_range(0, 10)),\n                \"B\": pa.Column(pa.Bool),\n            }\n        ),\n        session=Session.builder.getOrCreate(),\n        size=10,\n    )\n)\ndef test_my_function(df: DataFrame):\n    # Test your function here\n    ...\n```\n\n## Development\n\n### Set up a development environment\n\nTo set up a development environment, follow the steps below:\n\n1. Create a virtual environment using **venv** or **conda**. Replace \\\u003cenv-name\\\u003e with the name of your environment.\n\n    Using **venv**:\n\n    ```shell\n    python3.11 -m venv \u003cenv-name\u003e\n    source \u003cenv-name\u003e/bin/activate\n    ```\n\n    Using **conda**:\n\n    ```shell\n    conda create -n \u003cenv-name\u003e python=3.11\n    conda activate \u003cenv-name\u003e\n    ```\n\n2. Configure your IDE to use the previously created virtual environment:\n\n    * [Configuring a Python interpreter in PyCharm](https://www.jetbrains.com/help/pycharm/configuring-python-interpreter.html)\n    * [Configuring a Python interpreter in VS Code](https://code.visualstudio.com/docs/python/environments#_manually-specify-an-interpreter)\n\n3. Install the project dependencies:\n\n    ```shell\n    pip install hatch\n    pip install -e .\n    ```\n\n### Running Tests\n\nTo run tests, run the following command.\n\n```shell\nhatch run test:check\n```\n\n---\n\n# snowpark-checkpoints-configuration\n\n\n**snowpark-checkpoints-configuration** is a module for loading `checkpoint.json` and provides a model. \nThis module will work automatically with *snowpark-checkpoints-collector*  and *snowpark-checkpoints-validators*. This will try to read the configuration file from the current working directory.\n\n## Usage\n\nTo explicit load a file, you can import  `CheckpointMetadata` and create an instance as shown below:\n\n```python\nfrom snowflake.snowpark_checkpoints_configuration import CheckpointMetadata\n\nmy_checkpoint_metadata = CheckpointMetadata(\"path/to/checkpoint.json\")\n\ncheckpoint_model = my_checkpoint_metadata.get_checkpoint(\"my_checkpoint_name\")\n...\n```\n---\n\n## Run Demos\n\n### Requirements\n\n- Python \u003e= 3.9\n- OpenJDK 21.0.2\n- Snow CLI: The default connection needs to have a database and a schema. After running the app, a table called SNOWPARK_CHECKPOINTS_REPORT will be created.\n\n### Steps\n\n1. Create a Python environment with Python 3.9 or higher in the Demos dir.\n2. Build the Python snowpark-checkpoints and snowpark-checkpoints-collector packages. Learn more.\n\n```cmd\ncd package_dir\npip install -e .\npython3 -m pip install --upgrade build\npython3 -m build\n```\n\n3. In Demos dir, run:\n   pip install \"snowpark-checkpoints\"\n4. First, run the PySpark demo:\n   python demo_pyspark_pipeline.py\n   This will generate the JSON schema files. Then, run the Snowpark demo:\n   python demo_snowpark_pipeline.py\n\n## Contributing\nPlease refer to [CONTRIBUTING.md][contributing].\n\n------\n\n[source code]: https://github.com/snowflakedb/snowpark-checkpoints\n[Snowpark Checkpoints Developer guide]: https://docs.snowflake.com/en/developer-guide/snowpark/python/snowpark-checkpoints-library\n[Snowpark Checkpoints API references]: https://docs.snowflake.com/en/developer-guide/snowpark-checkpoints-api/reference/latest/index\n[contributing]: https://github.com/snowflakedb/snowpark-checkpoints/blob/main/CONTRIBUTING.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflakedb%2Fsnowpark-checkpoints","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowflakedb%2Fsnowpark-checkpoints","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowflakedb%2Fsnowpark-checkpoints/lists"}