{"id":14982292,"url":"https://github.com/ccao-data/service-spark-iasworld","last_synced_at":"2026-02-26T12:05:07.024Z","repository":{"id":252453305,"uuid":"840483273","full_name":"ccao-data/service-spark-iasworld","owner":"ccao-data","description":"Service for extracting tables from the CCAO system-of-record and uploading them to the Data Department's data warehouse","archived":false,"fork":false,"pushed_at":"2025-12-18T23:41:36.000Z","size":270,"stargazers_count":0,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-21T21:26:38.301Z","etag":null,"topics":["etl","iasworld","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ccao-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-08-09T20:15:08.000Z","updated_at":"2025-12-18T23:41:39.000Z","dependencies_parsed_at":"2024-08-19T22:49:28.767Z","dependency_job_id":"57999e5d-ab4e-42ba-b807-eb71829da164","html_url":"https://github.com/ccao-data/service-spark-iasworld","commit_stats":{"total_commits":168,"total_committers":3,"mean_commits":56.0,"dds":"0.36309523809523814","last_synced_commit":"bc430e995a0476480fe81bdcec2e054c58a6ac6f"},"previous_names":["ccao-data/service-spark-iasworld"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ccao-data/service-spark-iasworld","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Fservice-spark-iasworld","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Fservice-spark-iasworld/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Fservice-spark-iasworld/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Fservice-spark-iasworld/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ccao-data","download_url":"https://codeload.github.com/ccao-data/service-spark-iasworld/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ccao-data%2Fservice-spark-iasworld/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29858461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-26T08:51:08.701Z","status":"ssl_error","status_checked_at":"2026-02-26T08:50:19.607Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","iasworld","spark"],"created_at":"2024-09-24T14:05:05.160Z","updated_at":"2026-02-26T12:05:07.018Z","avatar_url":"https://github.com/ccao-data.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spark Extractor for iasWorld\n\nThis repository contains the dependencies and code necessary to run\n[Spark](https://spark.apache.org/docs/latest/) extract jobs targeting Cook\nCounty's iasWorld property system-of-record. It is a replacement for\n[`service-sqoop-iasworld`](https://github.com/ccao-data/service-sqoop-iasworld),\nwhich is now deprecated.\n\nEach Spark job pulls an iasWorld table (or part of a table) via\n[JDBC](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html) and\nwrites it as [Hive-partitioned](https://duckdb.org/docs/data/partitioning/hive_partitioning.html)\nParquet files to [AWS S3](https://aws.amazon.com/s3/). The Data Department then\nqueries the Parquet files using [AWS Athena](https://aws.amazon.com/athena),\nproviding a 1-1 mirror of the system-of-record for analytical queries.\n\nJobs are submitted in \"batches\" (called applications by Spark). Each batch may\ncontain multiple extract jobs. Once all jobs for a batch are complete, we also\n(optionally) trigger four additional processes. In order:\n\n1. Upload the extracted Parquet files to AWS S3. Uploads to the bucket and\n   prefix specified in the `SPARK_ENV` secrets file.\n2. Run an AWS Glue crawler to update table data types and/or partitions in\n   the Glue data catalog (which powers Athena). This process only occurs if\n   _new_ files are uploaded i.e. those not previously seen on S3.\n3. Run a [dbt testing workflow](https://github.com/ccao-data/data-architecture/blob/master/.github/workflows/test_dbt_models.yaml)\n   on GitHub Actions. This automatically tests the iasWorld data for issues and\n   outputs the results to various tables and reports.\n4. Upload the final logs to AWS CloudWatch.\n\n## Submitting job batches\n\n\u003e [!NOTE]\n\u003e Before attempting to submit batches to the cluster, first make sure the\n\u003e production Spark Docker Compose stack is active by running\n\u003e `docker compose --profile prod up -d` in the repository. Also, make sure all\n\u003e secret and driver files are populated, see\n\u003e [Files not included](#files-not-included) for more information.\n\n`service-spark-iasworld` job batches are submitted via JSON, either as a string\nor as a file. All batches should have the format below. Note that the name\nof the job itself (e.g. `job2` below) is arbitrary.\n\n```json\n{\n  \"addn\": {\n    \"table_name\": \"iasworld.addn\",\n    \"min_year\": 2020,\n    \"max_year\": 2024,\n    \"cur\": [\"Y\", \"D\"],\n    \"predicates_path\": \"default_predicates.sql\"\n  },\n  \"job2\": {\n    \"table_name\": \"iasworld.asmt_all\",\n    \"min_year\": 2021,\n    \"max_year\": 2021,\n    \"cur\": [\"Y\"],\n    \"predicates_path\": \"default_predicates.sql\"\n  }\n}\n```\n\n### Field definitions\n\n- `table_name (required)` - Name of the iasWorld table to extract, must be\n  prefixed with `iasworld.` (or `ias.` for test environment).\n- `min_year (optional)` - Minimum tax year (inclusive) to extract from the\n  table. Set to `null` in a job definition to ignore this column when filtering\n  and partitioning. Defaults to `1999`.\n- `max_year (optional)` - Maximum tax year (inclusive) to extract from the\n  table. To extract a single year, set `min_year` and `max_year` to the same\n  value. Set to `null` in a job definition to ignore this column when filtering\n  and partitioning. Defaults to the current year.\n- `cur (optional)` - Values of the `cur` column to extract from the table.\n  Must be an array. Set to `null` in a job definition to ignore this column\n  when filtering and partitioning. Defaults to `[\"Y\", \"N\", \"D\"]`.\n- `predicates_path (optional)` - String path to a SQL file within the `config/`\n  directory. The SQL file should define SQL BETWEEN expressions, where each\n  expression is one chunk that will be extracted by Spark during JDBC reads.\n  Expressions should not be overlapping. Set to `null` in a job definition\n  to disable using predicates completely. Defaults to `default_predicates.sql`.\n\n### Creating batch JSON\n\nThe example batch above contains two separate jobs, one per table. If you want\nto add additional tables/jobs to the batch, you can manually add the\ncorresponding table objects and modify the fields as\n[listed above](#field-definitions).\n\nIn practice, modifying JSON is a bit of a pain, so we store long-lived\nbatch and job definitions in YAML.\nThe directory `config/default_jobs/` contains definitions for four common job\nbatches:\n\n1. `weekday_jobs.yaml`: A daily batch that pulls the most recent 2 years of\n   each critical table.\n2. `weekend_jobs.yaml`: A weekend batch that pulls _all_ tables and years.\n3. `weekend_jobs_test.yaml`: A weekend batch that pulls all tables and years\n   from the test iasWorld instance. We only run this batch as needed, not on\n   a schedule, because it is resource-intensive and we rarely need its output.\n4. `test_jobs.yaml`: A test batch that pulls a subset of tables with\n   representative situations that are useful for testing our Spark code.\n   We only run this batch as needed, not on a schedule.\n\n### Submitting via the command line\n\nBatches are submitted to the Spark Docker cluster via the command line. The\nmain job submission argument is either `--json-string` or `--yaml-file`.\nFor example, to submit a job to extract all data in the `comdat` table\nstarting from 2023, run the following command using `--json-string`:\n\n```bash\ndocker exec spark-node-master-prod ./submit.sh \\\n    --json-string '{\"comdat\": {\"table_name\": \"iasworld.comdat\", \"min_year\": \"2023\"}}'\n```\n\nOr, run the weekday jobs using `--yaml-file`:\n\n```bash\ndocker exec spark-node-master-prod ./submit.sh \\\n    --yaml-file default_jobs/weekday_jobs.yaml\n```\n\nThe command line interface also has multiple optional flags:\n\n- `--extract-target` - iasWorld target environment to extract data from. Must\n  be one of `prod` or `test`. Defaults to `prod`.\n- `--run-github-workflow/--no-run-github-workflow` - Run the [`test_dbt_models`](https://github.com/ccao-data/data-architecture/blob/master/.github/workflows/test_dbt_models.yaml)\n  workflow on batch completion?\n- `--run-glue-crawler/--no-run-glue-crawler` - Run the iasWorld Glue crawler\n  on batch completion?\n- `--upload-data/--no-upload-data` - Upload extracted data to the iasWorld S3\n  bucket?\n- `--upload-logs/--no-upload-logs` - Upload batch logs to AWS CloudWatch?\n\nThe default values for these flags are set in the `config/default_settings.yaml`\nfile. The boolean flags are all `False` by default.\n\n## Additional notes\n\n### Data types\n\nSpark automatically attempts to mirror the data types within iasWorld using\nits own equivalent types. However, on occasion, it may use an incorrect or\nundesirable type. In such cases, this repository provides a hierarchical system\nof column-level schema/type overrides, with each type overriding the previous\none:\n\n1. By default, all `NUMBER` Oracle types are converted to `DECIMAL(10,0)`\n   and `TIMESTAMP` Oracle types are converted to `STRING`. This behavior is\n   ignored for columns with an override specified via the options below.\n2. Global schema overrides apply to all columns of a given name across all\n   tables. They can be specified even for columns that do not exist in every\n   table. They are defined in `config/default_settings.yaml`.\n3. Table schema overrides apply only to the columns of a single table. They\n   take precedence over all other overrides. They are defined in\n   `config/table_definitions.yaml`.\n\n\u003e [!WARNING]\n\u003e `NUMERIC` types are implicitly converted to `DECIMAL(10,0)` because as of\n\u003e 2024, all `NUMERIC` columns without a specified precision and scale are\n\u003e actually just integers. If this changes in the future, it's possible that\n\u003e we could begin to silently truncate numbers via this implicit type\n\u003e conversion. As such, stay on top of schema updates from the iasWorld team.\n\n### Constructing jobs\n\nPredicates, filters, and partitions are Spark concepts used to construct\nindividual jobs in a batch. They are mostly handled automatically, but you\nmay need to change them in rare cases. The list below outlines the role of each\nconcept and how to change them if needed:\n\n- **Predicates** are SQL statements used to chunk a table during reads\n  against the iasWorld database. The statements define mutually exclusive\n  queries that run in parallel (in order to speed up query execution).\n  Predicates are defined via a file of SQL statements in the `config/`\n  directory, then passed to each table job via a file path.\n- **Filters** are logic conditions included in queries to the database. Spark\n  uses [predicate pushdown](https://airbyte.com/data-engineering-resources/predicate-pushdown)\n  to compose the _predicates_ and _filter_ for each query into a single SQL\n  statement. Think of filters as a SQL WHERE clause applied across all the\n  predicate chunks specified above.\n  Filters are constructed automatically from any `min_year`, `max_year`,\n  and/or `cur` values passed as part of a job definition. If these values are\n  all null, then the entire table is returned.\n- **Partitions** define how the output Parquet files returned from each should\n  be broken up. We use Hive partitioning by default, which yields partitions\n  with the structure `$TABLE/taxyr=$YEAR/cur=$CUR_VALUE/part-0.parquet`.\n  Like filters, partitions are determined automatically via any `min_year`,\n  `max_year`, and/or `cur` values that are set. If these values are all null,\n  then the table is returned as a single file e.g. `$TABLE/part-0.parquet`.\n\n### Files not included\n\nSome necessary setup and credential files are not included in this repository\nfor security or licensing reasons. Templated versions are included for\ninstructional purposes. If you want to use this repository, you will need to\npopulate the following:\n\n- `drivers/ojdbc8.jar` - This is the JDBC driver for our Oracle backend and\n  can be found for free on [Oracle's site](https://www.oracle.com/ca-en/database/technologies/appdev/jdbc-downloads.html).\n- `secrets/` - These are credential files needed to connect to other systems.\n\n## Using the development environment\n\nThe Docker Compose stack we use to run Spark (via `docker compose up -d`)\nhas a separate development environment that can be used to test code changes\nwithout disrupting the production containers. The development environment is\nconfigured via a dedicated\n[Compose profile](https://docs.docker.com/compose/how-tos/profiles/) and is\nused by default (with no profile specified).\n\nTo start the development environment, run:\n\n```bash\n# Make sure to export your UID, or else Spark may run into permission problems\n# when trying to write to files in your local directory\nexport UID\n# Start the development services\ndocker compose up -d\n# OR pass in the --profiles flag to explicitly start development services (this\n# is equivalent to the command above)\ndocker compose --profile dev up -d\n```\n\nTo start the production environment:\n\n```bash\ndocker compose --profile prod up -d\n```\n\nTo submit a job to the development environment, change the container target\nof your command from `prod` to `dev`. For example:\n\n```bash\ndocker exec -it spark-node-master-dev ./submit.sh \\\n    --yaml-file default_jobs/weekday_jobs.yaml\n```\n\nA typical development workflow might look something like:\n\n1. Clone the repository to your own machine or home directory. Do _not_ use\n  the production `shiny-server` copy of the repository for development.\n2. Copy the secrets and drivers from the production setup to the development\n  repository. See [Files not included](#files-not-included). A simple example\n  would be: `cp $PROD_PROJECT/secrets/* $DEV_PROJECT/secrets/` followed by\n  `cp $PROD_PROJECT/drivers/* $DEV_PROJECT/drivers/`.\n3. Start the development environment using `docker compose up -d`.\n4. Make your code modifications. Changes in the `src/` directory are reflected\n  in the containers due to volume mounts (no need to rebuild).\n5. Submit a job to the development containers using `docker exec`, targeting\n  the development master node (`spark-node-master-dev`).\n6. Check the job status at `$SERVER_IP:8082`, instead of the production port\n  `$SERVER_IP:8080`.\n\n\u003e [!WARNING]\n\u003e The development environment shares the same targets as the production\n\u003e environment. That means it will write to the same S3 bucket/CloudWatch log\n\u003e group and trigger the same workflows/crawlers (though all these features are\n\u003e disabled by default). As such, use this environment carefully. If you mess up\n\u003e production data, you can run the production version of the code to re-fetch it.\n\n## Scheduling\n\nBatches are currently scheduled via\n[`cron`](https://man7.org/linux/man-pages/man8/cron.8.html). To edit the\nschedule file, use `crontab -e` as the main server user. The example crontab\nfile below schedules daily jobs for frequently updated tables and weekly ones\nfor rarely-updated tables.\n\n```bash\n# Extract recent years from frequently used tables on weekdays at 4 AM CST\n0 9 * * 1,2,3,4,5 docker exec spark-node-master-prod ./submit.sh --run-github-workflow --run-glue-crawler --upload-data --upload-logs --yaml-file default_jobs/weekday_jobs.yaml\n\n# Extract all tables on Saturday at 1 AM CST\n0 6 * * 6 docker exec spark-node-master-prod ./submit.sh --run-github-workflow --run-glue-crawler --upload-data --upload-logs --yaml-file default_jobs/weekend_jobs.yaml\n```\n\n## Structure\n\nHere's a breakdown of important files and the purpose of each one:\n\n```bash\n.\n├── docker-compose.yaml        # Defines the Spark nodes, environment, and networking\n├── Dockerfile                 # Defines dependencies bundled in each Spark node\n├── pyproject.toml             # Project metadata and tool settings\n├── README.md                  # This file!\n├── run.sh                     # Entrypoint shell script to create Spark jobs\n├── .github/                   # GitHub Actions workflows for linting, builds, etc.\n├── config/\n│   ├── default_jobs/          # Config files defining batches of common Spark jobs\n│   ├── default_predicates.sql # List of mutually exclusive SQL BETWEEN expressions\n│   ├── default_settings.yaml  # Runtime defaults and schema overrides\n│   ├── spark-defaults.conf    # Spark memory and driver settings\n│   └── table_definitions.yaml # Possible job values per table and schema overrides\n├── drivers/\n│   └── ojdbc8.jar             # Not included, but necessary to connect to iasWorld\n├── secrets/\n│   ├── AWS_CREDENTIALS_FILE   # AWS credentials config file specific to this job\n│   ├── GH_PEM                 # GitHub PEM file used to authorize workflow dispatch\n│   └── IPTS_PASSWORD          # Password file loaded at runtime into containers\n├── src/\n│   ├── submit_jobs.py         # Job submission entrypoint. Takes JSON as input\n│   ├── submit.sh              # Helper to launch jobs using spark-submit\n│   └── utils/\n│       ├── aws.py             # AWS client class for triggering Glue crawlers\n│       ├── github.py          # GitHub client class for running Actions workflows\n│       ├── helpers.py         # Miscellaneous helper functions\n│       └── spark.py           # Spark job and session classes\n└── target/\n    ├── final/                 # Landing directory after Parquet repartitioning\n    └── initial/               # Landing directory for initial JDBC read output\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccao-data%2Fservice-spark-iasworld","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fccao-data%2Fservice-spark-iasworld","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fccao-data%2Fservice-spark-iasworld/lists"}