{"id":50791781,"url":"https://github.com/thoven87/frames","last_synced_at":"2026-06-12T11:32:06.959Z","repository":{"id":331698064,"uuid":"1125548343","full_name":"thoven87/frames","owner":"thoven87","description":"A DataFrame library inspired by Scio and Polars","archived":false,"fork":false,"pushed_at":"2026-01-10T20:50:19.000Z","size":12,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-11T06:18:07.113Z","etag":null,"topics":["datapipelines","etl","etl-pipeline","swift-etl","swift-on-server"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thoven87.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-30T23:38:30.000Z","updated_at":"2026-01-10T20:50:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/thoven87/frames","commit_stats":null,"previous_names":["thoven87/frames"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/thoven87/frames","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoven87%2Fframes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoven87%2Fframes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoven87%2Fframes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoven87%2Fframes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thoven87","download_url":"https://codeload.github.com/thoven87/frames/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thoven87%2Fframes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34243051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datapipelines","etl","etl-pipeline","swift-etl","swift-on-server"],"created_at":"2026-06-12T11:32:06.852Z","updated_at":"2026-06-12T11:32:06.942Z","avatar_url":"https://github.com/thoven87.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Frames\n\nA Swift library for efficient data manipulation similar to Pandas/Polars, built for Swift 6.2. Frames is designed for server-side use with a focus on memory efficiency, streaming capabilities, and **compile-time type safety**.\n\n## Features\n\n## Installation\n\nAdd Frames to your `Package.swift` dependencies:\n\n```swift\ndependencies: [\n    .package(url: \"https://github.com/thoven87/frames.git\", from: \"0.2.0\")\n]\n```\n\n## Quick Start\n\n### Type-Safe API\n\nFrames provides a compile-time type-safe API inspired by Scio's SCollection pattern. All operations work directly with your Codable structs:\n\n```swift\nimport Frames\n\n// Define your data structure\nstruct User: Codable, Sendable {\n    let id: Int\n    let name: String\n    let age: Int\n    let email: String\n}\n\n// Read from CSV with type safety\nlet users = DataFrame\u003cUser\u003e(csvURL: fileURL)\n\n// All operations are type-safe - no string column names!\nlet adults = users\n    .filter { $0.age \u003e= 18 }  // Type-safe filter\n    .map { ($0.id, $0.name) }  // Type-safe projection\n    \nlet result = try await adults.collect()\n```\n\n**Benefits:**\n- **Compile-Time Safety**: All operations validated at compile time\n- **No String Column Names**: Work directly with strongly-typed Swift structs\n- **Better IDE Support**: Full autocomplete and refactoring support\n- **Streaming Native**: Built for efficient streaming and lazy evaluation\n\n### Reading Data\n\n#### From CSV\n\n```swift\nstruct Product: Codable, Sendable {\n    let id: Int\n    let name: String\n    let price: Double\n}\n\n// Read entire CSV into DataFrame (local file)\nlet products = DataFrame\u003cProduct\u003e(csvURL: productsFile)\nlet allProducts = try await products.collect()\n\n// Stream from HTTP URL (cloud storage: GCS, S3, Azure)\nlet httpURL = URL(string: \"https://storage.googleapis.com/my-bucket/products.csv\")!\nlet remoteProducts = DataFrame\u003cProduct\u003e(csvURL: httpURL)\nfor try await product in remoteProducts.stream() {\n    print(\"\\(product.name): $\\(product.price)\")\n}\n\n// Or stream rows one at a time (memory-efficient)\nlet reader = CSVReader(url: productsFile)\nfor try await product in reader.rows(as: Product.self) {\n    print(\"\\(product.name): $\\(product.price)\")\n}\n```\n\n#### From JSON\n\n```swift\n// Read JSON array from local file\nlet users = DataFrame\u003cUser\u003e(jsonURL: usersFile)\n\n// Read JSON from HTTP URL\nlet jsonURL = URL(string: \"https://api.example.com/users.json\")!\nlet remoteUsers = DataFrame\u003cUser\u003e(jsonURL: jsonURL)\n\n// Read JSONL (newline-delimited JSON) from HTTP URL\nlet logsURL = URL(string: \"https://storage.googleapis.com/logs/app.jsonl\")!\nlet logs = DataFrame\u003cLogEntry\u003e(jsonlURL: logsURL)\n\n// Stream JSONL for large files\nlet reader = JSONLReader(url: largeFile)\nfor try await entry in reader.rows() as JSONLRowSequence\u003cLogEntry\u003e {\n    processEntry(entry)\n}\n```\n\n#### From Text Files\n\n```swift\n// Read plain text file (lines become DataFrame elements)\nlet lines = DataFrame\u003cString\u003e(textURL: textFile)\nfor try await line in lines.stream() {\n    print(line)\n}\n\n// Read from path or Hugging Face URL\nlet logs = DataFrame\u003cString\u003e(textPath: \"/var/log/app.log\")\n\n// Stream large text files line by line\nlet reader = TextFileReader(url: largeFile)\nfor try await line in reader.lines() {\n    processLine(line)\n}\n\n// Filter and process text lines\nlet errorLines = try await lines\n    .filter { $0.contains(\"ERROR\") }\n    .collect()\n```\n\n#### From Arrays\n\n```swift\nlet users = [\n    User(id: 1, name: \"Alice\", age: 30, email: \"alice@example.com\"),\n    User(id: 2, name: \"Bob\", age: 25, email: \"bob@example.com\")\n]\n\nlet df = DataFrame(elements: users)\n```\n\n### Reading Multiple Files\n\nFrames supports reading multiple files using glob patterns, making it easy to process datasets split across multiple files:\n\n#### Multiple CSV Files\n\n```swift\nstruct Sale: Codable, Sendable {\n    let date: String\n    let product: String\n    let amount: Double\n}\n\n// Read all CSV files matching the pattern\n// Automatically concatenates: sales_2023.csv, sales_2024.csv, etc.\nlet sales = DataFrame\u003cSale\u003e(csvPath: \"/path/to/sales_*.csv\")\n\nlet totalRevenue = try await sales\n    .reduce(0.0) { total, sale in total + sale.amount }\n\nprint(\"Total revenue: $\\(totalRevenue)\")\n```\n\n#### Multiple JSON Files\n\n```swift\nstruct User: Codable, Sendable {\n    let id: Int\n    let name: String\n    let active: Bool\n}\n\n// Read all JSON files in directory\nlet users = DataFrame\u003cUser\u003e(jsonPath: \"/path/to/users_*.json\")\n\nlet activeCount = try await users\n    .filter { $0.active }\n    .count()\n```\n\n#### Multiple JSONL Files\n\n```swift\nstruct LogEntry: Codable, Sendable {\n    let timestamp: String\n    let level: String\n    let message: String\n}\n\n// Read all log files\nlet logs = DataFrame\u003cLogEntry\u003e(jsonlPath: \"/var/logs/app_*.jsonl\")\n\n// Count errors across all files\nlet errors = try await logs\n    .filter { $0.level == \"ERROR\" }\n    .count()\n```\n\n#### Multiple Text Files\n\n```swift\n// Read all text files matching the pattern\nlet allLogs = DataFrame\u003cString\u003e(textPath: \"/var/logs/*.log\")\n\n// Stream through all log lines\nfor try await line in allLogs.stream() {\n    if line.contains(\"ERROR\") {\n        print(line)\n    }\n}\n\n// Or use direct reader API\nlet stream = try TextFileReader.readMultiple(pattern: \"/data/*.txt\")\nfor try await line in stream {\n    process(line)\n}\n```\n\n**Glob Pattern Support:**\n- `*` - Matches any sequence of characters\n- `?` - Matches any single character\n- Files are automatically sorted and concatenated in alphabetical order\n- Works with CSV, JSON, JSONL, and Text formats\n\n**Memory-Efficient Streaming:**\n\nWhen using glob patterns, Frames now streams files one at a time instead of loading all files into memory. This is especially important for large datasets:\n\n```swift\n// Efficiently process large datasets split across many files\nlet largeDataset = DataFrame\u003cDataPoint\u003e(csvPath: \"/data/measurements_*.csv\")\n\n// Stream through all files without loading everything into memory\nvar total = 0.0\nvar count = 0\nfor try await point in largeDataset.stream() {\n    total += point.value\n    count += 1\n    // Only one file's worth of data is in memory at a time\n}\nlet average = total / Double(count)\n```\n\n**Streaming Behavior:**\n- Files are processed sequentially in alphabetical order\n- Only one file is held in memory at a time\n- Ideal for processing datasets larger than available RAM\n- Works seamlessly with filter, map, and other operations\n- Backward compatible with existing code\n\nFor direct streaming API access:\n```swift\n// Stream CSV files directly\nlet csvStream = try CSVReader.readMultiple(pattern: \"/data/*.csv\", hasHeader: true)\nfor try await record in csvStream as MultiCSVSequence\u003cRecord\u003e {\n    process(record)\n}\n\n// Stream JSONL files directly\nlet jsonlStream = try JSONLReader.readMultiple(pattern: \"/logs/*.jsonl\")\nfor try await entry in jsonlStream as MultiJSONLSequence\u003cLogEntry\u003e {\n    process(entry)\n}\n```\n\n### Hugging Face Datasets\n\nFrames supports reading datasets directly from Hugging Face using the `hf://` URL scheme:\n\n#### CSV from Hugging Face\n\n```swift\nstruct IrisData: Codable, Sendable {\n    let sepal_length: Double\n    let sepal_width: Double\n    let petal_length: Double\n    let petal_width: Double\n    let species: String\n}\n\n// Read directly from Hugging Face\nlet iris = DataFrame\u003cIrisData\u003e(\n    csvPath: \"hf://datasets/nameexhaustion/polars-docs/iris.csv\"\n)\n\nlet results = try await iris.collect()\nprint(\"Loaded \\(results.count) samples\")\n\n// Process with type-safe operations\nlet bySpecies = try await iris\n    .groupBy { $0.species }\n    .count()\n    .collect()\n```\n\n#### JSONL from Hugging Face\n\n```swift\n// Read JSONL from Hugging Face\nlet data = DataFrame\u003cYourType\u003e(\n    jsonlPath: \"hf://datasets/username/repo/data.jsonl\"\n)\n```\n\n#### With Specific Revision/Branch\n\n```swift\n// Specify a branch or revision with @\nlet data = DataFrame\u003cDataPoint\u003e(\n    csvPath: \"hf://datasets/username/repo@branch-name/file.csv\"\n)\n```\n\n**Hugging Face URL Format:**\n- `hf://BUCKET/REPOSITORY/PATH` - Default main branch\n- `hf://BUCKET/REPOSITORY@REVISION/PATH` - Specific branch/revision\n- **BUCKET**: `datasets` or `spaces`\n- **REPOSITORY**: `username/repo_name`\n- **REVISION**: Branch name or commit hash (defaults to `main`)\n- **PATH**: File path within the repository\n\n**Examples:**\n- `hf://datasets/nameexhaustion/polars-docs/iris.csv`\n- `hf://datasets/nameexhaustion/polars-docs@foods/data.csv`\n- `hf://spaces/nameexhaustion/polars-docs/orders.feather`\n\n### Type-Safe Joins\n\nJoins are fully type-safe with compile-time key validation:\n\n```swift\nstruct User: Codable, Sendable {\n    let id: Int\n    let name: String\n}\n\nstruct Order: Codable, Sendable {\n    let orderId: Int\n    let userId: Int\n    let amount: Double\n}\n\nlet users = DataFrame\u003cUser\u003e(csvURL: usersFile)\nlet orders = DataFrame\u003cOrder\u003e(csvURL: ordersFile)\n\n// Scio-style: First key both DataFrames, then join\nlet userKeyed = users.keyBy { $0.id }\nlet orderKeyed = orders.keyBy { $0.userId }\n\n// Inner join (only matching keys)\nlet joined = userKeyed.join(orderKeyed)\n\nfor try await item in joined.stream() {\n    print(\"User: \\(item.left.name), Amount: $\\(item.right.amount)\")\n}\n\n// Left join (all left elements preserved)\nlet leftJoin = userKeyed.leftJoin(orderKeyed)\n\n// Right join (all right elements preserved)\nlet rightJoin = userKeyed.rightJoin(orderKeyed)\n\n// Full outer join (all elements from both sides)\nlet outerJoin = userKeyed.fullOuterJoin(orderKeyed)\n```\n\n### Type-Safe GroupBy\n\nGroupBy operations work on typed elements:\n\n```swift\nstruct Sale: Codable, Sendable {\n    let category: String\n    let amount: Double\n    let quantity: Int\n}\n\nlet sales = DataFrame\u003cSale\u003e(csvURL: salesFile)\n\n// Group by category and count\nlet counts = try await sales\n    .groupBy { $0.category }\n    .count()\n    .collect()\n\n// Group by category and sum amounts\nlet totals = try await sales\n    .groupBy { $0.category }\n    .reduce(0.0) { acc, sale in acc + sale.amount }\n    .collect()\n\nfor total in totals {\n    print(\"Category \\(total.key): $\\(total.value)\")\n}\n\n// Map groups to custom values\nlet avgByCategory = try await sales\n    .groupBy { $0.category }\n    .mapGroups { group in\n        group.reduce(0.0) { $0 + $1.amount } / Double(group.count)\n    }\n    .collect()\n```\n\n### Window Functions\n\nApply window functions with partitioning and ordering:\n\n```swift\nstruct Sale: Codable, Sendable {\n    let category: String\n    let date: Int\n    let amount: Double\n}\n\nlet sales = DataFrame\u003cSale\u003e(csvURL: salesFile)\n\n// Add row numbers within each category, ordered by date\nlet numbered = try await sales\n    .window(partitionBy: { $0.category }, orderBy: { $0.date })\n    .rowNumber()\n    .collect()\n\n// Rank by amount (descending)\nlet ranked = try await sales\n    .window(orderBy: { -$0.amount })\n    .rank()\n    .collect()\n\n// Cumulative sum of amounts by category\nlet cumulative = try await sales\n    .window(partitionBy: { $0.category }, orderBy: { $0.date })\n    .cumSum { $0.amount }\n    .collect()\n\n// Moving average (3-period window)\nlet moving = try await sales\n    .window(orderBy: { $0.date })\n    .movingAverage(windowSize: 3) { $0.amount }\n    .collect()\n\n// Lag and lead values\nlet withLag = try await sales\n    .window(orderBy: { $0.date })\n    .lag(offset: 1) { $0.amount }\n    .collect()\n\nlet withLead = try await sales\n    .window(orderBy: { $0.date })\n    .lead(offset: 1) { $0.amount }\n    .collect()\n```\n\n**Available Window Functions:**\n- `rowNumber()` - Sequential row numbers within partition\n- `rank()` - Rank with gaps for ties\n- `denseRank()` - Rank without gaps for ties\n- `cumSum(_ getValue:)` - Cumulative sum\n- `movingAverage(windowSize:_:)` - Moving average over window\n- `lag(offset:_:)` - Previous row value\n- `lead(offset:_:)` - Next row value\n\n### Type-Safe Transformations\n\nChain type-safe transformations:\n\n```swift\nstruct Person: Codable, Sendable {\n    let name: String\n    let age: Int\n    let hobbies: [String]\n}\n\nlet people = DataFrame\u003cPerson\u003e(csvURL: peopleFile)\n\n// Extract all hobbies from all people\nlet allHobbies = try await people\n    .flatMap { $0.hobbies }\n    .collect()\n\n// Get names of adults\nlet adultNames = try await people\n    .filter { $0.age \u003e= 18 }\n    .map { $0.name }\n    .collect()\n\n// Take first 10 results\nlet first10 = try await people\n    .take(10)\n    .collect()\n\n// Skip first 5, then take next 10\nlet page2 = try await people\n    .skip(5)\n    .take(10)\n    .collect()\n\n// Sort by age\nlet sorted = try await people\n    .sort { $0.age \u003c $1.age }\n    .collect()\n```\n\n### Writing Data\n\n#### To CSV\n\n```swift\nlet users = DataFrame\u003cUser\u003e(elements: usersArray)\ntry await users.saveAsCSV(to: outputURL)\n```\n\n#### To JSON\n\n```swift\n// Save as JSON array\ntry await users.saveAsJSON(to: outputURL)\n\n// Save as pretty-printed JSON\ntry await users.saveAsJSON(to: outputURL, prettyPrinted: true)\n\n// Save as JSONL (one object per line)\ntry await users.saveAsJSONL(to: outputURL)\n```\n\n### Streaming for Large Files\n\nFor datasets that don't fit in memory, use streaming:\n\n```swift\n// Stream and process data in batches\nfor try await user in users.stream() {\n    await processUser(user)\n}\n\n// Reduce without materializing entire dataset\nlet totalAge = try await users.reduce(0) { acc, user in\n    acc + user.age\n}\n\n// forEach for side effects\ntry await users\n    .filter { $0.age \u003e= 18 }\n    .forEach { user in\n        await sendEmail(to: user.email)\n    }\n```\n\n### HTTP Streaming from Cloud Storage\n\nFrames supports streaming data directly from HTTP/HTTPS URLs, making it ideal for cloud environments:\n\n```swift\nstruct TaxiTrip: Codable, Sendable {\n    let tripId: Int\n    let pickupTime: String\n    let distance: Double\n    let fare: Double\n}\n\n// Stream from Google Cloud Storage (GCS)\nlet gcsURL = URL(string: \"https://storage.googleapis.com/my-bucket/taxi-data.csv\")!\nlet gcsTrips = DataFrame\u003cTaxiTrip\u003e(csvURL: gcsURL)\n\n// Stream from Amazon S3\nlet s3URL = URL(string: \"https://my-bucket.s3.us-east-1.amazonaws.com/taxi-data.csv\")!\nlet s3Trips = DataFrame\u003cTaxiTrip\u003e(csvURL: s3URL)\n\n// Stream from Azure Blob Storage\nlet azureURL = URL(string: \"https://myaccount.blob.core.windows.net/container/data.csv\")!\nlet azureTrips = DataFrame\u003cTaxiTrip\u003e(csvURL: azureURL)\n\n// Process with memory-efficient streaming\nvar totalFares = 0.0\nvar tripCount = 0\n\nfor try await trip in gcsTrips.stream() {\n    totalFares += trip.fare\n    tripCount += 1\n    \n    // Automatic backpressure handling\n    // Process in real-time without buffering entire dataset\n}\n\nprint(\"Average fare: $\\(totalFares / Double(tripCount))\")\n```\n\n**Benefits:**\n- ✅ No local storage required\n- ✅ Memory efficient for large datasets\n- ✅ Automatic backpressure handling\n- ✅ Works with presigned URLs\n- ✅ Perfect for serverless environments (Cloud Run, Lambda, etc.)\n- ✅ Supports all cloud providers (GCS, S3, Azure, etc.)\n\n## Advanced Examples\n\n### Complex Data Pipeline\n\n```swift\nstruct Transaction: Codable, Sendable {\n    let id: Int\n    let userId: Int\n    let amount: Double\n    let status: String\n    let category: String\n    let date: Date\n}\n\nlet transactions = DataFrame\u003cTransaction\u003e(csvURL: transactionsFile)\n\n// Find high-value completed transactions, grouped by category\nlet summary = try await transactions\n    .filter { $0.status == \"completed\" }\n    .filter { $0.amount \u003e 1000.0 }\n    .groupBy { $0.category }\n    .reduce(0.0) { acc, txn in acc + txn.amount }\n    .collect()\n\nfor item in summary {\n    print(\"Category \\(item.key): $\\(String(format: \"%.2f\", item.value))\")\n}\n```\n\n### Multi-Source Join\n\n```swift\nstruct Customer: Codable, Sendable {\n    let customerId: Int\n    let name: String\n    let email: String\n}\n\nstruct Order: Codable, Sendable {\n    let orderId: Int\n    let customerId: Int\n    let total: Double\n}\n\nlet customers = DataFrame\u003cCustomer\u003e(jsonURL: customersFile)\nlet orders = DataFrame\u003cOrder\u003e(csvURL: ordersFile)\n\nlet customerOrders = customers\n    .keyBy { $0.customerId }\n    .join(orders.keyBy { $0.customerId })\n\nlet highValueCustomers = try await customerOrders\n    .filter { $0.right.total \u003e 500.0 }\n    .map { (name: $0.left.name, email: $0.left.email, total: $0.right.total) }\n    .collect()\n```\n\n### ETL Pipeline with JSON and CSV\n\n```swift\n// Read from JSON\nlet rawData = DataFrame\u003cRawEvent\u003e(jsonlURL: inputFile)\n\n// Transform\nlet cleaned = rawData\n    .filter { $0.status == \"valid\" }\n    .map { CleanedEvent(from: $0) }\n\n// Write to CSV for analysis\ntry await cleaned.saveAsCSV(to: outputFile)\n```\n\n### Union of DataFrames\n\nCombine multiple DataFrames with the same element type:\n\n```swift\nstruct User: Codable, Sendable {\n    let id: Int\n    let name: String\n    let age: Int\n}\n\n// Load data from different sources\nlet users2023 = DataFrame\u003cUser\u003e(csvURL: file2023)\nlet users2024 = DataFrame\u003cUser\u003e(csvURL: file2024)\n\n// Combine both datasets\nlet allUsers = users2023.union(users2024)\n\n// Filter and analyze combined data\nlet adults = try await allUsers\n    .filter { $0.age \u003e= 18 }\n    .collect()\n\nprint(\"Total users: \\(adults.count)\")\n```\n\n### File Checksums for Data Integrity\n\nVerify file integrity using CRC32 checksums:\n\n```swift\nstruct Transaction: Codable, Sendable {\n    let id: Int\n    let amount: Double\n    let date: String\n}\n\n// Calculate checksum before processing\nlet csvFile = URL(fileURLWithPath: \"transactions.csv\")\nlet checksum = try DataFrame\u003cTransaction\u003e.checksum(fileURL: csvFile)\nprint(\"File checksum: 0x\\(String(format: \"%08X\", checksum))\")\n\n// Process the file\nlet transactions = DataFrame\u003cTransaction\u003e(csvURL: csvFile)\ntry await transactions\n    .filter { $0.amount \u003e 1000 }\n    .saveAsCSV(to: outputFile)\n\n// Verify output file\nlet outputChecksum = try DataFrame\u003cTransaction\u003e.checksum(fileURL: outputFile)\nprint(\"Output checksum: 0x\\(String(format: \"%08X\", outputChecksum))\")\n```\n\n**CRC32 Utility:**\n\nThe `CRC32` struct provides standalone checksum calculation:\n\n```swift\nimport Frames\n\n// From Data\nlet data = \"Hello, World!\".data(using: .utf8)!\nlet checksum1 = CRC32.checksum(data: data)\n\n// From String\nlet checksum2 = CRC32.checksum(string: \"Hello, World!\")\n\n// From Bytes\nlet bytes: [UInt8] = [72, 101, 108, 108, 111]\nlet checksum3 = CRC32.checksum(bytes: bytes)\n\n// From File\nlet fileChecksum = try CRC32.checksum(fileURL: fileURL)\n```\n\n### Processing Large Public Datasets\n\nFor a complete example of processing large CSV files from public datasets (NYC Taxi, Weather, COVID-19), see `Examples/LargeCSVProcessingExample.swift`.\n\nThis example demonstrates:\n- Memory-efficient streaming for large files\n- Data validation with checksums\n- Combining multiple datasets with union\n- Efficient filtering and aggregation\n- Writing processed results\n\n```swift\n// Example: Process large taxi trip data\nlet trips = DataFrame\u003cTaxiTrip\u003e(csvURL: largeTaxiFile)\n\n// Verify integrity\nlet checksum = try DataFrame\u003cTaxiTrip\u003e.checksum(fileURL: largeTaxiFile)\n\n// Stream and process without loading everything into memory\ntry await trips.forEach { trip in\n    await processTrip(trip)\n}\n\n// Combine multiple data sources\nlet station1 = DataFrame\u003cWeatherReading\u003e(csvURL: station1File)\nlet station2 = DataFrame\u003cWeatherReading\u003e(csvURL: station2File)\nlet allReadings = station1.union(station2)\n\n// Aggregate across combined datasets\nlet avgTemp = try await allReadings\n    .reduce(0.0) { acc, reading in acc + reading.temperature }\n```\n\n## API Reference\n\n### DataFrame\u003cElement\u003e\n\n```swift\npublic struct DataFrame\u003cElement: Codable \u0026 Sendable\u003e: Sendable\n```\n\n**Initialization:**\n- `init(elements: [Element])` - From array\n- `init(csvURL: URL, hasHeader: Bool = true, delimiter: Character = \",\")` - From single CSV file\n- `init(csvPath: String, hasHeader: Bool = true, delimiter: Character = \",\")` - From CSV file(s), glob pattern, or Hugging Face URL\n- `init(jsonURL: URL)` - From single JSON array file\n- `init(jsonPath: String)` - From JSON file(s), glob pattern, or Hugging Face URL\n- `init(jsonlURL: URL)` - From single JSONL file\n- `init(jsonlPath: String)` - From JSONL file(s), glob pattern, or Hugging Face URL\n- `init(textURL: URL)` - From single text file (where Element is String)\n- `init(textPath: String)` - From text file(s), glob pattern, or Hugging Face URL (where Element is String)\n\n**Transformations:**\n- `filter(_ predicate: (Element) -\u003e Bool) -\u003e DataFrame\u003cElement\u003e`\n- `map\u003cT\u003e(_ transform: (Element) -\u003e T) -\u003e DataFrame\u003cT\u003e`\n- `flatMap\u003cT\u003e(_ transform: (Element) -\u003e [T]) -\u003e DataFrame\u003cT\u003e`\n- `take(_ n: Int) -\u003e DataFrame\u003cElement\u003e`\n- `skip(_ n: Int) -\u003e DataFrame\u003cElement\u003e`\n- `sort(_ comparator: (Element, Element) -\u003e Bool) -\u003e DataFrame\u003cElement\u003e`\n- `union(_ other: DataFrame\u003cElement\u003e) -\u003e DataFrame\u003cElement\u003e` - Combine two DataFrames\n\n**Aggregations:**\n- `groupBy\u003cK\u003e(_ keyExtractor: (Element) -\u003e K) -\u003e GroupedDataFrame\u003cK, Element\u003e`\n- `keyBy\u003cK\u003e(_ keyExtractor: (Element) -\u003e K) -\u003e KeyedDataFrame\u003cK, Element\u003e`\n- `window\u003cPK, OK\u003e(partitionBy:orderBy:) -\u003e WindowSpec\u003c...\u003e`\n\n**Terminal Operations:**\n- `collect() async throws -\u003e [Element]`\n- `count() async throws -\u003e Int`\n- `reduce\u003cT\u003e(_ initialValue: T, _ combine: (T, Element) throws -\u003e T) async throws -\u003e T`\n- `forEach(_ action: (Element) async throws -\u003e Void) async throws`\n- `stream() -\u003e DataFrameStream\u003cElement\u003e`\n\n**I/O:**\n- `saveAsCSV(to url: URL) async throws`\n- `saveAsJSON(to url: URL, prettyPrinted: Bool = false) async throws`\n- `saveAsJSONL(to url: URL) async throws`\n- `saveAsText(to url: URL) async throws` - (for DataFrame\u003cString\u003e only)\n\n**Utilities:**\n- `static checksum(fileURL url: URL) throws -\u003e UInt32` - Calculate CRC32 checksum for a file\n\n### GroupedDataFrame\u003cKey, Element\u003e\n\n**Methods:**\n- `count() async throws -\u003e DataFrame\u003cGroupCount\u003cKey\u003e\u003e`\n- `reduce\u003cT\u003e(_ initialValue: T, _ combine: (T, Element) -\u003e T) async throws -\u003e DataFrame\u003cGroupValue\u003cKey, T\u003e\u003e`\n- `mapGroups\u003cT\u003e(_ transform: ([Element]) -\u003e T) async throws -\u003e DataFrame\u003cGroupValue\u003cKey, T\u003e\u003e`\n\n### KeyedDataFrame\u003cKey, Element\u003e\n\n**Methods:**\n- `join\u003cR\u003e(_ other: KeyedDataFrame\u003cKey, R\u003e) -\u003e DataFrame\u003cKeyedJoinResult\u003cKey, Element, R\u003e\u003e`\n- `leftJoin\u003cR\u003e(_ other: KeyedDataFrame\u003cKey, R\u003e) -\u003e DataFrame\u003cKeyedLeftJoinResult\u003cKey, Element, R\u003e\u003e`\n- `rightJoin\u003cR\u003e(_ other: KeyedDataFrame\u003cKey, R\u003e) -\u003e DataFrame\u003cKeyedRightJoinResult\u003cKey, Element, R\u003e\u003e`\n- `fullOuterJoin\u003cR\u003e(_ other: KeyedDataFrame\u003cKey, R\u003e) -\u003e DataFrame\u003cKeyedOuterJoinResult\u003cKey, Element, R\u003e\u003e`\n- `hashJoin\u003cR\u003e(_ other: KeyedDataFrame\u003cKey, R\u003e) -\u003e DataFrame\u003cKeyedJoinResult\u003cKey, Element, R\u003e\u003e` (alias for join)\n\n### WindowSpec\u003cElement, PartitionKey, OrderKey\u003e\n\n**Methods:**\n- `rowNumber() -\u003e DataFrame\u003cWindowedElement\u003cElement, Int\u003e\u003e`\n- `rank() -\u003e DataFrame\u003cWindowedElement\u003cElement, Int\u003e\u003e`\n- `denseRank() -\u003e DataFrame\u003cWindowedElement\u003cElement, Int\u003e\u003e`\n- `cumSum\u003cN: Numeric\u003e(_ getValue: (Element) -\u003e N) -\u003e DataFrame\u003cWindowedElement\u003cElement, Double\u003e\u003e`\n- `movingAverage\u003cN: Numeric\u003e(windowSize: Int, _ getValue: (Element) -\u003e N) -\u003e DataFrame\u003cWindowedElement\u003cElement, Double\u003e\u003e`\n- `lag\u003cV\u003e(offset: Int = 1, _ getValue: (Element) -\u003e V) -\u003e DataFrame\u003cWindowedElement\u003cElement, V?\u003e\u003e`\n- `lead\u003cV\u003e(offset: Int = 1, _ getValue: (Element) -\u003e V) -\u003e DataFrame\u003cWindowedElement\u003cElement, V?\u003e\u003e`\n\n### CSVReader\n\n```swift\npublic struct CSVReader: Sendable\n```\n\n**Initialization:**\n- `init(url: URL, hasHeader: Bool = true, delimiter: Character = \",\", bufferSize: Int = 8192)` - From URL\n- `init(path: String, hasHeader: Bool = true, delimiter: Character = \",\", bufferSize: Int = 8192)` - From path or Hugging Face URL\n\n**Methods:**\n- `rows() -\u003e CSVRowSequence` - Stream rows as string arrays\n- `rows\u003cT: Decodable\u003e(as type: T.Type) -\u003e CSVCodableSequence\u003cT\u003e` - Stream as Codable structs\n- `read\u003cT: Decodable \u0026 Sendable\u003e() async throws -\u003e [T]` - Read entire CSV into array\n- `static readMultiple\u003cT\u003e(pattern: String, hasHeader: Bool = true, delimiter: Character = \",\") async throws -\u003e [T]` - Read multiple CSV files matching glob pattern\n\n### JSONReader\n\n```swift\npublic struct JSONReader\n```\n\n**Initialization:**\n- `init(url: URL)` - From URL\n- `init(path: String)` - From path or Hugging Face URL\n\n**Methods:**\n- `read\u003cT: Decodable\u003e() async throws -\u003e [T]` - Read JSON array\n- `readDataFrame\u003cT: Codable \u0026 Sendable\u003e() async throws -\u003e DataFrame\u003cT\u003e` - Read into DataFrame\n- `static readMultiple\u003cT\u003e(pattern: String) async throws -\u003e [T]` - Read multiple JSON files matching glob pattern\n\n### JSONLReader\n\n```swift\npublic struct JSONLReader\n```\n\n**Initialization:**\n- `init(url: URL, bufferSize: Int = 8192)` - From URL\n- `init(path: String, bufferSize: Int = 8192)` - From path or Hugging Face URL\n\n**Methods:**\n- `rows\u003cT: Decodable\u003e() -\u003e JSONLRowSequence\u003cT\u003e` - Stream JSONL rows\n- `read\u003cT: Decodable\u003e() async throws -\u003e [T]` - Read all JSONL rows\n- `readDataFrame\u003cT: Codable \u0026 Sendable\u003e() async throws -\u003e DataFrame\u003cT\u003e` - Read into DataFrame\n- `static readMultiple\u003cT\u003e(pattern: String) async throws -\u003e [T]` - Read multiple JSONL files matching glob pattern\n\n### TextFileReader\n\n```swift\npublic struct TextFileReader: Sendable\n```\n\nA reader for plain text files that streams lines one at a time.\n\n**Initialization:**\n- `init(url: URL, bufferSize: Int = 8192)` - From URL\n- `init(path: String, bufferSize: Int = 8192)` - From path or Hugging Face URL\n\n**Methods:**\n- `lines() -\u003e TextLineSequence` - Stream text file lines\n- `read() async throws -\u003e [String]` - Read all lines into array\n- `readDataFrame() async throws -\u003e DataFrame\u003cString\u003e` - Read into DataFrame\n- `static readMultiple(pattern: String) throws -\u003e TextLineSequence` - Read multiple text files matching glob pattern\n\n### FileGlobbing\n\n```swift\npublic struct FileGlobbing\n```\n\nUtility for expanding glob patterns to file lists.\n\n**Methods:**\n- `static expandGlob(_ pattern: String) throws -\u003e [URL]` - Expand glob pattern (e.g., \"*.csv\") to list of matching file URLs\n\n### HuggingFaceURL\n\n```swift\npublic struct HuggingFaceURL\n```\n\nUtility for handling Hugging Face dataset URLs with `hf://` scheme.\n\n**Methods:**\n- `static parseHuggingFaceURL(_ urlString: String) throws -\u003e URL` - Parse `hf://` URL to standard HTTPS URL\n- `static isHuggingFaceURL(_ urlString: String) -\u003e Bool` - Check if string is a Hugging Face URL\n\n**URL Format:**\n- `hf://BUCKET/REPOSITORY/PATH` - Default main branch\n- `hf://BUCKET/REPOSITORY@REVISION/PATH` - Specific branch/revision\n- BUCKET: `datasets` or `spaces`\n- REPOSITORY: `username/repo_name`\n- REVISION: Branch name or commit (optional, defaults to `main`)\n\n### CRC32\n\n```swift\npublic struct CRC32\n```\n\n**Methods:**\n- `static checksum(data: Data) -\u003e UInt32` - Calculate CRC32 for Data\n- `static checksum(string: String, encoding: String.Encoding = .utf8) -\u003e UInt32?` - Calculate CRC32 for String\n- `static checksum(bytes: [UInt8]) -\u003e UInt32` - Calculate CRC32 for byte array\n- `static checksum(fileURL url: URL) throws -\u003e UInt32` - Calculate CRC32 for file\n\nThe CRC32 implementation uses the IEEE 802.3 polynomial (0xEDB88320) for standard CRC32 checksums. Useful for verifying data integrity of CSV, JSON, and other data files.\n\n## Requirements\n\n- Swift 6.2 or later\n- Linux or macOS\n\n## License\n\nMIT License - See LICENSE file for details\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoven87%2Fframes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthoven87%2Fframes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthoven87%2Fframes/lists"}