Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/thoven87/frames

A DataFrame library inspired by Scio and Polars
https://github.com/thoven87/frames

datapipelines etl etl-pipeline swift-etl swift-on-server

Last synced: about 1 month ago
JSON representation

A DataFrame library inspired by Scio and Polars

Awesome Lists containing this project

README 

          
# Frames



A Swift library for efficient data manipulation similar to Pandas/Polars, built for Swift 6.2. Frames is designed for server-side use with a focus on memory efficiency, streaming capabilities, and **compile-time type safety**.



## Features



## Installation



Add Frames to your `Package.swift` dependencies:



```swift

dependencies: [

    .package(url: "https://github.com/thoven87/frames.git", from: "0.2.0")

]

```



## Quick Start



### Type-Safe API



Frames provides a compile-time type-safe API inspired by Scio's SCollection pattern. All operations work directly with your Codable structs:



```swift

import Frames



// Define your data structure

struct User: Codable, Sendable {

    let id: Int

    let name: String

    let age: Int

    let email: String

}



// Read from CSV with type safety

let users = DataFrame(csvURL: fileURL)



// All operations are type-safe - no string column names!

let adults = users

    .filter { $0.age >= 18 }  // Type-safe filter

    .map { ($0.id, $0.name) }  // Type-safe projection

    

let result = try await adults.collect()

```



**Benefits:**

- **Compile-Time Safety**: All operations validated at compile time

- **No String Column Names**: Work directly with strongly-typed Swift structs

- **Better IDE Support**: Full autocomplete and refactoring support

- **Streaming Native**: Built for efficient streaming and lazy evaluation



### Reading Data



#### From CSV



```swift

struct Product: Codable, Sendable {

    let id: Int

    let name: String

    let price: Double

}



// Read entire CSV into DataFrame (local file)

let products = DataFrame(csvURL: productsFile)

let allProducts = try await products.collect()



// Stream from HTTP URL (cloud storage: GCS, S3, Azure)

let httpURL = URL(string: "https://storage.googleapis.com/my-bucket/products.csv")!

let remoteProducts = DataFrame(csvURL: httpURL)

for try await product in remoteProducts.stream() {

    print("\(product.name): $\(product.price)")

}



// Or stream rows one at a time (memory-efficient)

let reader = CSVReader(url: productsFile)

for try await product in reader.rows(as: Product.self) {

    print("\(product.name): $\(product.price)")

}

```



#### From JSON



```swift

// Read JSON array from local file

let users = DataFrame(jsonURL: usersFile)



// Read JSON from HTTP URL

let jsonURL = URL(string: "https://api.example.com/users.json")!

let remoteUsers = DataFrame(jsonURL: jsonURL)



// Read JSONL (newline-delimited JSON) from HTTP URL

let logsURL = URL(string: "https://storage.googleapis.com/logs/app.jsonl")!

let logs = DataFrame(jsonlURL: logsURL)



// Stream JSONL for large files

let reader = JSONLReader(url: largeFile)

for try await entry in reader.rows() as JSONLRowSequence {

    processEntry(entry)

}

```



#### From Text Files



```swift

// Read plain text file (lines become DataFrame elements)

let lines = DataFrame(textURL: textFile)

for try await line in lines.stream() {

    print(line)

}



// Read from path or Hugging Face URL

let logs = DataFrame(textPath: "/var/log/app.log")



// Stream large text files line by line

let reader = TextFileReader(url: largeFile)

for try await line in reader.lines() {

    processLine(line)

}



// Filter and process text lines

let errorLines = try await lines

    .filter { $0.contains("ERROR") }

    .collect()

```



#### From Arrays



```swift

let users = [

    User(id: 1, name: "Alice", age: 30, email: "alice@example.com"),

    User(id: 2, name: "Bob", age: 25, email: "bob@example.com")

]



let df = DataFrame(elements: users)

```



### Reading Multiple Files



Frames supports reading multiple files using glob patterns, making it easy to process datasets split across multiple files:



#### Multiple CSV Files



```swift

struct Sale: Codable, Sendable {

    let date: String

    let product: String

    let amount: Double

}



// Read all CSV files matching the pattern

// Automatically concatenates: sales_2023.csv, sales_2024.csv, etc.

let sales = DataFrame(csvPath: "/path/to/sales_*.csv")



let totalRevenue = try await sales

    .reduce(0.0) { total, sale in total + sale.amount }



print("Total revenue: $\(totalRevenue)")

```



#### Multiple JSON Files



```swift

struct User: Codable, Sendable {

    let id: Int

    let name: String

    let active: Bool

}



// Read all JSON files in directory

let users = DataFrame(jsonPath: "/path/to/users_*.json")



let activeCount = try await users

    .filter { $0.active }

    .count()

```



#### Multiple JSONL Files



```swift

struct LogEntry: Codable, Sendable {

    let timestamp: String

    let level: String

    let message: String

}



// Read all log files

let logs = DataFrame(jsonlPath: "/var/logs/app_*.jsonl")



// Count errors across all files

let errors = try await logs

    .filter { $0.level == "ERROR" }

    .count()

```



#### Multiple Text Files



```swift

// Read all text files matching the pattern

let allLogs = DataFrame(textPath: "/var/logs/*.log")



// Stream through all log lines

for try await line in allLogs.stream() {

    if line.contains("ERROR") {

        print(line)

    }

}



// Or use direct reader API

let stream = try TextFileReader.readMultiple(pattern: "/data/*.txt")

for try await line in stream {

    process(line)

}

```



**Glob Pattern Support:**

- `*` - Matches any sequence of characters

- `?` - Matches any single character

- Files are automatically sorted and concatenated in alphabetical order

- Works with CSV, JSON, JSONL, and Text formats



**Memory-Efficient Streaming:**



When using glob patterns, Frames now streams files one at a time instead of loading all files into memory. This is especially important for large datasets:



```swift

// Efficiently process large datasets split across many files

let largeDataset = DataFrame(csvPath: "/data/measurements_*.csv")



// Stream through all files without loading everything into memory

var total = 0.0

var count = 0

for try await point in largeDataset.stream() {

    total += point.value

    count += 1

    // Only one file's worth of data is in memory at a time

}

let average = total / Double(count)

```



**Streaming Behavior:**

- Files are processed sequentially in alphabetical order

- Only one file is held in memory at a time

- Ideal for processing datasets larger than available RAM

- Works seamlessly with filter, map, and other operations

- Backward compatible with existing code



For direct streaming API access:

```swift

// Stream CSV files directly

let csvStream = try CSVReader.readMultiple(pattern: "/data/*.csv", hasHeader: true)

for try await record in csvStream as MultiCSVSequence {

    process(record)

}



// Stream JSONL files directly

let jsonlStream = try JSONLReader.readMultiple(pattern: "/logs/*.jsonl")

for try await entry in jsonlStream as MultiJSONLSequence {

    process(entry)

}

```



### Hugging Face Datasets



Frames supports reading datasets directly from Hugging Face using the `hf://` URL scheme:



#### CSV from Hugging Face



```swift

struct IrisData: Codable, Sendable {

    let sepal_length: Double

    let sepal_width: Double

    let petal_length: Double

    let petal_width: Double

    let species: String

}



// Read directly from Hugging Face

let iris = DataFrame(

    csvPath: "hf://datasets/nameexhaustion/polars-docs/iris.csv"

)



let results = try await iris.collect()

print("Loaded \(results.count) samples")



// Process with type-safe operations

let bySpecies = try await iris

    .groupBy { $0.species }

    .count()

    .collect()

```



#### JSONL from Hugging Face



```swift

// Read JSONL from Hugging Face

let data = DataFrame(

    jsonlPath: "hf://datasets/username/repo/data.jsonl"

)

```



#### With Specific Revision/Branch



```swift

// Specify a branch or revision with @

let data = DataFrame(

    csvPath: "hf://datasets/username/repo@branch-name/file.csv"

)

```



**Hugging Face URL Format:**

- `hf://BUCKET/REPOSITORY/PATH` - Default main branch

- `hf://BUCKET/REPOSITORY@REVISION/PATH` - Specific branch/revision

- **BUCKET**: `datasets` or `spaces`

- **REPOSITORY**: `username/repo_name`

- **REVISION**: Branch name or commit hash (defaults to `main`)

- **PATH**: File path within the repository



**Examples:**

- `hf://datasets/nameexhaustion/polars-docs/iris.csv`

- `hf://datasets/nameexhaustion/polars-docs@foods/data.csv`

- `hf://spaces/nameexhaustion/polars-docs/orders.feather`



### Type-Safe Joins



Joins are fully type-safe with compile-time key validation:



```swift

struct User: Codable, Sendable {

    let id: Int

    let name: String

}



struct Order: Codable, Sendable {

    let orderId: Int

    let userId: Int

    let amount: Double

}



let users = DataFrame(csvURL: usersFile)

let orders = DataFrame(csvURL: ordersFile)



// Scio-style: First key both DataFrames, then join

let userKeyed = users.keyBy { $0.id }

let orderKeyed = orders.keyBy { $0.userId }



// Inner join (only matching keys)

let joined = userKeyed.join(orderKeyed)



for try await item in joined.stream() {

    print("User: \(item.left.name), Amount: $\(item.right.amount)")

}



// Left join (all left elements preserved)

let leftJoin = userKeyed.leftJoin(orderKeyed)



// Right join (all right elements preserved)

let rightJoin = userKeyed.rightJoin(orderKeyed)



// Full outer join (all elements from both sides)

let outerJoin = userKeyed.fullOuterJoin(orderKeyed)

```



### Type-Safe GroupBy



GroupBy operations work on typed elements:



```swift

struct Sale: Codable, Sendable {

    let category: String

    let amount: Double

    let quantity: Int

}



let sales = DataFrame(csvURL: salesFile)



// Group by category and count

let counts = try await sales

    .groupBy { $0.category }

    .count()

    .collect()



// Group by category and sum amounts

let totals = try await sales

    .groupBy { $0.category }

    .reduce(0.0) { acc, sale in acc + sale.amount }

    .collect()



for total in totals {

    print("Category \(total.key): $\(total.value)")

}



// Map groups to custom values

let avgByCategory = try await sales

    .groupBy { $0.category }

    .mapGroups { group in

        group.reduce(0.0) { $0 + $1.amount } / Double(group.count)

    }

    .collect()

```



### Window Functions



Apply window functions with partitioning and ordering:



```swift

struct Sale: Codable, Sendable {

    let category: String

    let date: Int

    let amount: Double

}



let sales = DataFrame(csvURL: salesFile)



// Add row numbers within each category, ordered by date

let numbered = try await sales

    .window(partitionBy: { $0.category }, orderBy: { $0.date })

    .rowNumber()

    .collect()



// Rank by amount (descending)

let ranked = try await sales

    .window(orderBy: { -$0.amount })

    .rank()

    .collect()



// Cumulative sum of amounts by category

let cumulative = try await sales

    .window(partitionBy: { $0.category }, orderBy: { $0.date })

    .cumSum { $0.amount }

    .collect()



// Moving average (3-period window)

let moving = try await sales

    .window(orderBy: { $0.date })

    .movingAverage(windowSize: 3) { $0.amount }

    .collect()



// Lag and lead values

let withLag = try await sales

    .window(orderBy: { $0.date })

    .lag(offset: 1) { $0.amount }

    .collect()



let withLead = try await sales

    .window(orderBy: { $0.date })

    .lead(offset: 1) { $0.amount }

    .collect()

```



**Available Window Functions:**

- `rowNumber()` - Sequential row numbers within partition

- `rank()` - Rank with gaps for ties

- `denseRank()` - Rank without gaps for ties

- `cumSum(_ getValue:)` - Cumulative sum

- `movingAverage(windowSize:_:)` - Moving average over window

- `lag(offset:_:)` - Previous row value

- `lead(offset:_:)` - Next row value



### Type-Safe Transformations



Chain type-safe transformations:



```swift

struct Person: Codable, Sendable {

    let name: String

    let age: Int

    let hobbies: [String]

}



let people = DataFrame(csvURL: peopleFile)



// Extract all hobbies from all people

let allHobbies = try await people

    .flatMap { $0.hobbies }

    .collect()



// Get names of adults

let adultNames = try await people

    .filter { $0.age >= 18 }

    .map { $0.name }

    .collect()



// Take first 10 results

let first10 = try await people

    .take(10)

    .collect()



// Skip first 5, then take next 10

let page2 = try await people

    .skip(5)

    .take(10)

    .collect()



// Sort by age

let sorted = try await people

    .sort { $0.age < $1.age }

    .collect()

```



### Writing Data



#### To CSV



```swift

let users = DataFrame(elements: usersArray)

try await users.saveAsCSV(to: outputURL)

```



#### To JSON



```swift

// Save as JSON array

try await users.saveAsJSON(to: outputURL)



// Save as pretty-printed JSON

try await users.saveAsJSON(to: outputURL, prettyPrinted: true)



// Save as JSONL (one object per line)

try await users.saveAsJSONL(to: outputURL)

```



### Streaming for Large Files



For datasets that don't fit in memory, use streaming:



```swift

// Stream and process data in batches

for try await user in users.stream() {

    await processUser(user)

}



// Reduce without materializing entire dataset

let totalAge = try await users.reduce(0) { acc, user in

    acc + user.age

}



// forEach for side effects

try await users

    .filter { $0.age >= 18 }

    .forEach { user in

        await sendEmail(to: user.email)

    }

```



### HTTP Streaming from Cloud Storage



Frames supports streaming data directly from HTTP/HTTPS URLs, making it ideal for cloud environments:



```swift

struct TaxiTrip: Codable, Sendable {

    let tripId: Int

    let pickupTime: String

    let distance: Double

    let fare: Double

}



// Stream from Google Cloud Storage (GCS)

let gcsURL = URL(string: "https://storage.googleapis.com/my-bucket/taxi-data.csv")!

let gcsTrips = DataFrame(csvURL: gcsURL)



// Stream from Amazon S3

let s3URL = URL(string: "https://my-bucket.s3.us-east-1.amazonaws.com/taxi-data.csv")!

let s3Trips = DataFrame(csvURL: s3URL)



// Stream from Azure Blob Storage

let azureURL = URL(string: "https://myaccount.blob.core.windows.net/container/data.csv")!

let azureTrips = DataFrame(csvURL: azureURL)



// Process with memory-efficient streaming

var totalFares = 0.0

var tripCount = 0



for try await trip in gcsTrips.stream() {

    totalFares += trip.fare

    tripCount += 1

    

    // Automatic backpressure handling

    // Process in real-time without buffering entire dataset

}



print("Average fare: $\(totalFares / Double(tripCount))")

```



**Benefits:**

- ✅ No local storage required

- ✅ Memory efficient for large datasets

- ✅ Automatic backpressure handling

- ✅ Works with presigned URLs

- ✅ Perfect for serverless environments (Cloud Run, Lambda, etc.)

- ✅ Supports all cloud providers (GCS, S3, Azure, etc.)



## Advanced Examples



### Complex Data Pipeline



```swift

struct Transaction: Codable, Sendable {

    let id: Int

    let userId: Int

    let amount: Double

    let status: String

    let category: String

    let date: Date

}



let transactions = DataFrame(csvURL: transactionsFile)



// Find high-value completed transactions, grouped by category

let summary = try await transactions

    .filter { $0.status == "completed" }

    .filter { $0.amount > 1000.0 }

    .groupBy { $0.category }

    .reduce(0.0) { acc, txn in acc + txn.amount }

    .collect()



for item in summary {

    print("Category \(item.key): $\(String(format: "%.2f", item.value))")

}

```



### Multi-Source Join



```swift

struct Customer: Codable, Sendable {

    let customerId: Int

    let name: String

    let email: String

}



struct Order: Codable, Sendable {

    let orderId: Int

    let customerId: Int

    let total: Double

}



let customers = DataFrame(jsonURL: customersFile)

let orders = DataFrame(csvURL: ordersFile)



let customerOrders = customers

    .keyBy { $0.customerId }

    .join(orders.keyBy { $0.customerId })



let highValueCustomers = try await customerOrders

    .filter { $0.right.total > 500.0 }

    .map { (name: $0.left.name, email: $0.left.email, total: $0.right.total) }

    .collect()

```



### ETL Pipeline with JSON and CSV



```swift

// Read from JSON

let rawData = DataFrame(jsonlURL: inputFile)



// Transform

let cleaned = rawData

    .filter { $0.status == "valid" }

    .map { CleanedEvent(from: $0) }



// Write to CSV for analysis

try await cleaned.saveAsCSV(to: outputFile)

```



### Union of DataFrames



Combine multiple DataFrames with the same element type:



```swift

struct User: Codable, Sendable {

    let id: Int

    let name: String

    let age: Int

}



// Load data from different sources

let users2023 = DataFrame(csvURL: file2023)

let users2024 = DataFrame(csvURL: file2024)



// Combine both datasets

let allUsers = users2023.union(users2024)



// Filter and analyze combined data

let adults = try await allUsers

    .filter { $0.age >= 18 }

    .collect()



print("Total users: \(adults.count)")

```



### File Checksums for Data Integrity



Verify file integrity using CRC32 checksums:



```swift

struct Transaction: Codable, Sendable {

    let id: Int

    let amount: Double

    let date: String

}



// Calculate checksum before processing

let csvFile = URL(fileURLWithPath: "transactions.csv")

let checksum = try DataFrame.checksum(fileURL: csvFile)

print("File checksum: 0x\(String(format: "%08X", checksum))")



// Process the file

let transactions = DataFrame(csvURL: csvFile)

try await transactions

    .filter { $0.amount > 1000 }

    .saveAsCSV(to: outputFile)



// Verify output file

let outputChecksum = try DataFrame.checksum(fileURL: outputFile)

print("Output checksum: 0x\(String(format: "%08X", outputChecksum))")

```



**CRC32 Utility:**



The `CRC32` struct provides standalone checksum calculation:



```swift

import Frames



// From Data

let data = "Hello, World!".data(using: .utf8)!

let checksum1 = CRC32.checksum(data: data)



// From String

let checksum2 = CRC32.checksum(string: "Hello, World!")



// From Bytes

let bytes: [UInt8] = [72, 101, 108, 108, 111]

let checksum3 = CRC32.checksum(bytes: bytes)



// From File

let fileChecksum = try CRC32.checksum(fileURL: fileURL)

```



### Processing Large Public Datasets



For a complete example of processing large CSV files from public datasets (NYC Taxi, Weather, COVID-19), see `Examples/LargeCSVProcessingExample.swift`.



This example demonstrates:

- Memory-efficient streaming for large files

- Data validation with checksums

- Combining multiple datasets with union

- Efficient filtering and aggregation

- Writing processed results



```swift

// Example: Process large taxi trip data

let trips = DataFrame(csvURL: largeTaxiFile)



// Verify integrity

let checksum = try DataFrame.checksum(fileURL: largeTaxiFile)



// Stream and process without loading everything into memory

try await trips.forEach { trip in

    await processTrip(trip)

}



// Combine multiple data sources

let station1 = DataFrame(csvURL: station1File)

let station2 = DataFrame(csvURL: station2File)

let allReadings = station1.union(station2)



// Aggregate across combined datasets

let avgTemp = try await allReadings

    .reduce(0.0) { acc, reading in acc + reading.temperature }

```



## API Reference



### DataFrame



```swift

public struct DataFrame: Sendable

```



**Initialization:**

- `init(elements: [Element])` - From array

- `init(csvURL: URL, hasHeader: Bool = true, delimiter: Character = ",")` - From single CSV file

- `init(csvPath: String, hasHeader: Bool = true, delimiter: Character = ",")` - From CSV file(s), glob pattern, or Hugging Face URL

- `init(jsonURL: URL)` - From single JSON array file

- `init(jsonPath: String)` - From JSON file(s), glob pattern, or Hugging Face URL

- `init(jsonlURL: URL)` - From single JSONL file

- `init(jsonlPath: String)` - From JSONL file(s), glob pattern, or Hugging Face URL

- `init(textURL: URL)` - From single text file (where Element is String)

- `init(textPath: String)` - From text file(s), glob pattern, or Hugging Face URL (where Element is String)



**Transformations:**

- `filter(_ predicate: (Element) -> Bool) -> DataFrame`

- `map(_ transform: (Element) -> T) -> DataFrame`

- `flatMap(_ transform: (Element) -> [T]) -> DataFrame`

- `take(_ n: Int) -> DataFrame`

- `skip(_ n: Int) -> DataFrame`

- `sort(_ comparator: (Element, Element) -> Bool) -> DataFrame`

- `union(_ other: DataFrame) -> DataFrame` - Combine two DataFrames



**Aggregations:**

- `groupBy(_ keyExtractor: (Element) -> K) -> GroupedDataFrame`

- `keyBy(_ keyExtractor: (Element) -> K) -> KeyedDataFrame`

- `window(partitionBy:orderBy:) -> WindowSpec<...>`



**Terminal Operations:**

- `collect() async throws -> [Element]`

- `count() async throws -> Int`

- `reduce(_ initialValue: T, _ combine: (T, Element) throws -> T) async throws -> T`

- `forEach(_ action: (Element) async throws -> Void) async throws`

- `stream() -> DataFrameStream`



**I/O:**

- `saveAsCSV(to url: URL) async throws`

- `saveAsJSON(to url: URL, prettyPrinted: Bool = false) async throws`

- `saveAsJSONL(to url: URL) async throws`

- `saveAsText(to url: URL) async throws` - (for DataFrame only)



**Utilities:**

- `static checksum(fileURL url: URL) throws -> UInt32` - Calculate CRC32 checksum for a file



### GroupedDataFrame



**Methods:**

- `count() async throws -> DataFrame>`

- `reduce(_ initialValue: T, _ combine: (T, Element) -> T) async throws -> DataFrame>`

- `mapGroups(_ transform: ([Element]) -> T) async throws -> DataFrame>`



### KeyedDataFrame



**Methods:**

- `join(_ other: KeyedDataFrame) -> DataFrame>`

- `leftJoin(_ other: KeyedDataFrame) -> DataFrame>`

- `rightJoin(_ other: KeyedDataFrame) -> DataFrame>`

- `fullOuterJoin(_ other: KeyedDataFrame) -> DataFrame>`

- `hashJoin(_ other: KeyedDataFrame) -> DataFrame>` (alias for join)



### WindowSpec



**Methods:**

- `rowNumber() -> DataFrame>`

- `rank() -> DataFrame>`

- `denseRank() -> DataFrame>`

- `cumSum(_ getValue: (Element) -> N) -> DataFrame>`

- `movingAverage(windowSize: Int, _ getValue: (Element) -> N) -> DataFrame>`

- `lag(offset: Int = 1, _ getValue: (Element) -> V) -> DataFrame>`

- `lead(offset: Int = 1, _ getValue: (Element) -> V) -> DataFrame>`



### CSVReader



```swift

public struct CSVReader: Sendable

```



**Initialization:**

- `init(url: URL, hasHeader: Bool = true, delimiter: Character = ",", bufferSize: Int = 8192)` - From URL

- `init(path: String, hasHeader: Bool = true, delimiter: Character = ",", bufferSize: Int = 8192)` - From path or Hugging Face URL



**Methods:**

- `rows() -> CSVRowSequence` - Stream rows as string arrays

- `rows(as type: T.Type) -> CSVCodableSequence` - Stream as Codable structs

- `read() async throws -> [T]` - Read entire CSV into array

- `static readMultiple(pattern: String, hasHeader: Bool = true, delimiter: Character = ",") async throws -> [T]` - Read multiple CSV files matching glob pattern



### JSONReader



```swift

public struct JSONReader

```



**Initialization:**

- `init(url: URL)` - From URL

- `init(path: String)` - From path or Hugging Face URL



**Methods:**

- `read() async throws -> [T]` - Read JSON array

- `readDataFrame() async throws -> DataFrame` - Read into DataFrame

- `static readMultiple(pattern: String) async throws -> [T]` - Read multiple JSON files matching glob pattern



### JSONLReader



```swift

public struct JSONLReader

```



**Initialization:**

- `init(url: URL, bufferSize: Int = 8192)` - From URL

- `init(path: String, bufferSize: Int = 8192)` - From path or Hugging Face URL



**Methods:**

- `rows() -> JSONLRowSequence` - Stream JSONL rows

- `read() async throws -> [T]` - Read all JSONL rows

- `readDataFrame() async throws -> DataFrame` - Read into DataFrame

- `static readMultiple(pattern: String) async throws -> [T]` - Read multiple JSONL files matching glob pattern



### TextFileReader



```swift

public struct TextFileReader: Sendable

```



A reader for plain text files that streams lines one at a time.



**Initialization:**

- `init(url: URL, bufferSize: Int = 8192)` - From URL

- `init(path: String, bufferSize: Int = 8192)` - From path or Hugging Face URL



**Methods:**

- `lines() -> TextLineSequence` - Stream text file lines

- `read() async throws -> [String]` - Read all lines into array

- `readDataFrame() async throws -> DataFrame` - Read into DataFrame

- `static readMultiple(pattern: String) throws -> TextLineSequence` - Read multiple text files matching glob pattern



### FileGlobbing



```swift

public struct FileGlobbing

```



Utility for expanding glob patterns to file lists.



**Methods:**

- `static expandGlob(_ pattern: String) throws -> [URL]` - Expand glob pattern (e.g., "*.csv") to list of matching file URLs



### HuggingFaceURL



```swift

public struct HuggingFaceURL

```



Utility for handling Hugging Face dataset URLs with `hf://` scheme.



**Methods:**

- `static parseHuggingFaceURL(_ urlString: String) throws -> URL` - Parse `hf://` URL to standard HTTPS URL

- `static isHuggingFaceURL(_ urlString: String) -> Bool` - Check if string is a Hugging Face URL



**URL Format:**

- `hf://BUCKET/REPOSITORY/PATH` - Default main branch

- `hf://BUCKET/REPOSITORY@REVISION/PATH` - Specific branch/revision

- BUCKET: `datasets` or `spaces`

- REPOSITORY: `username/repo_name`

- REVISION: Branch name or commit (optional, defaults to `main`)



### CRC32



```swift

public struct CRC32

```



**Methods:**

- `static checksum(data: Data) -> UInt32` - Calculate CRC32 for Data

- `static checksum(string: String, encoding: String.Encoding = .utf8) -> UInt32?` - Calculate CRC32 for String

- `static checksum(bytes: [UInt8]) -> UInt32` - Calculate CRC32 for byte array

- `static checksum(fileURL url: URL) throws -> UInt32` - Calculate CRC32 for file



The CRC32 implementation uses the IEEE 802.3 polynomial (0xEDB88320) for standard CRC32 checksums. Useful for verifying data integrity of CSV, JSON, and other data files.



## Requirements



- Swift 6.2 or later

- Linux or macOS



## License



MIT License - See LICENSE file for details



## Contributing



Contributions are welcome! Please feel free to submit a Pull Request.