{"id":37197040,"url":"https://github.com/beyondcivic/gocroissant","last_synced_at":"2026-01-14T22:55:05.524Z","repository":{"id":309730923,"uuid":"983307597","full_name":"beyondcivic/gocroissant","owner":"beyondcivic","description":"A Go implementation for working with the ML Commons Croissant metadata format.","archived":false,"fork":false,"pushed_at":"2025-10-07T09:24:03.000Z","size":7843,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-07T11:35:11.365Z","etag":null,"topics":["cli","croissant","go","json-ld","mlcommons"],"latest_commit_sha":null,"homepage":"https://mlcommons.org/croissant","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/beyondcivic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-14T07:31:37.000Z","updated_at":"2025-10-07T09:19:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"f8f10b95-4f32-4b81-b312-3ba04aaf0e0d","html_url":"https://github.com/beyondcivic/gocroissant","commit_stats":null,"previous_names":["beyondcivic/gocroissant"],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/beyondcivic/gocroissant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondcivic%2Fgocroissant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondcivic%2Fgocroissant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondcivic%2Fgocroissant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondcivic%2Fgocroissant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/beyondcivic","download_url":"https://codeload.github.com/beyondcivic/gocroissant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondcivic%2Fgocroissant/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28437228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T22:37:52.437Z","status":"ssl_error","status_checked_at":"2026-01-14T22:37:31.496Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","croissant","go","json-ld","mlcommons"],"created_at":"2026-01-14T22:55:02.690Z","updated_at":"2026-01-14T22:55:05.490Z","avatar_url":"https://github.com/beyondcivic.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gocroissant\n\n[![Version](https://img.shields.io/badge/version-v0.4.0-blue)](https://github.com/beyondcivic/gocroissant/releases/tag/v0.4.0)\n[![Go Version](https://img.shields.io/badge/Go-1.24+-00ADD8?logo=go)](https://golang.org/doc/devel/release.html)\n[![Go Reference](https://pkg.go.dev/badge/github.com/beyondcivic/gocroissant.svg)](https://pkg.go.dev/github.com/beyondcivic/gocroissant)\n[![License](https://img.shields.io/badge/license-MIT-green)](LICENSE)\n\nA Go implementation for working with the [ML Commons Croissant](https://github.com/mlcommons/croissant) metadata format - a standardized way to describe machine learning datasets using JSON-LD.\n\n## Overview\n\nCroissant is an open metadata standard designed to improve dataset documentation, searchability, and usage in machine learning workflows. This library simplifies the creation of Croissant-compatible metadata from CSV data sources by:\n\n- **Automatically inferring schema types** from dataset content\n- **Generating complete, valid JSON-LD metadata** compliant with the Croissant specification\n- **Providing comprehensive validation tools** to ensure compatibility\n- **Supporting metadata comparison** for schema compatibility checking\n- **Supporting the full Croissant specification** with extensible architecture\n\nThis project provides both a command-line interface and a Go library for working with Croissant metadata format.\n\n## Key Features\n\n- ✅ **Metadata Generation**: Convert CSV files to Croissant JSON-LD metadata\n- ✅ **Schema Validation**: Comprehensive validation with detailed error reporting\n- ✅ **Metadata Comparison**: Compare two metadata files for compatibility\n- ✅ **Type Inference**: Automatic detection of data types (Boolean, Integer, Float, Date, URL, Text)\n- ✅ **File Analysis**: Detailed CSV file inspection and statistics\n- ✅ **JSON-LD Support**: Full JSON-LD processing and validation\n- ✅ **CLI \u0026 Library**: Both command-line tool and Go library interfaces\n- ✅ **Cross-platform**: Works on Linux, macOS, and Windows\n\n## Getting Started\n\n### Prerequisites\n\n- Go 1.24 or later\n- Nix 2.25.4 or later (optional but recommended)\n- PowerShell v7.5.1 or later (for building)\n\n### Installation\n\n#### Option 1: Install from Source\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/beyondcivic/gocroissant.git\ncd gocroissant\n```\n\n2. Build the application:\n\n```bash\ngo build -o gocroissant .\n```\n\n#### Option 2: Using Nix (Recommended)\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/beyondcivic/gocroissant.git\ncd gocroissant\n```\n\n2. Prepare the environment using Nix flakes:\n\n```bash\nnix develop\n```\n\n3. Build the application:\n\n```bash\n./build.ps1\n```\n\n#### Option 3: Go Install\n\n```bash\ngo install github.com/beyondcivic/gocroissant@latest\n```\n\n## Quick Start\n\n### Command Line Interface\n\nThe `gocroissant` tool provides several commands for working with Croissant metadata:\n\n```bash\n# Generate metadata from CSV\ngocroissant generate data.csv -o metadata.jsonld\n\n# Validate existing metadata\ngocroissant validate metadata.jsonld\n\n# Compare two metadata files for compatibility\ngocroissant match reference.jsonld candidate.jsonld\n\n# Analyze CSV file structure\ngocroissant info data.csv\n\n# Show version information\ngocroissant version\n```\n\n### Go Library Usage\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/beyondcivic/gocroissant/pkg/croissant\"\n)\n\nfunc main() {\n\t// Generate metadata from CSV\n\toutputPath, err := croissant.GenerateMetadata(\"data.csv\", \"dataset.jsonld\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Error generating metadata: %v\", err)\n\t}\n\tfmt.Printf(\"Metadata saved to: %s\\n\", outputPath)\n\n\t// Validate metadata\n\tissues, err := croissant.ValidateFile(\"dataset.jsonld\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Validation error: %v\", err)\n\t}\n\n\tif issues.HasErrors() {\n\t\tfmt.Println(\"Validation failed:\")\n\t\tfmt.Println(issues.Report())\n\t} else {\n\t\tfmt.Println(\"Validation passed!\")\n\t}\n\n\t// Compare two metadata files\n\tref, err := croissant.LoadMetadataFromFile(\"reference.jsonld\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Error loading reference: %v\", err)\n\t}\n\n\tcand, err := croissant.LoadMetadataFromFile(\"candidate.jsonld\")\n\tif err != nil {\n\t\tlog.Fatalf(\"Error loading candidate: %v\", err)\n\t}\n\n\tresult := croissant.MatchMetadata(*ref, *cand)\n\tif result.IsMatch {\n\t\tfmt.Println(\"Files are compatible!\")\n\t} else {\n\t\tfmt.Printf(\"Compatibility issues found: %d missing, %d mismatched\\n\",\n\t\t\tlen(result.MissingFields), len(result.TypeMismatches))\n\t}\n}\n```\n\n## Detailed Command Reference\n\n### `generate` - Generate Metadata from CSV\n\nConvert a CSV file to Croissant metadata format with automatic type inference.\n\n```bash\ngocroissant generate [CSV_FILE] [OPTIONS]\n```\n\n**Options:**\n\n- `-o, --output`: Output file path (default: `[filename]_metadata.jsonld`)\n- `-v, --validate`: Validate generated metadata and show issues\n- `--strict`: Enable strict validation mode\n- `--check-files`: Check if referenced files exist during validation\n\n**Examples:**\n\n```bash\n# Basic generation\ngocroissant generate data.csv\n\n# With custom output path\ngocroissant generate data.csv -o my-dataset.jsonld\n\n# Generate and validate\ngocroissant generate data.csv -o metadata.jsonld --validate\n\n# Strict validation with file checking\ngocroissant generate data.csv --validate --strict --check-files\n```\n\n### `validate` - Validate Existing Metadata\n\nValidate a Croissant metadata file for compliance with the specification.\n\n```bash\ngocroissant validate [METADATA_FILE] [OPTIONS]\n```\n\n**Options:**\n\n- `--strict`: Enable strict validation mode\n- `--check-files`: Check if referenced files exist\n- `--check-urls`: Validate URLs by making HTTP requests\n\n**Examples:**\n\n```bash\n# Basic validation\ngocroissant validate metadata.jsonld\n\n# Strict validation with file and URL checking\ngocroissant validate metadata.jsonld --strict --check-files --check-urls\n```\n\n### `match` - Compare Metadata Compatibility\n\nCompare two Croissant metadata files to check schema compatibility. The candidate file can have additional fields, but must contain all fields from the reference with compatible data types.\n\n```bash\ngocroissant match [REFERENCE_FILE] [CANDIDATE_FILE] [OPTIONS]\n```\n\n**Options:**\n\n- `-v, --verbose`: Show detailed information including extra fields in candidate\n\n**Examples:**\n\n```bash\n# Basic compatibility check\ngocroissant match reference.jsonld candidate.jsonld\n\n# Verbose output showing all details\ngocroissant match reference.jsonld candidate.jsonld --verbose\n```\n\n**Exit Codes:**\n\n- `0`: Files are compatible\n- `1`: Files are incompatible or error occurred\n\n### `info` - Analyze CSV File\n\nDisplay detailed information about a CSV file's structure, columns, and inferred data types.\n\n```bash\ngocroissant info [CSV_FILE] [OPTIONS]\n```\n\n**Options:**\n\n- `--sample-size`: Number of rows to sample for type inference (default: 10)\n\n**Examples:**\n\n```bash\n# Basic file analysis\ngocroissant info data.csv\n\n# Analyze with larger sample size\ngocroissant info data.csv --sample-size 100\n```\n\n### `version` - Show Version Information\n\nDisplay version, build information, and system details.\n\n```bash\ngocroissant version\n```\n\n## Data Type Inference\n\nThe tool automatically detects and maps data types to schema.org types:\n\n| Detected Pattern           | Schema.org Type | Description     |\n| -------------------------- | --------------- | --------------- |\n| `true`, `false`, `1`, `0`  | `sc:Boolean`    | Boolean values  |\n| `123`, `-456`              | `sc:Integer`    | Whole numbers   |\n| `3.14`, `2.5e10`           | `sc:Float`      | Decimal numbers |\n| `2023-01-01`, `01/15/2023` | `sc:Date`       | Date values     |\n| `https://example.com`      | `sc:URL`        | Web URLs        |\n| Everything else            | `sc:Text`       | Text content    |\n\n## Configuration\n\nThe application supports configuration through environment variables:\n\n- `CROISSANT_OUTPUT_PATH`: Default output path for generated metadata\n\nIf no output path is provided, the default format is `[filename]_metadata.jsonld`.\n\n## Validation Features\n\nThe validation system provides comprehensive checking:\n\n### Error Types\n\n- **Mandatory fields**: Missing required properties\n- **Type validation**: Incorrect `@type` values\n- **Reference validation**: Invalid field or file references\n- **Data type validation**: Invalid `dataType` specifications\n- **JSON-LD structure**: Malformed JSON-LD documents\n\n### Validation Modes\n\n- **Standard mode**: Basic compliance checking\n- **Strict mode**: Enhanced validation with additional warnings\n- **File checking**: Verify referenced files exist\n- **URL validation**: Check URL accessibility (optional)\n\n## Schema Compatibility\n\nThe `match` command performs intelligent compatibility checking:\n\n### Compatible Scenarios\n\n- ✅ Candidate has additional fields (extra columns allowed)\n- ✅ Numeric type compatibility (`sc:Number` ↔ `sc:Float`, `sc:Integer`)\n- ✅ Exact field name and type matches\n\n### Incompatible Scenarios\n\n- ❌ Missing required fields from reference\n- ❌ Type mismatches (e.g., `sc:Text` vs `sc:Integer`)\n- ❌ Invalid JSON-LD structure\n\n## Examples\n\n### Example 1: Basic Metadata Generation\n\n```bash\n# Generate metadata for a simple CSV\n$ gocroissant generate sales_data.csv -o sales_metadata.jsonld\n\nGenerating Croissant metadata for 'sales_data.csv'...\n✓ Croissant metadata generated successfully and saved to: sales_metadata.jsonld\n```\n\n### Example 2: Generation with Validation\n\n```bash\n$ gocroissant generate sales_data.csv -o sales_metadata.jsonld --validate\n\nGenerating Croissant metadata for 'sales_data.csv'...\n✓ Validation passed with no issues.\n✓ Croissant metadata generated successfully and saved to: sales_metadata.jsonld\n```\n\n### Example 3: Metadata Validation\n\n```bash\n# Successful validation\n$ gocroissant validate sales_metadata.jsonld\nValidating Croissant metadata file 'sales_metadata.jsonld'...\n✓ Validation passed with no issues.\n\n# Validation with issues\n$ gocroissant validate ./samples_jsonld/missing_fields.jsonld\n\nFound the following 3 error(s) during the validation:\n  -  [Metadata(mydataset) \u003e FileObject(a-csv-table)] Property \"https://schema.org/contentUrl\" is mandatory, but does not exist.\n  -  [Metadata(mydataset) \u003e RecordSet(a-record-set) \u003e Field(first-field)] The field does not specify a valid http://mlcommons.org/croissant/dataType, neither does any of its predecessor. Got:\n  -  [Metadata(mydataset)] The current JSON-LD doesn't extend https://schema.org/Dataset.\n\nFound the following 1 warning(s) during the validation:\n  -  [Metadata(mydataset)] Property \"http://purl.org/dc/terms/conformsTo\" is recommended, but does not exist.\nexit status 1\n```\n\n### Example 4: Metadata Comparison\n\n```bash\n# Compatible schemas\n$ gocroissant match reference.jsonld candidate.jsonld\n\nComparing metadata files...\nReference: reference.jsonld\nCandidate: candidate.jsonld\n\n✓ Compatibility check PASSED\nThe candidate metadata is compatible with the reference.\n\nMatched fields (6):\n  ✓ transaction_id\n  ✓ timestamp\n  ✓ location\n  ✓ water_flow_rate\n  ✓ precipitation\n  ✓ turbidity\n\nSummary:\n  Matched: 6\n  Missing: 0\n  Type mismatches: 0\n  Extra fields: 0\n\n# Incompatible schemas\n$ gocroissant match reference.jsonld incompatible.jsonld\n\nComparing metadata files...\nReference: reference.jsonld\nCandidate: incompatible.jsonld\n\n✗ Compatibility check FAILED\nThe candidate metadata is NOT compatible with the reference.\n\nMissing fields (3):\n  ✗ transaction_id (required by reference but not found in candidate)\n  ✗ timestamp (required by reference but not found in candidate)\n  ✗ location (required by reference but not found in candidate)\n\nType mismatches (1):\n  ✗ water_flow_rate: reference expects 'sc:Float', candidate has 'sc:Text'\n\nCandidate has 2 additional field(s) (use --verbose to see details)\n\nSummary:\n  Matched: 2\n  Missing: 3\n  Type mismatches: 1\n  Extra fields: 2\n```\n\n### Example 5: CSV File Analysis\n\n```bash\n$ gocroissant info sample_data.csv --sample-size 20\n\nCSV File Information: sample_data.csv\n=====================================\nFile Size: 15247 bytes\nTotal Rows: 101 (including header)\nData Rows: 100\nColumns: 5\nSample Size: 20 rows\n\nColumn Information:\n-------------------\n1. id (sc:Integer)\n2. name (sc:Text)\n3. price (sc:Float)\n4. available (sc:Boolean)\n5. created_date (sc:Date)\n```\n\n## API Reference\n\n### Core Functions\n\n#### `GenerateMetadata(csvPath, outputPath string) (string, error)`\n\nGenerates Croissant metadata from a CSV file.\n\n#### `GenerateMetadataWithValidation(csvPath, outputPath string) (*Metadata, error)`\n\nGenerates metadata and returns the parsed Metadata struct for further processing.\n\n#### `ValidateFile(filePath string) (*Issues, error)`\n\nValidates a Croissant metadata file and returns validation issues.\n\n#### `ValidateJSON(data []byte) (*Issues, error)`\n\nValidates Croissant metadata from JSON bytes.\n\n#### `MatchMetadata(reference, candidate Metadata) *MatchResult`\n\nCompares two metadata objects for schema compatibility.\n\n#### `LoadMetadataFromFile(filePath string) (*Metadata, error)`\n\nLoads and parses a Croissant metadata file.\n\n### Data Structures\n\n#### `Metadata`\n\nRepresents the complete Croissant metadata structure.\n\n#### `MatchResult`\n\nContains the results of a metadata comparison:\n\n- `IsMatch bool`: Whether the schemas are compatible\n- `MissingFields []string`: Fields required by reference but missing in candidate\n- `TypeMismatches []FieldMismatch`: Fields with incompatible data types\n- `ExtraFields []string`: Additional fields in candidate\n- `MatchedFields []string`: Successfully matched fields\n\n#### `Issues`\n\nContains validation results with errors and warnings.\n\n## Architecture\n\nThe library is organized into several key components:\n\n### Core Package (`pkg/croissant`)\n\n- **Generation**: CSV parsing and metadata creation\n- **Validation**: JSON-LD and Croissant specification validation\n- **Matching**: Schema compatibility checking\n- **JSON-LD Processing**: JSON-LD document handling and validation\n- **Type Inference**: Automatic data type detection from CSV content\n\n### Command Line Interface (`cmd/gocroissant`)\n\n- **Cobra-based CLI** with subcommands for each major function\n- **Comprehensive help system** with detailed usage examples\n- **Flexible output options** and validation modes\n\n## Development\n\n### Adding New Data Types\n\nTo add support for new data types, modify the `InferDataType` function in `pkg/croissant/croissant.go`:\n\n```go\nfunc InferDataType(value string) string {\n\t// Existing data type detection...\n\n\t// Add your new data type detection here\n\tif myCustomTypeDetector(value) {\n\t\treturn \"sc:MyCustomType\"\n\t}\n\n\t// Default to Text\n\treturn \"sc:Text\"\n}\n```\n\n### Contributing\n\n1. Fork the repository\n2. Create a feature branch: `git checkout -b feature/new-feature`\n3. Make your changes and add tests\n4. Ensure all tests pass: `go test ./...`\n5. Commit your changes: `git commit -am 'Add new feature'`\n6. Push to the branch: `git push origin feature/new-feature`\n7. Submit a pull request\n\n### Testing\n\nRun the test suite:\n\n```bash\ngo test ./...\n```\n\nRun tests with coverage:\n\n```bash\ngo test -cover ./...\n```\n\n## Build Environment\n\n### Using Nix (Recommended)\n\nUse Nix flakes to set up the build environment:\n\n```bash\nnix develop\n```\n\n### Manual Build\n\nCheck the build arguments in `build.ps1`:\n\n```bash\n# Build static binary with version information\n$env:CGO_ENABLED = \"1\"\n$env:GOOS = \"linux\"\n$env:GOARCH = \"amd64\"\n```\n\nThen run:\n\n```bash\n./build.ps1\n```\n\nOr build manually:\n\n```bash\ngo build -o gocroissant .\n```\n\n## Related Projects\n\n- [ML Commons Croissant](https://github.com/mlcommons/croissant) - Official Croissant specification\n- [Croissant Editor](https://github.com/mlcommons/croissant/tree/main/editor) - Web-based metadata editor\n- [Python Croissant](https://github.com/mlcommons/croissant/tree/main/python) - Python implementation\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeyondcivic%2Fgocroissant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeyondcivic%2Fgocroissant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeyondcivic%2Fgocroissant/lists"}