{"id":43932434,"url":"https://github.com/usetero/policy-go","last_synced_at":"2026-02-19T02:00:53.259Z","repository":{"id":333917479,"uuid":"1139195611","full_name":"usetero/policy-go","owner":"usetero","description":"Go implementation for the Policy spec","archived":false,"fork":false,"pushed_at":"2026-02-17T16:47:31.000Z","size":2328,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-02-17T21:45:59.273Z","etag":null,"topics":["go","observability","policy","telemetry"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/usetero.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-21T16:40:23.000Z","updated_at":"2026-02-17T16:45:48.000Z","dependencies_parsed_at":"2026-01-22T08:03:12.538Z","dependency_job_id":null,"html_url":"https://github.com/usetero/policy-go","commit_stats":null,"previous_names":["usetero/policy-go"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/usetero/policy-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usetero%2Fpolicy-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usetero%2Fpolicy-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usetero%2Fpolicy-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usetero%2Fpolicy-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/usetero","download_url":"https://codeload.github.com/usetero/policy-go/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/usetero%2Fpolicy-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29600841,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T00:59:38.239Z","status":"online","status_checked_at":"2026-02-19T02:00:07.702Z","response_time":117,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","observability","policy","telemetry"],"created_at":"2026-02-07T00:17:33.950Z","updated_at":"2026-02-19T02:00:53.250Z","avatar_url":"https://github.com/usetero.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# policy-go\n\nA high-performance policy evaluation library for OpenTelemetry telemetry data in\nGo. Built with [Hyperscan](https://github.com/intel/hyperscan) for fast regex\nmatching and designed for hot-reload support.\n\n## Features\n\n- **High-Performance Matching**: Uses Intel Hyperscan for vectorized regex\n  evaluation\n- **Multi-Telemetry Support**: Evaluate logs, metrics, and traces with type-safe\n  APIs\n- **Hot Reload**: File-based policy providers support automatic reloading on\n  change\n- **Thread-Safe**: Immutable snapshots for concurrent evaluation\n- **Extensible**: Provider interface for custom policy sources (file, HTTP,\n  gRPC)\n- **Statistics**: Per-policy hit/drop/sample counters with atomic operations\n- **Rate Limiting**: Lock-free per-policy rate limiting with configurable\n  windows\n- **AND Semantics**: Multiple matchers in a policy are AND'd together\n\n## Installation\n\n```bash\ngo get github.com/usetero/policy-go\n```\n\n### Requirements\n\n- Go 1.21+\n- Hyperscan library (via [gohs](https://github.com/flier/gohs))\n\nOn macOS with Homebrew:\n\n```bash\nbrew install hyperscan\n```\n\nOn Ubuntu/Debian:\n\n```bash\napt-get install libhyperscan-dev\n```\n\n## Quick Start\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \"time\"\n\n    \"github.com/usetero/policy-go\"\n)\n\n// Define your log record type\ntype LogRecord struct {\n    Body               []byte\n    SeverityText       []byte\n    TraceID            []byte\n    LogAttributes      map[string]any\n    ResourceAttributes map[string]any\n}\n\n// Implement a match function to extract field values\nfunc matchLog(r *LogRecord, ref policy.LogFieldRef) []byte {\n    // Handle field lookups\n    if ref.IsField() {\n        switch ref.Field {\n        case policy.LogFieldBody:\n            return r.Body\n        case policy.LogFieldSeverityText:\n            return r.SeverityText\n        case policy.LogFieldTraceID:\n            return r.TraceID\n        default:\n            return nil\n        }\n    }\n\n    // Handle attribute lookups\n    var attrs map[string]any\n    switch {\n    case ref.IsResourceAttr():\n        attrs = r.ResourceAttributes\n    case ref.IsRecordAttr():\n        attrs = r.LogAttributes\n    default:\n        return nil\n    }\n    return traversePath(attrs, ref.AttrPath)\n}\n\nfunc traversePath(m map[string]any, path []string) []byte {\n    if len(path) == 0 || m == nil {\n        return nil\n    }\n    val, ok := m[path[0]]\n    if !ok {\n        return nil\n    }\n    if len(path) == 1 {\n        switch v := val.(type) {\n        case string:\n            return []byte(v)\n        case []byte:\n            return v\n        }\n        return nil\n    }\n    if nested, ok := val.(map[string]any); ok {\n        return traversePath(nested, path[1:])\n    }\n    return nil\n}\n\nfunc main() {\n    // Create a registry\n    registry := policy.NewPolicyRegistry()\n\n    // Create a file provider with hot reload\n    provider := policy.NewFileProvider(\"policies.json\",\n        policy.WithPollInterval(30*time.Second),\n        policy.WithOnError(func(err error) {\n            log.Printf(\"Policy error: %v\", err)\n        }),\n    )\n    defer provider.Stop()\n\n    // Register the provider\n    handle, err := registry.Register(provider)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer handle.Unregister()\n\n    // Create an engine\n    engine := policy.NewPolicyEngine(registry)\n\n    // Evaluate a log record\n    record := \u0026LogRecord{\n        Body:         []byte(\"debug trace message\"),\n        SeverityText: []byte(\"INFO\"),\n        LogAttributes: map[string]any{\n            \"http\": map[string]any{\n                \"method\": \"GET\",\n            },\n        },\n    }\n\n    result := policy.EvaluateLog(engine, record, matchLog)\n    fmt.Printf(\"Result: %s\\n\", result) // \"drop\" if matched by policy\n}\n```\n\n## Core Concepts\n\n### PolicyRegistry\n\nThe registry manages policies from multiple providers. When policies change, it\nautomatically recompiles the Hyperscan database and produces a new immutable\nsnapshot.\n\n```go\nregistry := policy.NewPolicyRegistry()\n\n// Register providers\nhandle, _ := registry.Register(fileProvider)\nhandle, _ := registry.Register(httpProvider)\n\n// Collect stats\nstats := registry.CollectStats()\n```\n\n### PolicyEngine\n\nThe engine evaluates telemetry against compiled policies. It holds a reference\nto the registry and automatically uses the latest snapshot for each evaluation.\n\n```go\nengine := policy.NewPolicyEngine(registry)\n\n// Evaluate logs\nresult := policy.EvaluateLog(engine, logRecord, matchLogFunc)\n\n// Evaluate metrics\nresult := policy.EvaluateMetric(engine, metricRecord, matchMetricFunc)\n\n// Evaluate traces/spans\nresult := policy.EvaluateTrace(engine, spanRecord, matchTraceFunc)\n\nswitch result {\ncase policy.ResultNoMatch:\n    // No policy matched - pass through\ncase policy.ResultKeep:\n    // Matched a keep policy (or under rate limit)\ncase policy.ResultDrop:\n    // Matched a drop policy (or over rate limit)\ncase policy.ResultSample:\n    // Sampled (for metrics without sample key)\n}\n```\n\n### Match Functions\n\nInstead of implementing an interface, you provide a match function that extracts\nfield values from your telemetry types. This allows maximum flexibility in how\nyou represent your data.\n\n```go\n// LogMatchFunc extracts values from log records\ntype LogMatchFunc[T any] func(record T, ref LogFieldRef) []byte\n\n// MetricMatchFunc extracts values from metrics\ntype MetricMatchFunc[T any] func(record T, ref MetricFieldRef) []byte\n\n// TraceMatchFunc extracts values from spans\ntype TraceMatchFunc[T any] func(record T, ref TraceFieldRef) []byte\n```\n\nExample match function for logs:\n\n```go\nfunc matchLog(r *MyLogRecord, ref policy.LogFieldRef) []byte {\n    // Handle field lookups\n    if ref.IsField() {\n        switch ref.Field {\n        case policy.LogFieldBody:\n            return r.Body\n        case policy.LogFieldSeverityText:\n            return r.SeverityText\n        default:\n            return nil\n        }\n    }\n\n    // Handle attribute lookups\n    var attrs map[string]any\n    switch {\n    case ref.IsResourceAttr():\n        attrs = r.ResourceAttributes\n    case ref.IsRecordAttr():\n        attrs = r.LogAttributes\n    case ref.IsScopeAttr():\n        attrs = r.ScopeAttributes\n    default:\n        return nil\n    }\n    return traversePath(attrs, ref.AttrPath)\n}\n```\n\n### Field References\n\nField references (`LogFieldRef`, `MetricFieldRef`, `TraceFieldRef`) describe\nwhat value to extract. Use helper methods to determine the reference type:\n\n```go\nref.IsField()        // Is this a direct field (body, name, etc.)?\nref.IsResourceAttr() // Is this a resource attribute?\nref.IsRecordAttr()   // Is this a record/span/datapoint attribute?\nref.IsScopeAttr()    // Is this a scope attribute?\nref.IsEventAttr()    // Is this an event attribute? (traces only)\nref.IsLinkAttr()     // Is this a link attribute? (traces only)\n\nref.Field            // The field enum value\nref.AttrPath         // The attribute path (e.g., [\"http\", \"method\"])\n```\n\n### Attribute Scopes\n\n- `AttrScopeResource`: Resource-level attributes (service.name, etc.)\n- `AttrScopeScope`: Instrumentation scope attributes\n- `AttrScopeRecord`: Record-level attributes (log attributes, span attributes,\n  datapoint attributes)\n- `AttrScopeEvent`: Span event attributes (traces only)\n- `AttrScopeLink`: Span link attributes (traces only)\n\n## Configuration\n\n### Config File\n\nUse a JSON configuration file to define providers:\n\n```json\n{\n  \"policy_providers\": [\n    {\n      \"type\": \"file\",\n      \"id\": \"local-policies\",\n      \"path\": \"/etc/tero/policies.json\",\n      \"poll_interval_secs\": 30\n    }\n  ]\n}\n```\n\n### Loading Config\n\n```go\nconfig, err := policy.LoadConfig(\"config.json\")\nif err != nil {\n    log.Fatal(err)\n}\n\nloader := policy.NewConfigLoader(registry).\n    WithOnError(func(err error) {\n        log.Printf(\"Provider error: %v\", err)\n    })\n\nproviders, err := loader.Load(config)\nif err != nil {\n    log.Fatal(err)\n}\ndefer policy.StopAll(providers)\ndefer policy.UnregisterAll(providers)\n```\n\n## Policy Format\n\nPolicies are defined in JSON format following the\n[Tero Policy Specification](https://buf.build/tero/policy):\n\n### Log Policies\n\n```json\n{\n  \"policies\": [\n    {\n      \"id\": \"drop-debug-logs\",\n      \"name\": \"Drop debug logs containing trace\",\n      \"log\": {\n        \"match\": [\n          { \"log_field\": \"body\", \"regex\": \"debug\" },\n          { \"log_field\": \"body\", \"regex\": \"trace\" }\n        ],\n        \"keep\": \"none\"\n      }\n    },\n    {\n      \"id\": \"drop-nginx-logs\",\n      \"name\": \"Drop nginx access logs\",\n      \"log\": {\n        \"match\": [{ \"log_attribute\": \"ddsource\", \"exact\": \"nginx\" }],\n        \"keep\": \"none\"\n      }\n    }\n  ]\n}\n```\n\n### Metric Policies\n\n```json\n{\n  \"policies\": [\n    {\n      \"id\": \"drop-internal-metrics\",\n      \"name\": \"Drop internal metrics\",\n      \"metric\": {\n        \"match\": [{ \"metric_field\": \"name\", \"starts_with\": \"internal.\" }],\n        \"keep\": false\n      }\n    },\n    {\n      \"id\": \"drop-histogram-metrics\",\n      \"name\": \"Drop histogram type metrics\",\n      \"metric\": {\n        \"match\": [{ \"metric_field\": \"type\", \"exact\": \"histogram\" }],\n        \"keep\": false\n      }\n    }\n  ]\n}\n```\n\n### Trace Policies\n\n```json\n{\n  \"policies\": [\n    {\n      \"id\": \"sample-traces\",\n      \"name\": \"Sample 10% of traces\",\n      \"trace\": {\n        \"match\": [{ \"span_field\": \"kind\", \"exact\": \"server\" }],\n        \"keep\": { \"percentage\": 10 }\n      }\n    },\n    {\n      \"id\": \"drop-health-checks\",\n      \"name\": \"Drop health check spans\",\n      \"trace\": {\n        \"match\": [{ \"span_field\": \"name\", \"exact\": \"/health\" }],\n        \"keep\": { \"percentage\": 0 }\n      }\n    }\n  ]\n}\n```\n\nTrace sampling uses OTel-compliant consistent probability sampling. The same\ntrace ID always produces the same sampling decision, ensuring all spans in a\ntrace are kept or dropped together.\n\n### Matcher Types\n\n#### Log Matchers\n\n| Field                | Description                                                    |\n| -------------------- | -------------------------------------------------------------- |\n| `log_field`          | Match on log fields: `body`, `severity_text`, `trace_id`, etc. |\n| `log_attribute`      | Match on log record attributes                                 |\n| `resource_attribute` | Match on resource attributes                                   |\n| `scope_attribute`    | Match on scope attributes                                      |\n\n#### Metric Matchers\n\n| Field                 | Description                                          |\n| --------------------- | ---------------------------------------------------- |\n| `metric_field`        | Match on metric fields: `name`, `type`, `unit`, etc. |\n| `datapoint_attribute` | Match on datapoint attributes                        |\n| `resource_attribute`  | Match on resource attributes                         |\n| `scope_attribute`     | Match on scope attributes                            |\n\n#### Trace Matchers\n\n| Field                | Description                                          |\n| -------------------- | ---------------------------------------------------- |\n| `span_field`         | Match on span fields: `name`, `kind`, `status`, etc. |\n| `span_attribute`     | Match on span attributes                             |\n| `resource_attribute` | Match on resource attributes                         |\n| `scope_attribute`    | Match on scope attributes                            |\n| `event_name`         | Match on span event names                            |\n| `event_attribute`    | Match on span event attributes                       |\n| `link_trace_id`      | Match on span link trace IDs                         |\n| `link_attribute`     | Match on span link attributes                        |\n\n#### Nested Attribute Access\n\nAttributes can be accessed using nested paths for structured data:\n\n```json\n{\n  \"log_attribute\": { \"path\": [\"http\", \"request\", \"method\"] },\n  \"exact\": \"POST\"\n}\n```\n\nShorthand forms are also supported:\n\n- Array: `\"log_attribute\": [\"http\", \"request\", \"method\"]`\n- String (single key): `\"log_attribute\": \"user_id\"`\n\n### Match Conditions\n\n| Condition          | Description                                               |\n| ------------------ | --------------------------------------------------------- |\n| `regex`            | Match if field matches the regex pattern                  |\n| `exact`            | Match if field equals the exact value                     |\n| `starts_with`      | Match if field starts with the literal prefix             |\n| `ends_with`        | Match if field ends with the literal suffix               |\n| `contains`         | Match if field contains the literal substring             |\n| `exists`           | Match if field exists (true) or doesn't exist (false)     |\n| `negate`           | Invert the match condition                                |\n| `case_insensitive` | Make the match case-insensitive (works with all matchers) |\n\nThe literal matchers (`starts_with`, `ends_with`, `contains`, `exact`) are\noptimized using Hyperscan and are more efficient than equivalent regex patterns.\n\n### Keep Actions\n\n| Action   | Description                        |\n| -------- | ---------------------------------- |\n| `\"all\"`  | Keep all matching records          |\n| `\"none\"` | Drop all matching records          |\n| `\"N%\"`   | Sample at N% (probabilistic)       |\n| `\"N/s\"`  | Rate limit to N records per second |\n| `\"N/m\"`  | Rate limit to N records per minute |\n\n### Sampling with Sample Key\n\nFor consistent sampling (same key always produces same decision), use\n`sample_key`:\n\n```json\n{\n  \"id\": \"sample-by-trace\",\n  \"name\": \"Sample 10% of logs by trace ID\",\n  \"log\": {\n    \"match\": [{ \"log_field\": \"body\", \"contains\": \"request\" }],\n    \"keep\": \"10%\",\n    \"sample_key\": {\n      \"log_field\": \"trace_id\"\n    }\n  }\n}\n```\n\nThe sample key can reference any field or attribute:\n\n- `log_field`: Use a log field (body, trace_id, span_id, etc.)\n- `log_attribute`: Use a log record attribute\n- `resource_attribute`: Use a resource attribute\n- `scope_attribute`: Use a scope attribute\n\nWhen a sample key is configured:\n\n- Records with the same key value always get the same keep/drop decision\n- This ensures consistent sampling across distributed systems\n- If the sample key field is empty/missing, the record is kept by default\n\n### Rate Limiting\n\nRate limiting allows you to cap the number of records kept per time window:\n\n```json\n{\n  \"id\": \"rate-limit-noisy-service\",\n  \"name\": \"Rate limit logs from noisy service to 100/s\",\n  \"log\": {\n    \"match\": [\n      { \"resource_attribute\": \"service.name\", \"exact\": \"noisy-service\" }\n    ],\n    \"keep\": \"100/s\"\n  }\n}\n```\n\nRate limiting features:\n\n- **Lock-free implementation**: Uses atomic operations for thread-safe access\n  without mutexes\n- **Per-policy rate limiters**: Each policy with rate limiting gets its own\n  limiter\n- **Automatic window reset**: Windows reset inline on first request after expiry\n- **Two time windows**: Use `/s` for per-second or `/m` for per-minute limits\n\nWhen the rate limit is exceeded, records are dropped (`ResultDrop`). When under\nthe limit, records are kept (`ResultKeep`).\n\n### AND Semantics\n\nAll matchers within a single policy are AND'd together. A policy only matches\nwhen ALL of its matchers match:\n\n```json\n{\n  \"match\": [\n    { \"log_field\": \"body\", \"regex\": \"debug\" },\n    { \"log_field\": \"body\", \"regex\": \"trace\" }\n  ]\n}\n```\n\nThis policy matches logs where the body contains BOTH \"debug\" AND \"trace\".\n\n## File Provider\n\nThe file provider loads policies from a JSON file and supports hot reload:\n\n```go\nprovider := policy.NewFileProvider(\"policies.json\",\n    policy.WithPollInterval(30*time.Second),  // Check every 30 seconds\n    policy.WithOnReload(func() {\n        log.Println(\"Policies reloaded\")\n    }),\n    policy.WithOnError(func(err error) {\n        log.Printf(\"Error: %v\", err)\n    }),\n)\ndefer provider.Stop()\n```\n\n## Statistics\n\nThe registry maintains per-policy statistics with atomic counters:\n\n```go\nstats := registry.CollectStats()\nfor _, s := range stats {\n    fmt.Printf(\"Policy %s: match_hits=%d match_misses=%d\\n\",\n        s.PolicyID, s.MatchHits, s.MatchMisses)\n}\n```\n\n## TODO\n\n### Zero-Allocation Optimizations\n\nThe current implementation allocates ~16-25 objects per evaluation. To achieve\nzero-allocation evaluation:\n\n- [x] Pool `matchCounts` slice in `PolicyEngine.Evaluate()`\n- [x] Pool `disqualified` slice in `PolicyEngine.Evaluate()`\n- [x] Pool Hyperscan scratch space per-database\n- [x] Pool match result slices from `db.Scan()`\n- [x] Use dense index arrays instead of maps for policy match tracking\n- [ ] Pre-allocate result slices in hot paths\n\n### Telemetry Type Support\n\n- [x] Log policies (`log` field)\n- [x] Metric policies (`metric` field)\n- [x] Trace policies (`trace` field) with OTel-compliant consistent sampling\n\n### Provider Support\n\n- [x] File provider with hot reload\n- [ ] HTTP provider (poll-based)\n- [ ] gRPC provider (streaming)\n\n### Additional Features\n\n- [x] Nested attribute path access (e.g., `http.request.method`)\n- [x] Optimized literal matchers (`starts_with`, `ends_with`, `contains`)\n- [x] Case-insensitive matching via Hyperscan flags\n- [x] Sampling with hash-based determinism via `sample_key`\n- [x] Rate limiting support (`N/s`, `N/m`) with lock-free implementation\n- [ ] Transform actions (keep with modifications)\n- [ ] Policy validation CLI tool\n- [ ] Prometheus metrics exporter for stats\n\n## License\n\nApache 2.0 - See [LICENSE](LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusetero%2Fpolicy-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fusetero%2Fpolicy-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fusetero%2Fpolicy-go/lists"}