{"id":18576184,"url":"https://github.com/deuill/farsight","last_synced_at":"2025-05-16T00:34:45.999Z","repository":{"id":57527630,"uuid":"59708085","full_name":"deuill/farsight","owner":"deuill","description":"Fetch, filter and store arbitrary data using struct types and tags","archived":false,"fork":false,"pushed_at":"2016-06-02T11:46:35.000Z","size":19,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-09T05:10:12.976Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deuill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-26T00:52:40.000Z","updated_at":"2022-03-16T02:28:53.000Z","dependencies_parsed_at":"2022-09-18T04:40:22.561Z","dependency_job_id":null,"html_url":"https://github.com/deuill/farsight","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deuill%2Ffarsight","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deuill%2Ffarsight/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deuill%2Ffarsight/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deuill%2Ffarsight/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deuill","download_url":"https://codeload.github.com/deuill/farsight/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254447994,"owners_count":22072755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T23:23:54.996Z","updated_at":"2025-05-16T00:34:45.955Z","avatar_url":"https://github.com/deuill.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Farsight - Fetch, filter, and store arbitrary data\n\n[![API Documentation][godoc-svg]][godoc-url] [![MIT License][license-svg]][license-url] [![Build Status][build-svg]][build-url]\n\nFarsight facilitates the fetching and transformation of data from arbitrary sources into pre-defined structures, which can be further processed and serialised into other formats (such as JSON, YAML etc.).\n\nA large amount of inspiration comes from the [GoStruct](https://github.com/bfontaine/gostruct) project.\n\n## Usage\n\nFarsight is very simple to use, and only exposes a single public method, `Fetch`. For example, this is the full package for a single-page HTML scraper:\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/deuill/farsight\"\n)\n\nvar url = \"https://deuill.org\"\n\ntype Post struct {\n\tTitle string `farsight:\"h2\"`\n\tText  string `farsight:\".post-text\"`\n}\n\ntype Data struct {\n\tIntro string `farsight:\".post-text:first-of-type\"`\n\tPosts []Post `farsight:\".post-summary\"`\n}\n\nfunc main() {\n\tdata := \u0026Data{}\n\tif err := farsight.Fetch(url, data, \"html\"); err != nil {\n\t\tpanic(\"Failed to fetch URL\")\n\t}\n\n\tfmt.Println(data.Posts[0].Title) // Returns the first post's title.\n}\n```\n\nCalling the `Fetch` method on the `Data` type fills each eligible field with the correct data, as matched by the specified CSS selectors.\n\n## Overview\n\nData sourcing is handled via generic `source` types, that correspond to URIs passed to `farsight.Fetch` and allow for transparent use of different types of sources (such as local files, HTTP endpoints etc.).\n\nData transformation is handled via `parser` types, that rely on specific struct tag fields in order to fill the destination structures with the correct data. Thus, there is a direct relationship between the type of data the user expects to query against, and the `parser` type used.\n\n## License\n\nFarsight is licensed under the MIT license, the terms of which can be found in the included LICENSE file.\n\n\n[godoc-url]: https://godoc.org/github.com/deuill/farsight\n[godoc-svg]: https://godoc.org/github.com/deuill/farsight?status.svg\n\n[license-url]: https://github.com/deuill/farsight/blob/master/LICENSE\n[license-svg]: https://img.shields.io/badge/license-MIT-blue.svg\n\n[build-url]: https://travis-ci.org/deuill/farsight\n[build-svg]: https://travis-ci.org/deuill/farsight.svg?branch=master\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeuill%2Ffarsight","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeuill%2Ffarsight","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeuill%2Ffarsight/lists"}