{"id":13575902,"url":"https://github.com/colinmarc/hdfs","last_synced_at":"2025-05-12T13:12:56.422Z","repository":{"id":21636701,"uuid":"24957315","full_name":"colinmarc/hdfs","owner":"colinmarc","description":"A native go client for HDFS","archived":false,"fork":false,"pushed_at":"2025-01-22T22:13:07.000Z","size":2216,"stargazers_count":1389,"open_issues_count":50,"forks_count":351,"subscribers_count":36,"default_branch":"master","last_synced_at":"2025-05-12T13:12:41.648Z","etag":null,"topics":["commandline","go","hdfs"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/colinmarc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-10-08T19:37:57.000Z","updated_at":"2025-05-11T22:41:37.000Z","dependencies_parsed_at":"2023-01-13T21:36:18.654Z","dependency_job_id":"4cefa5bd-5289-4d83-a9e6-07dfdcd5194a","html_url":"https://github.com/colinmarc/hdfs","commit_stats":{"total_commits":415,"total_committers":42,"mean_commits":9.880952380952381,"dds":0.3686746987951808,"last_synced_commit":"f19ae578cc5ebdc1bf16dcd4e774c6498f8f9121"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinmarc%2Fhdfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinmarc%2Fhdfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinmarc%2Fhdfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/colinmarc%2Fhdfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/colinmarc","download_url":"https://codeload.github.com/colinmarc/hdfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253745195,"owners_count":21957319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commandline","go","hdfs"],"created_at":"2024-08-01T15:01:05.257Z","updated_at":"2025-05-12T13:12:56.312Z","avatar_url":"https://github.com/colinmarc.png","language":"Go","funding_links":[],"categories":["Go","Libraries and Tools","Repositories"],"sub_categories":[],"readme":"\u003cimg src=\"docs/gopher.png\" alt=\"gopher\" align=\"right\" width=\"200\"/\u003e\n\nHDFS for Go\n===========\n\n[![GoDoc](https://godoc.org/github.com/colinmarc/hdfs/web?status.svg)](https://godoc.org/github.com/colinmarc/hdfs/v2) [![build](https://github.com/colinmarc/hdfs/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/colinmarc/hdfs/actions/workflows/tests.yml)\n\n\u003e [!IMPORTANT]\n\u003e Maintainer(s) wanted! If you or your company uses this in production (I don't\n\u003e anymore) and is interested in taking the project over, please get in touch via\n\u003e email.\n\nThis is a native golang client for hdfs. It connects directly to the namenode using\nthe protocol buffers API.\n\nIt tries to be idiomatic by aping the stdlib `os` package, where possible, and\nimplements the interfaces from it, including `os.FileInfo` and `os.PathError`.\n\nHere's what it looks like in action:\n\n```go\nclient, _ := hdfs.New(\"namenode:8020\")\n\nfile, _ := client.Open(\"/mobydick.txt\")\n\nbuf := make([]byte, 59)\nfile.ReadAt(buf, 48847)\n\nfmt.Println(string(buf))\n// =\u003e Abominable are the tumblers into which he pours his poison.\n```\n\nFor complete documentation, check out the [Godoc][1].\n\nThe `hdfs` Binary\n-----------------\n\nAlong with the library, this repo contains a commandline client for HDFS. Like\nthe library, its primary aim is to be idiomatic, by enabling your favorite unix\nverbs:\n\n\n    $ hdfs --help\n    Usage: hdfs COMMAND\n    The flags available are a subset of the POSIX ones, but should behave similarly.\n\n    Valid commands:\n      ls [-lah] [FILE]...\n      rm [-rf] FILE...\n      mv [-fT] SOURCE... DEST\n      mkdir [-p] FILE...\n      touch [-amc] FILE...\n      chmod [-R] OCTAL-MODE FILE...\n      chown [-R] OWNER[:GROUP] FILE...\n      cat SOURCE...\n      head [-n LINES | -c BYTES] SOURCE...\n      tail [-n LINES | -c BYTES] SOURCE...\n      du [-sh] FILE...\n      checksum FILE...\n      get SOURCE [DEST]\n      getmerge SOURCE DEST\n      put SOURCE DEST\n\nSince it doesn't have to wait for the JVM to start up, it's also a lot faster\n`hadoop -fs`:\n\n    $ time hadoop fs -ls / \u003e /dev/null\n\n    real  0m2.218s\n    user  0m2.500s\n    sys 0m0.376s\n\n    $ time hdfs ls / \u003e /dev/null\n\n    real  0m0.015s\n    user  0m0.004s\n    sys 0m0.004s\n\nBest of all, it comes with bash tab completion for paths!\n\nInstalling the commandline client\n---------------------------------\n\nGrab a tarball from the [releases page](https://github.com/colinmarc/hdfs/releases)\nand unzip it wherever you like.\n\nTo configure the client, make sure one or both of these environment variables\npoint to your Hadoop configuration (`core-site.xml` and `hdfs-site.xml`). On\nsystems with Hadoop installed, they should already be set.\n\n    $ export HADOOP_HOME=\"/etc/hadoop\"\n    $ export HADOOP_CONF_DIR=\"/etc/hadoop/conf\"\n\nTo install tab completion globally on linux, copy or link the `bash_completion`\nfile which comes with the tarball into the right place:\n\n    $ ln -sT bash_completion /etc/bash_completion.d/gohdfs\n\nBy default on non-kerberized clusters, the HDFS user is set to the\ncurrently-logged-in user. You can override this with another environment\nvariable:\n\n    $ export HADOOP_USER_NAME=username\n\nUsing the commandline client with Kerberos authentication\n---------------------------------------------------------\n\nLike `hadoop fs`, the commandline client expects a `ccache` file in the default\nlocation: `/tmp/krb5cc_\u003cuid\u003e`. That means it should 'just work' to use `kinit`:\n\n    $ kinit bob@EXAMPLE.com\n    $ hdfs ls /\n\nIf that doesn't work, try setting the `KRB5CCNAME` environment variable to\nwherever you have the `ccache` saved.\n\nCompatibility\n-------------\n\nThis library uses \"Version 9\" of the HDFS protocol, which means it should work\nwith hadoop distributions based on 2.2.x and above, as well as 3.x.\n\nAcknowledgements\n----------------\n\nThis library is heavily indebted to [snakebite][3].\n\n[1]: https://godoc.org/github.com/colinmarc/hdfs\n[2]: https://golang.org/doc/install\n[3]: https://github.com/spotify/snakebite\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinmarc%2Fhdfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcolinmarc%2Fhdfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcolinmarc%2Fhdfs/lists"}