https://github.com/fasouto/webhdfspy
Python wrapper to access Hadoop HDFS REST API
https://github.com/fasouto/webhdfspy
hadoop-filesystem hdfs python wrapper
Last synced: 9 months ago
JSON representation
Python wrapper to access Hadoop HDFS REST API
- Host: GitHub
- URL: https://github.com/fasouto/webhdfspy
- Owner: fasouto
- License: mit
- Created: 2012-03-07T02:06:17.000Z (almost 14 years ago)
- Default Branch: master
- Last Pushed: 2016-10-26T20:24:23.000Z (about 9 years ago)
- Last Synced: 2025-03-29T12:11:25.180Z (10 months ago)
- Topics: hadoop-filesystem, hdfs, python, wrapper
- Language: Python
- Homepage: http://webhdfspy.readthedocs.org/en/latest/
- Size: 38.1 KB
- Stars: 8
- Watchers: 3
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
=========
webhdfspy
=========
.. image:: https://badge.fury.io/py/webhdfspy.svg
:target: http://badge.fury.io/py/webhdfspy
A Python 2/3 wrapper library to access `Hadoop WebHDFS REST API `_
Installation
============
To install webhdfspy from PyPI::
$ pip install webhdfspy
Python versions
===============
webhdfspy supports Python 2.7 and 3.4
Usage
=====
::
>>> import webhdfspy
>>> webHDFS = webhdfspy.WebHDFSClient("localhost", 50070, "username")
>>> print(webHDFS.listdir('/'))
[]
>>> webHDFS.mkdir('/foo')
True
>>> print(webHDFS.listdir('/'))
[{u'group': u'supergroup', u'permission': u'755', u'blockSize': 0, u'accessTime': 0, u'pathSuffix': u'foo', u'modificationTime': 1429805040695, u'replication': 0, u'length': 0, u'childrenNum': 0, u'owner': u'username', u'storagePolicy': 0, u'type': u'DIRECTORY', u'fileId': 16387}]
>>> print webHDFS.create('/foo/foo.txt', "just put some text here", True)
True
>>> print webHDFS.open('/pywebhdfs_test/foo.txt')
just put some text here
>>> webHDFS.remove('/foo')
True
>>> print(webHDFS.listdir('/'))
[]
Documentation
=============
http://webhdfspy.readthedocs.org/en/latest/
Hadoop configuration
====================
To enable WebHDFS in Hadoop add this to your $HADOOP_DIR/conf/hdfs-site.xml: ::
dfs.webhdfs.enabled
true
To enable append on HDFS you need to configure your hdfs-site.xml as follows: ::
dfs.support.append
true
More about WebHDFS: https://hadoop.apache.org/docs/r1.0.4/webhdfs.html