https://github.com/mghalix/storix
Storage Unix Style for Developers
https://github.com/mghalix/storix
azure cli python sdk storage
Last synced: about 1 month ago
JSON representation
Storage Unix Style for Developers
- Host: GitHub
- URL: https://github.com/mghalix/storix
- Owner: mghalix
- License: apache-2.0
- Created: 2025-06-24T21:20:23.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-12-14T20:33:55.000Z (7 months ago)
- Last Synced: 2026-01-20T23:26:30.460Z (6 months ago)
- Topics: azure, cli, python, sdk, storage
- Language: Python
- Homepage: https://pypi.org/project/storix/
- Size: 4.55 MB
- Stars: 2
- Watchers: 0
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Storix: Storage for Unix Lovers
**A blazing-fast, secure, and developer-friendly storage abstraction for
Python.**
[](https://pypi.org/project/storix/)
[](https://github.com/mghalix/storix)
[](https://github.com/mghalix/storix/blob/main/LICENSE)
---
## 🎬 Demo
_Watch storix in action: navigating files, creating directories, and using the
interactive shell mode._
---
## 🚀 Getting Started
### 1. Install Storix
Choose the installation that fits your needs:
```bash
# Basic (local filesystem only)
uv add storix
# With CLI tools
uv add "storix[cli]"
# With Azure support
uv add "storix[azure]"
# Everything included
uv add "storix[all]"
```
### 2. Configure (Optional)
Create a `.env` file for custom settings:
```bash
cp env.example .env
# Edit .env with your preferences
```
### 3. Start Using
```python
from storix import get_storage
fs = get_storage()
fs.touch("hello.txt", "Hello, Storix!")
```
---
## 💡 Quick Examples
```python
# Basic file operations
from storix import get_storage
fs = get_storage()
fs.touch("hello.txt", "Hello, World!")
content = fs.cat("hello.txt").decode()
print(content) # Hello, World!
# List files
files = fs.ls("/")
print(files) # [StorixPath('hello.txt')]
# Create directories
fs.mkdir("mydata", parents=True)
fs.touch("mydata/config.json", '{"key": "value"}')
```
```python
# Async operations
import asyncio
from storix.aio import get_storage
async def main():
fs = get_storage()
await fs.touch("async.txt", "Async is easy!")
content = await fs.cat("async.txt")
print(content.decode()) # Async is easy!
asyncio.run(main())
```
### More operations
```python
from storix import get_storage
fs = get_storage()
fs.mkdir("logs", parents=True)
# Stream writes: pass any iterable/generator of chunks
fs.echo((f"line {i}\n" for i in range(3)), "logs/app.log")
fs.echo("another line\n", "logs/app.log", mode="a")
# Metadata and sizes
info = fs.stat("logs/app.log")
print(info.file_kind, info.size) # file ...
print(fs.du("logs/app.log")) # bytes
# Tree view
print(fs.tree("/", abs=False))
```
---
## 🛠️ Quickstart
### Install
```bash
# Basic installation (local filesystem only)
uv add storix
# or
pip install storix
# With CLI support
uv add "storix[cli]"
# or
pip install "storix[cli]"
# With Azure support
uv add "storix[azure]"
# or
pip install "storix[azure]"
# With everything (CLI + Azure)
uv add "storix[all]"
# or
pip install "storix[all]"
```
### Development Installation
```bash
# Clone the repository
git clone https://github.com/mghalix/storix.git
cd storix
# Install dependencies and setup development environment
uv sync
uv run pip install -e .
```
### CLI Configuration (Optional)
Create a `.env` file to customize your storage settings. Storix will
automatically find your `.env` file in:
1. **Current directory** (where you run the command)
2. **Parent directories** (searches up the directory tree)
3. **Home directory** (`~/.storix/.env`)
```bash
# Copy the example configuration
cp env.example .env
```
Then edit `.env` with your preferences:
```env
# Storage Provider (local or azure)
STORAGE_PROVIDER=local
# Initial paths for different providers
STORAGE_INITIAL_PATH=.
STORAGE_INITIAL_PATH_LOCAL=/path/to/your/data
STORAGE_INITIAL_PATH_AZURE=/your/azure/path
# Azure credentials (only needed for azure provider)
ADLSG2_CONTAINER_NAME=my-container
ADLSG2_ACCOUNT_NAME=my-storage-account
ADLSG2_TOKEN=your-sas-token-or-account-key
```
**Smart Discovery**: You can run `sx` from any subdirectory of your project and
it will find your `.env` file automatically!
**Path Configuration**:
- `STORAGE_INITIAL_PATH` is the shared default path for all providers
- `STORAGE_INITIAL_PATH_LOCAL` and `STORAGE_INITIAL_PATH_AZURE` override the
shared path for specific providers
- Provider-specific paths take precedence over the shared path
### CLI Usage
```bash
# List files in current directory
sx ls
# Interactive shell
sx
# Switch providers
sx -p local ls
# Create directory
sx mkdir mydata
# Upload file
sx upload local_file.txt remote_file.txt
```
### Basic Usage
```python
from storix import LocalFilesystem
fs = LocalFilesystem("/tmp/mydata")
fs.touch("hello.txt", "Hello, world!")
print(fs.cat("hello.txt").decode())
```
### Sandboxed Usage
```python
from storix import LocalFilesystem
fs = LocalFilesystem("/tmp/sandbox", sandboxed=True)
fs.touch("/secret.txt", "sandboxed!")
print(fs.ls("/")) # [StorixPath('secret.txt')]
```
### Async Usage
```python
from storix.aio import LocalFilesystem
fs = LocalFilesystem("/tmp/mydata")
await fs.touch("async.txt", "Async is easy!")
print((await fs.cat("async.txt")).decode())
```
---
## 🗄️ Supported Backends
| Backend | Sync | Async | Sandboxed | Status |
| -------------------------------- | ---- | ----- | --------- | ------- |
| **Local Filesystem** | ✅ | ✅ | ✅ | Beta |
| **Azure Data Lake Storage Gen2** | ✅ | ✅ | ✅ | Beta |
| **AWS S3** | ❌ | ❌ | ❌ | Planned |
| **Google Cloud Storage** | ❌ | ❌ | ❌ | Planned |
## 🔌 Integrations
### FastAPI Integration
Storix integrates seamlessly with FastAPI for dependency injection and async
file operations:
```python
from typing import AsyncGenerator
from fastapi import Depends, FastAPI
from typing_extensions import Annotated
import storix as sx
app = FastAPI()
async def get_fs() -> AsyncGenerator[sx.Storage, None]:
"""Dependency to provide storage service."""
async with sx.get_storage() as fs:
yield fs
StorageDep = Annotated[sx.Storage, Depends(get_fs)]
@app.get("/files/{file_path:path}")
async def read_file(file_path: str, fs: StorageDep):
"""Read a file from storage."""
content = await fs.cat(file_path)
return {"content": content.decode()}
@app.post("/files/{file_path:path}")
async def write_file(file_path: str, content: str, fs: StorageDep):
"""Write content to a file."""
success = await fs.touch(file_path, content)
return {"success": success}
```
**Configuration**: Set `STORAGE_PROVIDER` in your `.env` file to choose your backend:
```sh
# For local filesystem
STORAGE_PROVIDER=local
STORAGE_INITIAL_PATH_LOCAL=/path/to/local/data
# For Azure Data Lake
STORAGE_PROVIDER=azure
STORAGE_INITIAL_PATH_AZURE=/your/azure/path
ADLSG2_ACCOUNT_NAME=your_account
ADLSG2_TOKEN=your_token
ADLSG2_CONTAINER_NAME=your_container
```
> Notes
>
> > **Local:**
> >
> > - if `STORAGE_INITIAL_PATH` set to "~", gets mapped to your home directory
> > - if `STORAGE_INITIAL_PATH` set to ".", gets mapped to your current working directory
>
> > **Azure:**
> >
> > - if `STORAGE_INITIAL_PATH` set to "~", gets mapped to root "/"
> > - if `STORAGE_INITIAL_PATH` set to ".", gets mapped to root "/"
## 🖼️ Media Support
Storix makes it effortless to store and retrieve images or any binary media
files—just use Python's `bytes` type.
### Write an Image to Storage
```python
from storix import get_storage
fs = get_storage()
with open("my_photo.jpg", "rb") as img_file:
img_bytes = img_file.read()
fs.touch("photos/profile.jpg", img_bytes)
```
### Read an Image from Storage
```python
from storix import get_storage
fs = get_storage()
img_bytes = fs.cat("photos/profile.jpg")
# Now you can use img_bytes with PIL, OpenCV, or send it in a web response...
```
### Async Example
```python
from storix.aio import get_storage
fs = get_storage()
async with fs:
img_bytes = await fs.cat("photos/profile.jpg")
await fs.touch("photos/backup.jpg", img_bytes)
```
**It's that simple:**
- No special APIs for media—just use `bytes` for any file type.
- Works for images, audio, video, or any binary data.
- Seamless support for both sync and async code.
## ✨ Features
- **Sync & Async APIs:**
Use `from storix import LocalFilesystem` or `from storix.aio import
LocalFilesystem` — just add `await` for async!
- **Sandboxing:**
Restrict all file operations to a virtual root, blocking path traversal and
symlink escapes.
- **Consistent Path Handling:**
Absolute and relative paths, `cd`, `ls`, and more — just like a shell.
- **Decorator Support:**
Automatic path conversion for your own functions.
- **Easy Migration:**
Switch from sync to async in seconds.
- **CLI Tool:**
Manage files and sandboxes from your terminal.
---
## 🚀 Why storix
- **Unified API:** Seamless sync and async support with identical interfaces.
- **Rock-solid sandboxing:** Secure your file operations with robust path
traversal and symlink protection.
- **Plug-and-play:** Instantly switch between local and cloud (Azure) backends.
- **CLI included:** Script and automate storage tasks from the command line.
- **Extensible:** Clean, modern codebase ready for new providers and features.
- **Tested & Secure:** Comprehensive test suite and security-first design.
---
## 🔒 Security
- **Path Traversal Protection:**
All `../` and symlink escapes are blocked.
- **Virtual Root:**
Sandboxed mode makes `/` map to your chosen directory.
- **Symlink Safety:**
Symlinks are resolved and validated before access.
---
## 🙋 Need Help or Found a Bug?
If you run into any issues, have questions, or spot a bug, please [open an
issue](https://github.com/mghalix/storix/issues) on the GitHub repository.
We welcome all feedback and contributions—your input helps make Storix better
for everyone!
---
## 🧑💻 Contributing
Contributions are welcome and appreciated. If you have ideas for improvements,
new features, or spot something that could be better, feel free to open an
issue or a pull request.
Storix aims to be robust, secure, and easy to use. We value clear code, good
documentation, and thoughtful discussion. Whether you want to add a new
provider, enhance the CLI, share real-world examples, or review security, your
input is valued.
To get started:
- Star and fork the repository
- Open an issue to discuss your idea or report a bug
- Submit a pull request with your proposed changes
For more details, see the documentation or join the discussion on GitHub.
---
## 📣 Spread the Word
If you like `storix` ⭐ star the repo, share it, and help us grow the community!
---
## 📚 Documentation
- [Sandbox Implementation Details](docs/SANDBOX_IMPLEMENTATION.md)
- [Async Migration Guide](docs/ASYNC_MIGRATION.md)
- [Release Notes](release-notes.md)
---
## 🧪 Testing
```bash
# Run all tests
uv run pytest
# Run with coverage
uv run pytest --cov=storix
# Run specific test categories
uv run pytest -m "not slow" # Skip slow tests
uv run pytest -m integration # Run only integration tests
```
## 🗺️ Roadmap
Planned and upcoming features:
- **`mv` support for directories in Azure sync/async providers**
- **`du` (Disk Usage):**
- Calculate and display the size of files and directories, similar to the
Unix `du` command.
- **`stat` (File Metadata):**
- Retrieve and display detailed file metadata (size, permissions, timestamps, etc.),
similar to the Unix `stat` command.
- Normalize the result using a shared model between the sync/async interfaces.
- `touch`, `mkdir`, `rm`, `rmdir` should take **`*paths`** instead of a single
path arg
- **Storage Tree Structure (`tree`):**
- Not just for visualization—enables programmatic operations such as copying,
iterating, or transforming entire directory trees easily.
- Will likely use a tree data structure (e.g.,
[anytree](https://anytree.readthedocs.io/), TBD) to represent and manipulate
storage hierarchies efficiently.
- Should be streaming for performance (leverage generators)
- **Advanced CLI Features:**
- Enhanced command-line tools for scripting and automation.
- **Performance Improvements:**
- Further optimize for speed and scalability.
- **Additional Cloud Providers:**
- Add support for more cloud storage backends (e.g., S3, GCS). Local
filesystem support is already included.
- **Auto Completions in the storix REPL (Interactive Shell)**
- **Improve validation strategy**
- Reduce code duplication in validation logic between sync/async providers
- Evaluate shared utility functions, mixins, or decorator patterns
- Focus on common operations like `rm`, `rmdir`, `touch` validation
- Ensure consistent error messages and behavior across providers
- **Storage Connection Pool**
- Implement connection pooling for cloud storage providers to improve performance
- Reuse connections across requests to reduce latency and overhead
- Configurable pool sizes and connection timeouts
---
## 📝 License
Storix is licensed under the Apache 2.0 License