{"id":26895179,"url":"https://github.com/pyfunc/memfs","last_synced_at":"2026-05-16T17:31:34.277Z","repository":{"id":285077051,"uuid":"956996063","full_name":"pyfunc/memfs","owner":"pyfunc","description":"Moduł memfs implementuje wirtualny system plików w pamięci. Ten moduł zapewnia interfejs zgodny z modułem os i zapewnia operacje na plikach i katalogach przechowywanych w pamięci RAM, a nie na dysku.","archived":false,"fork":false,"pushed_at":"2025-03-29T12:43:05.000Z","size":88,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-03-27T21:37:39.236Z","etag":null,"topics":["cli","filesystem","memfs","shell","storage","virual"],"latest_commit_sha":null,"homepage":"http://memfs.pyfunc.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pyfunc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-29T09:58:52.000Z","updated_at":"2025-06-29T23:16:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"1eeee56d-0d93-4333-aa01-d7e39d3c9cf0","html_url":"https://github.com/pyfunc/memfs","commit_stats":null,"previous_names":["pyfunc/memfs"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/pyfunc/memfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyfunc%2Fmemfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyfunc%2Fmemfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyfunc%2Fmemfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyfunc%2Fmemfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pyfunc","download_url":"https://codeload.github.com/pyfunc/memfs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pyfunc%2Fmemfs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33111613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","filesystem","memfs","shell","storage","virual"],"created_at":"2025-04-01T01:54:25.160Z","updated_at":"2026-05-16T17:31:34.271Z","avatar_url":"https://github.com/pyfunc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# memfs - A Python Virtual File System in Memory\n\nA Python module that implements a virtual file system in memory. This module provides an interface compatible with the standard `os` module and enables operations on files and directories stored in RAM rather than on disk.\n\n## Overview\n\n`memfs` is designed to provide a fast, isolated file system environment for applications that need temporary file operations without the overhead of disk I/O. It's particularly useful for testing, data processing pipelines, and applications that need to manipulate files without affecting the host system.\n\n## Features\n\n- Complete in-memory file system implementation\n- API compatible with Python's standard `os` module\n- File and directory operations (create, read, write, delete, rename)\n- Path manipulation and traversal\n- File-like objects with context manager support\n- gRPC service generation for pipeline components\n- Encryption and compression support via extended filesystem\n- State persistence between CLI invocations\n- No disk I/O overhead\n- Isolated from the host file system\n\n## Installation\n\n```bash\npip install memfs\n```\n\nOr install from source:\n\n```bash\ngit clone https://github.com/pyfunc/memfs.git\ncd memfs\n```\n\nFor development setup:\n\n```bash\n# Create a virtual environment\npython3 -m venv .venv\nsource .venv/bin/activate  # On Linux/macOS\n# .venv\\Scripts\\activate  # On Windows\n\n# Install in development mode\npip install --upgrade pip\npip install setuptools wheel\npip install -e .\n\n# Build the package\npython -m build\n```\n\n## Basic Usage Examples\n\n### Basic File Operations\n\n```python\nfrom memfs import create_fs\n\n# Create a file system instance\nfs = create_fs()\n\n# Write to a file\nfs.writefile('/hello.txt', 'Hello, world!')\n\n# Read from a file\ncontent = fs.readfile('/hello.txt')\nprint(content)  # Outputs: Hello, world!\n\n# Check if a file exists\nif fs.exists('/hello.txt'):\n    print('File exists!')\n\n# Create directories\nfs.makedirs('/path/to/directory')\n\n# List directory contents\nfiles = fs.listdir('/path/to')\n```\n\n### Using File-Like Objects\n\n```python\nfrom memfs import create_fs\n\nfs = create_fs()\n\n# Write using a file-like object\nwith fs.open('/data.txt', 'w') as f:\n    f.write('Line 1\\n')\n    f.write('Line 2\\n')\n\n# Read using a file-like object\nwith fs.open('/data.txt', 'r') as f:\n    for line in f:\n        print(line.strip())\n```\n\n### Directory Operations\n\n```python\nfrom memfs import create_fs\n\nfs = create_fs()\n\n# Create nested directories\nfs.makedirs('/a/b/c')\n\n# Walk the directory tree\nfor root, dirs, files in fs.walk('/'):\n    print(f\"Directory: {root}\")\n    print(f\"Subdirectories: {dirs}\")\n    print(f\"Files: {files}\")\n```\n\n## Advanced Usage Examples\n\n### Data Processing Pipeline\n\n```python\nfrom memfs import create_fs\nimport json\nimport csv\n\nfs = create_fs()\n\n# Create directories\nfs.makedirs('/data/raw', exist_ok=True)\nfs.makedirs('/data/processed', exist_ok=True)\n\n# Write CSV data\nwith fs.open('/data/raw/input.csv', 'w', newline='') as f:\n    writer = csv.writer(f)\n    writer.writerows([\n        ['id', 'name', 'value'],\n        [1, 'Alpha', 100],\n        [2, 'Beta', 200]\n    ])\n\n# Process CSV to JSON\nwith fs.open('/data/raw/input.csv', 'r', newline='') as f:\n    reader = csv.DictReader(f)\n    data = [row for row in reader]\n\n# Transform and save the data\nfor item in data:\n    item['value'] = int(item['value'])\n    item['double_value'] = item['value'] * 2\n\nwith fs.open('/data/processed/output.json', 'w') as f:\n    json.dump(data, f, indent=2)\n```\n\n### Parallel Processing\n\n```python\nfrom memfs import create_fs\nimport json\nimport concurrent.futures\n\nfs = create_fs()\nfs.makedirs('/parallel/input', exist_ok=True)\nfs.makedirs('/parallel/output', exist_ok=True)\n\n# Create input files\nfor i in range(10):\n    fs.writefile(f'/parallel/input/file_{i}.json', json.dumps({'id': i}))\n\ndef process_file(filename):\n    with fs.open(f'/parallel/input/{filename}', 'r') as f:\n        data = json.loads(f.read())\n    \n    # Process data\n    data['processed'] = True\n    \n    with fs.open(f'/parallel/output/processed_{filename}', 'w') as f:\n        f.write(json.dumps(data, indent=2))\n    \n    return data['id']\n\n# Process files in parallel\nwith concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:\n    futures = {executor.submit(process_file, f): f for f in fs.listdir('/parallel/input')}\n    for future in concurrent.futures.as_completed(futures):\n        file_id = future.result()\n        print(f\"Processed file ID: {file_id}\")\n```\n\n### Encrypted and Compressed Files\n\n```python\nfrom memfs.examples.custom_filesystem import create_extended_fs\n\n# Create extended filesystem with encryption and compression\nfs = create_extended_fs()\n\n# Write to an encrypted file\nwith fs.open_encrypted('/secret.txt', 'w', password='mysecret') as f:\n    f.write('This is sensitive information')\n\n# Read from an encrypted file\nwith fs.open_encrypted('/secret.txt', 'r', password='mysecret') as f:\n    content = f.read()\n    print(content)\n\n# Write to a compressed file (good for large text)\nwith fs.open_compressed('/compressed.txt', 'w', compression_level=9) as f:\n    f.write('This content will be compressed ' * 1000)\n\n# Check the file sizes\nnormal_size = len(fs.readfile('/secret.txt'))\ncompressed_size = len(fs._FS_DATA['files']['/compressed.txt'])\nprint(f\"Compression ratio: {normal_size / compressed_size:.2f}x\")\n```\n\n### gRPC Service Pipeline\n\n```python\nfrom memfs import create_fs\nfrom memfs.api import DynamicgRPCComponent, PipelineOrchestrator\n\n# Define transformation functions\ndef transform_data(data):\n    if isinstance(data, dict):\n        data['transformed'] = True\n    return data\n\ndef format_data(data):\n    if isinstance(data, dict):\n        data['formatted'] = True\n    return data\n\n# Create virtual directories\nfs = create_fs()\nfs.makedirs('/proto/transform', exist_ok=True)\nfs.makedirs('/proto/format', exist_ok=True)\nfs.makedirs('/generated/transform', exist_ok=True)\nfs.makedirs('/generated/format', exist_ok=True)\n\n# Create components\ntransform_component = DynamicgRPCComponent(\n    transform_data,\n    proto_dir=\"/proto/transform\",\n    generated_dir=\"/generated/transform\",\n    port=50051\n)\n\nformat_component = DynamicgRPCComponent(\n    format_data,\n    proto_dir=\"/proto/format\",\n    generated_dir=\"/generated/format\",\n    port=50052\n)\n\n# Create and execute pipeline\npipeline = PipelineOrchestrator()\npipeline.add_component(transform_component)\npipeline.add_component(format_component)\n\nresult = pipeline.execute_pipeline({\"input\": \"data\"})\nprint(result)  # {\"input\": \"data\", \"transformed\": true, \"formatted\": true}\n```\n\n## Command-line Interface\n\n`memfs` provides a command-line interface for basic file operations. The CLI maintains state between invocations by default, storing filesystem data in `~/.memfs_state.json`.\n\n### Basic CLI Usage\n\n```bash\n# Initialize a new filesystem (clears any existing state)\nmemfs init\n\n# Display filesystem as a tree\nmemfs tree /\n\n# Create a directory with parents\nmemfs mkdir -p /data/subdir\n\n# Create an empty file\nmemfs touch /data/hello.txt\n\n# Write content to a file\nmemfs write /data/hello.txt \"Hello, virtual world!\"\n\n# Read file content\nmemfs read /data/hello.txt\n\n# Dump filesystem content as JSON\nmemfs dump\n```\n\n### Interactive Shell Mode\n\nFor a more interactive experience, you can use the shell mode:\n\n```bash\nmemfs shell\n```\n\nThis launches an interactive shell where you can run multiple commands without restarting the CLI:\n\n```\nmemfs\u003e mkdir -p /data\nmemfs\u003e touch /data/hello.txt\nmemfs\u003e write /data/hello.txt \"Hello from shell mode!\"\nmemfs\u003e tree /\nmemfs\u003e exit\n```\n\n### CLI State Management\n```\nmkdir -p /data\ntouch /data/hello.txt\nwrite /data/hello.txt \"Hello from shell mode!\"\ntree /\nexit\n```\n\n### CLI State Management\n\nThe CLI stores state in `~/.memfs_state.json`. If you're experiencing issues with state persistence:\n\n```bash\n# Check if state file exists\nls -la ~/.memfs_state.json\n\n# Reset state by initializing a new filesystem\nmemfs init\n\n# Or manually create an empty state file\necho '{\"files\": {}, \"dirs\": [\"/\"]}' \u003e ~/.memfs_state.json\n```\n\n### Creating a Custom CLI Command\n\nYou can create a custom script to use memfs in a single process:\n\n```python\n#!/usr/bin/env python\nfrom memfs import create_fs\n\nfs = create_fs()\nfs.makedirs('/data', exist_ok=True)\nfs.writefile('/data/hello.txt', 'Hello, world!')\n\nprint(\"Filesystem contents:\")\nfor root, dirs, files in fs.walk('/'):\n    print(f\"Directory: {root}\")\n    for d in dirs:\n        print(f\"  Dir: {d}\")\n    for f in files:\n        print(f\"  File: {f}\")\n```\n\n## Project Structure\n\n```\nmemfs/\n├── setup.py          # Package installation configuration\n├── setup.cfg         # Setup configuration\n├── README.md         # Project documentation\n├── src/              # Source code\n│   └── memfs/        # Main package\n│       ├── __init__.py     # Basic component imports\n│       ├── _version.py     # Version information\n│       ├── memfs.py        # Virtual filesystem implementation\n│       ├── api.py          # gRPC service generation module\n│       └── cli.py          # Command-line interface\n├── tests/            # Unit tests\n│   ├── __init__.py\n│   ├── test_memfs.py       # Tests for memfs module\n│   └── test_api.py         # Tests for API module\n└── examples/         # Usage examples\n    ├── basic_usage.py      # Basic operations\n    └── advanced_usage.py   # Advanced scenarios\n```\n\n## API Reference\n\n### MemoryFS Class\n\n- `open(path, mode='r')` - Open a file\n- `makedirs(path, exist_ok=False)` - Create directories recursively\n- `mkdir(path, mode=0o777)` - Create a directory\n- `exists(path)` - Check if a path exists\n- `isfile(path)` - Check if a path is a file\n- `isdir(path)` - Check if a path is a directory\n- `listdir(path)` - List directory contents\n- `walk(top)` - Walk through directories recursively\n- `remove(path)` - Remove a file\n- `rmdir(path)` - Remove an empty directory\n- `rename(src, dst)` - Rename a file or directory\n- `readfile(path)` - Read an entire file\n- `writefile(path, data)` - Write data to a file\n- `readfilebytes(path)` - Read a file's contents as bytes\n- `writefilebytes(path, data)` - Write binary content to a file\n\n### Extended MemoryFS Class\n\n- `open_encrypted(path, mode='r', password='')` - Open an encrypted file\n- `open_compressed(path, mode='r', compression_level=9)` - Open a compressed file\n- `set_metadata(path, metadata)` - Set metadata for a file\n- `get_metadata(path)` - Get metadata for a file\n- `find(pattern, start_path='/')` - Find files matching a pattern\n- `search_content(text, extensions=None, start_path='/')` - Search for files containing text\n- `backup(path, backup_dir='/backup')` - Create a backup of a file or directory\n\n### API Module\n\n- `DynamicgRPCComponent` - Create a gRPC service from a function\n- `PipelineOrchestrator` - Orchestrate multiple components into a pipeline\n- `ApiFuncConfig` - Configuration for gRPC services\n- `ApiFuncFramework` - Framework for creating gRPC services\n\n## Use Cases\n\n- **Unit testing** - Test file operations without touching the disk\n- **Data processing pipelines** - Process data through multiple stages in memory\n- **Microservices** - Create gRPC services from Python functions\n- **Sandboxed environments** - Run file operations in an isolated environment\n- **Performance optimization** - Avoid disk I/O overhead for temporary operations\n- **Secure storage** - Encrypt sensitive data in memory\n- **Containerized applications** - Reduce container size by using in-memory storage\n\n## License\n\nApache-2.0","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyfunc%2Fmemfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpyfunc%2Fmemfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpyfunc%2Fmemfs/lists"}