{"id":21823672,"url":"https://github.com/godcong/mmap","last_synced_at":"2026-02-22T07:34:03.695Z","repository":{"id":224723860,"uuid":"761072235","full_name":"godcong/mmap","owner":"godcong","description":"the go mmap and share memory package","archived":false,"fork":false,"pushed_at":"2025-04-07T06:53:34.000Z","size":72,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-07T07:34:23.032Z","etag":null,"topics":["darwin","file","go","io","linux","memory","mmap","share","tcp","udp","windows"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/godcong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-21T07:26:01.000Z","updated_at":"2025-04-07T06:53:37.000Z","dependencies_parsed_at":"2024-04-07T09:27:09.382Z","dependency_job_id":"c9f76c36-ebba-44d9-8380-b9233d79c0e0","html_url":"https://github.com/godcong/mmap","commit_stats":null,"previous_names":["godcong/mmap"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/godcong%2Fmmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/godcong%2Fmmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/godcong%2Fmmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/godcong%2Fmmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/godcong","download_url":"https://codeload.github.com/godcong/mmap/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248823928,"owners_count":21167339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["darwin","file","go","io","linux","memory","mmap","share","tcp","udp","windows"],"created_at":"2024-11-27T17:36:02.125Z","updated_at":"2026-02-22T07:34:03.689Z","avatar_url":"https://github.com/godcong.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MMAP\n\n[![GitHub release](https://img.shields.io/github/release/godcong/mmap.svg)](https://github.com/godcong/mmap/releases)\n[![go.dev reference](https://img.shields.io/badge/go.dev-reference-007d9c?logo=go\u0026logoColor=white\u0026style=flat)](https://pkg.go.dev/github.com/godcong/mmap)\n[![codecov](https://codecov.io/gh/godcong/mmap/branch/main/graph/badge.svg)](https://codecov.io/gh/godcong/mmap)\n[![GoDoc](https://godoc.org/github.com/godcong/mmap?status.svg)](http://godoc.org/github.com/godcong/mmap)\n[![License](https://img.shields.io/github/license/godcong/mmap.svg)](https://github.com/godcong/mmap/blob/main/LICENSE)\n[![Go Report Card](https://goreportcard.com/badge/github.com/godcong/mmap)](https://goreportcard.com/report/github.com/godcong/mmap)\n\nThe `MMAP` package provides a safe and efficient syscall interface for memory mapping operations, supporting both file mapping and shared memory.\n\n## Features\n\n- **File Memory Mapping**: Memory-mapped files with read/write access\n- **Shared Memory**: Cross-process shared memory support\n- **Cross-Platform**: Supports Darwin (macOS), Linux, and Windows\n- **Standard Interfaces**: Implements Go's `io.Reader`, `io.Writer`, `io.ReaderAt`, `io.WriterAt`, `io.Seeker`, and `io.Closer` interfaces\n- **High Performance**: Optimized for high-throughput data operations\n\n## Installation\n\n```bash\ngo get github.com/godcong/mmap@latest\n```\n\n## Requirements\n\n- Go 1.23 or later\n- Supported platforms: Windows, Linux, macOS (Darwin)\n\n## Quick Start\n\n### File Memory Mapping\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \"github.com/godcong/mmap\"\n)\n\nfunc main() {\n    // Read-only file mapping\n    file, err := mmap.Open(\"example.txt\")\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer file.Close()\n\n    // Read data\n    data := make([]byte, 1024)\n    n, err := file.Read(data)\n    if err != nil {\n        log.Fatal(err)\n    }\n    \n    fmt.Printf(\"Read %d bytes: %s\\n\", n, string(data[:n]))\n}\n```\n\n### Read-Write File Mapping\n\n```go\npackage main\n\nimport (\n    \"log\"\n    \"os\"\n    \"github.com/godcong/mmap\"\n)\n\nfunc main() {\n    // Open file for reading and writing\n    file, err := mmap.OpenFile(\"data.bin\", os.O_RDWR|os.O_CREATE, 0644)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer file.Close()\n\n    // Write data at specific position\n    _, err = file.WriteAt([]byte(\"Hello, World!\"), 0)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // Read data back\n    buf := make([]byte, 13)\n    _, err = file.ReadAt(buf, 0)\n    if err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\n### Shared Memory\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n    \"github.com/godcong/mmap\"\n)\n\nfunc main() {\n    // Create shared memory (writer)\n    writer, err := mmap.OpenMem(mmap.MapMemKeyInvalid, 1024)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer writer.Close()\n\n    // Write data to shared memory\n    _, err = writer.Write([]byte(\"Shared data\"))\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // Open shared memory for reading (using the ID from writer)\n    reader, err := mmap.OpenMem(writer.ID(), 1024)\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer reader.Close()\n\n    // Read data from shared memory\n    buf := make([]byte, 12)\n    _, err = reader.ReadAt(buf, 0)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    fmt.Printf(\"Shared memory content: %s\\n\", string(buf))\n}\n```\n\n## API Reference\n\n### File Mapping\n\n#### `Open(filename string) (*MapFile, error)`\nOpens the named file for read-only memory mapping.\n\n#### `OpenFile(filename string, flag int, mode os.FileMode) (*MapFile, error)`\nOpens the named file with specified flags and permissions for memory mapping.\n\n#### `OpenFileS(filename string, flag int, mode os.FileMode, size int) (*MapFile, error)`\nSimilar to `OpenFile` but with explicit size specification.\n\n### Shared Memory\n\n#### `OpenMem(id int, size int) (*MapMem, error)`\nOpens or creates shared memory with the specified ID and size.\n\n#### `OpenMemS(id int) (*MapMem, error)`\nOpens shared memory with system-defined size.\n\n### Constants\n\n- `MapMemKeyInvalid` (-1): Used to create new shared memory instances\n- Standard file flags: `os.O_RDONLY`, `os.O_WRONLY`, `os.O_RDWR`, `os.O_CREATE`\n\n### Interfaces\n\nBoth `MapFile` and `MapMem` implement:\n- `io.Reader`\n- `io.Writer` (when writable)\n- `io.ReaderAt`\n- `io.WriterAt` (when writable)\n- `io.Seeker`\n- `io.Closer`\n- `io.ByteReader`\n- `io.ByteWriter` (when writable)\n\n## Examples\n\nSee the [`examples`](examples/) folder for more detailed usage examples including:\n- Basic file mapping\n- Read-write operations\n- Shared memory communication\n- Error handling\n\n## Performance\n\nThis package is optimized for performance with benchmark results showing:\n- Shared Memory: Up to 1.8 GB/s throughput\n- File Mapping: Efficient zero-copy operations\n- Low latency memory access\n\nSee the [Benchmark Results](#benchmark-results) section below for detailed performance data.\n\n## Platform Support\n\n| Platform | File Mapping | Shared Memory | Status |\n|----------|-------------|---------------|---------|\n| Windows  | ✅ | ✅ | Fully Supported |\n| Linux    | ✅ | ✅ | Fully Supported |\n| macOS    | ✅ | ✅ | Fully Supported |\n\n## Platform-Specific Implementation\n\n### Windows\n- Uses `CreateFileMapping` and `MapViewOfFile` for file mapping\n- Uses `CreateFileMapping` with `INVALID_HANDLE_VALUE` for shared memory\n- Supports both anonymous and named shared memory\n\n### Linux\n- Uses `mmap()` system call for file mapping\n- Uses `shm_open()` and `mmap()` for shared memory\n- Follows POSIX standards\n\n### macOS (Darwin)\n- Uses `mmap()` system call for file mapping\n- Uses `shm_open()` and `mmap()` for shared memory\n- Compatible with BSD-style memory mapping\n\n## Error Handling\n\nThe package defines specific error types for common scenarios:\n\n```go\nvar (\n    ErrInvalid     = errors.New(\"invalid argument\")\n    ErrBadFileDesc = errors.New(\"bad file descriptor\")\n    ErrClosed      = errors.New(\"file closed\")\n    ErrShortWrite  = errors.New(\"short write\")\n    EOF            = errors.New(\"end of file\")\n)\n```\n\n## Best Practices\n\n### Memory Management\n- Always call `Close()` to properly unmap memory\n- Use `defer` statements to ensure cleanup\n- Be mindful of memory limits when mapping large files\n\n### Shared Memory Usage\n- Use consistent sizes when opening the same shared memory ID\n- Implement proper synchronization for concurrent access\n- Handle the case where shared memory might not exist yet\n\n### Performance Tips\n- Use `ReadAt` and `WriteAt` for random access patterns\n- For sequential access, use `Read` and `Write` methods\n- Consider buffer sizes that align with system page size\n\n## Advanced Usage\n\n### Large File Handling\n```go\n// Map a portion of a large file\nfile, err := mmap.OpenFile(\"largefile.bin\", os.O_RDONLY, 0)\nif err != nil {\n    log.Fatal(err)\n}\ndefer file.Close()\n\n// Seek to position and read\n_, err = file.Seek(1024*1024, io.SeekStart) // 1MB offset\nif err != nil {\n    log.Fatal(err)\n}\n\nbuf := make([]byte, 4096)\nn, err := file.Read(buf)\n```\n\n### Multi-Process Communication\n```go\n// Process 1: Writer\nwriter, _ := mmap.OpenMem(mmap.MapMemKeyInvalid, 4096)\ndefer writer.Close()\nid := writer.ID()\n\n// Process 2: Reader (needs to obtain the ID)\nreader, _ := mmap.OpenMem(id, 4096)\ndefer reader.Close()\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Permission Denied**: Ensure file permissions are adequate\n2. **Memory Allocation Failed**: Check available system memory\n3. **Invalid File Descriptor**: Verify file exists and is accessible\n4. **Mapping Size Mismatch**: Ensure consistent sizes when reopening shared memory\n\n### Debug Mode\n\nEnable debug logging for troubleshooting:\n\n```go\n// Enable debug logging (import debug_on.go instead of debug.go)\nimport _ \"github.com/godcong/mmap/debug_on\"\n```\n\n## Similar Packages\n\n- **golang.org/x/exp/mmap**: Experimental mmap package from Go team\n- **github.com/riobard/go-mmap**: Alternative mmap implementation\n- **launchpad.net/gommap**: Go memory mapping library\n- **github.com/edsrzf/mmap-go**: Cross-platform mmap implementation\n\n### Why Choose This Package?\n\n- **Complete Feature Set**: Both file mapping and shared memory\n- **Modern Go**: Uses Go 1.23+ features and best practices\n- **Production Ready**: Extensive testing and benchmarking\n- **Cross-Platform**: Consistent API across Windows, Linux, and macOS\n- **Performance Optimized**: Efficient zero-copy operations\n\n## Benchmark Results\n\nThe following benchmarks were conducted on Windows 11 with an Intel Core i7-12700H CPU:\n\n### Shared Memory Performance\n\n| Size (Bytes) | Operations/sec | Throughput (MB/s) | Latency (ns/op) |\n|-------------|---------------|------------------|-----------------|\n| 1,024       | 220,854       | 37.65            | 27,198          |\n| 4,096       | 142,830       | 530.60           | 7,720           |\n| 16,384      | 68,470        | 1,052.54         | 15,566          |\n| 65,536      | 26,972        | 1,506.76         | 43,495          |\n| 262,144     | 8,418         | 1,843.62         | 142,190         |\n| 1,048,576   | 1,506         | 1,206.79         | 868,899         |\n\n### TCP Local Communication Performance\n\n| Size (Bytes) | Operations/sec | Throughput (MB/s) | Latency (ns/op) |\n|-------------|---------------|------------------|-----------------|\n| 1,024       | 0.2           | ~0.00            | ~4,949,659,100  |\n| 4,096       | 100           | 0.08             | 49,502,445      |\n| 16,384      | 100           | 0.33             | 49,498,112      |\n| 65,536      | 0.2           | ~0.01            | ~4,950,349,400  |\n| 262,144     | 0.2           | ~0.05            | ~4,950,514,800  |\n| 1,048,576   | 0.2           | 0.21             | ~4,949,555,300  |\n\n### Pipe Communication Performance\n\n| Size (Bytes) | Operations/sec | Throughput (MB/s) | Latency (ns/op) |\n|-------------|---------------|------------------|-----------------|\n| 1,024       | 52,519        | 47.04            | 21,769          |\n| 4,096       | 53,853        | 162.11           | 25,266          |\n| 16,384      | 40,072        | 526.27           | 31,132          |\n| 65,536      | 22,765        | 1,322.70         | 49,547          |\n| 262,144     | 9,231         | 1,610.32         | 162,790         |\n| 1,048,576   | 3,385         | 2,505.86         | 418,449         |\n\n### Memory Copy Performance (Baseline)\n\n| Size (Bytes) | Operations/sec | Throughput (MB/s) | Latency (ns/op) |\n|-------------|---------------|------------------|-----------------|\n| 1,024       | 100,000,000   | 86,907.92        | 11.78           |\n| 4,096       | 33,314,269    | 120,091.04       | 34.11           |\n| 16,384      | 11,565,373    | 155,608.58       | 105.3           |\n| 65,536      | 823,875       | 41,620.35        | 1,575           |\n| 262,144     | 200,730       | 46,547.26        | 5,632           |\n| 1,048,576   | 20,812        | 19,646.77        | 53,371          |\n\n### Concurrent Shared Memory Performance (4KB data)\n\n| Concurrency | Operations/sec | Throughput (MB/s) | Latency (ns/op) |\n|-------------|---------------|------------------|-----------------|\n| 1           | 182,282       | 659.32           | 6,212           |\n| 2           | 157,710       | 554.99           | 7,380           |\n| 4           | 151,671       | 611.26           | 6,701           |\n| 8           | 123,558       | 442.67           | 9,253           |\n| 16          | 138,694       | 463.51           | 8,837           |\n\n### Performance Analysis\n\n1. **Shared Memory** provides excellent performance for medium to large data sizes, with peak throughput around 1.8 GB/s for 256KB data.\n2. **Memory Copy** serves as the theoretical maximum baseline, achieving up to 155 GB/s throughput.\n3. **Pipe Communication** shows good scalability and reaches 2.5 GB/s for large data transfers.\n4. **TCP Local** has the highest overhead due to network stack processing, making it suitable for network scenarios but not optimized for local communication.\n5. **Concurrent Access** shows diminishing returns beyond 4 concurrent threads due to memory bandwidth limitations.\n\n### Running Benchmarks\n\n```bash\n# Run all benchmarks\ngo test -run=^$ -bench=. -benchtime=1s\n\n# Run specific benchmark\ngo test -run=^$ -bench=BenchmarkSharedMemory -benchtime=1s\n\n# Run with longer duration for more stable results\ngo test -run=^$ -bench=. -benchtime=10s\n```\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.\n\n### Development Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/godcong/mmap.git\ncd mmap\n\n# Run tests\ngo test ./...\n\n# Run benchmarks\ngo test -bench=. -benchtime=1s\n\n# Run with race detection\ngo test -race ./...\n```\n\n### Running Tests\n\n```bash\n# Run all tests\ngo test -v ./...\n\n# Run specific test\ngo test -v -run TestSharedMemory\n\n# Run benchmarks\ngo test -bench=BenchmarkSharedMemory -benchtime=5s\n\n# Run coverage\ngo test -cover ./...\n```\n\n## Roadmap\n\n- [ ] TCP transmits data through shared memory between threads\n- [ ] More elegant shared memory cleanup mechanisms\n- [ ] Additional platform support (BSD, AIX)\n- [ ] Memory-mapped I/O for device files\n- [ ] Stream-based shared memory API\n\n## Memory Map Service(TODO) Flow\nSee [`ServiceMemMap`](docs/ServiceMemMap.mmd)\n\n## Version History\n\nSee [CHANGELOG.md](CHANGELOG.md) for detailed version information.\n\n### Recent Updates\n- **v1.x.x**: Added comprehensive shared memory support\n- **v0.x.x**: Initial file mapping implementation\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Citation\n\nIf you use this package in your research or production code, please cite:\n\n```bibtex\n@software{godcong_mmap,\n  author = {godcong},\n  title = {MMAP: High-performance memory mapping library for Go},\n  year = {2024},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/godcong/mmap}}\n}\n```\n\n## Support\n\n- **Issues**: [GitHub Issues](https://github.com/godcong/mmap/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/godcong/mmap/discussions)\n- **Documentation**: [Go.dev Reference](https://pkg.go.dev/github.com/godcong/mmap)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgodcong%2Fmmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgodcong%2Fmmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgodcong%2Fmmap/lists"}