Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/brosnanyuen/raybnn_dataloader
Data Loader for RayBNN
https://github.com/brosnanyuen/raybnn_dataloader
arrayfire cpu csv csv-parser cuda data-structures gpu-computing oneapi opencl parallel parallel-computing rust
Last synced: 2 days ago
JSON representation
Data Loader for RayBNN
- Host: GitHub
- URL: https://github.com/brosnanyuen/raybnn_dataloader
- Owner: BrosnanYuen
- License: gpl-3.0
- Created: 2023-09-02T22:39:00.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-10-30T22:44:16.000Z (about 1 year ago)
- Last Synced: 2024-04-25T06:43:33.163Z (9 months ago)
- Topics: arrayfire, cpu, csv, csv-parser, cuda, data-structures, gpu-computing, oneapi, opencl, parallel, parallel-computing, rust
- Language: Rust
- Homepage:
- Size: 62.5 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# RayBNN_DataLoader
Data Loader for RayBNN
Read CSV, numpy, and binary files to Rust vectors of f16, f32, f64, u8, u16, u32, u64, i8, i16, i32, i64
Read CSV, numpy, and binary files to Arrayfire GPU arrays of f16, f32, f64, u8, u16, u32, u64, i8, i16, i32, i64
# Install Arrayfire
Install the Arrayfire 3.9.0 binaries at [https://arrayfire.com/binaries/](https://arrayfire.com/binaries/)
or build from source
[https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire](https://github.com/arrayfire/arrayfire/wiki/Getting-ArrayFire)
# Add to Cargo.toml
```
arrayfire = { version = "3.8.1", package = "arrayfire_fork" }
rayon = "1.10.0"
num = "0.4.3"
num-traits = "0.2.19"
half = { version = "2.4.1" , features = ["num-traits"] }
npyz = "0.8.3"
RayBNN_DataLoader = "2.0.3"
```
# List of Examples
# Read a CSV file to a floating point 64 bit CPU Vector
The vector is completely flat
```
let (mut cpu_vector,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_vec_cpu::(
"./test_data/read_test.dat"
);
```
# Read a CSV file to a integer 64 bit CPU Vector
The vector is completely flat
```
let (mut cpu_vector,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_vec_cpu::(
"./test_data/read_test2.dat"
);
```
# Read a CSV file to a floating point 64 bit arrayfire
The array is 2D existing in GPU or OpenCL
```
let read_test = RayBNN_DataLoader::Dataset::CSV::file_to_arrayfire::(
"./test_data/read_test.dat"
);
```
# Read a CSV file to a floating point 64 bit HashMap
```
let hashdata = RayBNN_DataLoader::Dataset::CSV::file_to_hash_cpu::(
"./test_data/dataloader.csv",
7,
5
);
```
# Read a CSV file to a floating point 64 bit HashMap with Arrayfire
```
let arr_dims = arrayfire::Dim4::new(&[5, 7, 1, 1]);
let (hashdata,metadata) = RayBNN_DataLoader::Dataset::CSV::file_to_hash_arrayfire::(
"./test_data/dataloader.csv",
arr_dims,
);
```
# Write a float 32 bit CPU vector to CSV file
```
let mut metadata: HashMap = HashMap::new();
metadata.insert("dim0", 11);
metadata.insert("dim1", 3);
RayBNN_DataLoader::Dataset::CSV::write_vec_cpu_to_csv::(
"./randvec2.csv",
&randvec,
&metadata
);
```
# Write a float 64 bit arrayfire to CSV file
```
RayBNN_DataLoader::Dataset::CSV::write_arrayfire_to_csv::(
"./randvec.csv",
&arr
);
```
# Read YAML Model Information
```
let mut modeldata_string: HashMap = HashMap::new();
let mut modeldata_float: HashMap = HashMap::new();
let mut modeldata_int: HashMap = HashMap::new();
RayBNN_DataLoader::Model::YAML::read(
"./test_data/test.yaml",
&mut modeldata_string,
&mut modeldata_float,
&mut modeldata_int,
);
```
# Write YAML Model Information
```
RayBNN_DataLoader::Model::YAML::write(
"./sample.yaml",
&modeldata_string,
&modeldata_float,
&modeldata_int,
);
```