https://github.com/sergeyklay/clusterium
QA Dataset Toolkit for clustering and analysis
https://github.com/sergeyklay/clusterium
clustering data-science datasets llm qa-dataset semantic-segmentation
Last synced: 7 months ago
JSON representation
QA Dataset Toolkit for clustering and analysis
- Host: GitHub
- URL: https://github.com/sergeyklay/clusterium
- Owner: sergeyklay
- License: mit
- Created: 2025-03-08T11:07:56.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-03-11T19:25:59.000Z (over 1 year ago)
- Last Synced: 2025-03-11T19:39:56.447Z (over 1 year ago)
- Topics: clustering, data-science, datasets, llm, qa-dataset, semantic-segmentation
- Language: Python
- Homepage:
- Size: 568 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Clusterium
[](https://github.com/sergeyklay/clusterium/actions/workflows/ci.yml)
[](https://codecov.io/gh/sergeyklay/clusterium)
A toolkit for clustering, analyzing, and benchmarking text data using state-of-the-art embedding models and clustering algorithms.
## Features
- **Dirichlet Process Clustering**: Implements the Dirichlet Process for text clustering
- **Pitman-Yor Process Clustering**: Implements the Pitman-Yor Process for text clustering with improved performance
- **Evaluation**: Evaluates clustering results using a variety of metrics, including Silhouette Score, Davies-Bouldin Index, and Power-law Analysis
- **Visualization**: Generates plots of cluster size distributions
## Installation
For detailed installation instructions, please see [INSTALL.md](INSTALL.md).
### Quick Start
```bash
git clone https://github.com/sergeyklay/clusterium.git
cd clusterium
poetry install
```
## Usage
For detailed usage instructions, use cases, examples, and advanced configuration options, please see [USAGE.md](USAGE.md).
### Quick Start
```bash
# Run clustering
clusx --input your_data.csv --column your_column --output clusters.csv
# Evaluate clustering results and generate visualizations
clusx evaluate \
--input input.csv \
--column your_column \
--dp-clusters output_dp.csv \
--pyp-clusters output_pyp.csv \
--plot
```
### Python API Example
```python
from clusx.clustering import DirichletProcess
from clusx.clustering.utils import load_data_from_csv, save_clusters_to_json
# Load data
texts, data = load_data_from_csv("your_data.csv", column="your_column")
# Perform clustering
dp = DirichletProcess(alpha=1.0)
clusters, params = dp.fit(texts)
# Save results
save_clusters_to_json("clusters.json", texts, clusters, "DP", data)
```
## License
This project is licensed under the MIT License - see the LICENSE file for details.