https://github.com/joscha/shardymcshardface

Sharding things for CI systems
https://github.com/joscha/shardymcshardface

ci continuous-integration parallelization sharding test testing

Last synced: 8 days ago
JSON representation

Sharding things for CI systems

• Host: GitHub
• URL: https://github.com/joscha/shardymcshardface
• Owner: joscha
• Created: 2019-08-12T15:49:42.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2024-08-08T12:55:23.000Z (6 months ago)
• Last Synced: 2025-01-09T22:52:11.561Z (12 days ago)
• Topics: ci, continuous-integration, parallelization, sharding, test, testing
• Language: TypeScript
• Homepage:
• Size: 617 KB
• Stars: 4
• Watchers: 3
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ShardyMcShardFace

Package to shard a set of items based on CI parallelization parameters, e.g. `YOUR_CI_SYSTEM_INDEX` and `YOUR_CI_SYSTEM_COUNT`.

## Features:

-   Shards as evenly as possible, uneven splits will end up in the tail shards

-   Supports sharding less items than parallelization count; tail shards will be empty

-   Distributes items into shards based on a given seed for a random number generator to provide random, but stable distribution.

-   Fully typed in Typescript

## Installation

```bash

yarn add shardy-mc-shard-face

```

## Usage

```

shardyMcShardFace(items: any[] [, options: { throwOnEmpty: boolean, throwWhenNotSharding: boolean, seed: string }]): shard[];

```

You can get debug output by setting the environment variable `DEBUG=ShardyMcShardFace:*`.

## Example

```ts

import { shard as shardyMcShardFace } from 'shardy-mc-shard-face';

const shard = shardyMcShardFace([1, 2, 3, 4]);

// shard is an array that contains items based on YOUR_CI_SYSTEM_INDEX and YOUR_CI_SYSTEM_COUNT

```

## CLI example

```bash

cat items | shardy shard

```

Input is expected to be newline-separated to `stdin`. Output is also newline-separated to `stdout`.

On a CI run this would look like this:

```bash

# CI_NODE_INDEX=1 (set by your CI system)

# CI_NODE_TOTAL=2 (set by your CI system)

echo "A\nB" | shardy shard

# Will print "A" (w/o quotes) to stdout

```

you can control the seed as well via `-s `.

For a full list of options, please run `shardy --help`.

## CI system support

CI systems supported are the ones supported by [ci-parallel-vars](https://github.com/jamiebuilds/ci-parallel-vars#supports). Feel free to open a pull request there and I will be happy to bump the dependency.