Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tysonandre/pecl-teds

Tentative Extra Data Structures for php
https://github.com/tysonandre/pecl-teds

data-structures pecl php

Last synced: about 2 months ago
JSON representation

Tentative Extra Data Structures for php

Awesome Lists containing this project

README 

        
# Teds



## Introduction



[![Build Status](https://github.com/TysonAndre/pecl-teds/actions/workflows/main.yml/badge.svg?branch=main)](https://github.com/TysonAndre/pecl-teds/actions/workflows/main.yml?query=branch%3Amain)

[![Build Status (Windows)](https://ci.appveyor.com/api/projects/status/j5s46tetcg4101g8?svg=true)](https://ci.appveyor.com/project/TysonAndre/pecl-teds)



Teds is a another collection of data structures.

(Tentative Extra Data Structures)



## Installation



This extension requires php 8.0 or newer.



```

phpize

./configure

make install

```



On Windows, see https://wiki.php.net/internals/windows/stepbystepbuild_sdk_2 instead



## Overview

`Teds` contains the following types of `Teds\Collection` instances as well as various [iterable functionality](#iterable-functions):



### Teds\Sequence - A Collection of values with keys 0..n-1



`Teds\Sequence` is an interface representing a Collection of values with keys `0..n-1` and no gaps. (also known as a list in other languages)

It is implemented by the following classes.



- `Teds\Vector` - a memory-efficient representation of a list of values that can easily grow/shrink.

- `Teds\Deque` - a memory-efficient representation of a list of values with amortized constant time push/pop/pushFront/popFront.



`Teds` also includes several specializations of `Teds\Sequence` for memory-efficiency:



- `Teds\IntVector` - a memory-efficient specialization of lists of integers, typically with faster serialization requiring less memory.

- `Teds\LowMemoryVector` - exposes the same API as `Teds\Vector` but uses less memory for specializations (exclusively bool/null, exclusively integers, exclusively floats)

- `Teds\BitVector` - exposes the same API as `Teds\Vector` but uses significantly less memory for booleans. Provides ways to read/write raw bytes/words.



And immutable versions:



- `Teds\ImmutableSequence` - represents an immutable sequence.

- `Teds\EmptySequence::INSTANCE` (PHP 8.1+, uses enum) - represents an immutable empty sequence.



### Teds\Set - a Collection of unique values



The `Teds\Set` interface is implemented by the following classes:



- `Teds\StrictHashSet` - a hash table based set of unique values providing efficient average-time lookup/insertion/removal. Uses `Teds\strict_hash`.

- `Teds\StrictTreeSet` - a binary tree-based set of **sorted** unique values providing good worst-case time. Uses `Teds\stable_compare` for stable ordering.

- `Teds\StrictSortedVectorSet` - provides a similar API to `Teds\StrictTreeSet` but is represented internally as a Vector. This has reduced memory usage and faster construction time, and can be useful in cases where modification is infrequent (e.g. more common to unserialize without modifying)



#### Low-memory Sets



Some specializations are provided for reduced memory usage:



- `Teds\SortedIntVectorSet` - a mutable sorted set of integers, represented internally as a Vector. This has reduced memory usage and can be useful in cases where modification is infrequent (e.g. more common to unserialize without modifying)



`Teds` also provides immutable specializations of common data types offering faster serialization/unserialization:



- `Teds\ImmutableSortedStringSet` - a sorted set of strings sorted by strcmp. Note that internally, this is backed by a single string with the data that would be used for `__unserialize`/`__serialize`.



  As a result, this is faster to unserialize and uses less memory than an array  ([see benchmark](https://github.com/TysonAndre/pecl-teds/blob/main/benchmarks/benchmark_string_set.php)), but when iterating over the string, temporary copies of the strings that are members of the set are returned.

- `Teds\ImmutableSortedIntSet` - a sorted set of integers.

- `Teds\EmptySet::INSTANCE` (PHP 8.1+, uses enum) - represents an immutable empty set.



### Teds\Map - a Collection mapping unique keys to values



The `Teds\Map` interface is implemented by the following classes:



- `Teds\StrictHashMap` - a hash table based map of unique keys to values providing efficient average-time lookup/insertion/removal. Uses `Teds\strict_hash`.

- `Teds\StrictTreeMap` - a binary tree-based map of **sorted** unique keys to values providing slower average-case time but good worst-case time. Uses `Teds\stable_compare` for stable ordering.

- `Teds\StrictSortedVectorMap` - provides a similar API to `Teds\StrictTreeMap` but is represented internally like a Vector. This has reduced memory usage and faster construction time and can be useful in cases where modification is infrequent (e.g. more common to unserialize without modifying)



`Teds` also provides the following map types:



- `Teds\EmptyMap::INSTANCE` (PHP 8.1+, uses enum) - represents an immutable empty map.



### Teds\Collections - others



- `Teds\StrictMinHeap` and `Teds\StrictMaxHeap` are heaps where elements can be added (allowing duplicates) and are removed in order (or reverse order) of `Teds\stable_compare`.

- `Teds\CachedIterable` is a `Collection` that lazily evaluates iterables such as Generators and stores the exact keys and values to allow future retrieval and iteration.

- `Teds\ImmutableIterable` is a `Collection` that eagerly evaluates iterables such as Generators and stores the exact keys and values to allow future retrieval and iteration.

- `Teds\MutableIterable` is a `Collection` that allows construction and modification of iterables, setting keys and values and allowing duplicate keys in any order.



## Functionality



### Teds\ImmutableIterable



[`Teds\ImmutableIterable` API](./teds_immutableiterable.stub.php)



Currently, PHP does not provide a built-in way to store the state of an arbitrary iterable for reuse later (when the iterable has arbitrary keys, or when keys might be repeated). There are use cases for this, such as:



1. Creating a rewindable copy of a non-rewindable `Traversable` (e.g. a `Generator`) before passing that copy to a function that consumes an `iterable`/`Traversable`. (new ImmutableIterable(my_generator()))

2. Generating an `IteratorAggregate` from a class still implementing `Iterator` (e.g. `SplObjectStorage`) so that code can independently iterate over the key-value sequences. (e.g. `foreach ($immutableImmutableIterable as $k1 => $v1) { foreach ($immutableImmutableIterable as $k2 => $v2) { /* process pairs */ } }`)

3. Providing helpers such as `iterable_flip(iterable $iterable)`, `iterable_take(iterable $iterable, int $limit)`, `iterable_chunk(iterable $iterable, int $chunk_size)` that act on iterables with arbitrary key/value sequences and have return values including iterables with arbitrary key/value sequences

4. Providing constant time access to both keys and values of arbitrary key-value sequences at any offset (for binary searching on keys and/or values, etc.)



Having this implemented as a native class allows it to be much more efficient than a userland solution (in terms of time to create, time to iterate over the result, and total memory usage, depending on the representation).



Objects within this data structure or references in arrays in this data structure can still be mutated.



Traversables are eagerly iterated over in the constructor.



### Teds\CachedIterable



[`Teds\CachedIterable` API](./teds_cachediterable.stub.php)



This is similar to `Teds\ImmutableIterable` but lazily evaluates Traversables instead of eagerly evaluating Traversables.

(E.g. Generators will only run until the last offset used from a CachedIterable. See [tests/CachedIterable/lazy.phpt](./tests/CachedIterable/lazy.phpt) and [tests/CachedIterable/selfReferential.phpt](./tests/CachedIterable/selfReferential.phpt) for examples.)



This can be used to cache results of generators without fetching more elements than needed (e.g. Generators that repeatedly call databases or services to paginate).



### Teds\ImmutableSequence



[`Teds\ImmutableSequence` API](./teds_immutablesequence.stub.php)



Similar to SplFixedArray or Ds\Sequence, but immutable.

This stores a sequence of values with the keys 0, 1, 2....



### Teds\LowMemoryVector



[`Teds\LowMemoryVector` API](./teds_lowmemoryvector.stub.php)



This exposes the same API as a Vector, but with a more memory-efficient representation if the LowMemoryVector has only ever included a type that it could optimize.



Benefits:



- For collections of exclusively int, exclusively floats, or exclusively null/true/false, this uses less memory when serialized compared to vectors/arrays.

- For collections of exclusively int, exclusively floats, or exclusively null/true/false, this uses less memory when serialized compared to vectors/arrays.



  Note that adding other types will make this use as much memory as a Vector, e.g. adding any non-float (including int) to a collection of floats.

- Has faster checks for contains/indexOf (if values can have an optimized representation)

- Has faster garbage collection (if values can have an optimized representation due to int/float/bool/null not needing reference counting).

- Interchangeable with `Vector` or other collections without configuration - this will silently fall back to the default `mixed` representation if a more efficient representation is not supported.



Drawbacks:

- Slightly more overhead when types aren't specialized.

- Adding a different type to the collection will permanently make it used the less efficient `mixed` representation.



In 64-bit builds, the following types are supported, with the following amounts of memory (plus constant overhead to represent the LowMemoryVector itself, and extra capacity for growing the LowMemoryVector):



1. null/bool : 1 byte per value.



   (In serialize(), this is even less, adding 1 to 2 bits per value (2 bits if null is included)).

2. signed 8-bit int (-128..127): 1 byte per value. Adding a larger int to any of these n-bit types will convert them to the collection of that larger int type.

3. signed 16-bit int: 2 bytes per value.

4. signed 32-bit int: 4 bytes per value.

5. signed 64-bit int: 8 bytes per value. (64-bit php builds only)

6. signed PHP `float`:  8 bytes per value. (C `double`)

7. `mixed` or combinations of the above: 16 bytes per value. (Same as `Vector`)



In comparison, in 64-bit builds of PHP, PHP's arrays take at least 16 bytes per value in php 8.2, and at least 32 bytes per value before php 8.1, at the time of writing.



Example benchmarks: [benchmarks/benchmark_vector_bool.php](benchmarks/benchmark_vector_bool.php) and [benchmarks/benchmark_vector_unserialize.phpt](benchmarks/benchmark_vector_unserialize.phpt).



### Teds\IntVector



[`Teds\IntVector` API](./teds_intvector.stub.php)



Similar to `Teds\LowMemoryVector` but throws a TypeError on attempts to add non-integers.



### Teds\BitVector



[`Teds\BitVector` API](./teds_bitset.stub.php)



Similar to `Teds\LowMemoryVector`/`Teds\IntVector` but throws a TypeError on attempts to add non-booleans.

This can be used as a memory-efficient vector of booleans.



This uses only a single bit per value for large bit sets in memory and when serializing (around 128 times less memory than arrays, for large arrays of booleans)



### Teds\Vector



[`Teds\Vector` API](./teds_vector.stub.php)



Similar to SplFixedArray or Ds\Vector.

This stores a mutable sequence of values with the keys 0, 1, 2...

It can be appended to with `push()`, and elements can be removed from the end with `pop()`



This is implemented based on SplFixedArray/ImmutableSequence.

There are plans to add more methods.



### Teds\MutableIterable



[`Teds\MutableIterable` API](./teds_mutableiterable.stub.php)



Similar to `Teds\Vector` and `Teds\ImmutableIterable`.

This stores a mutable vector of keys and values with the keys 0, 1, 2...

It can be resized with `setSize()`.



### Teds\Deque



[`Teds\Deque` API](./teds_deque.stub.php)



Similar to SplDoublyLinkedList but backed by an array instead of a linked list.

Much more efficient in memory usage and random access than SplDoublyLinkedList.



(Also similar to `Ds\Deque`)



### Teds\StrictTreeMap and Teds\StrictTreeSet



[`Teds\StrictTreeMap` API](./teds_stricthashmap.stub.php)



This is a map where entries for keys of any type can be inserted if `Teds\stable_compare !== 0`.



This currently uses a balanced [red-black tree](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree) to ensure logarithmic time is needed for insertions/removals/lookups.



Removing a `Teds\StrictTreeMap`/`Teds\StrictTreeSet` entry will move iterators pointing to that entries to the entry before the removed entry (as of 1.2.1).



This uses [`Teds\stable_compare`](#stable-comparison) internally.



The [`Teds\StrictTreeSet` API](./teds_stricttreeset.stub.php) implementation is similar, but does not associate values with keys. Also, `StrictTreeSet` does not implement ArrayAccess and uses different method names.



### Teds\StrictHashMap and Teds\StrictHashSet



[`Teds\StrictHashMap` API](./teds_stricthashmap.stub.php)



This is a map where entries for keys of any type can be inserted if they are `!==` to other keys.

This uses [`Teds\strict_hash`](#strict-hashing) internally.



The [`Teds\StrictHashSet` API](./teds_stricthashset.stub.php) implementation is similar, but does not associate values with keys and does not implement ArrayAccess and uses different method names.



NOTE: The floats `0.0` and [`-0.0` (negative zero)](https://en.wikipedia.org/wiki/Signed_zero) have the same hashes and are treated as the same entries, because `0.0 === -0.0` in php.

NOTE: The float `NAN` (Not a Number) is deliberately treated as equivalent to itself by `Teds\strict_hash` and  `StrictHashSet`/`StrictHashMap`, despite having `NAN !== $x` in php for any $x, including `NAN`. This is done to avoid duplicate or unremovable entries.



Removing an entry from a hash map/set will move iterators pointing to that entry to the entry prior to the removed entry.



### Teds\StrictMinHeap and Teds\StrictMaxHeap



[`Teds\Strict*Heap` API](./teds_strictheap.stub.php)



This uses `Teds\stable_compare` instead of PHP's unstable default comparisons.

Sorting logic can be customized by inserting `[$priority, $value]` instead of `$value`.

(Or by subclassing `SplMinHeap`/`SplMaxHeap` and overriding `compare` manually).



```php

php > $x = new SplMinHeap();

php > foreach (['19', '9', '2b', '2'] as $v) { $x->insert($v); }

php > foreach ($x as $value) { echo json_encode($value).","; } echo "\n"; // unpredictable order

"2","19","2b","9",



php > $x = new Teds\StrictMinHeap();

php > foreach (['19', '9', '2b', '2'] as $v) { $x->add($v); }

php > foreach ($x as $value) { echo json_encode($value).","; } echo "\n"; // lexicographically sorted

"19","2","2b","9",

```



### Teds\EmptySequence and Teds\EmptySet and Teds\EmptyMap



This provides empty immutable collections for php 8.1+ based on single-case enums.



[Empty Immutable Collection API](./teds_emptycollection.stub.php)



### Common interfaces



[teds_interfaces.stub.php](./teds_interfaces.stub.php)



**NOTE: This is currently being revised, and new methods may be added to these interfaces in 0.x releases or new major releases.** More methods are currently being added.



These provide common interfaces for accessing the lists, sorted/hash sets, sorted/hash maps, sequences, and key value sequences that are provided by `Teds\`.



```php

 the list of values in the collection */

    public function values(): array {}



    /**

     * Returns a list or associative array,

     * typically attempting to cast keys to array key types (int/string) to insert elements of the collection into the array, or throwing.

     *

     * When this is impossible for the class in general,

     * the behavior depends on the class implementation (e.g. throws \Teds\UnsupportedOperationException, returns an array with representations of key/value entries of the Collection)

     */

    public function toArray(): array {}



    /** Returns true if count() would be 0 */

    public function isEmpty(): bool {}



    /** Returns true if this contains a value identical to $value. */

    public function contains(mixed $value): bool {}

}



/**

 * This represents a Collection that can be used like a list without gaps.

 * E.g. get()/set() will work for is_int($offset) && 0 <= $offset and $offset < $list->count().

 */

interface Sequence extends Collection, \ArrayAccess {

    public function get(int $offset): mixed {}

    public function set(int $offset, mixed $value): void {}



    public function push(mixed ...$values): void {}

    public function pop(): mixed {}

}



/**

 * A Map is a type of Collection mapping unique keys to values.

 *

 * Implementations should either coerce unsupported key types or throw TypeError when using keys.

 *

 * Implementations include

 *

 * 1. Teds\StrictHashMap, a hash table with amortized constant time operations

 * 2. Teds\StrictTreeMap, a sorted binary tree

 */

interface Map extends Collection, \ArrayAccess {

    /**

     * Returns true if there exists a key identical to $key according to the semantics of the implementing collection.

     * Typically, this is close to the definition of `===`, but may be stricter or weaker in some implementations, e.g. for NAN, negative zero, etc.

     *

     * containsKey differs from offsetExists, where implementations of offsetExists usually return false if the key was found but the corresponding value was null.

     * (This is analogous to the difference between array_key_exists($key, $array) and isset($array[$key]))

     */

    public function containsKey(mixed $value): bool {}

}



/**

 * A Set is a type of Collection representing a set of unique values.

 * Implementations include Teds\StrictHashSet and Teds\StrictTreeSet.

 *

 * 1. Teds\StrictHashSet, a hash table with amortized constant time operations

 * 2. Teds\StrictTreeSet, a sorted binary tree

 */

interface Set extends Collection {

    /**

     * Returns true if $value was added to this Set and was not previously in this Set.

     */

    public function add(mixed $value): bool {}



    /**

     * Returns true if $value was found in this Set before being removed from this Set.

     */

    public function remove(mixed $value): bool {}

}

```



### iterable functions



This PECL contains a library of native implementations of various functions acting on iterables.

See [`teds.stub.php`](./teds.stub.php) for function signatures.



**The behavior is equivalent to the following polyfill**

(similarly to array_filter, the native implementation is likely faster than the polyfill with no callback, and slower with a callback)



```php

namespace Teds;



/**

 * Determines whether any element of the iterable satisfies the predicate.

 *

 * If the value returned by the callback is truthy

 * (e.g. true, non-zero number, non-empty array, truthy object, etc.),

 * this is treated as satisfying the predicate.

 *

 * @param iterable $iterable

 * @param null|callable(mixed):mixed $callback

 */

function any(iterable $iterable, ?callable $callback = null): bool {

    foreach ($iterable as $v) {

        if ($callback !== null ? $callback($v) : $v) {

            return true;

        }

    }

    return false;

}

/**

 * Determines whether all elements of the iterable satisfy the predicate.

 *

 * If the value returned by the callback is truthy

 * (e.g. true, non-zero number, non-empty array, truthy object, etc.),

 * this is treated as satisfying the predicate.

 *

 * @param iterable $iterable

 * @param null|callable(mixed):mixed $callback

 */

function all(iterable $iterable, ?callable $callback = null): bool {

    foreach ($iterable as $v) {

        if (!($callback !== null ? $callback($v) : $v)) {

            return false;

        }

    }

    return true;

}



/**

 * Determines whether no element of the iterable satisfies the predicate.

 *

 * If the value returned by the callback is truthy

 * (e.g. true, non-zero number, non-empty array, truthy object, etc.),

 * this is treated as satisfying the predicate.

 *

 * @param iterable $iterable

 * @param null|callable(mixed):mixed $callback

 */

function none(iterable $iterable, ?callable $callback = null): bool {

	return !any($iterable, $callback);

}



// Analogous to array_reduce but with mandatory default

function fold(iterable $iterable, callable $callback, mixed $default): bool {

	foreach ($iterable as $value) {

		$default = $callback($default, $value);

	}

	return $default;

}



/**

 * Returns the first value for which $callback($value) is truthy.

 */

function find(iterable $iterable, callable $callback, mixed $default = null): bool {

	foreach ($iterable as $value) {

		if ($callback($value)) {

			return $value;

		}

	}

	return $default;

}



/**

 * Similar to in_array($value, $array, true) but also works on Traversables.

 */

function includes_value(iterable $iterable, mixed $value): bool {

	foreach ($iterable as $other) {

		if ($other === $value) {

			return true;

		}

	}

	return false;

}



/**

 * Returns a list of unique values in order of occurrence,

 * using a hash table with `Teds\strict_hash` to deduplicate values.

 */

function unique_values(iterable $iterable): array {

	// Without Teds installed, this takes quadratic time instead of linear time.

	$result = [];

	foreach ($iterable as $value) {

		if (!in_array($value, $result, true)) {

			$result[] = $value;

		}

	}

	return $result;

}

```



## Stable comparison



`Teds\stable_compare` is a function that can be used to compare arbitrary values in a stable order.



This exists because php's `<` operator is not stable. `'10' < '0a' < '1b' < '9' < '10'`.

`Teds\stable_compare` fixes that by strictly ordering:



- `null < false < true < int,float < string < array < object < resource`.

- objects are compared by class name with strcmp, then by spl_object_id.

- resources are compared by id.

- arrays are compared recursively. Smaller arrays are less than larger arrays.

- int/float are compared numerically. If an int is equal to a float, then the int is first.

- strings are compared with strcmp.



## Strict hashing



`Teds\strict_hash` provides a hash based on value identity.

Before a final step to improve accidental hash collisions:



- Objects are hashed based only on `spl_object id`.

  Different objects will have different hashes for the lifetime of the hash.

- Resources are hashed based on `get_resource_id`.

- Strings are hashed

- References are dereferenced and hashed the same way as the value.

- Integers are used directly.

- Floats are hashed in a possibly platform-specific way.

- Arrays are hashed recursively. If $a1 === $a2 then they will have the same hash.



This may vary based on php release, OS, CPU architecture, or Teds release

and should not be saved/loaded outside of a given php process.

(and `spl_object_id`/`get_resource_id` are unpredictable)



## Binary search



`Teds\binary_search(array $values, mixed $target, callable $comparer = null, bool $useKey=false)`

can be used to [binary search](https://en.wikipedia.org/wiki/Binary_search_algorithm) on arrays that are sorted by key (ksort, uksort) or value (sort, usort, uasort).

(even if keys were unset).



This will have unpredictable results if the array is out of order. See [`Teds\stable_sort`](#stable-comparison) for ways to sort even arbitrary values in a stable order.



This is faster for very large sorted arrays. See [benchmarks](benchmarks/).



This returns the key and value of the first entry `<=` $needle according to the comparer, and whether an entry comparing equal was found.

By default, php's default comparison behavior (`<=>`) is used.



```php

php > $values = [1 => 100, 3 => 200, 4 => 1000];

php > echo json_encode(Teds\binary_search($values, 1));

{"found":false,"key":null,"value":null}

php > echo json_encode(Teds\binary_search($values, 100));

{"found":true,"key":1,"value":100}

php > echo json_encode(Teds\binary_search($values, 201));

{"found":false,"key":3,"value":200}

php > echo json_encode(Teds\binary_search($values, 99));

{"found":false,"key":null,"value":null}

php > echo json_encode(Teds\binary_search($values, 1, useKey: true));

{"found":true,"key":1,"value":100}

```



## Array functionality



[teds.stub.php](./teds.stub.php)



- `Teds\is_same_array_handle(array $array1, array $array2)` - check if two arrays have the same handle, for [infinite recursion detection](tests/array/is_same_array_handle.phpt).

- `Teds\array_value_first(array $array)`, `Teds\array_value_last(array $array)` - Return the first/last value of an array without creating references or moving the internal array pointer. Similar to `$array[array_key_first($array)] ?? null`.



## Motivation



This contains functionality and data structures that may be proposed for inclusion into PHP itself (under a different namespace) at a future date, reimplemented using [SPL's source code](https://github.com/php/php-src/tree/master/ext/spl) as a starting point.



Providing this as a PECL first makes this functionality easier to validate for correctness, and make it more practical to change APIs before proposing including them in PHP if needed.



## License



See [COPYING](./COPYING)



## Changelog



See [package.xml](./package.xml)



## Related Projects



- https://www.php.net/spl is built into php

- https://www.php.net/manual/en/book.ds.php