{"id":15196991,"url":"https://github.com/hansalemaos/stridesduplicatefinder","last_synced_at":"2026-03-07T13:03:26.232Z","repository":{"id":193769300,"uuid":"689463909","full_name":"hansalemaos/stridesduplicatefinder","owner":"hansalemaos","description":"Calculate overlapping values between two arrays and return the results as a DataFrame","archived":false,"fork":false,"pushed_at":"2023-09-09T22:21:49.000Z","size":25,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-01T15:58:32.597Z","etag":null,"topics":["duplicates","fast","numexpr","numpy","strides"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/stridesduplicatefinder/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hansalemaos.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-09T22:21:16.000Z","updated_at":"2023-09-09T22:22:28.000Z","dependencies_parsed_at":"2023-09-09T22:45:01.732Z","dependency_job_id":null,"html_url":"https://github.com/hansalemaos/stridesduplicatefinder","commit_stats":null,"previous_names":["hansalemaos/stridesduplicatefinder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hansalemaos/stridesduplicatefinder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fstridesduplicatefinder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fstridesduplicatefinder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fstridesduplicatefinder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fstridesduplicatefinder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hansalemaos","download_url":"https://codeload.github.com/hansalemaos/stridesduplicatefinder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fstridesduplicatefinder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30214618,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T12:15:00.571Z","status":"ssl_error","status_checked_at":"2026-03-07T12:15:00.217Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["duplicates","fast","numexpr","numpy","strides"],"created_at":"2024-09-28T00:22:22.112Z","updated_at":"2026-03-07T13:03:26.216Z","avatar_url":"https://github.com/hansalemaos.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Calculate overlapping values between two arrays and return the results as a DataFrame\r\n\r\n## Tested against Windows 10 / Python 3.10 / Anaconda\r\n\r\n## pip install stridesduplicatefinder\r\n\r\n#### Problem: you have to lists of different sizes and want to find the overlapping values. \r\n\r\n## Using pure Python - working, but slow\r\n\r\n#### all indices / same values\r\n\r\n```python\r\na1=[1,2,3,4,5,6,7]\r\na2=[0,0,3,1,5,6,8,1,32,]\r\nres1=[(index1, index2, value1, value2) for index1, value1 in enumerate(a1) for index2, value2 in enumerate(a2) if value1 == value2]\r\nprint(res1)\r\n# [(0, 3, 1, 1), (0, 7, 1, 1), (2, 2, 3, 3), (4, 4, 5, 5), (5, 5, 6, 6)]\r\n```\r\n\r\n#### same indices / same values\r\n\r\n```python\r\nres2=[(index1, index2, value1, value2) for index1, value1 in enumerate(a1) for index2, value2 in enumerate(a2) if value1 == value2 and index1==index2]\r\nprint(res2)\r\n# [(2, 2, 3, 3), (4, 4, 5, 5), (5, 5, 6, 6)]\r\n```\r\n\r\n## Using stridesduplicatefinder - numpy or numexpr\r\n\r\n```python\r\nfrom stridesduplicatefinder import get_overlapping\r\n\r\ndef test_numexpr():\r\n    start = perf_counter()\r\n\r\n    _ = get_overlapping(\r\n        fu=\"a==b\", a=a1, b=a2, numpy_or_numexpr=\"numexpr\", same_index_required=False\r\n    )\r\n    print(f\"numexpr test: {perf_counter() - start}\")\r\n    print(_)\r\n\r\n\r\ndef test_numpy():\r\n    start = perf_counter()\r\n    _ = get_overlapping(\r\n        fu=lambda a, b: a == b,\r\n        a=a1,\r\n        b=a2,\r\n        numpy_or_numexpr=\"numpy\",\r\n        same_index_required=False,\r\n    )\r\n    print(f\"numpy test: {perf_counter() - start}\")\r\n    print(_)\r\n\r\n\r\ndef python_test():\r\n    start = perf_counter()\r\n    _ = [(i1, i2, a, b) for i2, a in enumerate(a1) for i1, b in enumerate(a2) if a == b]\r\n    print(f\"python test: {perf_counter() - start}\")\r\n    print(_[:10])\r\n\r\n\r\n\r\na1 = np.random.randint(1, 100, size=(19000,),dtype=np.int64)\r\na2 = np.random.randint(1, 100, size=(7777,),dtype=np.int64)\r\nfrom time import perf_counter\r\n\r\npython_test()\r\n# python test: 13.229658300006122\r\n\r\ntest_numpy()\r\n# numpy test: 0.5666937999994843\r\n\r\ntest_numexpr()\r\n# numexpr test: 0.48387080000247806\r\n\r\n\r\n```\r\n\r\n\r\n```python\r\n\r\nCalculate overlapping values between two arrays and return the results as a DataFrame.\r\n\r\nParameters:\r\n- fu: function or string to be evaluated as a condition for overlap.\r\n- a: First input array.\r\n- b: Second input array.\r\n- numpy_or_numexpr: 'numpy' or 'numexpr' indicating the evaluation method.\r\n- same_index_required: If True, only return rows where index1 == index2.\r\n\r\nReturns:\r\n- A DataFrame with columns 'index1', 'value1', 'index2', 'value2' containing\r\n  information about overlapping values.\r\n\r\nExample Usage:\r\n- To find overlapping values between two NumPy arrays:\r\n  \r\n  a1 = np.random.randint(1, 10, size=(100000,))\r\n  a2 = np.random.randint(1, 10, size=(100,))\r\n  df1 = get_overlapping(\r\n\t  fu=\"a==b\", a=a1, b=a2, numpy_or_numexpr=\"numexpr\", same_index_required=True\r\n  )\r\n  print(df1)\r\n  \r\n\r\n- To find overlapping values using a custom function:\r\n  \r\n  a1 = np.random.randint(1, 10, size=(100000,))\r\n  a2 = np.random.randint(1, 10, size=(100,))\r\n  df2 = get_overlapping(\r\n\t  fu=lambda a, b: a == b,\r\n\t  a=a1,\r\n\t  b=a2,\r\n\t  numpy_or_numexpr=\"numpy\",\r\n\t  same_index_required=False,\r\n  )\r\n  print(df2)\r\n  \r\n\r\n- To find overlapping values between two arrays of strings:\r\n  \r\n  a1 = np.array([\"aa\", \"b\", \"c\", \"d\", \"ee11\", \"f\", \"gg\", \"h\", \"i\", \"j\"])\r\n  a1 = np.repeat(a1, 1000)\r\n  a2 = np.array([\"aa\", \"b\", \"c\", \"ee11\", \"f\", \"gg\"])\r\n  a2 = np.repeat(a2, 1000)\r\n  np.random.shuffle(a1)\r\n  np.random.shuffle(a2)\r\n  df3 = get_overlapping(\r\n\t  fu=\"a == b\",\r\n\t  a=np.char.array(a1).encode(\"utf-8\"),\r\n\t  b=np.char.array(a2).encode(\"utf-8\"),\r\n\t  numpy_or_numexpr=\"numexpr\",\r\n\t  same_index_required=True,\r\n  )\r\n  print(df3)\r\n  \r\n\t\t#     index1  value1  index2  value2\r\n\t# 0        5       1       5       1\r\n\t# 1       20       8      20       8\r\n\t# 2       33       5      33       5\r\n\t# 3       34       1      34       1\r\n\t# 4       41       5      41       5\r\n\t# 5       43       2      43       2\r\n\t# 6       51       7      51       7\r\n\t# 7       52       1      52       1\r\n\t# 8       55       7      55       7\r\n\t# 9       57       1      57       1\r\n\t# 10      70       2      70       2\r\n\t# 11      74       8      74       8\r\n\r\n\r\n\t#          index1  value1  index2  value2\r\n\t# 0             0       4       8       4\r\n\t# 1             0       4      12       4\r\n\t# 2             0       4      13       4\r\n\t# 3             0       4      26       4\r\n\t# 4             0       4      53       4\r\n\t#          ...     ...     ...     ...\r\n\t# 1112213   99999       9      47       9\r\n\t# 1112214   99999       9      62       9\r\n\t# 1112215   99999       9      72       9\r\n\t# 1112216   99999       9      81       9\r\n\t# 1112217   99999       9      96       9\r\n\t# [1112218 rows x 4 columns]\r\n\r\n\r\n\t#          index1 value1  index2 value2\r\n\t# 0             1     gg       4     gg\r\n\t# 1             1     gg       5     gg\r\n\t# 2             1     gg      10     gg\r\n\t# 3             1     gg      13     gg\r\n\t# 4             1     gg      17     gg\r\n\t#          ...    ...     ...    ...\r\n\t# 5999995    9999      c    5978      c\r\n\t# 5999996    9999      c    5979      c\r\n\t# 5999997    9999      c    5990      c\r\n\t# 5999998    9999      c    5992      c\r\n\t# 5999999    9999      c    5995      c\r\n\t# [6000000 rows x 4 columns]\r\n\r\n\r\n\t#      index1   value1  index2   value2\r\n\t# 0        31    b'aa'      31    b'aa'\r\n\t# 1        40     b'b'      40     b'b'\r\n\t# 2        46    b'aa'      46    b'aa'\r\n\t# 3        47    b'gg'      47    b'gg'\r\n\t# 4        65     b'b'      65     b'b'\r\n\t# ..      ...      ...     ...      ...\r\n\t# 626    5966    b'aa'    5966    b'aa'\r\n\t# 627    5982     b'f'    5982     b'f'\r\n\t# 628    5985  b'ee11'    5985  b'ee11'\r\n\t# 629    5995     b'c'    5995     b'c'\r\n\t# 630    5996    b'gg'    5996    b'gg'\r\n\t# [631 rows x 4 columns]\r\n\r\nThe function computes the overlapping values based on the specified condition (function or string)\r\nand returns a DataFrame with the results. If `same_index_required` is set to True, it filters\r\nthe results to include only rows where the indices match.\r\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fstridesduplicatefinder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhansalemaos%2Fstridesduplicatefinder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fstridesduplicatefinder/lists"}