https://github.com/benhoyt/py-1brc

Optimising the One Billion Row Challenge (1BRC) in Python
https://github.com/benhoyt/py-1brc

1brc optimization python

Last synced: 2 months ago
JSON representation

Optimising the One Billion Row Challenge (1BRC) in Python

• Host: GitHub
• URL: https://github.com/benhoyt/py-1brc
• Owner: benhoyt
• License: mit
• Created: 2024-03-21T10:23:15.000Z (9 months ago)
• Default Branch: master
• Last Pushed: 2024-03-25T07:18:07.000Z (9 months ago)
• Last Synced: 2024-06-15T06:42:52.621Z (6 months ago)
• Topics: 1brc, optimization, python
• Language: Python
• Homepage:
• Size: 11.7 KB
• Stars: 5
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# py-1brc

This repo contains the source code for a talk I gave on Python performance at the Christchurch Python User Group. My talk looked at increasingly faster solutions to the [One Billion Row Challenge](https://www.morling.dev/blog/one-billion-row-challenge/), showing how to profile and optimise Python as we went.

Some of the solutions import [line_profiler](https://pypi.org/project/line-profiler/), so to run them, first set up a virtual environment and install that package:

```

$ python3 -m venv venv             # create the virtual environment

$ source venv/bin/activate         # active it

$ pip install -r requirements.txt  # install modules (line_profiler)

```

Then run a solution as follows (r1 through r7):

```

$ time python3 r7.py measurements.txt >results.txt

```