https://github.com/dogsheep/hacker-news-to-sqlite
Create a SQLite database containing data pulled from Hacker News
https://github.com/dogsheep/hacker-news-to-sqlite
datasette datasette-io datasette-tool dogsheep hacker-news
Last synced: 2 months ago
JSON representation
Create a SQLite database containing data pulled from Hacker News
- Host: GitHub
- URL: https://github.com/dogsheep/hacker-news-to-sqlite
- Owner: dogsheep
- License: apache-2.0
- Created: 2020-03-21T04:02:05.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2023-03-26T22:22:39.000Z (over 2 years ago)
- Last Synced: 2025-06-26T01:41:33.439Z (3 months ago)
- Topics: datasette, datasette-io, datasette-tool, dogsheep, hacker-news
- Language: Python
- Homepage:
- Size: 18.6 KB
- Stars: 51
- Watchers: 2
- Forks: 7
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# hacker-news-to-sqlite
[](https://pypi.org/project/hacker-news-to-sqlite/)
[](https://github.com/dogsheep/hacker-news-to-sqlite/releases)
[](https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing data fetched from [Hacker News](https://news.ycombinator.com/).
## How to install
$ pip install hacker-news-to-sqlite
## Usage
$ hacker-news-to-sqlite user hacker-news.db your-username
Importing items: 37%|███████████ | 845/2297 [05:09<11:02, 2.19it/s]
Imports all of your Hacker News submissions and comments into a SQLite database called `hacker-news.db`.
$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218
Fetches the entire comments tree in which any of those content IDs appears.
## Browsing your data with Datasette
You can use [Datasette](https://datasette.readthedocs.org/) to browse your data. Install Datasette like this:
$ pip install datasette
Now run it against your `hacker-news.db` file like so:
$ datasette hacker-news.db
Visit `https://localhost:8001/` to search and explore your data.
You can improve the display of your data usinng the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) and [datasette-render-html](https://github.com/simonw/datasette-render-html) plugins. Install them like this:
$ pip install datasette-render-timestamps datasette-render-html
Now save the following configuration in a file called `metadata.json`:
```json
{
"databases": {
"hacker-news": {
"tables": {
"items": {
"plugins": {
"datasette-render-html": {
"columns": [
"text"
]
},
"datasette-render-timestamps": {
"columns": [
"time"
]
}
}
},
"users": {
"plugins": {
"datasette-render-timestamps": {
"columns": [
"created"
]
}
}
}
}
}
}
}
```
Run Datasette like this:
$ datasette -m metadata.json hacker-news.db
The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.