https://github.com/hironsan/wiki-article-dataset

Wikipedia article dataset
https://github.com/hironsan/wiki-article-dataset

dataset machine-learning natural-language-processing

Last synced: 12 months ago
JSON representation

Wikipedia article dataset

• Host: GitHub
• URL: https://github.com/hironsan/wiki-article-dataset
• Owner: Hironsan
• License: mit
• Created: 2019-05-06T10:12:18.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2019-05-10T03:54:08.000Z (about 7 years ago)
• Last Synced: 2025-04-02T23:12:07.264Z (over 1 year ago)
• Topics: dataset, machine-learning, natural-language-processing
• Language: Jupyter Notebook
• Homepage:
• Size: 52.7 KB
• Stars: 12
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# wiki-article-dataset

wiki-article-dataset is a text corpus generated from japanese wikipedia(20181220 dump).

You can download this corpus from the following link:

* [ja.wikipedia_250k.zip](https://s3-ap-northeast-1.amazonaws.com/dev.tech-sketch.jp/chakki/public/ja.wikipedia_250k.zip)

## Format

Each line represents an article and it contains sentences divided by tab:

```python

ニューヨーク 市 の 建築 ニューヨーク 市 の 建築 ...

...

```

Each sentence is tokenized by MeCab and IPADIC.

## Example use cases

### Learning Sentence Embeddings

Learning the mapping between sentences. Given the embedding of one sentence, one can find semantically similar/relevant sentences.

![](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2018/11/07/ss1.gif)

Ref: 

## Requirements

* Python 3.6+

* MeCab

* pipenv

## Make corpus by yourself

You can download this corpus. But you can make the corpus by yourself.

Simply run:

```commandline

$ ./build.sh

```