https://github.com/ivan-kleshnin/paqmind-ptf

https://github.com/ivan-kleshnin/paqmind-ptf

Last synced: 6 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/ivan-kleshnin/paqmind-ptf
• Owner: ivan-kleshnin
• Created: 2020-09-10T05:32:15.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2020-09-15T14:51:28.000Z (about 5 years ago)
• Last Synced: 2025-03-27T19:18:56.289Z (8 months ago)
• Language: JavaScript
• Size: 2.93 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Paqmind-PTF

Create `dev.env`, filling the credentials as necessary:

```

export GITHUB_TOKEN=???

```

## Log of the steps, taken

### Step-0

```

@ download db1.csv from Google Drive 

$ . dev.env 

```

### Step-1

```

$ cat db0.csv | node src/1.reformat.js > db1.csv

```

### Step-2

```

$ cat db1.csv 1 | pv -q -L 1k | node src/2.scrape.js > db2.csv -- throttle up to 1Kib per second

```

### Step-3

```

$ node src/3.scrape-more.js | tail -n +2 >> db2.csv -- append to db2.csv without header

```