https://github.com/druzsan/justetf-scraping

Scraping the justETF
https://github.com/druzsan/justetf-scraping

etf justetf parsing python scraping stocks

Last synced: 3 months ago
JSON representation

Scraping the justETF

Host: GitHub
URL: https://github.com/druzsan/justetf-scraping
Owner: druzsan
License: mit
Created: 2022-05-26T08:40:23.000Z (about 3 years ago)
Default Branch: main
Last Pushed: 2025-01-18T15:23:58.000Z (5 months ago)
Last Synced: 2025-02-27T07:51:08.578Z (4 months ago)
Topics: etf, justetf, parsing, python, scraping, stocks
Language: Python
Homepage:
Size: 4.29 MB
Stars: 48
Watchers: 3
Forks: 15
Open Issues: 3
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# 💹 justETF Scraping

Scrape the [justETF](https://www.justetf.com).

## 🛠️ Installation

To use justETF scraping package in your project, install the actual version from GitHub:

```shell
pip install git+https://github.com/druzsan/justetf-scraping.git
```

If you are going to play [notebooks](./notebooks) through, use the following installation:

```shell
pip install justetf-scraping[all]@git+https://github.com/druzsan/justetf-scraping.git
```

## 🚀 Usage

### 📋 Scrape the [justETF Screener](https://www.justetf.com/en/find-etf.html)

Load overviews for all available (over 3400 at the moment) ETFs (requires a request for all ETF type: long-only, active, short & leveraged):

```python
import justetf_scraping

df = justetf_scraping.load_overview()
df
```

wkn
ticker
valor
name
inception_date
age_in_days
age_in_years
strategy
domicile_country
currency
...
last_year_volatility
last_three_years_volatility
last_five_years_volatility
last_year_return_per_risk
last_three_years_return_per_risk
last_five_years_return_per_risk
max_drawdown
last_year_max_drawdown
last_three_years_max_drawdown
last_five_years_max_drawdown

isin

IE00B5BMR087
A0YEDG
SXR8
10737041
iShares Core S&P 500 UCITS ETF USD (Acc)
2010-05-19
5415
14.835616
Long-only
Ireland
USD
...
15.73
18.00
20.09
0.67
0.67
0.88
-33.71
-13.39
-15.92
-17.11

IE00B4L5Y983
A0RPWH
EUNL
10608388
iShares Core MSCI World UCITS ETF USD (Acc)
2009-09-25
5651
15.482192
Long-only
Ireland
USD
...
13.79
15.27
17.02
0.73
0.72
0.97
-33.91
-11.24
-15.01
-16.88

IE00B3XXRP09
A1JX53
VUSA
18575508
Vanguard S&P 500 UCITS ETF (USD) Distributing
2012-05-22
4681
12.824658
Long-only
Ireland
USD
...
15.95
18.20
20.19
0.66
0.66
0.87
-33.70
-13.43
-16.10
-17.28

IE00B3YCGJ38
A1CYW7
P500
11358996
Invesco S&P 500 UCITS ETF
2010-05-20
5414
14.832877
Long-only
Ireland
USD
...
15.87
18.08
20.16
0.69
0.68
0.89
-33.69
-13.61
-16.06
-17.20

IE00BKM4GZ66
A111X9
IS3N
24209517
iShares Core MSCI Emerging Markets IMI UCITS E...
2014-05-30
3943
10.802740
Long-only
Ireland
USD
...
14.27
14.24
15.79
0.64
0.36
0.54
-34.34
-10.36
-17.53
-23.61

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

IE00BF4TW453
A2F4WK
XMWH
<NA>
WisdomTree Bund 30Y 3x Daily Short
2017-12-08
2655
7.273973
Short & Leveraged
Ireland
EUR
...
43.83
58.07
NaN
0.85
0.65
NaN
-49.46
-30.58
-49.46
NaN

XS2427363036
A3GWVS
2FLY
<NA>
WisdomTree STOXX Europe Travel & Leisure 2x Da...
2022-03-02
1110
3.041096
Short & Leveraged
Ireland
EUR
...
33.07
43.18
NaN
-0.46
0.19
NaN
-42.03
-36.10
-42.03
NaN

JE00B24DKJ77
A0V6Y0
4RTE
<NA>
WisdomTree Nickel 1x Daily Short
2008-02-22
6232
17.073973
Short & Leveraged
Jersey
USD
...
25.45
46.59
49.48
0.56
0.51
-0.51
-96.99
-23.84
-47.83
-89.44

XS2842095759
<NA>
3SBB
<NA>
GraniteShares 3x Short Alibaba Daily ETP
2022-02-02
1138
3.117808
Short & Leveraged
Ireland
USD
...
130.77
154.54
NaN
-0.73
-0.58
NaN
-99.88
-95.97
-99.88
NaN

XS2427474023
A3GWVN
O2IG
<NA>
WisdomTree STOXX Europe Oil & Gas 2x Daily Short
2022-03-02
1110
3.041096
Short & Leveraged
Ireland
EUR
...
32.07
41.70
NaN
-0.29
-0.59
NaN
-63.42
-23.03
-61.78
NaN

3486 rows × 42 columns

Further enrich the data with additional information (asset class, region, exchanges and instrument, it requires further requests):

```python
df = justetf_scraping.load_overview(enrich=True)
```

Load long-only ETFs (only requires one single request):

```python
df = justetf_scraping.load_overview(strategy="epg-longOnly")
```

Load MSCI World ETFs:

```python
df = justetf_scraping.load_overview(strategy="epg-longOnly", index="MSCI World")
```

### 📈 Scrape ETF Chart Data from justETF ([e.g.](https://www.justetf.com/en/etf-profile.html?isin=IE00B0M62Q58#chart))

Load the whole history of a chosen ETF by its ISIN:

```python
df = justetf_scraping.load_chart("IE00B0M62Q58")
df
```

quote
relative
dividends
cumulative_dividends
quote_with_dividends
relative_with_dividends
reinvested_dividends
quote_with_reinvested_dividends
relative_with_reinvested_dividends

date

2005-10-28
20.60
0.000000
0.0
0.00
20.60
0.000000
0.000000
20.600000
0.000000

2005-10-29
20.60
0.000000
0.0
0.00
20.60
0.000000
0.000000
20.600000
0.000000

2005-10-30
20.60
0.000000
0.0
0.00
20.60
0.000000
0.000000
20.600000
0.000000

2005-10-31
20.99
1.893204
0.0
0.00
20.99
1.893204
0.000000
20.990000
1.893204

2005-11-01
21.03
2.087379
0.0
0.00
21.03
2.087379
0.000000
21.030000
2.087379

...
...
...
...
...
...
...
...
...
...

2025-03-10
74.54
261.844660
0.0
10.22
84.76
311.456311
24.032814
98.572814
378.508807

2025-03-11
73.96
259.029126
0.0
10.22
84.18
308.640777
23.845814
97.805814
374.785503

2025-03-12
74.37
261.019417
0.0
10.22
84.59
310.631068
23.978004
98.348004
377.417494

2025-03-13
70.64
242.912621
0.0
10.22
80.86
292.524272
22.775396
93.415396
353.472795

2025-03-14
71.49
247.038835
0.0
10.22
81.71
296.650485
23.049449
94.539449
358.929362

7078 rows × 9 columns

Compare

```python
df = justetf_scraping.compare_charts(
{
"IE00B0M62Q58": justetf_scraping.load_chart("IE00B0M62Q58"),
"IE00B0M63177": justetf_scraping.load_chart("IE00B0M63177"),
},
input_value="quote_with_dividends"
)
df
```

IE00B0M62Q58
IE00B0M63177

date

2005-11-18
0.000000
0.000000

2005-11-19
0.000000
0.000000

2005-11-20
0.000000
0.000000

2005-11-21
-0.539811
-0.887436

2005-11-22
0.629780
-0.934143

...
...
...

2025-03-10
281.286550
147.874825

2025-03-11
278.677463
147.220925

2025-03-12
280.521817
147.594582

2025-03-13
263.742690
138.953760

2025-03-14
267.566352
141.569360

7057 rows × 2 columns

For further exploration examples, see [Jupyter Notebooks](notebooks/)

## ⚒️ Development Setup

To setup locally cloned project, first install [Poetry](https://python-poetry.org/):

```shell
pip install poetry
```

In the local project folder, install all dependencies and extras:

```shell
poetry install --all-extras
```

Activate local Poetry environment:

```shell
poetry shell
```

Setup [pre-commit hooks](https://pre-commit.com/):

```shell
pre-commit install
```

## Similar Projects

- https://github.com/AshNL/justETF-overview-scraper
- https://github.com/SimonMandlik/etf_filter

## Thanks

This project was inspired by
[this](https://stackoverflow.com/questions/64813023/scraping-dynamic-datatable-of-many-pages-but-same-url)
Stack Overflow question.

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/druzsan/justetf-scraping

Awesome Lists containing this project

README