Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/druzsan/justetf-scraping

Scraping the justETF
https://github.com/druzsan/justetf-scraping

etf justetf parsing python scraping stocks

Last synced: 3 months ago
JSON representation

Scraping the justETF

Awesome Lists containing this project

README 

        
# 💹 justETF Scraping



Scrape the [justETF](https://www.justetf.com).



## 🛠️ Installation



To use justETF scraping package in your project, install the actual version from GitHub:



```shell

pip install git+https://github.com/druzsan/justetf-scraping.git

```



If you are going to play [notebooks](./notebooks) through, use the following installation:



```shell

pip install justetf-scraping[all]@git+https://github.com/druzsan/justetf-scraping.git

```



## 🚀 Usage



### 📋 Scrape the [justETF Screener](https://www.justetf.com/en/find-etf.html)



Load overviews for all available (over 3300 at the moment) ETFs (requires a request for all ETF type: long-only, active, short & leveraged):



```python

import justetf_scraping



df = justetf_scraping.load_overview()

df

```



  

    

      

      wkn

      ticker

      valor

      name

      inception_date

      age_in_days

      age_in_years

      strategy

      domicile_country

      currency

      hedged

      securities_lending

      dividends

      ter

      replication

      size

      is_sustainable

      number_of_holdings

      yesterday

      last_week

      last_month

      last_three_months

      last_six_months

      last_year

      last_three_years

      last_five_years

      2023

      2022

      2021

      2020

      last_dividends

      last_year_dividends

      last_year_volatility

      last_three_years_volatility

      last_five_years_volatility

      last_year_return_per_risk

      last_three_years_return_per_risk

      last_five_years_return_per_risk

      max_drawdown

      last_year_max_drawdown

      last_three_years_max_drawdown

      last_five_years_max_drawdown

      asset_class

      instrument

      region

      exchange

    

    

      isin

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

      

    

  

  

    

      IE00B5BMR087

      A0YEDG

      SXR8

      10737041

      iShares Core S&P 500 UCITS ETF (Acc)

      2010-05-19

      5198

      14.241096

      Long-only

      Ireland

      USD

      False

      True

      Accumulating

      0.07

      Full replication

      78449

      False

      503

      13.67

      -1.20

      -5.42

      0.61

      5.04

      21.13

      34.36

      99.91

      21.54

      -13.30

      39.07

      8.04

      NaN

      NaN

      12.86

      17.99

      21.63

      1.64

      0.57

      0.69

      -33.71

      -9.09

      -17.11

      -33.71

      Equity

      ETF

      

      gettex, XETRA, London, Stuttgart, SIX Swiss Exchange, Borsa Italiana, Euronext Amsterdam

    

    

      IE00B4L5Y983

      A0RPWH

      EUNL

      10608388

      iShares Core MSCI World UCITS ETF USD (Acc)

      2009-09-25

      5434

      14.887671

      Long-only

      Ireland

      USD

      False

      True

      Accumulating

      0.20

      Optimized sampling

      67967

      False

      1429

      10.85

      -1.06

      -4.81

      -0.59

      4.28

      17.87

      25.66

      79.25

      19.55

      -12.96

      32.10

      6.13

      NaN

      NaN

      11.17

      15.25

      18.27

      1.59

      0.52

      0.68

      -33.91

      -8.75

      -16.88

      -33.91

      Equity

      ETF

      World

      gettex, XETRA, London, Stuttgart, SIX Swiss Exchange, Borsa Italiana, Euronext Amsterdam

    

    

      IE00B3XXRP09

      A1JX53

      VUSA

      18575508

      Vanguard S&P 500 UCITS ETF

      2012-05-22

      4464

      12.230137

      Long-only

      Ireland

      USD

      False

      False

      Distributing

      0.07

      Full replication

      35981

      False

      498

      13.63

      -1.23

      -5.45

      0.57

      5.01

      21.08

      34.31

      99.85

      21.54

      -13.29

      39.08

      8.05

      1.11

      1.33

      12.89

      18.13

      21.70

      1.63

      0.57

      0.68

      -33.70

      -9.09

      -17.28

      -33.70

      Equity

      ETF

      

      gettex, XETRA, London, Euronext Paris, Stuttgart, SIX Swiss Exchange, Borsa Italiana, Euronext Amsterdam

    

  



Further enrich the data with additional information (asset class, region, exchanges and instrument, it requires further requests):



```python

df = justetf_scraping.load_overview(enrich=True)

```



Load long-only ETFs (only requires one single request):



```python

df = justetf_scraping.load_overview(strategy="epg-longOnly")

```



Load MSCI World ETFs:



```python

df = justetf_scraping.load_overview(strategy="epg-longOnly", index="MSCI World")

```



### 📈 Scrape ETF Chart Data from justETF ([e.g.](https://www.justetf.com/en/etf-profile.html?isin=IE00B0M62Q58#chart))



Load the whole history of a chosen ETF by its ISIN:



```python

df = justetf_scraping.load_chart("IE00B0M62Q58")

df

```



  

    

      

      quote

      relative

      dividends

      cumulative_dividends

      quote_with_dividends

      relative_with_dividends

      reinvested_dividends

      quote_with_reinvested_dividends

      relative_with_reinvested_dividends

    

    

      date

      

      

      

      

      

      

      

      

      

    

  

  

    

      2005-10-28

      20.60

      0.000000

      0.0

      0.0

      20.60

      0.000000

      0.000000

      20.600000

      0.000000

    

    

      2005-10-29

      20.60

      0.000000

      0.0

      0.0

      20.60

      0.000000

      0.000000

      20.600000

      0.000000

    

    

      2005-10-30

      20.60

      0.000000

      0.0

      0.0

      20.60

      0.000000

      0.000000

      20.600000

      0.000000

    

    

      2005-10-31

      20.99

      1.893204

      0.0

      0.0

      20.99

      1.893204

      0.000000

      20.990000

      1.893204

    

    

      2005-11-01

      21.03

      2.087379

      0.0

      0.0

      21.03

      2.087379

      0.000000

      21.030000

      2.087379

    

    

      ...

      ...

      ...

      ...

      ...

      ...

      ...

      ...

      ...

      ...

    

    

      2024-08-05

      64.90

      215.048544

      0.0

      9.9

      74.80

      263.106796

      20.643909

      85.543909

      315.261692

    

    

      2024-08-06

      65.95

      220.145631

      0.0

      9.9

      75.85

      268.203883

      20.977901

      86.927901

      321.980101

    

    

      2024-08-07

      65.74

      219.126214

      0.0

      9.9

      75.64

      267.184466

      20.911102

      86.651102

      320.636419

    

    

      2024-08-08

      66.80

      224.271845

      0.0

      9.9

      76.70

      272.330097

      21.248276

      88.048276

      327.418814

    

    

      2024-08-09

      67.05

      225.485437

      0.0

      9.9

      76.95

      273.543689

      21.327798

      88.377798

      329.018435

    

  



6861 rows × 9 columns



Compare



```python

df = justetf_scraping.compare_charts(

    {

        "IE00B0M62Q58": justetf_scraping.load_chart("IE00B0M62Q58"),

        "IE00B0M63177": justetf_scraping.load_chart("IE00B0M63177"),

    },

    input_value="quote_with_dividends"

)

df

```



  

    

      

      IE00B0M62Q58

      IE00B0M63177

    

    

      date

      

      

    

  

  

    

      2005-11-18

      0.000000

      0.000000

    

    

      2005-11-19

      0.000000

      0.000000

    

    

      2005-11-20

      0.000000

      0.000000

    

    

      2005-11-21

      -0.539811

      -0.887436

    

    

      2005-11-22

      0.629780

      -0.934143

    

    

      ...

      ...

      ...

    

    

      2024-08-05

      236.482231

      120.784680

    

    

      2024-08-06

      241.205578

      123.960766

    

    

      2024-08-07

      240.260909

      127.136852

    

    

      2024-08-08

      245.029240

      127.697338

    

    

      2024-08-09

      246.153846

      128.304531

    

  



6840 rows × 2 columns



For further exploration examples, see [Jupyter Notebooks](notebooks/)



## ⚒️ Development Setup



To setup locally cloned project, first install [Poetry](https://python-poetry.org/):



```shell

pip install poetry

```



In the local project folder, install all dependencies and extras:



```shell

poetry install --all-extras

```



Activate local Poetry environment:



```shell

poetry shell

```



Setup [pre-commit hooks](https://pre-commit.com/):



```shell

pre-commit install

```



## Similar Projects



- https://github.com/AshNL/justETF-overview-scraper

- https://github.com/SimonMandlik/etf_filter



## Thanks



This project was inspired by

[this](https://stackoverflow.com/questions/64813023/scraping-dynamic-datatable-of-many-pages-but-same-url)

Stack Overflow question.