Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sayantikabanik/fp2
End-to-end AutoML based, price forecasting framework. All the tools and technologies used are open source.
https://github.com/sayantikabanik/fp2
dagster etl-pipeline flask heroku mlflow model-monitoring modeling pycaret
Last synced: 1 day ago
JSON representation
End-to-end AutoML based, price forecasting framework. All the tools and technologies used are open source.
- Host: GitHub
- URL: https://github.com/sayantikabanik/fp2
- Owner: sayantikabanik
- License: bsd-2-clause
- Created: 2021-12-12T06:04:41.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2022-02-10T10:16:28.000Z (almost 3 years ago)
- Last Synced: 2024-05-02T04:07:18.115Z (9 months ago)
- Topics: dagster, etl-pipeline, flask, heroku, mlflow, model-monitoring, modeling, pycaret
- Language: Jupyter Notebook
- Homepage: https://price-map-fp2.herokuapp.com/
- Size: 9.11 MB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Crop price forecasting
### Motivation
- 15 to 16 % of GDP is contributed by agriculture and its allied sectors
- 55 to 60 of Indian households are depending on income generated by Agri sectors: it emphasizes how big is the base of issue.
- Price of agricultural products, such as preservable and non-preservable are volatile throughout seasons. Unstable prices are heart-breaking for producers and painful for consumers Also, head-ache for serving governments: on the supply side, managing of input material & on procurement, ensuring of MSPs (Minimum supporting prices)
### Possible solution
- A better estimation of prices commodities on long (12 to 36 months) and short term
(3 to 5 months) can direct farmers towards the cultivation of remunerative crops
- In turn, the Price managing strategy will eases the burden on incentives provided by
governments. Also helps in the planning of input material
- Price forecasting and demand estimation help in planning input supplies (Either raw
or ready-to-consume) efficiently. In turn, the availability of products at affordable prices
to consumers.
### Data sources
- [Indian government data repository](data.gov.in)
- [National Agriculture Market](https://agmarknet.gov.in/)
- [DEPARTMENT OF CONSUMER AFFAIRS](https://fcainfoweb.nic.in/Reports/Report_Menu_Web.aspx)
- [India enviroment Portal](http://www.indiaenvironmentportal.org.in/media/iep/infographics/Rainfall%20in%20India/112%20years%20of%20rainfall.html)
### Useful Resources to read and implement
- [GitHub CI/CD](https://resources.github.com/ci-cd/)
- [PyTest](https://realpython.com/pytest-python-testing/)
- [Env management using conda](https://towardsdatascience.com/manage-your-python-virtual-environment-with-conda-a0d2934d5195)
- [Pre-Commit hooks](https://pre-commit.com/)
### Installing miniconda/light version of anaconda
- [Info + details](https://docs.conda.io/en/latest/miniconda.html)
### Commands to create and use conda environment
We are pinning versions of the packages
"conda list" look and feel after you complete the below steps
```python
# Name Version Build Channel
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h3422bc3_6 conda-forge
brotli-bin 1.0.9 h3422bc3_6 conda-forge
brotlipy 0.7.0 py38hea4295b_1003 conda-forge
ca-certificates 2021.10.8 h4653dfc_0 conda-forge
certifi 2021.10.8 py38h10201cd_1 conda-forge
cffi 1.15.0 py38hc67bbb8_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
charset-normalizer 2.0.9 pyhd8ed1ab_0 conda-forge
cryptography 36.0.1 py38h10d4710_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
distlib 0.3.4 pyhd8ed1ab_0 conda-forge
editdistance-s 1.0.0 py38h1670459_2 conda-forge
filelock 3.4.2 pyhd8ed1ab_0 conda-forge
fonttools 4.28.5 py38hea4295b_0 conda-forge
freetype 2.10.4 h17b34a0_1 conda-forge
identify 2.3.7 pyhd8ed1ab_0 conda-forge
idna 3.1 pyhd3deb0d_0 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
jbig 2.1 h3422bc3_2003 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9d h27ca646_0 conda-forge
kiwisolver 1.3.2 py38h1670459_1 conda-forge
lcms2 2.12 had6a04f_0 conda-forge
lerc 3.0 hbdafb3b_0 conda-forge
libblas 3.9.0 12_osxarm64_openblas conda-forge
libbrotlicommon 1.0.9 h3422bc3_6 conda-forge
libbrotlidec 1.0.9 h3422bc3_6 conda-forge
libbrotlienc 1.0.9 h3422bc3_6 conda-forge
libcblas 3.9.0 12_osxarm64_openblas conda-forge
libcxx 12.0.1 h168391b_0 conda-forge
libdeflate 1.8 h3422bc3_0 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libgfortran 5.0.0.dev0 11_0_1_hf114ba7_23 conda-forge
libgfortran5 11.0.1.dev0 hf114ba7_23 conda-forge
liblapack 3.9.0 12_osxarm64_openblas conda-forge
libopenblas 0.3.18 openmp_h5dd58f0_0 conda-forge
libpng 1.6.37 hf7e6567_2 conda-forge
libtiff 4.3.0 h74060c4_2 conda-forge
libwebp-base 1.2.1 h3422bc3_0 conda-forge
libzlib 1.2.11 hee7b306_1013 conda-forge
llvm-openmp 12.0.1 hf3c4609_1 conda-forge
lz4-c 1.9.3 hbdafb3b_1 conda-forge
matplotlib 3.5.1 py38h150bfb4_0 conda-forge
matplotlib-base 3.5.1 py38hb140015_0 conda-forge
more-itertools 8.12.0 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
ncurses 6.2 h9aa5885_4 conda-forge
nodeenv 1.6.0 pyhd8ed1ab_0 conda-forge
numpy 1.21.5 py38hb29071a_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 h062765e_1 conda-forge
openssl 1.1.1l h3422bc3_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py38h3777fb4_0 conda-forge
patsy 0.5.2 pyhd8ed1ab_0 conda-forge
pillow 8.4.0 py38h02acf36_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 py38h10201cd_2 conda-forge
pre-commit 2.16.0 py38h10201cd_0 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 py38h10201cd_4 conda-forge
pytest 6.2.5 py38h10201cd_1 conda-forge
python 3.8.12 hab31e5c_2_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.8 2_cp38 conda-forge
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py38hea4295b_3 conda-forge
readline 8.1 hedafd6a_0 conda-forge
requests 2.26.0 pyhd8ed1ab_1 conda-forge
scikit-learn 1.0.2 py38h2cd4032_0 conda-forge
scipy 1.7.3 py38hd0c9ec0_0 conda-forge
seaborn 0.11.2 hd8ed1ab_0 conda-forge
seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge
setuptools 60.1.1 py38h10201cd_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.37.0 h72a2b83_0 conda-forge
statsmodels 0.13.1 py38h691f20f_0 conda-forge
threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge
tk 8.6.11 he1e0b03_1 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py38hea4295b_2 conda-forge
unicodedata2 14.0.0 py38hea4295b_0 conda-forge
urllib3 1.26.7 pyhd8ed1ab_0 conda-forge
virtualenv 20.4.7 py38h10201cd_1 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xz 5.2.5 h642e427_1 conda-forge
yaml 0.2.5 h642e427_0 conda-forge
zlib 1.2.11 hee7b306_1013 conda-forge
zstd 1.5.1 h861e0a7_0 conda-forge
```
```shell
conda env create --file environment.yml
```
```shell
conda activate fp2
```
```shell
conda list
```
```shell
conda info
```
```shell
conda deactivate
```
### Installing the package in local
```shell
pip install -e .
```
### Basic flow how to make best use of the workflow
- Clone the repo
- Create the env using the above commands
- Install the package
- All the tests goes into the `tests` directory
- Any test experiements eg- `pickle files generated from autoML` goes under `experiments` directory
- All modelling and related details into `forecasting_framework`, create subdirectories as required
- Under forecasting_framework there are three submodules `utils`, `model`, `data`
- `utils` reusable code components
- `model` all modelling aspects (python scripts are highly encouraged)
- `data` pipepine and raw data
### How to contribute to the repo
- Create a separate branch for your usecase
- Raise PR (dont commit to main under any circumstance)
### Running the data pipline
- `python pipeline.py` - returns the processed data in ~/data directory
- `dagit -f pipeline.py` - Dagster UI
### Running tests
Install pytest (it is not part of environment.yml/package)
It should be installed locally
- `pip install pytest==6.2.5`
- `pytest tests`