Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ahmedshahriar/bd-ponno
Scrapy Mongodb Djongo integrated API that scrapes popular e-commerce sites (10+) from Bangladesh
https://github.com/ahmedshahriar/bd-ponno
django django-rest-framework djongo mongodb mongodb-atlas python python37 restful-api scrapy web-scraping
Last synced: 2 days ago
JSON representation
Scrapy Mongodb Djongo integrated API that scrapes popular e-commerce sites (10+) from Bangladesh
- Host: GitHub
- URL: https://github.com/ahmedshahriar/bd-ponno
- Owner: ahmedshahriar
- License: apache-2.0
- Created: 2021-02-22T19:44:46.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-09-08T23:00:46.000Z (2 months ago)
- Last Synced: 2024-09-08T23:49:50.012Z (2 months ago)
- Topics: django, django-rest-framework, djongo, mongodb, mongodb-atlas, python, python37, restful-api, scrapy, web-scraping
- Language: Python
- Homepage:
- Size: 91.8 KB
- Stars: 5
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# bd-ponno
Scrapy MongoDB Django integrated API that scrapes popular e-commerce sites (10+) from Bangladesh.\
Added some extra functionalities for the API, ex-custom price, and product tag filter
**The project is no longer maintained.**
## Built With
```
Django==3.1.7
django-cors-headers==3.7.0
django-filter==2.4.0
django-heroku==0.3.1
djangorestframework==3.12.2
djongo==1.3.4
Scrapy==2.4.1
scrapy-djangoitem==1.1.1
pymongo==3.11.3
```
### Remarks
MongoDB Atlas is not a good option for djongo or remote web scraping, because djongo is just a connector for the underlying relational mapper with Django which causes additional delay, hence bad performance for remote scraping with Atlas. Also, you might run into lots of unintentional issues like - ['list' object has no attribute '_meta'](https://github.com/doableware/djongo/issues/136#issue-325361852) while checking the product details view in the admin panel (`ArrayReferenceField` class (`Category` model)).
### API preview
