{"id":21711547,"url":"https://github.com/developerdavi/meli-crawler","last_synced_at":"2025-04-12T17:41:29.058Z","repository":{"id":40858582,"uuid":"240030754","full_name":"developerdavi/meli-crawler","owner":"developerdavi","description":"Basic web crawler API for getting products from MercadoLibre (BRL | MLB)","archived":false,"fork":false,"pushed_at":"2023-01-05T16:24:31.000Z","size":1654,"stargazers_count":4,"open_issues_count":13,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T00:06:35.276Z","etag":null,"topics":["api","crawler","meli-crawler","mercadolibre","mercadolibre-sdk","mercadolivre","mercadolivre-sdk","nextjs","now","products","react","zeit"],"latest_commit_sha":null,"homepage":"https://meli-crawler.devdavi.com.br","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developerdavi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-12T14:18:43.000Z","updated_at":"2023-04-04T17:16:11.000Z","dependencies_parsed_at":"2023-02-04T07:32:08.259Z","dependency_job_id":null,"html_url":"https://github.com/developerdavi/meli-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developerdavi%2Fmeli-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developerdavi%2Fmeli-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developerdavi%2Fmeli-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developerdavi%2Fmeli-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developerdavi","download_url":"https://codeload.github.com/developerdavi/meli-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248607885,"owners_count":21132625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","crawler","meli-crawler","mercadolibre","mercadolibre-sdk","mercadolivre","mercadolivre-sdk","nextjs","now","products","react","zeit"],"created_at":"2024-11-25T23:27:24.617Z","updated_at":"2025-04-12T17:41:29.036Z","avatar_url":"https://github.com/developerdavi.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Meli Crawler - Get products from MercadoLibre :brazil:\n\n### :rocket: Technologies\n\n- NodeJS\n- NextJS\n- MercadoLibre SDK\n- [ZEIT Now](https://zeit.co)\n\n### :book: Docs\n\nGet Insomnia workspace [here](https://raw.githubusercontent.com/developerdavi/meli-crawler/master/docs/Insomnia_2020-02-12.json), import it and start using right now :octocat:\n\n### :eyes: Example:\n\n#### Request\n\nBASE URL: `https://meli-crawler.devdavi.com.br`\n\nPOST `/api/products` \n```json\n{\n  \"search\": \"Notebook\",\n  \"limit\": 10\n}\n```\n\n*Ps.: Fetching products data from the same route using GET method returns cached results from crawler with improved perfomance (from 1s to 40ms).*\n**Sellers' data is always cached to improve reusable and *almost immutable* data usage.**\n\n#### Response\n\n```json\n[\n  {\n    \"name\": \"Notebook Dell Inspiron 3583-u05p Pentium 4gb 500gb 15.6 Linux\",\n    \"link\": \"https://produto.mercadolivre.com.br/MLB-1291517506-notebook-dell-inspiron-3583-u05p-pentium-4gb-500gb-156-linux-_JM\",\n    \"price\": 1719,\n    \"store\": \"DELLCOMPUTADORES\",\n    \"state\": \"São Paulo\"\n  },\n]\n```\n### :question: How it works?\n\nThe app uses NextJS to do server-side rendering and ZEIT Now to cache results to improve searches performance and save a lot of time.\n\nNextJS has routes API that allows us doing things like we do using ExpressJS, for example. Inside `/pages` we have our routes and their respective handlers. \n\nWe have to export there a React component or a custom response to clients. As we are creating an crawler, we just need to return json data, using `response.json(data)`, just like we do using the ExpressJS API. So, the learning curve is as minimum as possible.\n\n### :v: Contribution\n\nPlease, if you want to contribute:\n- Leave a star :star:\n- Do a fork\n- Create a `my-feature` branch\n- Create a pull request\n- Smile :smile: \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloperdavi%2Fmeli-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeveloperdavi%2Fmeli-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloperdavi%2Fmeli-crawler/lists"}