An open API service indexing awesome lists of open source software.

https://github.com/okfn-brasil/querido-diario

📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
https://github.com/okfn-brasil/querido-diario

civic-tech data-science digital-public-goods dpg governments-gazettes govtech hacktoberfest open-data politics scraping sdg-16 spider

Last synced: 28 days ago
JSON representation

📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.

Awesome Lists containing this project

README

          

**English (US)** | [Português (BR)](README.md)


Querido Diário

Querido Diário



Tests


License


Version

# Querido Diário
Within the [Querido Diário ecosystem](https://docs.queridodiario.ok.org.br/en/latest/contributing/contribution-guide.html#ecosystem), this repository is responsible for **scraping official gazettes publishing sites**

Find out more about [technologies](https://queridodiario.ok.org.br/tecnologia) and [history](https://queridodiario.ok.org.br/sobre) of the project on the [Querido Diário website](https://queridodiario.ok.org.br)

# Summary
- [How to contribute](#how-to-contribute)
- [Development Environment](#development-environment)
- [How to run](#how-to-run)
- [Troubleshooting](#troubleshooting)
- [Support](#support)
- [Thanks](#thanks)
- [Open Knowledge Brazil](#open-knowledge-brazil)
- [Complete Documentation](#complete-documentation)
- [License](#license)

# How to contribute



catarse

Thank you for considering contributing to Querido Diário! :tada:

You can find how to do it at [CONTRIBUTING-en-US.md](docs/CONTRIBUTING-en-US.md)!

Also, check the [Querido Diário documentation](https://docs.queridodiario.ok.org.br/en/latest/) to help you.

# Development Environment
You need to have [uv](https://docs.astral.sh/uv/) installed, which takes care of installing the right Python version and managing the project's virtual environment.

The commands below set it up. They consist of installing the dependencies (including development ones) from the `querido_diario_raspadores` directory and the code standardization tool `pre-commit`.

``` console
cd querido_diario_raspadores
uv sync --all-groups
cd ..
uv tool install pre-commit
pre-commit install
```

> Configuration on other operating systems is available at ["how to setup the development environment"](docs/CONTRIBUTING-en-US.md#how-to-setup-the-development-environment), including more details for those who want to contribute to the repository.

# How to run
To try running a scraper already integrated into the project or to test what you are developing, follow the commands:

1. Go to the `querido_diario_raspadores` directory:
```console
cd querido_diario_raspadores
```
2. Check the available scrapers list:
```console
uv run scrapy list
```
3. Run a listed scraper:
```console
uv run scrapy crawl //example: scrapy crawl ba_acajutiba
```
4. The official gazettes collected from scraping will be saved in the `querido_diario_raspadores/data` folder

5. When executing item 3, the scraper will collect all official gazettes from the publishing site of that municipality since the first digital edition. For smaller runs, use flags in the run command:

- `start=YYYY-MM-DD`: will set the collecting start date.
```console
uv run scrapy crawl -a start=
```
- `end=YYYY-MM-DD`: will set the collecting end date. If omitted, it will assume the date of the day it is being executed.
```console
uv run scrapy crawl -a end=
```

# Troubleshooting
Check out the [troubleshooting](docs/TROUBLESHOOTING-en-US.md) file to resolve the most common issues with project environment setup.

# Community



Discord Invite

Join our [community server](https://go.ok.org.br/discord) for exchanges about projects, questions, requests for help with contributions and talk about civic innovation in general.

# Thanks
This project is maintained by Open Knowledge Brazil and made possible thanks to the technical communities, volunteers and financial donors, in addition to partner universities, companies supporters and funders.

Meet [who supports Querido Diario](https://queridodiario.ok.org.br/apoie#quem-apoia).

# Open Knowledge Brazil



Bluesky Follow


Instagram Follow


LinkedIn Follow


Mastodon Follow

[Open Knowledge Brazil](https://ok.org.br/) is a non-profit civil society organization whose mission is to use and develop civic tools, projects, public policy analysis and data journalism to promote free knowledge in the various fields of society.

All work produced by OKBR is openly and freely available.

# Complete Documentation
Here is a complete list of all documentation available in the `docs/` folder:

- [Code of Conduct](docs/CODE_OF_CONDUCT.md) | [Code of Conduct (EN)](docs/CODE_OF_CONDUCT-en-US.md)
- [Contribution Guide](docs/CONTRIBUTING.md) | [Contribution Guide (EN)](docs/CONTRIBUTING-en-US.md)
- [Support](docs/SUPPORT.md) | [Support (EN)](docs/SUPPORT-en-US.md)
- [Troubleshooting](docs/TROUBLESHOOTING.md) | [Troubleshooting (EN)](docs/TROUBLESHOOTING-en-US.md)

# License

Code licensed under the [MIT License](LICENSE.md).