{"id":51576831,"url":"https://github.com/quantum-software-development/fii-investor-intelligence-platform","last_synced_at":"2026-07-11T01:01:18.362Z","repository":{"id":357284408,"uuid":"1236255037","full_name":"Quantum-Software-Development/fii-investor-intelligence-platform","owner":"Quantum-Software-Development","description":"Big Data and NLP platform for Real Estate Investment Fund (FII) intelligence, sentiment analysis, and distributed analytics.","archived":false,"fork":false,"pushed_at":"2026-07-02T02:49:48.000Z","size":12181,"stargazers_count":2,"open_issues_count":5,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-02T04:26:14.849Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://github.com/Quantum-Software-Development/fii-investor-intelligence-platform","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Quantum-Software-Development.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":"docs/governance/data_dictionary.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"Quantum-Software-Development","Custom":"https://github.com/sponsors/Quantum-Software-Development/card"}},"created_at":"2026-05-12T04:52:12.000Z","updated_at":"2026-07-02T02:54:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Quantum-Software-Development/fii-investor-intelligence-platform","commit_stats":null,"previous_names":["quantum-software-development/5-cybersecurity-social-engineering-project-fii-marketing-intelligence-platform","quantum-software-development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform","quantum-software-development/5-cybersecurity-social-engineering-fii-investor-intelligence-platform"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Quantum-Software-Development/fii-investor-intelligence-platform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantum-Software-Development%2Ffii-investor-intelligence-platform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantum-Software-Development%2Ffii-investor-intelligence-platform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantum-Software-Development%2Ffii-investor-intelligence-platform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantum-Software-Development%2Ffii-investor-intelligence-platform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Quantum-Software-Development","download_url":"https://codeload.github.com/Quantum-Software-Development/fii-investor-intelligence-platform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Quantum-Software-Development%2Ffii-investor-intelligence-platform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35347554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-11T01:01:16.353Z","updated_at":"2026-07-11T01:01:18.354Z","avatar_url":"https://github.com/Quantum-Software-Development.png","language":"HTML","funding_links":["https://github.com/sponsors/Quantum-Software-Development","https://github.com/sponsors/Quantum-Software-Development/card"],"categories":[],"sub_categories":[],"readme":"\n\u003c!-- ======================================= ⚡️ Start DEFAULT HEADER ===========================================  --\u003e\n\u003c!-- ========= START LANGUAGE BUTTON ========= --\u003e\n**\\[[🇧🇷 Português](README.pt_BR.md)\\] \\[**[🇬🇧 English](README.md)**\\]**\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END LANGUAGE BUTTON ========= --\u003e\n\n\n\u003c!-- ========= START REPO TITLE ========= --\u003e\n# \u003cp align=\"center\"\u003e [Investor Intelligence Platform  🇧🇷  Brazilian FIIs]() \n\n### \u003cp align=\"center\"\u003e Real Estate Investment Funds (FIIs) - Market Intelligence \u0026 Behavioral Analytics\n\n\u003cbr\u003e\n\n$$\\Huge {\\textbf{\\color{green} CRISP-DM} \\space \\textbf{\\color{white} •} \\space \\textbf{\\color{yellow}  Data Lakehouse} \\space \\textbf{\\color{white} •} \\space \\textbf{\\color{green} NLP} \\space \\textbf{\\color{white} •} \\space \\textbf{\\color{yellow} Responsible AI} \\space \\textbf{\\color{white} •} \\space \\textbf{\\color{green} Regulatory Alignment}}$$\n\n\u003cbr\u003e\n\n### \u003cp align=\"center\"\u003e ***An institutional-grade intelligence platform for monitoring, structuring, ranking, and interpreting Brazilian Real Estate Investment Fund (FII) signals from financial media, research portals, and investor communities.***\n\n\u003cbr\u003e\n\n#### \u003cp align=\"center\"\u003e [Big Data]() • [PySpark]() • [MapReduce Word Count]() • [NLP]() • [TF-IDF]() • [BM25 Ranking]() • [Hybrid Retrieval]() • [FAISS + Multilingual Embeddings]() • [Web Scraping]() • [TOFU/MOFU/BOFU]() • [CRISP-DM]() • [FastAPI]() • [Streamlit]() • [Docker]() • [Responsible AI]() • [LGPD]() • [EU AI Act Alignment]()\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END REPO TITLE ========= --\u003e\n\n\u003c!-- ========= START SPONSOR BADGES ========= --\u003e\n### \u003cp align=\"center\"\u003e [![Sponsor Quantum Software Development](https://img.shields.io/badge/Sponsor-Quantum%20Software%20Development-brightgreen?logo=GitHub)](https://github.com/sponsors/Quantum-Software-Development)\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END SPONSOR BADGES ========= --\u003e\n\n\u003c!-- ========= START DEMO VIDEO ========= --\u003e\n\u003cp align=\"center\"\u003e\n   \u003cimg src=\"https://github.com/user-attachments/assets/791a69e2-d09a-429f-9257-f6667fff5c04 \" /\u003e\n\n \u003c/p\u003e\n\n\u003c!--\n#### 🖤 Creative Direction, Music Curation \u0026 Editing by Fab⚡️  \n##### 🎶 [Soundtrack:]() \"Canon in D\" — Johann Pachelbel\n--\u003e\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END DEMO VIDEO ========= --\u003e\n\n\n\u003c!-- ========= START Institutional INFO ========= --\u003e\n## 🎓 Academic \n\n\u003cbr\u003e\n\n[**Institution:**]() Pontifical Catholic University of São Paulo (PUC-SP)  \u003cbr\u003e\n[**School**:]() FACEI — Faculty of Exact Sciences and Informatics \u003cbr\u003e\n[**Bachelor’s Program:**]() Human-Centerd AI \u0026 Data Science • 5th Semester • 2026   \u003cbr\u003e\n[**Course:**]() AI Security, Cybersecurity \u0026 Social Engineering   \u003cbr\u003e\n[**Methodology:**]()  CRISP-DM (Cross-Industry Standard Process for Data Mining)  \u003cbr\u003e\n**Professors** [✨ Carlos Eduardo Paes](https://www.linkedin.com/in/carlos-eduardo-de-barros-paes-ph-d-7b137a4/)  and  [✨ Eduardo Savino Gomes]() \u003cbr\u003e\n**Project Author:** [Fabiana ⚡️ Campanari](https://linktr.ee/fabianacampanari) \n\n\u003cbr\u003e\u003cbr\u003e\n\n#\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END Institutional INFO ========= --\u003e\n\n\n\u003c!-- ========= START Dashboard Streamlit ========= --\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cimg \n      src=\"https://img.shields.io/badge/FIIs_Marketing_Intelligence_Dashboard-Streamlit-0f172a?style=for-the-badge\u0026logo=streamlit\u0026logoColor=white\" \n      alt=\"FIIs Marketing Intelligence Dashboard\"\n      style=\"height: 32px; width: auto;\"\n    /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003c!-- ========= END Dashboard Streamlit ========= --\u003e\n\n\u003c!-- ========= START REACT APP ========= --\u003e\n\u003cp align=\"center\"\u003e\n\n  \u003ca href=\"https://euphonious-churros-b68a51.netlify.app\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cimg \n      src=\"https://img.shields.io/badge/React-Interactive%20FIIs%20Analytics%20Slides-14532d?style=for-the-badge\u0026logo=react\u0026logoColor=white\" \n      alt=\"React Interactive FIIs Analytics Slides\"\n      style=\"height: 30px; width: auto;\"\n    /\u003e\n  \u003c/a\u003e\n  \u003c!-- ========= END REACT APP ========= --\u003e\n\n\u003c!-- ========= START PPTX ========= --\u003e\n  \u003ca href=\"\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cimg \n      src=\"https://img.shields.io/badge/FIIs_Strategic_Presentation-PPTX-0f766e?style=for-the-badge\u0026logo=microsoftpowerpoint\u0026logoColor=white\" \n      alt=\"FIIs Strategic Presentation PPTX\"\n      style=\"height: 30px; width: auto;\"\n    /\u003e\n  \u003c/a\u003e\n\n\u003c/p\u003e\n\u003c!-- ========= END PPTX ========= --\u003e\n\n\u003c!-- ========= START DATA ANALYSING REPORT ========= --\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"\"\u003e\n    \u003cimg \n      src=\"https://img.shields.io/badge/FIIs_Market_Analysis-Executive%20Report-134e4a?style=for-the-badge\u0026logo=googleanalytics\u0026logoColor=white\u0026labelColor=022c22\" \n      alt=\"FIIs Market Analysis Executive Report\"\n    /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\u003cbr\u003e\n\n#\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END DATA ANALYSING REPORT ========= --\u003e\n\u003c!-- ===================== END BADGE GROUP 1 ===================== --\u003e\n\n\n\u003c!-- ========= START NOTE ========= --\u003e\n\u003e [!TIP]\n\u003e \n\u003e ↗ **[Explore the Full Course Repository](https://github.com/Quantum-Software-Development/1-Cybersecurity-SocialEngineering_Hub))**\n\u003e\n\u003e ###  Real Estate Investment Funds (FIIs) 🇧🇷 Market Intelligence \u0026 Behavioral Analytics \u003cbr\u003e\u003cbr\u003e\n\u003e \n\u003e A scalable platform combining [**Big Data**](), [**PySpark**](), [**MapReduce**](), [**Word Count**](), [**NLP**](), [**TF-IDF**](), [**BM25**](), [**FAISS**](), [p**Multilingual Embeddings**](), [**Web Scraping**](), [**Hybrid RAG**]() and [**AI-Assisted Analytics**]() to transform large-scale financial discussions into actionable insights for FIIs.\n\u003e\n\u003e \u003cbr\u003e\n\u003e\n\u003e  #\n\u003e\n\u003e \u003cbr\u003e\u003cbr\u003e\n\u003e\n\u003e $$\\Huge {\\textbf{\\color{green} Where market discussions become investment narratives…}}$$\n\u003e\n\u003e $$\\Huge {\\textbf{\\color{yellow} because markets talk a lot...}}$$\n\u003e \n\u003e $$\\Huge {\\textbf{\\color{green} intelligent systems just listen better}}$$\n\u003e\n\u003e ### \u003cp align=\"center\"\u003e ⚡\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n#\n\n\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END NOTE ========= --\u003e\n\n\u003c!-- ========= START !WARNING] ========= --\u003e\n\u003e [!WARNING]\n\u003e\n\u003e \u003cbr\u003e\n\u003e ⚠️ Projects may be publicly shared when permitted.  \n\u003e The focus is on applied, hands-on learning with real datasets in AI governance and security contexts.  \n\u003e All sensitive content remains protected in private repositories when required.  \u003cbr\u003e\u003cbr\u003e \n\u003e \n\u003e ⚠️ Disclaimer\n\u003e Plataforma exclusivamente educacional e analítica. Não constitui recomendação de investimento.\n\n\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\u003cbr\u003e\n\u003c!-- ========= END!WARNING]========= --\u003e\n\n## Table of Contents\n\n1. [Academic Context](#-academic-context)\n2. [Overview](#overview)\n3. [System Overview](#system-overview)\n4. [What This Platform Delivers](#-what-this-platform-delivers)\n5. [Why This Matters](#-why-this-matters)\n6. [Architecture and Pipeline](#-architecture-and-pipeline)\n7. [Notebooks Executive Summary](#-notebooks-executive-summary)\n8. [Notebooks NB00–NB07: Technical Report](#-notebooks-nb00nb07-technical-report)\n9. [The 3 Core Techniques + FAISS Semantic Layer](#-the-3-core-techniques--faiss-semantic-layer-mapreduce--tf-idf--bm25--embeddings)\n10. [Data Sources — 21 Sources](#-data-sources--21-monitored-sources)\n11. [Big Data Infrastructure](#-big-data-infrastructure)\n12. [Dependencies and requirements.txt](#-dependencies-and-requirementstxt)\n13. [CRISP-DM Methodology](#-crisp-dm-methodology)\n14. [Marketing Funnel: TOFU, MOFU and BOFU in the Project](#-marketing-funnel-tofu-mofu-and-bofu-in-the-project)\n15. [RAG Chatbot: Groq + Gemini (Automatic Fallback)](#-rag-chatbot-groq--gemini-automatic-fallback)\n16. [Governance](#-governance)\n17. [Folder Structure](#-folder-structure)\n18. [How to Run](#-how-to-run)\n19. [Deployment \u0026 Automation Workflow](#deployment--automation-workflow)\n20. [Which requirements*.txt to Use in Each Scenario](#-which-requirementstxt-to-use-in-each-scenario)\n21. [Makefile — Full Reference](#-makefile--full-reference)\n22. [Expected Outputs](#-expected-outputs)\n23. [Technical Glossary](#-technical-glossary)\n24. [Repository and Project Links](#-repository-and-project-links)\n25. [References](#-references)\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [🎓 Academic and Institutional Context]()\n\n\u003cbr\u003e\n\nThis project was developed at PUC-SP in the courses of cybersecurity, social engineering, data engineering and Big Data analytics applied to financial markets. The original requirement focused on demonstrating a distributed word count solution using PySpark and the MapReduce paradigm.\n\nFrom this starting point, the repository was extended to incorporate more advanced analytical techniques (TF-IDF, BM25, contextual sentiment), a serving architecture with FastAPI + RAG and a structured pipeline oriented towards FII marketing intelligence.\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [Academic Requirements Met]()\n\n\u003cbr\u003e\n\n| [Requirement]() | [Implementation]() |\n| :-- | :-- |\n| [Distributed Computing]() | PySpark · RDD MapReduce · SparkSession |\n| [Big Data Architecture]() | Medallion (Bronze → Silver → Gold) |\n| [Machine Learning]() | TF-IDF · BM25 · Semantic Embeddings · Sentiment Analysis |\n| [Vector Search]() | FAISS · Dense Index · Multilingual PT-BR Embeddings |\n| [NLP]() | PT-BR Tokenization · FII Lexicon · Signal Flags |\n| [Data Governance]() | LGPD · EU AI Act · Responsible AI · XAI |\n| [REST API]() | FastAPI · Uvicorn |\n| [RAG / LLM]() | Groq (openai/gpt-oss-20b, primary) + Gemini 2.5 Flash (automatic fallback) |\n| [Visualization]() | Streamlit · Plotly |\n| [Cybersecurity]() | Narrative surface analysis · Social Engineering awareness |\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [Product Overview and Definition]()\n\n\u003cbr\u003e\n\nThe [**Investor Intelligence Platform 🇧🇷 FIIs Brazil**]() is not just an academic Big Data exercise. It is an investor intelligence platform for Brazilian Real Estate Investment Funds (FIIs), designed to transform fragmented public financial discussions into structured, searchable, explainable and decision-oriented market intelligence.\n\nInstead of being a simple dashboard, the system operates as an end-to-end analytical environment that:\n\n[-]() collects data from 21 sources (RSS · scraping · Reddit)\u003cbr\u003e\n[-]()organizes them in a Bronze/Silver/Gold architecture \u003cbr\u003e\n[-]() enriches them with hybrid retrieval (TF-IDF + BM25 + FAISS semantic search with multilingual PT-BR embeddings), FII PT-BR sentiment and explainable marketing intelligence signals   \u003cbr\u003e\n[-]() exposes results via [**FastAPI + RAG + Groq chatbot + Streamlit**]()\n\n\u003cbr\u003e\n\n```mermaid\n%%{init:{\n'theme':'dark',\n'themeVariables':{\n'background':'#090d13',\n'primaryTextColor':'#F5F7FA',\n'lineColor':'#2dd4bf'\n}}}%%\n\ngraph LR\n\nSRC[\"21 SOURCES\u003cbr/\u003eRSS • Scraping • Social\"]:::setup\n\nPIPE[\"NLP PIPELINE\u003cbr/\u003ePySpark • MapReduce\u003cbr/\u003eTF-IDF • BM25 • FAISS\"]:::gold\n\nGOLD[\"GOLD LAYER\u003cbr/\u003eParquet artifacts\"]:::bronze\n\nAPI[\"FASTAPI\u003cbr/\u003eREST + RAG\"]:::dash\n\nDASH[\"STREAMLIT\u003cbr/\u003eAnalytics Dashboard\"]:::dash\n\nLLM[\"LLM LAYER\u003cbr/\u003eGroq + Gemini\"]:::llm\n\nSRC --\u003e PIPE --\u003e GOLD\nGOLD --\u003e API\nGOLD --\u003e DASH\nAPI --\u003e LLM\nDASH --\u003e LLM\n\nclassDef setup fill:#0d2137,stroke:#00d2ff,color:#F5F7FA,stroke-width:2.5px;\nclassDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;\nclassDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;\nclassDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;\nclassDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;\nclassDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## [🎯 Objectives]()\n\n\u003cbr\u003e\n\n1. [**Complete end-to-end pipeline**]() — from ingestion to analytical output. \u003cbr\u003e\n2. [**Distributed processing + NLP**]() — PySpark MapReduce combined with TF-IDF, BM25 and contextual sentiment. \u003cbr\u003e\n3. [**RAG over FII corpus**]() — hybrid retrieval via TF-IDF, BM25 and FAISS-backed multilingual PT-BR embeddings, followed by contextual generation via Groq. \u003cbr\u003e\n4. [**Cybersecurity and Social Engineering**]() — security perspective in interpreting channels and narratives.\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [👥 Target Audience]()\n\n\u003cbr\u003e\n\n[-]() Asset and fund managers who monitor investor perception \u003cbr\u003e\n[-]() Financial analysts who track market narratives \u003cbr\u003e\n[-]() Marketing teams interested in FII visibility and engagement \u003cbr\u003e\n[-]() Academic evaluators assessing Big Data, Spark, NLP and RAG \u003cbr\u003e\n[-]() Recruiters and technical portfolio reviewers\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [ Why This Matters ❓]()\n\n\u003cbr\u003e\n\nAnalysts, managers and financial communication teams face:\n\n[-]() information dispersed across dozens of portals and communities \u003cbr\u003e\n[-]() high noise-to-signal ratio in market discussions \u003cbr\u003e\n[-]() difficulty tracking how sentiment and narratives evolvev \u003cbr\u003e\n[-]()  lack of transparent tools aligned with LGPD and the EU AI Act\n\n\u003cbr\u003e\n\n\u003e [!TIP]\n\u003e This platform addresses this gap with 21 monitored sources, a Bronze/Silver/Gold pipeline and reproducible, interpretable analytics.\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## [ Source Coverage]()\n\nThe platform monitors a curated set of editorial and behavioral sources relevant to the Brazilian FII ecosystem. Instead of treating all inputs as an undifferentiated corpus, the project distinguishes:\n\n\u003cbr\u003e\n\n[-]() [**Editorial RSS sources**]() — collected via structured feeds \u003e\u003cbr\u003e\n[-]() [**Editorial portals via scraping**]() — controlled extraction of public metadata \u003e\u003cbr\u003e\n[-]() [**Behavioral social sources**]() — Reddit as a community sentiment layer\n\n\u003cbr\u003e\n\n\u003e [!IMPORTANT]\n\u003e Detailed documentation per source: [`docs/data_sources.md`](https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform/blob/2b697bb54a78f4d31424ecd334466f9fc4a8d6e0/docs/data_sources.md)\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## [Collection Strategy by Source Type]()\n\n\u003cbr\u003e\n\n### [***RSS-First***]()\n\n\nWhen available, [**RSS**]() is preferred: lower extraction cost, native structured metadata, no risk of breakage due to HTML layout changes, reliable scheduling.\n\n\u003cbr\u003e\n\n### [***Scraping as Controlled Fallback***]()\n\nWhen RSS is unavailable or unstable, controlled HTML extraction of public pages. It does not simulate human navigation — it collects observable metadata (titles, links, timestamps, categories, excerpts).\n\n\n### [***Collection of Social Sources***]()\n\nReddit follows a separate logical path because it represents conversational and community-based data.\nIt is treated as a behavioral and discursive input layer that complements editorial coverage with public sentiment and emerging narratives.\n\n\n### [***3-level strategy:***](docs/data_collection.md)\n\n\u003cbr\u003e\n\n| [Level]() | [Method]() | [Requires]() |\n|---|---|---|\n| [1]() | PRAW (Python Reddit API Wrapper) | `REDDIT_CLIENT_ID` + `REDDIT_CLIENT_SECRET` |\n| [2]() | Public API `/new.json` + `/hot.json` | None |\n| [3]() | Committed frozen Parquet | None |\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n###  [ Official Data Sources — 21 Monitored Sources]()\n\n\u003cbr\u003e\n\n| #  | [Source]()                                    | [Category]()  | [Primary Method]() | [Fallback]() | [Endpoint]()                        |\n| -- | --------------------------------------------- | ------------- | ------------------ | ------------ | ----------------------------------- |\n| 1  | [InfoMoney]()                                 | Editorial     | RSS                | —            | infomoney.com.br/feed/              |\n| 2  | [Empiricus]()                                 | Editorial     | RSS                | Scraping     | empiricus.com.br/feed/              |\n| 3  | [Money Times]()                               | Editorial     | RSS                | —            | moneytimes.com.br/feed/             |\n| 4  | [Seu Dinheiro]()                              | Editorial     | RSS                | —            | seudinheiro.com/feed/               |\n| 5  | [Exame Invest]()                              | Editorial     | RSS                | —            | exame.com/feed/                     |\n| 6  | [CNN Brasil Business ]()                      | Editorial     | RSS                | —            | cnnbrasil.com.br/feed/              |\n| 7  | [Suno Research]()                             | Editorial     | RSS (Secondary)    | —            | sunoresearch.com.br/feed/           |\n| 8  | [E-Investidor]()                              | Editorial     | RSS (Secondary)    | —            | einvestidor.estadao.com.br/feed     |\n| 9  | [NeoFeed]()                                   | Editorial     | RSS (Secondary)    | —            | neofeed.com.br/feed/                |\n| 10 | [Toro Investimentos]()                        | Editorial     | RSS                | Scraping     | blog.toroinvestimentos.com.br/feed/ |\n| 11 | [Funds Explorer]()                            | Portal        | Scraping           | —            | fundsexplorer.com.br                |\n| 12 | [Status Invest]()                             | Portal        | Scraping           | —            | statusinvest.com.br                 |\n| 13 | [Clube FII]()                                 | Portal        | Scraping           | —            | clubefii.com.br                     |\n| 14 | [FIIs.com.br]()                               | Portal        | Scraping           | —            | fiis.com.br                         |\n| 15 | [Portal do FII]()                             | Portal        | Scraping           | RSS          | portaldofii.com.br                  |\n| 16 | [Investidor10]()                              | Portal        | Scraping           | —            | investidor10.com.br                 |\n| 17 | [Eu Quero Investir]()                         | Portal        | Scraping           | —            | euqueroinvestir.com                 |\n| 18 | [Bora Investir (B3)]()                        | Institutional | Scraping           | —            | borainvestir.b3.com.br              |\n| 19 | [XP Conteúdos]()                              | Institutional | Scraping           | —            | conteudos.xpi.com.br                |\n| 20 | [Investing Brasil]()                          | Portal        | Scraping           | —            | br.investing.com                    |\n| 21| [**Reddit / Google News (Fallback)**]() | [**Social / Behavioral**]() | [**PRAW (when available) + Google News RSS (fallback)**]() | `r/investimentos` · `r/farialimabets` · news.google.com |\n\n\u003cbr\u003e\n\n\n\u003e [!TIP]\n\u003e The original behavioral source uses Reddit subreddits (`r/investimentos` and `r/farialimabets`) as a [**social intelligence and market narrative layer**]().  \n\u003e Following changes to Reddit’s public API policy in April 2023 (HTTP 403 restrictions), the pipeline was redesigned to operate across three levels:\n\n\u003cbr\u003e\n\n### [***Source 21 — Reddit / Google News (Fallback)***]()\n\n\u003cbr\u003e\n\n1. [**Level 1 — PRAW**]() (when `REDDIT_API_AVAILABLE = True`) \n \n   Uses the authenticated Reddit API to collect recent posts from the target subreddits.\n\n   \u003cbr\u003e\n\n2. [**Level 2 — Google News RSS PT-BR (fallback)*]()\n   \n   - When Level 1 is unavailable (e.g., missing `REDDIT_CLIENT_ID` in `.env` or public API restrictions), NB01 triggers `collect_google_news_rss()`, which:\n   - queries Google News in Portuguese using FII-specific search terms,\n   - filters content using FII-related keywords (`FII_FILTER_TERMS`),\n   - stores articles with `source='news.google.com'`, `source_type='reddit'`, `tags='google_news_rss'`, and `ingestion_method='feedparser_google_news'`.\n  \n    \u003cbr\u003e\n\n3. [**Level 3 — Frozen Parquet (Resilient Snapshot)** ]()\n\n   For reproducible evaluations and operational resilience, Source 21 data can be frozen in `data/external/` and reused without issuing new requests.\n\nIn the documented reference execution, the [Google News RSS fallback]() generated [**351 FII-related articles**]() for [Source 2]()1, preserving continuity of the behavioral intelligence layer even when direct access to Reddit’s public API was unavailable. [page:46]\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## [🏗️ High-Level Architecture]()\n\n\u003cbr\u003e\n\n```mermaid\n%%{init:{\n'theme':'dark',\n'themeVariables':{\n'background':'#090d13',\n'primaryTextColor':'#F5F7FA',\n'lineColor':'#2dd4bf'\n}}}%%\n\ngraph TD\n\nNB00[\"NB00\u003cbr/\u003e21 DATA SOURCES\u003cbr/\u003eRSS • Scraping • Reddit\"]:::bronze\n\nNB01[\"BRONZE LAYER\u003cbr/\u003eIngestion\u003cbr/\u003efeedparser • BS4 • PRAW\"]:::bronze\n\nNB02[\"SILVER LAYER\u003cbr/\u003eCleaning \u0026 Normalization\u003cbr/\u003eQuality Gates\"]:::silver\n\nNB03[\"SILVER LAYER\u003cbr/\u003eMapReduce Word Count\"]:::silver\n\nNB04[\"GOLD LAYER\u003cbr/\u003eTF-IDF + BM25\u003cbr/\u003eRetrieval Index\"]:::gold\n\nNB05[\"GOLD LAYER\u003cbr/\u003eSentiment Analysis\u003cbr/\u003ePT-BR Lexicon\"]:::gold\n\nNB06[\"GOLD LAYER\u003cbr/\u003eMarketing Intelligence\u003cbr/\u003eSignals • Funnel • Insights\"]:::gold\n\nNB07[\"SERVING LAYER\u003cbr/\u003eDashboard Dataset\"]:::dash\n\nAPI[\"FASTAPI\u003cbr/\u003eServing Layer\u003cbr/\u003eREST API\"]:::dash\n\nST[\"STREAMLIT\u003cbr/\u003eServing Layer\u003cbr/\u003eDashboard\"]:::dash\n\nBOT[\"GROQ CHATBOT\u003cbr/\u003eLLM Layer\u003cbr/\u003eGPT-OSS-20B\"]:::llm\n\nNB00 --\u003e NB01 --\u003e NB02\n\nNB02 --\u003e NB03\nNB02 --\u003e NB04\nNB02 --\u003e NB05\n\nNB03 --\u003e NB06\nNB04 --\u003e NB06\nNB05 --\u003e NB06\n\nNB06 --\u003e NB07\nNB07 --\u003e API\nNB07 --\u003e ST\n\nAPI --\u003e BOT\nST --\u003e BOT\n\nclassDef bronze fill:#2a1512,stroke:#a85a4a,color:#F5F7FA,stroke-width:2.5px;\nclassDef silver fill:#1b2430,stroke:#b0b7c3,color:#F5F7FA,stroke-width:2.5px;\nclassDef gold fill:#2a2208,stroke:#e6c35a,color:#F5F7FA,stroke-width:2.5px;\nclassDef dash fill:#06363d,stroke:#2dd4bf,color:#F5F7FA,stroke-width:2.5px;\nclassDef llm fill:#231433,stroke:#b56cff,color:#F5F7FA,stroke-width:2.5px;\n```\n\n\u003cbr\u003e\n\n\n\u003e [!TIP]\n\u003e \n\u003e Detailed architecture diagram → [docs/architecture.md](https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform/blob/5e7c18a109c56f765ea7cdbf16b8a65ad41a0e2a/docs/architecture.md)\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 4. [Project Structure]()\n\n```text\napp/\n├── main.py\n├── api/\n│   └── routes.py\n├── services/\n│   ├── retrieval.py\n│   ├── embeddings.py\n│   ├── llm.py\n├── models/\n│   └── schemas.py\n├── db/\n│   └── vector_store.py\n├── core/\n│   └── config.py\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 5. [API Layer (FastAPI)]()\n\n\u003cbr\u003e\n\n```python\nfrom fastapi import FastAPI\nfrom app.api.routes import router\n\napp = FastAPI(\n    title=\"Market Intelligence API\",\n    description=\"RAG-powered financial intelligence system\",\n    version=\"1.0.0\"\n)\n\napp.include_router(router)\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 6. [Core Endpoint — Semantic Query]()\n\n\u003cbr\u003e\n\n```python\n@router.post(\"/query\")\nasync def query_system(question: str):\n    \n    context = retrieve_context(question)\n    answer = generate_answer(question, context)\n\n    return {\n        \"question\": question,\n        \"context\": context,\n        \"answer\": answer\n    }\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 7. [Retrieval Layer (RAG)]()\n\n\u003cbr\u003e\n\n```python\ndef retrieve_context(query: str, k: int = 5):\n    query_embedding = embed_query(query)\n    results = search_vectors(query_embedding, k=k)\n    return [r[\"text\"] for r in results]\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 8. [Embeddings Layer]()\n\n\u003cbr\u003e\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\ndef embed_query(text: str):\n    return model.encode(text)\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 9. [Vector Store (FAISS)]()\n\n\u003cbr\u003e\n\n```python\nindex = faiss.IndexFlatL2(384)\n\ndef search_vectors(query_embedding, k=5):\n    D, I = index.search(np.array([query_embedding]), k)\n    return [{\"text\": f\"doc_{i}\"} for i in I[0]]\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 10. [LLM Generation Layer]()\n\n\u003cbr\u003e\n\n```python\ndef generate_answer(question, context):\n    prompt = f\"\"\"\n    Context:\n    {context}\n\n    Question:\n    {question}\n\n    Answer:\n    \"\"\"\n    return call_llm(prompt)\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 11. [End-to-End Flow]()\n\n\u003cbr\u003e\n\n| [Layer]()   | [Function]()                         |\n| ------- | -------------------------------- |\n| 🥉 [Bronze]()  | Raw ingestion and storage        |\n| 🥈 [Silver]()  | Data cleaning and NLP processing |\n| 🥇 [Gold]()    | Signal generation and ranking    |\n| [RAG ]()    | Semantic retrieval               |\n| [FastAPI]() | API interface                    |\n| [LLM]()    | Natural language reasoning       |\n\n\u003cbr\u003e\u003cbr\u003e\n\n## 12. [Example Query]()\n\n\u003cbr\u003e\n\n```json\n{\n  \"question\": \"What is the current investor sentiment on logistics REITs?\"\n}\n```\n\n\u003cbr\u003e\n\n➠ [**Response:**]()\n\n```json\n{\n  \"answer\": \"Recent data indicates a moderately positive sentiment driven by stable dividend yields and occupancy rates.\"\n}\n```\n\n\u003cbr\u003e\n\n## [Final Note]()\n\nThis architecture transforms a traditional data pipeline into a **full-stack AI intelligence system**, enabling:\n\n[*]() semantic search \u003cbr\u003e\n[*]()  investor sentiment  \u003cbr\u003e\n[*]()  real-time insights \u003cbr\u003e\n[*]()  natural language interaction\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n\n\n\n\n\n\n\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\u003cbr\u003e\u003cbr\u003e\n\n\n## [How to run this project locally]()\n\n### [Prerequisites]()\n\n[-]() Python 3.10+ installed\n[-]() Git installed\n[-]() (Optional) Python virtual environment (venv) to isolate dependencies\n\n\u003cbr\u003e\n\n### [Clone the repository]()\n\n```bash\ngit clone https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform.git\ncd 5-cybersecurity-social-engineering-fii-marketing-intelligence-platform\n```\n\n\u003cbr\u003e\n\n### [Create and activate the virtual environment]]()\n\n```bash\n# macOS / Linux\npython3 -m venv .venv\nsource .venv/bin/activate\n\n# Windows (PowerShell)\npython -m venv .venv\n.\\.venv\\Scripts\\Activate.ps1\n```\n\n\u003e Note: the `.venv/` folder is already ignored in `.gitignore`, so the virtual environment will not be versioned. \n\n\u003cbr\u003e\n\n### [Install dependencies]()\n\n```bash\npip install --upgrade pip\npip install -r requirements.txt\n```\n\n\u003cbr\u003e\n\n### [Run notebooks / scripts]()\n\n- Open the notebooks in the `2-FIIs_Final` folder in Jupyter Notebook, JupyterLab, or VS Code.\n- Make sure the selected kernel is the `.venv` virtual environment.\n- Adjust data paths if needed (under the `data/` directory). Local data layers such as `data/bronze`, `data/silver`, and `data/gold` are git-ignored by default.\n\n\u003cbr\u003e\n\n### [Whenever you add or remove dependencies:]()\n\n```bash\npip freeze \u003e requirements.txt\ngit add requirements.txt\ngit commit -m \"Update project dependencies\"\n```\n\n\u003cbr\u003e\u003cbr\u003e\n\n## [References]()\n\n- Barocas, S., \u0026 Selbst, A. D. (2016). Big Data’s Disparate Impact. *California Law Review*, 104(3), 671–732.\n- Blei, D. M., Ng, A. Y., \u0026 Jordan, M. I. (2003). Latent Dirichlet Allocation. *Journal of Machine Learning Research (JMLR)*, 3, 993–1022.\n- Brasil. (2018). *Lei nº 13.709, de 14 de agosto de 2018: Lei Geral de Proteção de Dados Pessoais (LGPD)*.\n- Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., \u0026 Wirth, R. (2000). *CRISP-DM 1.0: Step-by-step data mining guide*. SPSS.\n- European Commission. (2019). *Ethics Guidelines for Trustworthy AI*. Brussels: High-Level Expert Group on Artificial Intelligence.\n- Goodfellow, I., Bengio, Y., \u0026 Courville, A. (2016). *Deep Learning*. MIT Press.\n- Jurafsky, D., \u0026 Martin, J. H. (2025). *Speech and Language Processing* (3rd ed.). Stanford University.\n- Manning, C. D., Raghavan, P., \u0026 Schütze, H. (2008). *Introduction to Information Retrieval*. Cambridge University Press.\n- Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., \u0026 Gebru, T. (2019). Model Cards for Model Reporting. In *Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT)* (pp. 220–229).\n- Molnar, C. (2022). *Interpretable Machine Learning* (2nd ed.). Lulu.com.\n- Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M., \u0026 Gatford, M. (1995). Okapi at TREC-3. In *Text REtrieval Conference (TREC-3)*. NIST.\n- Robertson, S. E., \u0026 Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. *Foundations and Trends in Information Retrieval*, 3(4), 333–389.\n- Russell, S., \u0026 Norvig, P. (2021). *Artificial Intelligence: A Modern Approach* (4th ed.). Pearson.\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n\n\u003c!-- ======================================= Start DEFAULT Footer ===========================================  --\u003e\n\n\u003cbr\u003e\u003cbr\u003e\n\n\n## 💌 [Let the data flow... Ping Me !](mailto:fabicampanari@proton.me)\n\n\u003cbr\u003e\n\n\n#### \u003cp align=\"center\"\u003e  🛸๋ My Contacts [Hub](https://linktr.ee/fabianacampanari)\n\n\n\u003cbr\u003e\n\n### \u003cp align=\"center\"\u003e \u003cimg src=\"https://github.com/user-attachments/assets/517fc573-7607-4c5d-82a7-38383cc0537d\" /\u003e\n\n\n\u003cbr\u003e\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e  ────────────── ⊹🔭๋ ──────────────\n\n\u003c!--\n\u003cp align=\"center\"\u003e  ────────────── 🛸๋*ੈ✩* 🔭*ੈ₊ ──────────────\n--\u003e\n\n\u003cbr\u003e\n\n\u003cp align=\"center\"\u003e ➣➢➤ \u003ca href=\"#top\"\u003eBack to Top \u003c/a\u003e\n  \n\n  \n#\n \n##### \u003cp align=\"center\"\u003eCopyright 2026 Quantum Software Development. Code released under the  [MIT license.](https://github.com/Quantum-Software-Development/5-cybersecurity-social-engineering-fii-marketing-intelligence-platform/blob/64d9e815b5abcee1658cf8aaa9f44af11c60b6c6/LICENSE)\n\u003c!-- ======================================= End  DEFAULT Footer ===========================================  --\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquantum-software-development%2Ffii-investor-intelligence-platform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquantum-software-development%2Ffii-investor-intelligence-platform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquantum-software-development%2Ffii-investor-intelligence-platform/lists"}