{"id":29811665,"url":"https://github.com/aitorres/csca-5028-final-project","last_synced_at":"2025-07-28T17:09:27.963Z","repository":{"id":305641885,"uuid":"1015065746","full_name":"aitorres/csca-5028-final-project","owner":"aitorres","description":"Repository for the final project of CSCA-5028: Applications of Software Architecture Patterns for Big Data - CU Boulder's MSCS","archived":false,"fork":false,"pushed_at":"2025-07-21T07:49:45.000Z","size":626,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-21T09:29:14.264Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aitorres.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-07T00:24:15.000Z","updated_at":"2025-07-21T07:49:48.000Z","dependencies_parsed_at":"2025-07-21T09:43:23.003Z","dependency_job_id":null,"html_url":"https://github.com/aitorres/csca-5028-final-project","commit_stats":null,"previous_names":["aitorres/csca-5028-final-project"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/aitorres/csca-5028-final-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aitorres%2Fcsca-5028-final-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aitorres%2Fcsca-5028-final-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aitorres%2Fcsca-5028-final-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aitorres%2Fcsca-5028-final-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aitorres","download_url":"https://codeload.github.com/aitorres/csca-5028-final-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aitorres%2Fcsca-5028-final-project/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267552097,"owners_count":24106000,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-28T17:09:25.170Z","updated_at":"2025-07-28T17:09:27.943Z","avatar_url":"https://github.com/aitorres.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSCA-5028 - Applications of Software Architecture for Big Data - Final Project\n\n![Screenshot of the application's web interface](./images/homepage.png)\n\n\u003e NOTE: this GitHub repository has been set to public for the duration of the course as the project report links\n\u003e to the files hosted in the repository. The repository will be set to private after the course ends.\n\nThis is the final project for the CSCA-5028 course (\"Applications of Software Architecture for Big Data\"),\ntaken as part of University of Colorado-Boulder's Master of Science in Computer Science online program,\nby Andres Torres.\n\nThis report includes a high-level description of the project, a whiteboard architecture diagram with its description,\nand recount of the design decisions made during the project. It will also include a list of implemented features, with\npointers to the relevant code sections.\n\nThe up-to-date code (as well as this report) is available at the [GitHub repository](https://github.com/aitorres/csca-5028-final-project), and\nthe deployed application is available as an [Azure Container App](https://web.mangoisland-c02529cd.westus3.azurecontainerapps.io/) (available during the semester).\n\n\u003e Note: the project implements logic to filter out inappropriate content from messages collected from Bluesky,\n\u003e but it might not be 100% effective. The application is intended for educational purposes only.\n\n## How to run\n\nTo run the application locally, you will require [Docker Compose installed on your machine](https://docs.docker.com/compose/install/).\n\nThe [`docker-compose.yaml`](https://github.com/aitorres/csca-5028-final-project/blob/main/docker-compose.yaml) file\ncontains the required services to run the application (including all 3 main components as described below, as well as the database and message queue).\n\nTo run everything:\n\n```bash\ndocker compose up --build\n```\n\nOnce all services are up and running, you can access the web application at `http://localhost:8080`.\n\nData will be persisted to the `postgres-data` directory between sessions.\n\n## Project description\n\n**What's up in Vancouver?** is an application that tracks and displays information about social media\n(and user-submitted) trends and opinions related to the city of Vancouver, Canada.\n\nThis application is targeted towards residents (and people interested in) the city of Vancouver who\nwant to stay informed about the public opinion and trends in the city, accessible in a single place\nthrough a web application interface, and with sentiment analysis details.\n\nThe product's unique value is that data is sourced in an automated way by (anonymously) fetching posts from Bluesky, a\ndistributed social media platform, in real time. End-users can also submit their own posts, content or opinions about\nthe city through a form on the application's web interface. All posts are then analyzed for sentiment in real time and\ndisplayed in a table.\n\nCollected data is filtered to remove unwanted content, and a sentiment analysis library is used to enrich\nthe posts' data with a category (positive, negative, neutral).\n\n## Architecture\n\nThe following whiteboard architecture diagram illustrates the high-level architecture of the project (including data stores, communication and external services):\n\n![Architecture diagram](./images/architecture.png)\n\nThe three main components of the architecture are:\n\n- The **social media collector** application (code available here: [collector](https://github.com/aitorres/csca-5028-final-project/tree/main/collector)), a Python application that connects to the Bluesky social media platform through their posts stream (Jetstream) using web sockets, and collects posts that mention Vancouver in real time.\n- The **post analyzer** application (code available here: [analyzer](https://github.com/aitorres/csca-5028-final-project/tree/main/analyzer)), a Python application that processes received posts to filter out unwanted content, enrich them with sentiment analysis, and stores them in a database.\n- The **web application** (code available here: [web](https://github.com/aitorres/csca-5028-final-project/tree/main/web)), a web application built in Flask (back-end) and HTML / CSS / JavaScript (front-end) that displays the posts in a table as well as a few visualizations, and allows users to submit their own posts that are then sent to the post analyzer for processing.\n\nThe architecture is designed to be cloud-native, modular and scalable, with each component being independently deployable and scalable based on demand.\n\nCommunication between components is done through a RabbitMQ message queue system, with the **web** and **collector** sending messages to the **analyzer** for processing.\n\nData is persisted in a PostgreSQL database, which is used by the **analyzer** to store processed posts and by the **web** application to read them.\n\nAs described above, data is collected from an external source (Bluesky) as well as user-submitted posts, and processed in real time.\n\n## Design decisions\n\nThe following is a recount of the most significant design decisions made during the project:\n\n### Usage of a database for data persistence\n\nTo persist collected data, the project uses a PostgreSQL database. The current schema is very simple, with a single\ntable to store posts (with content, creation and insertion date, source and sentiment analysis category).\n\nGiven the simplicity of the data model, any robust and persistent data store would have sufficed. However, PostgreSQL was chosen for its robustness and scalability, in anticipation of new features that could require relationships between data entities in the future (e.g. storing information about users and tying posts to each author).\n\nAnother reason for choosing PostgreSQL is that it is widely used and supported in the industry, with most cloud providers offering managed PostgreSQL services, and most programming languages having libraries to interact with it.\n\nThe database schema can be found in the [migrations](https://github.com/aitorres/csca-5028-final-project/tree/main/migrations) directory.\n\n### Usage of a message queue for communication between components\n\nGiven the modular nature of the architecture and the fact that each component has a very specific\nresponsibility, RabbitMQ was chosen as the message queue system to facilitate communication between components.\n\nRabbitMQ allows for asynchronous communication between components, ideal for the architecture's design. Other features\nof RabbitMQ, such as message persistence and delivery guarantees, are also beneficial for the architecture\nto ensure that messages are not lost in case of failures. This is an advantage over using e.g. a REST API for communication, which would require more complex error handling and retries.\n\nUsing RabbitMQ also allows for easy scaling of components, as new instances of each component can be created\nand destroyed independently without requiring to notify other components, as long as they are connected to the same message queue.\n\n### Usage of Python as the main programming language\n\nAll main components of the application are written in Python. This language was chosen for its simplicity\nand quick development cycle, as well as its rich ecosystem of libraries and frameworks that facilitate\nboth web applications and data analysis.\n\nFor the **web** component, Flask was chosen as a lightweight web framework, as most of the application logic\nis implemented through REST APIs and rendering a template. More complex web frameworks such as Django would\nbe useful for larger, more complex applications.\n\nFor the data processing component, **nltk** was chosen as the natural language processing library that's been\nbattle-tested and includes a native sentiment analysis model. For more complex tasks, Python\noffers a wide variety of libraries such as scikit-learn and spaCy.\n\nOther relevant libraries used in the project include **pika** for RabbitMQ communication, **psycopg2** for PostgreSQL database interaction, and **websockets** for connecting to the Bluesky posts stream.\n\nThe project uses [Poetry](https://python-poetry.org/) for dependency management and packaging, which simplifies the process of managing dependencies and creating virtual environments.\n\n### Containerization for portability and scalability\n\nAll components of the application are containerized using Docker, allowing for easy deployment in both\nlocal environments (through Docker Compose) and cloud environments (e.g. through Azure Container Apps, or\nan orchestration system like Kubernetes).\n\nAll application components are stateless so they can be scaled horizontally by adding more instances of each component as needed, and deleted without losing any data.\n\n### Infrastructure as code\n\nThe project uses [Terraform](https://developer.hashicorp.com/terraform) to define the infrastructure required to run the application in the cloud. This includes deployment of all three main components (collector, analyzer, web), one virtual machine to run the PostgreSQL database and RabbitMQ instances, and the necessary networking and security configurations.\n\nDefining the infrastructure as code allows for easy deployment and management of the application in the cloud,\nas well as versioning and tracking of changes to the infrastructure. To set up the infrastructure from scratch,\none would only need to set up the Terraform variables (as shown [here](https://github.com/aitorres/csca-5028-final-project/blob/main/infrastructure/variables.tf)), then apply the Terraform configuration.\n\nThe only manual step required after the first deployment is to run the PostgreSQL migrations and apply the RabbitMQ definitions. Once that's done, all subsequent deployments can be done by running `terraform apply` again, which will only apply the changes made to the infrastructure.\n\nThe Terraform configuration files can be found in the [infrastructure](https://github.com/aitorres/csca-5028-final-project/tree/main/infrastructure) directory.\n\n#### Cloud provider\n\nAll Terraform resources are defined to be deployed on Microsoft Azure, using Azure Container Apps for the three main components of the application, and a virtual machine for the PostgreSQL database and RabbitMQ instances due to the simplicity of the model.\n\nThe files are easily modifiable to deploy the PostgreSQL and RabbitMQ instances in a managed service, such as Azure Database for PostgreSQL and Azure Service Bus, requiring only to change environment variables\nin the Azure Container Apps definitions for this to work.\n\n## Features\n\nThe following is a brief list of the features implemented in the project, with pointers to the relevant code sections (note that some of these were described in the architecture section above):\n\n### Web application with a basic form and reporting\n\nThe [web application](https://github.com/aitorres/csca-5028-final-project/tree/main/web) is built using Flask (back-end) and HTML/CSS/JavaScript (front-end). It includes data reporting through a data table and pie charts, and a form to submit user posts for processing.\n\n![Screenshot of the web form](./images/webform.png)\n\n### Data collection\n\n[Automated data collection](https://github.com/aitorres/csca-5028-final-project/tree/main/collector) from the Bluesky social media platform is implemented using web sockets to listen to Bluesky's posts stream (Jetstream) in real-time.\n\n### Data analyzer\n\n[Data analysis](https://github.com/aitorres/csca-5028-final-project/tree/main/analyzer) is implemented using a Python application that processes received posts (both from Bluesky and user-submitted) to perform sentiment analysis (through **nltk**) and store content in a PostgreSQL database.\n\n### Unit tests\n\nAll components of the application include unit tests to ensure correctness of the code. Tests are implemented using `pytest`, and can be run using the following command in each directory:\n\n```bash\npoetry run pytest\n```\n\nTest coverage includes most of the main application logic, testing for both valid and invalid inputs, and edge cases.\n\nTests can be found here: [web tests](https://github.com/aitorres/csca-5028-final-project/tree/main/web/tests), [collector tests](https://github.com/aitorres/csca-5028-final-project/tree/main/collector/tests) and [analyzer tests](https://github.com/aitorres/csca-5028-final-project/tree/main/analyzer/tests).\n\n### Data persistence\n\nProcessed and analyzed posts, with metadata and sentiment category, are persisted in a PostgreSQL database. Only the **analyzer** application writes to the database, and only the **web** application reads from it.\n\nData schema consists of only one table for **posts** and the schema can be found in the [migrations](https://github.com/aitorres/csca-5028-final-project/tree/main/migrations) directory. It includes enumerated types for known values, and indexes for efficient querying.\n\n### Internal REST APIs\n\nThe **web** application front-end interacts with the back-end through a REST API, implemented in Flask. The API contains the following endpoints:\n\n- `GET /api/health`: a simple health check endpoint that returns a 200 OK response if the application is running, used for Azure Container Apps health checks.\n- `GET /api/posts`: retrieves all posts from the database.\n- `POST /api/posts`: submits a new post to the database, which is then processed by the **analyzer** application.\n- `GET /api/posts/count`: retrieves the total number of posts in the database, used by the front-end to let a current user know there are new posts available.\n- `GET /api/posts/statistics/sentiment`: retrieves the sentiment statistics for the posts in the database, used by the front-end to display pie charts.\n- `GET /api/posts/statistics/sources`: retrieves the source statistics for the posts in the database, used by the front-end to display pie charts.\n\n### Product environment\n\nThe application is implemented with a production-first approach, with a focus on security, performance and scalability. No sensitive data is hardcoded in the codebase, and all environment configuration values are read from environment variables.\n\n### Integration tests\n\nAn integration test script is implemented in the [integration](https://github.com/aitorres/csca-5028-final-project/tree/main/integration) directory, to serve as a smoke test for the entire application. It pulls up the whole stack via Docker Compose, ensures the web health-check endpoint is reachable, and then submits a post through the web form, checking that it appears in the API response after a few seconds.\n\nThe integration test script can be run using the following command:\n\n```bash\npython integration/integration_tests.py\n```\n\n### Using mocks and spy objects\n\nThe project's unit tests use the `pytest-mock` library to create mock objects and spy on function calls,\nto quickly test the application's logic without relying on external services or databases.\n\nMocks and spy objects are used particularly to simulate the behavior of the RabbitMQ message queue and the PostgreSQL database. An example of this can be found in the `test_get_post_source_statistics` code [here](https://github.com/aitorres/csca-5028-final-project/blob/6b908c8283a2d09d13227f68d006a894ac389081/web/tests/test_main.py#L171C31-L172C1).\n\n### Continuous integration and continuous delivery (CI/CD)\n\n![Continous Integration and Delivery using Github Actions](./images/cicd.png)\n\nThe project uses a GitHub Actions workflow to implement continuous integration and continuous delivery (CI/CD). The workflow is triggered on every push to the `main` branch and is described in the [`continuous-integration.yaml`](https://github.com/aitorres/csca-5028-final-project/blob/main/.github/workflows/continuous-integration.yaml) file. A sample run can be found [here](https://github.com/aitorres/csca-5028-final-project/actions/runs/16409467599).\n\nThe workflow includes the following jobs that run in sequence:\n\n- Three jobs that each build the Docker image and run tests for the **web**, **collector** and **analyzer** components. If any of the tests fail, the workflow will stop and report the failure. If each job succeeds, each Docker image is pushed to a private Azure Container Registry (ACR) repository.\n- A job that runs the integration tests using Docker Compose. If the integrations tests fail, the workflow will stop and report the failure.\n- A job that deploys the infrastructure to Azure via Terraform, updating only the resources that have changed since the last deployment. This also takes care of updating the Docker images used by the Azure Container Apps to the latest versions pushed to the ACR repository.\n- A job that creates a GitHub deployment entry, for quick access to the deployed application URL.\n\nThis workflow ensures that tests are always run before deploying any changes to the application, and that the application is always deployed with the latest code changes, for rapid iteration and development.\n\n### Production monitoring\n\nAll Python components are integrated with [Sentry](https://sentry.io/) for error tracking and monitoring. Sentry is configured to capture unhandled exceptions and performance issues, and to alert the development team (me!) via email when errors occur in production.\n\n### Event messaging and collaboration\n\nThe project uses RabbitMQ as the message queue system to facilitate communication between components. The **web** and **collector** applications send messages to the **analyzer** application for processing, allowing for asynchronous communication and decoupling of components.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faitorres%2Fcsca-5028-final-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faitorres%2Fcsca-5028-final-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faitorres%2Fcsca-5028-final-project/lists"}