{"id":18376662,"url":"https://github.com/manojpatra1991/log-analysis","last_synced_at":"2026-05-08T13:18:34.166Z","repository":{"id":104745840,"uuid":"96661174","full_name":"MANOJPATRA1991/LOG-ANALYSIS","owner":"MANOJPATRA1991","description":"A reporting tool for a postgresql database","archived":false,"fork":false,"pushed_at":"2017-07-11T02:54:08.000Z","size":11768,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-15T22:29:05.576Z","etag":null,"topics":["pep8","postgresql","pylint","python3","sql","udacity-fullstack-nanodegree","vagrant","virtualbox-vm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MANOJPATRA1991.png","metadata":{"files":{"readme":"README.md","changelog":"newsdata.py","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-09T04:33:53.000Z","updated_at":"2020-05-14T06:41:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"3868531f-59f6-413f-8311-f632cff02d28","html_url":"https://github.com/MANOJPATRA1991/LOG-ANALYSIS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MANOJPATRA1991%2FLOG-ANALYSIS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MANOJPATRA1991%2FLOG-ANALYSIS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MANOJPATRA1991%2FLOG-ANALYSIS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MANOJPATRA1991%2FLOG-ANALYSIS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MANOJPATRA1991","download_url":"https://codeload.github.com/MANOJPATRA1991/LOG-ANALYSIS/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345605,"owners_count":21088311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pep8","postgresql","pylint","python3","sql","udacity-fullstack-nanodegree","vagrant","virtualbox-vm"],"created_at":"2024-11-06T00:24:18.472Z","updated_at":"2026-05-08T13:18:29.142Z","avatar_url":"https://github.com/MANOJPATRA1991.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# LOGS ANALYSIS \n\nThis project is a part of Part 3 *The Backend: Databases \u0026 Applications* of [Udacity Full Stack Nanodegree course](https://in.udacity.com/course/full-stack-web-developer-nanodegree--nd004/).\n\n## Table of Contents\n1. [Installation](#installation)\n2. [Description](#description)\n3. [Reference](#reference)\n4. [License](#license)\n\n### Installation\n\n**NOTE**: \n\nThis project was built on **Windows 10** OS. All the interaction with the Virtual Machine was done through **Command Prompt on Windows**.\n\n(**Do not use Git Bash for this project. It simply won't work**.)\n\n1. Python\n\nThe source code for this project is written in [Python v3.6.1](https://www.python.org/downloads/) programming language.\nFor direct download of version 3.6.1 [click here](https://www.python.org/ftp/python/3.6.1/python-3.6.1.exe).\n\n2. Code Validators\n\nThe source code was checked against bugs and quality using **Pylint** tool, **PEP8** tool and [PEP8 online check](http://pep8online.com).\n\nTo install Pylint:\n\n```\npip3 install pylint\n```\n\nTo check Python file using Pylint:\n\n```\npylint fileName.py\n```\n\nTo install pep8:\n\n```\npip3 install pep8\n```\n\n\nTo check Python file using pep8:\n\n```\npep8 fileName.py\n```\n\n3. Virtual Box\n\nTo run the Virtual Machine, first, we need to download it and then install it.\nVirtual Box can be downloaded from [here](https://www.virtualbox.org/wiki/Downloads).\n\n4. Vagrant\n\nVagrant is the software that configures the VM and lets you share files between your host computer and the VM's filesystem. \nVagrant can be downloaded from [here](https://www.vagrantup.com/downloads.html).\n\n5. `psycopg2` python module is required. To install:\n\n```\npip3 install psycopg2\n```\n\n### Description\n\nThis project is an information reporting tool which provides information regarding the most popular articles, the most popular authors and the most logged errors in a day from a news database.\n\nFollowing are the views that were created as part of the **news** database:\n\n1. MostViewedPaths\n\n```\ncreate view MostViewedPaths as (\nselect path, count(*) as views\nfrom log\nwhere path like '/_%'\ngroup by path\norder by views desc);\n```\n\n2. articleShortInfo\n\n```\ncreate view articleShortInfo as (\nselect a.title, c.name, b.views\nfrom articles as a join MostViewedPaths as b\non concat('/article/', a.slug) = b.path\njoin authors as c\non a.author = c.id\norder by views desc);\n```\n\n3. LogRequests\n\n```\ncreate view LogRequests as (\nselect time::timestamp::date, count(*) as total\nfrom log\ngroup by time::timestamp::date\norder by total desc);\n```\n\n4. ErrorRequests\n\n```\ncreate view ErrorRequests as (\nselect time::timestamp::date, count(*) as errors\nfrom log\nwhere status like '4%' or status like '5%'\ngroup by time::timestamp::date\norder by errors desc);\n```\n\nThe **newsdb.py** file contains the implementations for the three functions:\n\n1. `get_most_popular_articles()`\n2. `get_most_popular_authors()`\n3. `get_most_logged_errors()`\n\nIn each of the these functions, a new Connection object and Cursor object are created as:\n\n```\nDBNAME = \"news\"\nconn = psycopg2.connect(database=DBNAME)\ncursor = conn.cursor()\n```\n\nThen, the query is run using the `cursor.execute()` function.\nThe data is fetched using `cursor.fetchall()` and stored in a local variable which is returned by the respective functions.\n\nFinally, the connection is closed using `conn.close()` within each function.\n\n##### Executing the program\n\n1. Copy the python files to the folder which contains the newsdata.sql file.\n2. If the folder doesn't already contain a Vagrantfile. Run the following command to create one.\n        \n```\nvagrant init\n```\n    \n3. To start the virtual machine, from your local directory, run the following command:\n        \n```\nvagrant up\n```\n\n4. Then to drop a full-fledged SSH session, run the following command:\n        \n```\nvagrant ssh\n```\n        \n5. Type `psql` to switch to the interactive terminal for working with PostgreSQL.\n6. Now create a new database(if it doesn't exist already):\n        \n```\ncreate database news;\n```\n        \n7. Then exit with **Ctrl + D**.\n\n8. Run `psql -d news -f newsdata.sql` to create the tables **authors**, **articles** and **log**. This will exit the psql terminal.\n \n9. Start the psql terminal with `psql` and move into the **news** database with `\\c news`.\n\n10. Now open up another Command Prompt. Move to the project directory. Run `vagrant ssh` to move into the VM.\n\n11. Run the **newsdata.py** file with the following command after moving into the file's location to get the output:\n\n```\npython newsdata.py\n```\n\nTo know how the output looks [check here](log_analysis_output.txt).\n\n### Reference\n1. [Python Documentation](https://docs.python.org/3/)\n2. [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n3. [PEP8](https://www.python.org/dev/peps/pep-0008/)\n4. [Vagrant](https://www.vagrantup.com/)\n5. [Oracle VM Virtual Box](https://www.virtualbox.org/)\n6. [PostgreSQL](https://www.postgresql.org/)\n\n### License\nThe content of this repository is licensed under [MIT](https://choosealicense.com/licenses/mit/).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojpatra1991%2Flog-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanojpatra1991%2Flog-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanojpatra1991%2Flog-analysis/lists"}