Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/tafara-n/alx-backend-storage

Introduction to Backend Storage.
https://github.com/tafara-n/alx-backend-storage

mysql-advanced no-sql redis-basic

Last synced: 5 months ago
JSON representation

Introduction to Backend Storage.

Awesome Lists containing this project

README 

          
# Introduction to Backend Storage: MySQL, MongoDB and Redis.



## Table of Content

- [Author](#author)

- [Description](#description)

___



- [MySQL Advanced](0x00-MySQL_Advanced/README.md)

- [NoSQL: MongoDB](0x01-NoSQL/README.md)

- [NoSQL: Redis](0x02-redis_basic/README.md)

___



# Description



1. # MySQL Advanced



## Concepts



*For this project, we expect you to look at this concept:*



# MySQL



- [Advanced SQL](https://intranet.alxswe.com/concepts/555)



## Resources



**Read or watch:**

- [MySQL cheatsheet](https://devhints.io/mysql)

- [MySQL Performance: How To Leverage MySQL Database Indexing](https://www.liquidweb.com/blog/mysql-optimization-how-to-leverage-mysql-database-indexing/)

- [Stored Procedure](https://www.w3resource.com/mysql/mysql-procedure.php)

- [Triggers](https://www.w3resource.com/mysql/mysql-triggers.php)

- [Views](https://www.w3resource.com/mysql/mysql-views.php)

- [Functions and Operators](https://dev.mysql.com/doc/refman/5.7/en/functions.html)

- [Trigger Syntax and Examples](https://dev.mysql.com/doc/refman/5.7/en/trigger-syntax.html)

- [CREATE TABLE Statement](https://dev.mysql.com/doc/refman/5.7/en/create-table.html)

- [CREATE PROCEDURE and CREATE FUNCTION Statements](https://dev.mysql.com/doc/refman/5.7/en/create-procedure.html)

- [CREATE INDEX Statement](https://dev.mysql.com/doc/refman/5.7/en/create-index.html)

- [CREATE VIEW Statement](https://dev.mysql.com/doc/refman/5.7/en/create-view.html)



# Learning Objectives

At the end of this project, you are expected to be able to [explain to anyone](https://fs.blog/feynman-learning-technique/), **without the help of Google:**



## General

- How to create tables with constraints

- How to optimize queries by adding indexes

- What is and how to implement stored procedures and functions in MySQL

- What is and how to implement views in MySQL

- What is and how to implement triggers in MySQL



# Requirements



##General

- All your files will be executed on Ubuntu 18.04 LTS using `MySQL 5.7` (version `5.7.30`)

- All your files should end with a new line

- All your SQL queries should have a comment just before (i.e. syntax above)

- All your files should start by a comment describing the task

- All SQL keywords should be in uppercase (`SELECT`, `WHERE`…)

- A `README.md` file, at the root of the folder of the project, is mandatory

- The length of your files will be tested using `wc`



# More Info



**Comments for your SQL file:**



```bash

$ cat my_script.sql

-- 3 first students in the Batch ID=3

-- because Batch 3 is the best!

SELECT id, name FROM students WHERE batch_id = 3 ORDER BY created_at DESC LIMIT 3;

$

```



**Use “container-on-demand” to run MySQL**



- Ask for container `Ubuntu 18.04 - Python 3.7`

- Connect via SSH

- Or via the WebTerminal

- In the container, you should start MySQL before playing with it:



```bash

$ service mysql start

 * MySQL Community Server 5.7.30 is started

$

$ cat 0-list_databases.sql | mysql -uroot -p my_database

Enter password:

Database

information_schema

mysql

performance_schema

sys

$

```



**In the container, credentials are `root/root`**



## How to import a SQL dump



```bash

$ echo "CREATE DATABASE hbtn_0d_tvshows;" | mysql -uroot -p

Enter password:

$ curl "https://s3.amazonaws.com/intranet-projects-files/holbertonschool-higher-level_programming+/274/hbtn_0d_tvshows.sql" -s | mysql -uroot -p hbtn_0d_tvshows

Enter password:

$ echo "SELECT * FROM tv_genres" | mysql -uroot -p hbtn_0d_tvshows

Enter password:

id  name

1   Drama

2   Mystery

3   Adventure

4   Fantasy

5   Comedy

6   Crime

7   Suspense

8   Thriller

$

```



2. # NoSQL



# Resources



**Read or watch:**

- [NoSQL Databases Explained](https://riak.com/resources/nosql-databases/)

- [What is NoSQL ?](https://www.youtube.com/watch?v=qUV2j3XBRHc)

- [MongoDB with Python Crash Course - Tutorial for Beginners](https://www.youtube.com/watch?v=E-1xI85Zog8)

- [MongoDB Tutorial 2 : Insert, Update, Remove, Query](https://www.youtube.com/watch?v=CB9G5Dvv-EE)

- [Aggregation](https://www.mongodb.com/docs/manual/aggregation/)

- [Introduction to MongoDB and Python](https://realpython.com/introduction-to-mongodb-and-python/)

- [mongo Shell Methods](https://www.mongodb.com/docs/manual/reference/method/)

- [Mongosh](https://www.mongodb.com/docs/mongodb-shell/#mongodb-binary-bin.mongosh)



# Learning Objectives

At the end of this project, you are expected to be able to [explain to anyone](https://fs.blog/feynman-learning-technique/), **without the help of Google:**



## General

- What NoSQL means

- What is difference between SQL and NoSQL

- What is ACID

- What is a document storage

- What are NoSQL types

- What are benefits of a NoSQL database

- How to query information from a NoSQL database

- How to insert/update/delete information from a NoSQL database

- How to use MongoDB



# Requirements



## MongoDB Command File

- All your files will be interpreted/compiled on Ubuntu 18.04 LTS using `MongoDB` (version `4.2`)

- All your files should end with a new line

- The first line of all your files should be a comment: `// my comment`

- A `README.md` file, at the root of the folder of the project, is mandatory

- The length of your files will be tested using `wc`



## Python Scripts

- All your files will be interpreted/compiled on Ubuntu 18.04 LTS using `python3` (version `3.7`) and `PyMongo` (version `3.10`)

- All your files should end with a new line

- The first line of all your files should be exactly `#!/usr/bin/env python3`

- A `README.md` file, at the root of the folder of the project, is mandatory

- Your code should use the `pycodestyle` style (version `2.5.*`)

- The length of your files will be tested using `wc`

- All your modules should have a documentation ( `python3 -c 'print(__import__("my_module").__doc__)'` )

- All your functions should have a documentation ( `python3 -c 'print(__import__("my_module").my_function.__doc__)'`)

- Your code should not be executed when imported (by using `if __name__ == "__main__" :`)



# More Info



## Install MongoDB 4.2 in Ubuntu 18.04



- [Official installation guide](https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/)



```bash

$ wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | apt-key add -

$ echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2

multiverse" > /etc/apt/sources.list.d/mongodb-org-4.2.list

$ sudo apt-get update

$ sudo apt-get install -y mongodb-org

...

$ sudo service mongod status

mongod start/running, process 3627

$ mongo --version

MongoDB shell version v4.2.8

git version: 43d25964249164d76d5e04dd6cf38f6111e21f5f

OpenSSL version: OpenSSL 1.1.1 11 Sep 2018

allocator: tcmalloc

modules: none

build environment:

distmod: ubuntu1804

distarch: x86_64

target_arch: x86_64

$

$ pip3 install pymongo

$ python3

>>> import pymongo

>>> pymongo.__version__

'3.10.1'

```



Potential issue if documents creation doesn’t work or this error: `Data directory /data/db not found., terminating` ([source](https://bryantson.medium.com/fixing-data-db-not-found-error-in-macos-x-when-starting-mongodb-d7b82abb2479) and [source](https://stackoverflow.com/questions/37702957/mongodb-data-db-not-found))



```bash

$ sudo mkdir -p /data/db

```



Or if `/etc/init.d/mongod` is missing, please find here an example of the file:



```bash

#!/bin/sh

### BEGIN INIT INFO

# Provides:          mongod

# Required-Start:    $network $local_fs $remote_fs

# Required-Stop:     $network $local_fs $remote_fs

# Should-Start:      $named

# Should-Stop:

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# Short-Description: An object/document-oriented database

# Description:       MongoDB is a high-performance, open source, schema-free

#                    document-oriented data store that's easy to deploy, manage

#                    and use. It's network accessible, written in C++ and offers

#                    the following features:

#

#                       * Collection oriented storage - easy storage of object-

#                         style data

#                       * Full index support, including on inner objects

#                       * Query profiling

#                       * Replication and fail-over support

#                       * Efficient storage of binary data including large

#                         objects (e.g. videos)

#                       * Automatic partitioning for cloud-level scalability

#

#                    High performance, scalability, and reasonable depth of

#                    functionality are the goals for the project.

### END INIT INFO



PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

DAEMON=/usr/bin/mongod

DESC=database



NAME=mongod

# Defaults.  Can be overridden by the /etc/default/$NAME

# Other configuration options are located in $CONF file. See here for more:

# http://dochub.mongodb.org/core/configurationoptions

CONF=/etc/mongod.conf

PIDFILE=/var/run/$NAME.pid

ENABLE_MONGOD=yes



# Include mongodb defaults if available.

# All variables set before this point can be overridden by users, by

# setting them directly in the defaults file. Use this to explicitly

# override these values, at your own risk.

if [ -f /etc/default/$NAME ] ; then

        . /etc/default/$NAME

fi



# Handle NUMA access to CPUs (SERVER-3574)

# This verifies the existence of numactl as well as testing that the command works

NUMACTL_ARGS="--interleave=all"

if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null

then

    NUMACTL="`which numactl` -- $NUMACTL_ARGS"

    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}

else

    NUMACTL=""

    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}

fi



if test ! -x $DAEMON; then

    echo "Could not find $DAEMON"

    exit 0

fi



if test "x$ENABLE_MONGOD" != "xyes"; then

    exit 0

fi



. /lib/lsb/init-functions



STARTTIME=1

DIETIME=10                  # Time to wait for the server to die, in seconds

                            # If this value is set too low you might not

                            # let some servers to die gracefully and

                            # 'restart' will not work



DAEMONUSER=${DAEMONUSER:-mongodb}

DAEMONGROUP=${DAEMONGROUP:-mongodb}



set -e



running_pid() {

# Check if a given process pid's cmdline matches a given name

    pid=$1

    name=$2

    [ -z "$pid" ] && return 1

    [ ! -d /proc/$pid ] &&  return 1

    cmd=`cat /proc/$pid/cmdline | tr "\000" "\n"|head -n 1 |cut -d : -f 1`

    # Is this the expected server

    [ "$cmd" != "$name" ] &&  return 1

    return 0

}



running() {

# Check if the process is running looking at /proc

# (works for all users)



    # No pidfile, probably no daemon present

    [ ! -f "$PIDFILE" ] && return 1

    pid=`cat $PIDFILE`

    running_pid $pid $DAEMON || return 1

    return 0

}



start_server() {

            # Start the process using the wrapper

            start-stop-daemon --background --start --quiet --pidfile $PIDFILE \

                        --make-pidfile --chuid $DAEMONUSER:$DAEMONGROUP \

                        --exec $NUMACTL $DAEMON $DAEMON_OPTS

            errcode=$?

        return $errcode

}



stop_server() {

# Stop the process using the wrapper

            start-stop-daemon --stop --quiet --pidfile $PIDFILE \

                        --retry 300 \

                        --user $DAEMONUSER \

                        --exec $DAEMON

            errcode=$?

        return $errcode

}



force_stop() {

# Force the process to die killing it manually

        [ ! -e "$PIDFILE" ] && return

        if running ; then

                kill -15 $pid

        # Is it really dead?

                sleep "$DIETIME"s

                if running ; then

                        kill -9 $pid

                        sleep "$DIETIME"s

                        if running ; then

                                echo "Cannot kill $NAME (pid=$pid)!"

                                exit 1

                        fi

                fi

        fi

        rm -f $PIDFILE

}



case "$1" in

  start)

        log_daemon_msg "Starting $DESC" "$NAME"

        # Check if it's running first

        if running ;  then

            log_progress_msg "apparently already running"

            log_end_msg 0

            exit 0

        fi

        if start_server ; then

            # NOTE: Some servers might die some time after they start,

            # this code will detect this issue if STARTTIME is set

            # to a reasonable value

            [ -n "$STARTTIME" ] && sleep $STARTTIME # Wait some time

            if  running ;  then

                # It's ok, the server started and is running

                log_end_msg 0

            else

                # It is not running after we did start

                log_end_msg 1

            fi

        else

            # Either we could not start it

            log_end_msg 1

        fi

        ;;

  stop)

        log_daemon_msg "Stopping $DESC" "$NAME"

        if running ; then

            # Only stop the server if we see it running

                        errcode=0

            stop_server || errcode=$?

            log_end_msg $errcode

        else

            # If it's not running don't do anything

            log_progress_msg "apparently not running"

            log_end_msg 0

            exit 0

        fi

        ;;

  force-stop)

        # First try to stop gracefully the program

        $0 stop

        if running; then

            # If it's still running try to kill it more forcefully

            log_daemon_msg "Stopping (force) $DESC" "$NAME"

                        errcode=0

            force_stop || errcode=$?

            log_end_msg $errcode

        fi

        ;;

  restart|force-reload)

        log_daemon_msg "Restarting $DESC" "$NAME"

                errcode=0

        stop_server || errcode=$?

        # Wait some sensible amount, some server need this

        [ -n "$DIETIME" ] && sleep $DIETIME

        start_server || errcode=$?

        [ -n "$STARTTIME" ] && sleep $STARTTIME

        running || errcode=$?

        log_end_msg $errcode

        ;;

  status)



        log_daemon_msg "Checking status of $DESC" "$NAME"

        if running ;  then

            log_progress_msg "running"

            log_end_msg 0

        else

            log_progress_msg "apparently not running"

            log_end_msg 1

            exit 1

        fi

        ;;

  # MongoDB can't reload its configuration.

  reload)

        log_warning_msg "Reloading $NAME daemon: not implemented, as the daemon"

        log_warning_msg "cannot re-read the config file (use restart)."

        ;;



  *)

        N=/etc/init.d/$NAME

        echo "Usage: $N {start|stop|force-stop|restart|force-reload|status}" >&2

        exit 1

        ;;

esac



exit 0

```



## Use “container-on-demand” to run MongoDB



- Ask for container `Ubuntu 18.04 - MongoDB`

- Connect via SSH

- Or via the WebTerminal

- In the container, you should start MongoDB before playing with it:



```bash

$ service mongod start

* Starting database mongod [ OK ]

$

$ cat 0-list_databases | mongo

MongoDB shell version v4.2.8

connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb

Implicit session: session { "id" : UUID("70f14b38-6d0b-48e1-a9a4-0534bcf15301") }

MongoDB server version: 4.2.8

admin 0.000GB

config 0.000GB

local 0.000GB

bye

$

```



3. # Redis



![Redis is a fancy Hash Table!!](redis_is_fancy_hash_table.png)



# Resources



**Read or watch:**

- [Redis Crash Course Tutorial](https://www.youtube.com/watch?v=Hbt56gFj998)

- [Redis commands](https://redis.io/docs/latest/commands/)

- [Redis python client](https://redis-py.readthedocs.io/en/stable/)

- [How to Use Redis With Python](https://realpython.com/python-redis/)



## Learning Objectives

- Learn how to use redis for basic operations

- Learn how to use redis as a simple cache



# Requirements



- All of your files will be interpreted/compiled on Ubuntu 18.04 LTS using - `python3` (version `3.7`)

- All of your files should end with a new line

- A `README.md` file, at the root of the folder of the project, is mandatory

- The first line of all your files should be exactly `#!/usr/bin/env python3`

- Your code should use the `pycodestyle` style (version 2.5)

- All your modules should have documentation (`python3 -c 'print(__import__("my_module").__doc__)'`)

- All your classes should have documentation (`python3 -c 'print(__import__("my_module").MyClass.__doc__)'`)

- All your functions and methods should have documentation (`python3 -c 'print(__import__("my_module").my_function.__doc__)'` and `python3 -c 'print(__import__("my_module").MyClass.my_function.__doc__)'`)

- A documentation is not a simple word, it’s a real sentence explaining what’s the purpose of the module, class or method (the length of it will be verified)

- All your functions and coroutines must be type-annotated.



## Install Redis on Ubuntu 18.04



```bash

$ sudo apt-get -y install redis-server

$ pip3 install redis

$ sed -i "s/bind .*/bind 127.0.0.1/g" /etc/redis/redis.conf

```



## Use Redis in a container



Redis server is stopped by default - when you are starting a container, you should start it with: `service redis-server start`



## Author



**Tafara Nyamhunga - [Github](https://github.com/tafara-n) / [Twitter](https://twitter.com/tafaranyamhunga)**