https://github.com/phomint/gowork

Library to help track your credentials and database engines
https://github.com/phomint/gowork

athena databases mysql python

Last synced: 3 months ago
JSON representation

Library to help track your credentials and database engines

• Host: GitHub
• URL: https://github.com/phomint/gowork
• Owner: Phomint
• Created: 2021-12-21T20:55:08.000Z (about 3 years ago)
• Default Branch: master
• Last Pushed: 2024-04-16T19:24:44.000Z (9 months ago)
• Last Synced: 2024-05-16T20:21:24.882Z (8 months ago)
• Topics: athena, databases, mysql, python
• Language: Python
• Homepage: https://pypi.org/project/gowork/
• Size: 27.3 KB
• Stars: 2
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

README

        
GoWork: Easily Works

=======================

GoWork it's a nice tool to your daily analysis, to be easy connect throw your databases and keep your credentials saved and secured.

Every query that you receive is built on `pandas `_, a commun tool in Data Science and Data Analysis.

**First of all, some steps to work easy**

You need to save your credential by this examples above:

**REMEMBER: You can include many kinds of connection and you need to configure just once**

.. code-block:: python

    from gowork.safe.configs import Credentials

    c = Credentials()

**Athena**

.. code-block:: python

    c.insert(name='datalake',

         connector='athena',

         credentials={'aws_access_key_id': 'XXXXXXXXXXXX',

                     'aws_secret_access_key': 'XXXXXXXXXXXXX',

                     's3_staging_dir': 's3://XXXXXXXXXXX/',

                     'region_name': 'us-east-1'},

         encode=['aws_access_key_id', 'aws_secret_access_key']

        )

**Mysql**

.. code-block:: python

    c.insert(name='dw_prod',

         connector='mysql',

         credentials={'host': 'XXXXXXXXX',

                      'database': 'XX',

                      'password': 'XXXXXXXXXX',

                      'port': 3306,

                      'user': 'XXXXXXXXXX'},

         encode=['password', 'user']

         )

**MongoDB**

.. code-block:: python

    c.insert(name='atlas',

         connector='mongodb',

         credentials={'database': 'XXXXXXX',

                      'url': 'mongodb+srv://XXXXXXXXXXXXXXXXXXXXXXXXX'},

         encode=['url']

         )

[IMPORTANT]: the field ``encode`` is what sets it has to be encrypted or not !!

----

**Let's work**

.. code-block:: python

    from gowork.databases import AthenaGo

    a = AthenaGo('datalake')

    a.read_sql('''SELECT * FROM "dataset"."prices" p''')

When you are working in a big query or in a big project that uses many queries, there another tool to help you.

You need just say where your queries are keeped, then you need just to select by the name of the file, using **GoQuery**.

.. image:: img/folder.png

  :width: 400

.. code-block:: python

    from gowork.databases import AthenaGo, GoQuery

    a = AthenaGo('datalake')

    q = GoQuery('../inputs/queries')

    df = a.read_sql(q.use('gather'))

----

 Observation about MongoDB

We are looking at an unstructured data, it's works like Json so see that example when you are filtering a document in other document

.. code-block:: python

    from gowork.databases import MongoGo

    m = MongoGo('FATEC')

    m.read_sql('SELECT id_cliente, forma_envio FROM pedidos')

In order to get just a specific field in ``forma_envio`` like ``tipo``

.. code-block:: python

    m.read_sql('SELECT id_cliente, forma_envio.tipo FROM pedidos')