{"id":17383665,"url":"https://github.com/paularnaud2/partools","last_synced_at":"2026-04-16T18:01:54.403Z","repository":{"id":57450971,"uuid":"344529875","full_name":"paularnaud2/ParTools","owner":"paularnaud2","description":"This package contains a bunch of Python tools developed for Support, Test and Automation IT Engineers","archived":false,"fork":false,"pushed_at":"2022-09-17T14:17:16.000Z","size":552,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-27T21:14:45.332Z","etag":null,"topics":["automation","big-files","csv","logging","mail","mailer","oracle","shell-script","sql","support","test","xml"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paularnaud2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-04T15:59:11.000Z","updated_at":"2024-05-31T08:59:22.000Z","dependencies_parsed_at":"2022-09-26T17:31:22.757Z","dependency_job_id":null,"html_url":"https://github.com/paularnaud2/ParTools","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paularnaud2%2FParTools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paularnaud2%2FParTools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paularnaud2%2FParTools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paularnaud2%2FParTools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paularnaud2","download_url":"https://codeload.github.com/paularnaud2/ParTools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245924515,"owners_count":20694730,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","big-files","csv","logging","mail","mailer","oracle","shell-script","sql","support","test","xml"],"created_at":"2024-10-16T07:43:26.817Z","updated_at":"2026-04-16T18:01:54.334Z","avatar_url":"https://github.com/paularnaud2.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ParTools\n \nThis package contains a bunch of Python tools I have developed and used as an IT Test / Support / Automation Engineer.  \nYou can mainly use them for:\n \n- Performing multi threaded SQL queries on an Oracle DB (__sql__)\n- Performing multi threaded SQL queries on a given perimeter on an Oracle DB (__rl__)\n- Comparing and sorting potentially big files (__dq__)\n- Reading and searching potentially big files (__tools.bf__)\n- Parsing potentially big XML files (__tools.xml__)\n- Searching and removing duplicates (__tools.dup__)\n- Filtering potentially big files with flexible conditions (__tools.filter__)\n- Splitting potentially big files (__tools.split__)\n- Extracting doc string from Python code (__tools.extract_doc__)\n- Sending mails with gmail, outlook or without authentication (__mail__)\n- Simple logging or logging in loops (``log`` and ``step_log``), manipulating files and strings, running shell commands, generating threaded message boxes and many others (__utils__)\n \n## Quickstart\n  \n    pip install partools\n \nCreate a python file containing:\n\n    import partools.utils as u\n\n    u.init_log('TEST', True)\n    u.log('Hello World')\n\nAnd run it. You should then see something like this in the console:\n \n    ...\n    Checkout the README.md on GitHub: https://github.com/paularnaud2/ParTools\n    Get started here c:\\Git\\Test\\venv\\lib\\site-packages\\partools\\quickstart\n    Set up your conf here: c:\\Git\\Test\\venv\\lib\\site-packages\\partools\\conf.py\n    Happy scripting!\n    ...\n    07:51:26 - Hello World\n \nThis gives you the links to the __quickstart__ folder where you'll find examples of use and descriptions of the different available packages and functions. You'll also find here the link to the __conf.py__ file (see below).\n \nIf you want to use the __cx_Oracle__ dependant packages (__sql__ and __rl__), you'll need an [Oracle instant client](https://www.oracle.com/uk/database/technologies/instant-client/downloads.html) whose directory you can set in the __conf.py__ file.\n \n### The conf.py file\n \nThe __partools/conf.py__ file contains the __main user settings__ for the __partools__ package such as the path to the Oracle instant client (``ORACLE_CLIENT``). If needed, you can create a __PTconf.py__ file at the root whose attributes will overwrite default (found in __partools/conf.py__). Similarly, you can also create a __PTconf_perso.py__ file that will take over PTconf.py. This can be useful if you __work on a shared repository__ but still want/need to have your own local configurations.\n \n### The global variable files gl.py and main functions inputs\n \nEach package has a __gl.py__ file which sets its __global variables and constants__. Each of these variables can be passed to the main package function (e.g. rl.reqlist for the rl package) and if so, overwrites the value defined in the __gl.py__ file. In that respect, __constants defined in the gl.py file can be seen as default input values__.\n\n## The utils package\n \nThe __partools__ package includes a __utils__ package which provides generic functions used by the other packages. As you may want to use some of them for your own code, I recommend you to check out the list of those functions in __partools/utils/\\_\\_init\\_\\_.py__. Here are a few examples:\n \n- `save_list`: saves a list into a text file\n- `load_txt`: loads a text file into a string or a list\n- `list_files`: returns the list of the files in a folder\n- `like`: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '\\*'). It returns a re.match object giving you access to the matched wildcards.  \nExample: ``m = like('Hello World', 'He\\*o w\\*d')``, m.group(1) =\u003e 'll'\n- `big_number`: converts a potentially big number into a lisible string. For example big_number(10000000) returns '10 000 000'.\n- `get_duration_string`: outputs a string representing the time elapsed since the input ``start_time``. For example ``get_duration_string(0, end_time=200)`` returns '3 minutes and 20 seconds'.\n- ``run_cmd``: runs a Windows shell command (__sTools__)\n- ``run_sqlplus``: connects to sqlplus as sysdba and runs a sql script (__sTools__)\n- ``msg_box``: opens a threaded message box containing the 'msg' input string. This can be used as a end-process notification (__sTools__)\n \n### Logging with the utils package\n \nIf you want the `log` function to actually fill a log file, you have to run `init_log()` before using it, otherwise it will just print out the log info in the console.  \nYou can specify a ``log_format`` for the log timestamp which by default is ``'%H:%M:%S -'`` (conf.LOG_FORMAT setting). Here is what a default log line looks like:\n \n    19:45:04 - This line has been generated by the partools.utils.log function\n \nThe `step_log` function allows you to log some information only when the input ``counter`` is a multiple of the input ``step``. Thus, `step_log` is to be used in loops to __track the progress of long processes__ such as reading or writing millions of lines in a file. The ``what`` input expects a description of what is being counted. It's default value is  ``'lines written'``.  \nIn order to correctly measure the elapsed time for the first log line, the ``step_log`` function has to be initialised by running ``init_sl_time()``.  \nSo for example, if you input ``step=500`` and don't input any ``what`` value, you should get something like this:\n \n    19:45:04 - 500 lines written in 3 ms. 500 lines written in total.\n    19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.\n    19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.\n \nCheckout the __utils_log.py__ file in __partools/quickstart__ for a simple example of use.\n \n### The PT folder\n \nWhen first used, the __utils__ package gets initialised by creating a __PT__ directory (which you can set in __conf.py__). It is intended to contain the __log files and the temporary files__ generated by the different ParTools's scripts. It also has __in__ and __out__ directories used by the __test__ package (and of course that you can also use for your own scripts using ParTools).\n \n## The sql package\n \nThe __sql__ package provides you with three main functions:\n- ``download``: executes multithreaded SELECT or COUNT queries (see partools/quickstart/sql_download.py)\n- ``upload``: executes mass insert queries (see partools/quickstart/sql_upload.py)\n- ``execute``: executes PL/SQL procedures or SQL scripts (see partools/quickstart/sql_execute.py)\n \nThe ``download`` function has three main cases of use:\n- Writing the result of a simple SELECT/COUNT query in a csv file\n- Processing a list of SELECT/COUNT queries in parallel\n- Processing a range query ie. a variabilized query which is executed in parallel for a specified range of KEYs/IDs\n\nEach of these cases is detailed in __partools/quickstart/sql_download.py__\n \n## The rl package\n \nThe __rl__ package has two main functions:\n- ``reqlist``: executes a multithreaded SELECT query on a perimeter given by a csv file. The SQL output can be joint to the input file (see partools/quickstart/reqlist.py)\n- ``left_join_files``: performs a joint between two csv files. This function is used by reqlist to joint the SQL result to the input file.\n\n## Sending emails\n \nParTools is provided with a __mail__ package allowing you to easily send HTML emails with Python. Three functions are available corresponding to different infrastructure/environment usages:\n \n- ``gmail``: for personal computer/network use\n- ``no_auth``: for business computer/network use (if an internal no authentication smtp server is available)\n- ``outlook``: for business computer/network use (uses your Outlook application)\n \nIn order for the mail package to be working, you have to initialise a mail folder by running the 'init_mail' function. This will create a folder in the MAIL_DIR defined in 'conf.py' ('mails/' by default).\n \nIf you want to use the gmail or no_auth functions, you have to set a confidential.txt file. This file must be saved at the CFI_PATH defined in the partools/conf.py file (root by default) using the example provided in the initialised folder. \n \nThe initialised folder contains the mail folders corresponding to the mail_name passed in the mail function. As you'll notice, it initially contains a 'test' folder, allowing you to quickly test the function and to provide you with an example of what a 'mail_name' folder is expected to contain. USER_GMAIL and PWD_GMAIL are only to be set if you use the ``gmail`` function.\n \nSo you'll see two files in the mails/test folder:\n- template.html: the HTML template for the body of your mail. It can contain variables delimited by @@ (in the example @@NAME@@ and @@DATE@@) which are replaced using the ``var_dict`` passed in input.\n- recipients.txt: the list of recipients here containing three fictitious recipients. For your test, it is advised to just let one line with your address.\n \nYou can also directly input a recipients list and a HTMLbody. In that case, setting the template.html and recipients.html is not needed. If you input a ``HTMLbody`` and a ``var_dict``, your ``HTMLbody`` input will be seen as a template whose variables will be replaced using the ``var_dict``.\n \nFor the no_auth function, you'll need to set the host in the conf file (``HOST_NO_AUTH``).\n \n \n## Recover functionalities\n \n``sql.download``, ``rl.reqlist`` and ``sql.execute`` have a __recovery functionality__. This means that if the process is unexpectedly stopped (e.g. because of network issues), then when relaunched, the script __automatically restarts from where it stopped__. When you run long processes (e.g. extracting millions of lines from a database), this can save you a significant amount of time if something goes wrong (especially when close to the end!).  \nThe reliability of these recovery mechanisms is ensured by automated tests using the ``multiprocessing`` library to simulate the unexpected process interruption.\n \nHappy scripting!\n \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaularnaud2%2Fpartools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaularnaud2%2Fpartools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaularnaud2%2Fpartools/lists"}