Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/derb12/mcetl

A small-scale Extract-Transform-Load framework focused on materials characterization
https://github.com/derb12/mcetl

materials-characterization materials-engineering materials-science python

Last synced: 2 months ago
JSON representation

A small-scale Extract-Transform-Load framework focused on materials characterization

Awesome Lists containing this project

README 

          
=====

mcetl

=====



.. image:: https://github.com/derb12/mcetl/raw/main/docs/images/logo.png

    :alt: mcetl Logo

    :align: center



.. image:: https://img.shields.io/pypi/v/mcetl.svg

    :target: https://pypi.python.org/pypi/mcetl

    :alt: Most Recent Version



.. image:: https://readthedocs.org/projects/mcetl/badge/?version=latest

    :target: https://mcetl.readthedocs.io

    :alt: Documentation Status



.. image:: https://img.shields.io/pypi/pyversions/mcetl.svg

    :target: https://pypi.python.org/pypi/mcetl

    :alt: Supported Python versions



.. image:: https://img.shields.io/badge/license-BSD%203--Clause-blue.svg

    :target: https://github.com/derb12/mcetl/tree/main/LICENSE.txt

    :alt: BSD 3-clause license



mcetl is a small-scale, GUI-based Extract-Transform-Load framework focused on materials characterization.



* For Python 3.7+

* Open Source: BSD 3-Clause License

* Source Code: https://github.com/derb12/mcetl

* Documentation: https://mcetl.readthedocs.io.



mcetl is focused on reducing the time required to repeatedly process data files and

write the results to Excel. It does this by allowing the user to define DataSource objects

for each separate source of data. Each DataSource contains information such as the

options needed to import data from files, the calculations that will be performed

on the data, and the options for writing the data to Excel. Once a DataSource is created,

it can be selected within mcetl's main user interface.



In addition, mcetl provides fitting and plotting user interfaces that

can be used without any prior setup.



.. contents:: **Contents**

    :depth: 1



Introduction

------------



Purpose

~~~~~~~



The aim of mcetl is to ease the repeated processing of data files. Contrary to its name, mcetl

can process any tabulated files (txt, csv, tsv, xlsx, etc.), and does not require that the files originate

from materials characterization. However, the focus on materials characterization was selected because:



* Most data files from materials characterization are relatively small in size (a few kB or MB).

* Materials characterization files are typically cleanly tabulated and do not require handling

  messy or missing data.

* It was the author's area of usage and naming things is hard...



mcetl requires only a very basic understanding of Python to use, and allows a single person to

create a tool that their entire group can use to process data and produce Excel files with a

consistent style. mcetl can create new Excel files when processing data or saving peak fitting

results, or it can append to an existing Excel file to easily work with already created files.



Limitations

~~~~~~~~~~~



* Data from files is fully loaded into memory for processing, so mcetl is not

  suited for processing files whose total memory size is large (e.g. cannot

  load a 10 GB file on a computer with only 8 GB of RAM).

* mcetl does not provide any resources for processing data files directly from

  characterization equipment (such as .XRDML, .PAR, etc.). Other libraries such

  as `xylib `_ already exist and are capable of

  converting many such files to a format mcetl can use (txt, csv, etc.).

* The peak fitting and plotting modules in mcetl are not as feature-complete as

  other alternatives such as `Origin `_,

  `fityk `_, `SciDAVis `_,

  etc. The modules are included in mcetl in case those better alternatives are not

  available, and the author highly recommends using those alternatives over mcetl if available.



Installation

------------



Dependencies

~~~~~~~~~~~~



mcetl requires `Python `_ version 3.7 or later and the following libraries:



* `asteval `_

* `lmfit `_ (>= 1.0)

* `Matplotlib `_ (>= 3.1)

* `NumPy `_ (>= 1.8)

* `openpyxl `_ (>= 2.4)

* `pandas `_ (>= 0.25)

* `PySimpleGUI `_ (>= 4.29)

* `SciPy `_



All of the required libraries should be automatically installed when installing mcetl

using either of the two installation methods below.



Additionally, mcetl can optionally use `Pillow `_

to allow for additional options when saving figures in the plotting GUI.



Stable Release

~~~~~~~~~~~~~~



mcetl is easily installed using `pip `_, simply by running

the following command in your terminal:



.. code-block:: console



    pip install --upgrade mcetl



This is the preferred method to install mcetl, as it will always install the

most recent stable release.



Development Version

~~~~~~~~~~~~~~~~~~~



The sources for mcetl can be downloaded from the `Github repo`_.



The public repository can be cloned using:



.. code-block:: console



    git clone https://github.com/derb12/mcetl.git



Once the repository is downloaded, it can be installed with:



.. code-block:: console



    cd mcetl

    python setup.py install



.. _Github repo: https://github.com/derb12/mcetl



Quick Start

-----------



The sections below give a quick introduction to using mcetl, requiring no setup.

For a more detailed introduction, refer to the `tutorials section`_ of mcetl's

documentation.



.. _tutorials section: https://mcetl.readthedocs.io/en/latest/tutorials/index.html



Note: on Windows operating systems, the GUIs can appear blurry due to how dpi

scaling is handled. To fix, simply do:



.. code-block:: python



    import mcetl

    mcetl.set_dpi_awareness()



The above code **must** be called before opening any GUIs, or else the dpi scaling

will be incorrect.



Main GUI

~~~~~~~~



The main GUI for mcetl contains options for processing data, fitting, plotting,

writing data to Excel, and moving files.



Before using the main GUI, DataSource objects must be created. Each DataSource

contains the information for reading files for that DataSource (such as what

separator to use, which rows and columns to use, labels for the columns, etc.),

the calculations that will be performed on the data, and the options for writing

the data to Excel (formatting, placement in the worksheet, etc.).



The following will create a DataSource named 'tutorial' with the default settings,

and will then open the main GUI.



.. code-block:: python



    import mcetl



    simple_datasource = mcetl.DataSource(name='tutorial')

    mcetl.launch_main_gui([simple_datasource])



Fitting Data

~~~~~~~~~~~~



To use the fitting module in mcetl, simply do:



.. code-block:: python



    from mcetl import fitting

    fitting.launch_peak_fitting_gui()



A window will then appear to select the data file(s) to be fit and the Excel file for saving the results.

No other setup is required for doing fitting.



After doing the fitting, the fit results and plots will be saved to Excel.



Plotting

~~~~~~~~



To use the plotting module in mcetl, simply do:



.. code-block:: python



    from mcetl import plotting

    plotting.launch_plotting_gui()



Similar to fitting, a window will then appear to select the data file(s) to be plotted,

and no other setup is required for doing plotting.



When plotting, the image of the plots can be saved to all formats supported by

`Matplotlib `_, including tiff, jpg, png, svg, and pdf.



In addition, the layout of the plots can be saved to apply to other figures later, and the data

for the plots can be saved so that the entire plot can be recreated.



To reopen a figure saved through mcetl, do:



.. code-block:: python



    plotting.load_previous_figure()



Generating Example Data

~~~~~~~~~~~~~~~~~~~~~~~



Files for example data from characterization techniques can be created using:



.. code-block:: python



    from mcetl import raw_data

    raw_data.generate_raw_data()



Data produced by the generate_raw_data function covers the following characterization techniques:



* X-ray diffraction (XRD)

* Fourier-transform infrared spectroscopy (FTIR)

* Raman spectroscopy

* Thermogravimetric analysis (TGA)

* Differential scanning calorimetry (DSC)

* Rheometry

* Uniaxial tensile tests

* Pore size measurements



Example Programs

~~~~~~~~~~~~~~~~



`Example programs`_  are available to show basic usage of mcetl. The examples include:



* Generating raw data

* Using the main GUI

* Using the fitting GUI

* Using the plotting GUI

* Reopening a figure saved with the plotting GUI



The example program for using the main GUI contains all necessary inputs for processing the

example raw data generated by the generate_raw_data function as described above and is an

excellent resource for creating new DataSource objects.



.. _Example programs: https://github.com/derb12/mcetl/tree/main/examples



Future Plans

------------



Planned features for later releases:



* Develop tests for all modules in the package.

* Switch from print statements to logging.

* Switch GUI backend from PySimpleGUI to wxPython or something web-based.

* Add more plot types to the plotting gui, including bar charts, categorical plots, and 3d plots.

* Make fitting more flexible by allowing more options or user inputs.

* Potentially add support for importing data from more file types.

* Improve overall look and usability of all GUIs.



Contributing

------------



Contributions are welcomed and greatly appreciated. For information on submitting bug reports,

pull requests, or general feedback, please refer to the `contributing guide`_.



.. _contributing guide: https://github.com/derb12/mcetl/tree/main/docs/contributing.rst



Changelog

---------



Refer to the changelog_ for information on mcetl's changes.



.. _changelog: https://github.com/derb12/mcetl/tree/main/CHANGELOG.rst



License

-------



mcetl is open source and available under the BSD 3-clause license.

For more information, refer to the license_.



.. _license: https://github.com/derb12/mcetl/tree/main/LICENSE.txt



Author

------



* Donald Erb 



Gallery

-------



Images of the various GUIs can be found on the `gallery section`_ of

mcetl's documentation.



.. _gallery section: https://mcetl.readthedocs.io/en/latest/gallery.html