Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/multimeric/csvschema


https://github.com/multimeric/csvschema

Last synced: 12 months ago
JSON representation

Awesome Lists containing this project

README 

          
==========

CsvSchema

==========



CsvSchema is an easy-to-use module designed to make CSV file checking easier. It allows for the fast creation of complex

validation rules faster using some predefined building blocks.



Basics

------

Fundamentally, CsvSchema allows you to describe what a correct CSV file should look like.

In order to do this you will need to implement a subclass of ``csv_schema.structure.BaseCsvStructure``.



Example::



  from csv_schema.structure.base import BaseCsvStructure

  from csv_schema.columns import (

    IntColumn,

    DecimalColumn,

    StringColumn,

  )



  class TestCsvStructure(BaseCsvStructure):



     a = StringColumn()

     b = IntColumn()

     c = DecimalColumn()



In the example above, we defined a CSV schema class that represents a file with three columns. The first column may

contain any kind of characters, the second allows only numerical values, and third one may contain only decimal values.



**NOTE**:

   The order of attributes in the schema class is important. If you put ``b`` before ``a`` then column ``b`` in CSV file will be the first one.

   Of course this will change validation and first column will no longer allow values like e.g ``'Python'``



After defining your schema you can use it like this::



   schema = TestCsvStructure(['A', '6', ''], 1)

   schema.is_valid()



The first constructor parameter is a list of data representing a single line form CSV file. The second is its position in

 the file from which it was taken.



**NOTE**:

   You can use whatever CSV reading method you like. Just make sure that each CSV line is transformed into list.



If ``is_valid`` return ``False`` you can see errors that has been found in ``schema.errors``. Each error message is formatted in two ways:



- *Line : * when error message applies to whole line

- *Line , : * when error message applies to particular column



More about columns

------------------

There are three types of columns. Their behavior can be altered by some additional keyword arguments:



:StringColumn([blank, min_length, max_length, permissible_values]):

   - blank

       If set to ``True`` column does not has to be filled

   - min_length

       Value can not be shorter that ``min_length``

   - max_length

       Maximal lenght of value

   - permissible_values

       List of allowed values



:IntColumn([blank, min_exclusive, max_exclusive, min_inclusive, max_inclusive]):

   - blank

       If set to ``True`` column does not has to be filled

   - min_exclusive

       Minimum allowed value, exclusive

   - max_exclusive

       Maximal allowed value, exclusive

   - min_inclusive

       Minimum allowed value, inclusive

   - max_inclusive

       Maximal allowed value, inclusive



:DecimalColumn([blank, min_exclusive, max_exclusive, min_inclusive, max_inclusive, fraction_digits, total_digits]):

   - blank

       If set to ``True`` column does not has to be filled

   - min_exclusive

       Minimum allowed value, exclusive

   - max_exclusive

       Maximal allowed value, exclusive

   - min_inclusive

       Minimum allowed value, inclusive

   - max_inclusive

       Maximal allowed value, inclusive

   - fraction_digits

       Number of digits before comma

   - total_digits

       Total number of digits in whole value (before and after comma)



:DateTimeColumn([earliest, latest, format]):

   - earliest

       The earliest date this column will accept, inclusive

   - latest

       The latest date this column will accept, inclusive

   - format

       The datetime format string indicating what kind of date is valid. Refer to the

       `datetime.strptime documentation `_



**NOTE**:

   ``DecimalColumn`` operates on ``decimal.Decimal`` objects. Have that in mind when you will be setting ``min_exclusive``, ``max_exclusive``,

   ``min_inclusive`` or ``max_inclusive``.



Remember that you can always make your own columns by simply subclassing ``csv_schema.columns.base.BaseColumn``::



   from csv_schema.columns.base import BaseColumn

   from csv_schema.exceptions import ImproperValueRestrictionException



   class MyColumn(BaseColumn):



      value_template = ''  # Regular expression describing how proper value should look like in CSV file



      def convert(self, raw_val):  # This method is called in order to transform raw value into Python object

         return None



      def check_restriction(self, value):  # This method is optional. It allows you to specify keyword arguments that can alter column behavior.

         required_value = self.options.get('required_value', None)

         if required_value is not None:

            if required_value != value:

               # Message from ImproperValueRestrictionException will be added to structure errors

               raise ImproperValueRestrictionException('That is not the value you are looking for...')



Column set

----------

Till now you have seen how to use CsvSchema for simple CSV file description. Sometimes specifying types of columns and their behavior just is not enough.

What if you would like to describe more complex validation rules? Let's say that you want a validation rule that says: you have to fill

column A or column B or both of them. This is the situation when you need ``Cs`` objects.



``Cs`` stands for *Column Set* and allows you to express more complex validation rules by simply combining ``Cs`` with use of some logic operators.

Let's consider simple validation rule that we mentioned earlier: you have to fill column A or column B or both of them::



   from csv_schema.structure.set import Cs



   class TestCsvStructure(BaseCsvStructure):



      a = IntColumn(blank=True)

      b = IntColumn(blank=True)



      class Rules(object):

         a_or_b_rule = Cs('a') | Cs('b')



*Changed in 1.1.0: CsvSchema will now store rules in special inner class - Rules*



**NOTE**:

   If you are going to use column sets remeber to set columns used in ``Cs`` instances as **blank**.



Each ``Cs`` instance has assigned columns that needs to be filled in order to *evaluate* ``Cs`` as *true*. In our example each ``Cs`` instance has

only one column but you can assign them as many as you need. For example, if you create ``Cs`` instance like this::



   Cs('a', 'b')



will mean that you want **both** column, ``a`` and ``b`` to be filled because ``Cs`` will *evaluate true* only if **every** column in set is filled.

We used ``|`` operator to combine two ``Cs``. ``|`` can be referred as rule that demands at least one ``Cs`` instance to be evaluated as *true*.

``Cs`` supports also ``^`` operator. It is used to express rule that demands **only one** ``Cs`` instance to be filled. If you create rule ``Cs('a') ^ Cs('b')``

and fill both columns the whole expression will be evaluated as *false*.



**NOTE**:

   Defined rules are evaluated during ``is_valid()`` call and their error messages are added to structure ``errors`` attribute. If custom error message

   is not appropriate to your needs you can override it by calling ``error`` method on whole rule::



      ...

      class Rules(object):

         a_or_b_rule = (Cs('a') | Cs('b')).error('Column A or B needs to be filled')

      ...



If you want to define more than one rule in single structure class you can do it like this::



   ...

   class Rules(object):

      rule_1 = Cs('a') | Cs('b')

      rule_2 = Cs('c') ^ (Cs('d') | Cs('e'))

   ...



Similarly as columns, ``Cs`` behavior can be altered by keyword arguments::



   ...

   class Rules(object):

      rule = Cs('a', b='B')

   ...



In above example ``Cs`` instance will be evaluated *true* if column ``a`` is filled and column ``b`` has value equal to ``'B'``.

Table below shows possible ``Cs`` states depending on different data and settings:



+-------------------+--------------+--------------+------------+

|      Setting      | Column ``a`` | Column ``b`` | Evaluation |

+===================+==============+==============+============+

|      Cs('a')      |      ''      |      ''      |    False   |

+-------------------+--------------+--------------+------------+

|      Cs('a')      |      'A'     |      ''      |    True    |

+-------------------+--------------+--------------+------------+

|      Cs('a')      |      ''      |      'B'     |    False   |

+-------------------+--------------+--------------+------------+

|      Cs('a')      |      'A'     |      'B'     |    True    |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      ''      |      ''      |    True    |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      'A'     |      ''      |    False   |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      ''      |      'B'     |    False   |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      'A'     |      'B'     |    True    |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      ''      |      'C'     |    True    |

+-------------------+--------------+--------------+------------+

|   Cs('a', b='B')  |      'A'     |      'C'     |    False   |

+-------------------+--------------+--------------+------------+



Notice that when column ``b`` is empty or has wrong value column ``a`` can not be filled.



**NOTE**:

   Rememer that you can have more than one value condition in ``Cs``. Creating object like this::



      Cs('a', b='B', c='C')



   will make it *true* if ``b`` is equal to ``'B'`` **and** ``c`` is equal to ``'C'`` (and of course, ``a`` is not empty).

   You can even demand that particular column has to have specific value::



      Cs('a', a='A')