https://github.com/relativitydev/relativity-import-samples

Repository provides useful samples how Relativity Import API can be utilized.
https://github.com/relativitydev/relativity-import-samples

Last synced: 2 months ago
JSON representation

Repository provides useful samples how Relativity Import API can be utilized.

• Host: GitHub
• URL: https://github.com/relativitydev/relativity-import-samples
• Owner: relativitydev
• License: other
• Created: 2022-08-11T07:27:54.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2024-09-10T12:56:49.000Z (over 1 year ago)
• Last Synced: 2025-01-28T12:26:24.673Z (about 1 year ago)
• Language: HTML
• Size: 755 KB
• Stars: 6
• Watchers: 3
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Codeowners: CODEOWNERS

Awesome Lists containing this project

README

          
# relativity-import

### Table of Contents

### **[Introduction](#introduction)**

**[Prerequisites](#prerequisites)**


**[Glossary](#glossary)**


### **[Getting Started](#getting-started-1)**

**[NuGet Libraries](#relativityimportsdk)**


**[Authorization](#authorization)**


**[Permissions](#permissions)**


**[Builders](#builders)**


**[General Import flow description ](#general-import-flow-description)**


**[Example Import flows](#example-of-simple-import-documents-flow)**


**[API Documentation](#rest-api)**


**[API Response](#api-response)**


**[ImportJob & DataSource States](#rest-api)**


**[Errors](#errors)**


### **[Code Samples](#samples)**

**[.NET Console Application - How-to](#net-code-samples---how-to)**


**[.NET Framework & Kepler Console Application - How-to](#keplerclient-code-samples---how-to)**


**[Powershell scripts - How-to](#powershell-script-samples---how-to)**


### **[Performance Best Practices](#performance-best-practices)**

**[Import Job Settings](#import-job-settings)**


# Introduction

The ***Relativity Import Service API***  is a Kepler service that provides functionality for importing large numbers of documents, images, and Relativity Dynamic Objects (RDOs) into a Relativity workspace. 

The import process operates on structured data sets that are described by load file and located in a place accessible for workspace.  

The main principle of operation is based on creating managed importing job with a list of data sets (intended for import) assigned to it.




Thanks to RESTful API you are able to easily create import job, configure it and run it. 

Dataset (containing structured data) that you want to import can be then added as a source to the job. The system will take care of it in the background by adding this source to the queue, scheduling and finally starting the import data to destination workspace, and if necessary, resuming the import process. All that remains for the user is to monitor the status of import and current progress - all using  provided API.

Job and data sources configurations allow you to flexibly adjust the import to your needs. In addition, the adopted error handling helps you to identify the source of potential problems. 

 NOTE: Import Service (*Import*) is delivered as a RAP application installed in Relativity One.

# Prerequisites

1. The following Relativity applications must be installed:

    | application name     | application Guid                      | where installed     |

    |----------------------|---------------------------------------|---------------------|

    |*Import*              | 21F65FDC-3016-4F2B-9698-DE151A6186A2  |  workspace          |

    |*DataTransfer.Legacy* | 9f9d45ff-5dcd-462d-996d-b9033ea8cfce  |  instance           |

2. Appropriate user [permissions](#permissions) need to be set. 

3. Data set - load files, source files (native documents, images, text files) - need to be placed in the destination fileshare location accessible to workspace. 

   You can use TAPI (https://platform.relativity.com/RelativityOne/#Transfer_API/Relativity_Transfer_SDK.htm) to upload all files to destination location.

4. The following packages need to be installed in client application:

   *NOTE*: Required only when Kepler .NET client is used.

   - [Relativity.Import.SDK](#relativityimportsdk)

   - [Relativity.Import.Models.SDK](#relativityimportsdk) (added automatically as dependency to Relativity.Import.SDK)

   - [Relativity.Kepler.Client.SDK](https://www.nuget.org/packages/Relativity.Kepler.Client.SDK)

   (added automatically as dependency to Relativity.Import.SDK)

***

## Glossary

**Data importing** - Functionality that makes that structured data set are uploaded into destination workspace.

**Dataset** - Structured data containing metadata, native documents, images, text files described by load file or opticon file.

Such a dataset can be pointed during data source configuration and MUST be located in place accessible for workspace. 

**ImportJob** - It is the main object in import service taking part in import flow. It represents single import entity described by its configuration which decides about import behavior e.g. import type, overlay mode, fields mapping, etc.  

In addition, ImportJob object holds the information about its current state and import progress.

Import jobs aggregates dataSources - single import job can consists of many sources.

**DataSource**  - It is an object that corresponds to single set of data to be imported. Each data source has own configuration that indicates the physical location of data set (load file). Data set configuration affects also how data in load file are read.

In addition, data source stores the information about current state and import progress of particular source.

**Kepler service** - API service created but using the Relativity Kepler framework. This framework provides you with the ability to build custom REST Endpoints via a .NET interface. Additionally, the Kepler framework includes a client proxy that you can use when interacting with the services through .NET. [See more information](https://platform.relativity.com/RelativityOne/index.htm#Kepler_framework/Kepler_framework.htm#Client-s)

**Item Error** - An error that may occur during the import process and concerns only one imported record from the load file. A common reason of these errors are data validation or already existing records in the workspace.

---

# Getting Started

Import Service is built as a standard Relativity Kepler Service. It provides sets of endpoints that must be called sequentially in order to execute import.

The following sections outline how to make calls to import service.

    

> HTTP clients 

 You can make calls to a import service using any standard REST or HTTP client, because all APIs (Keplers APIs) are exposed over the HTTP protocol. You need to set the required X-CSRF-Header. [more details](https://platform.relativity.com/RelativityOne/index.htm#Kepler_framework/Kepler_framework.htm#Client-s) 

        HttpClient client = new HttpClient();

        client.DefaultRequestHeaders.Add("X-CSRF-Header", "-");

        createImportJobUri = $"{host}/Relativity.REST/api/import.service/v1....

        

        var response = await httpClient.PostAsJsonAsync(createImportJobUri, payload);

In case of using .NET client the [Relativity.Import.Models.SDK](#relativityimportmodelssdk) package containing contract models would be used.

  Please look at dedicated [code samples](#samples) for .NET 7 or for PowerShell scripts.

> Kepler .NET client

You can access Kepler service from any .NET language using the client library provided as part of the Kepler framework. It exposes a factory class that you can use to create the client proxy by passing URIs to import services and credentials. Then use .NET proxy to interact with a import service as a set of .NET objects. When you call a member method, the proxy makes a corresponding HTTP request to the respective service endpoint. [more details](https://platform.relativity.com/RelativityOne/index.htm#Kepler_framework/Kepler_framework.htm#Client-s) 

Kepler contract for import service are exposed in [Relativity.Import.SDK](relativityimportsdk) package.

        using (Relativity.Import.V1.Services.IImportJobController importJobController =_serviceFactory.CreateProxy())

        {

            // Create import job.

            Response response = await importJobController.CreateAsync(

                importJobID: importId,

                workspaceID: workspaceId,

                applicationName: "Import-service-sample-app",

                correlationID: "Sample-job-0001");

            }

        }

  Please look at dedicated [code samples](#samples) for .NET 4.6.2 with Kepler.

---

## Relativity.Import.SDK ###

Relativity.Import.SDK is a .NET library that contains kepler interfaces for import service.

It provides and simplifies executing import in client application.

Relativity.Import.SDK targets .NET Framework 4.6.2

**NOTE: Use this package when your application USE Kepler.**

[![Version](https://img.shields.io/nuget/v/Relativity.Import.SDK.svg?color=royalblue)](https://www.nuget.org/packages/Relativity.Import.SDK)

[![Downloads](https://img.shields.io/nuget/dt/Relativity.Import.SDK?color=green)](https://www.nuget.org/packages/Relativity.Import.SDK)


  

### **Installing via NuGet** 

        Install-Package Relativity.Import.SDK 

## Relativity.Import.Models.SDK ###

Relativity.Import.Models.SDK is a .NET library that contains contract models for API and [builders](#builders) which help user to prepare payloads in correct and consistent way.

Relativity.Import.Models.SDK targets .NET Standard 2.0. The NuGet package also includes direct targets for .NET Framework 4.6.2.


  

**NOTE:**

This package is automatically installed as dependency when using Relativity.Import.SDK.

**NOTE:** You can install this package directly when your application does not use Kepler.


 

[![Version](https://img.shields.io/nuget/v/Relativity.Import.Models.SDK.svg?color=royalblue)](https://www.nuget.org/packages/Relativity.Import.Models.SDK)

[![Downloads](https://img.shields.io/nuget/dt/Relativity.Import.Models.SDK?color=green)](https://www.nuget.org/packages/Relativity.Import.Models.SDK)

### **Installing via NuGet** 

        Install-Package Relativity.Import.Models.SDK

---

## Authorization

> HTTP clients 

Import Service API conforms to the same authentication rules as other Relativity REST APIs. 

The more details can be found under the following link:

[REST_API_authentication](https://platform.relativity.com/RelativityOne/Content/REST_API/REST_API_authentication.htm)

> Kepler .NET client

The Kepler framework uses a proxy to handle client requests. The more details can be found under the following link:

[Proxies_and_authentication](https://platform.relativity.com/RelativityOne/Content/Kepler_framework/Proxies_and_authentication.htm#Service)

---

## Permissions

The following Relativity permissions are required to use import features provided in Import Service API.

| Object Security section| Permission |

| :---- | :----: |

| •	Document |	View, Add, Edit |

| •	Relativity Import Job: |	View, Add, Edit |

| •	Relativity Import Data Source |	View, Add, Edit |

| Tab Visibility|

| :---- |

| •	Documents |

| Admin Operation|

| :---- |

| •	Allow Import |

---


  

## Builders

Builders provided in Relativity.Import.Models.SDK package help to create settings for import job and data source in correct and consistent way. It is highly recommended to prepare these objects in such a way in .NET application. They are implemented in fluent api pattern so it is very easy to use them. Moreover, using them in client application will avoid the risk of incorrect and inconsistent configuration

which may lead to errors during import process.

*ImportDocumentsSettingsBuilder* - builds ImportDocumentsSettings used for import job configuration (documents import).

*ImportRdoSettingsBuilder* - builds ImportRdoSettings used for import job configuration (rdos import).

*DataSourceSettingsBuilder* - builds DataSourceSettings used for data source configuration.

> C#

    // Example of using ImportDocumentSettingsBuilder to create ImportDocumentSettings.

    ImportDocumentSettingsBuilder.Create()

        .WithOverlayMode(x => x

            .WithKeyField(overlayKeyField)

            .WithMultiFieldOverlayBehaviour(MultiFieldOverlayBehaviour.MergeAll))

        .WithNatives(x => x

            .WithFilePathDefinedInColumn(filePathColumnIndex)

            .WithFileNameDefinedInColumn(fileNameColumnIndex))

        .WithoutImages()

        .WithFieldsMapped(x => x

            .WithField(controlNumberColumnIndex, "Control Number")

            .WithExtractedTextInSeparateFiles(f => f

		.WithEncoding("UTF-16")

		.WithFileSizeDefinedInColumn(fileSizeColumnIndex))))

        .WithFolders(f => f

            .WithRootFolderID(rootFolderId, r => r

                .WithFolderPathDefinedInColumn(folderPathColumnIndex)));

> C#

    // Example of using DataSourceSettingsBuilder to create DataSourceSettings.

    DataSourceSettings dataSourceSettings = DataSourceSettingsBuilder.Create()

        .ForLoadFile(loadFile01Path)

        .WithDelimiters(d => d

            .WithColumnDelimiters('|')

            .WithQuoteDelimiter('^')

            .WithNewLineDelimiter('#')

            .WithNestedValueDelimiter('&')

            .WithMultiValueDelimiter('$'))

        .WithFirstLineContainingHeaders()

        .WithEndOfLineForWindows()

        .WithStartFromBeginning()

        .WithDefaultEncoding()

        .WithDefaultCultureInfo();

NOTE: Please review the samples to find more about builders.

***

## General Import flow description

The general flow includes several steps consisted in sending appropriate HTTP requests.

1. **Create Import Job** 

   Creates import job entity in particular workspace. Job is defined by its unique Id generated by user and provided in the request which is used in the next steps.

2. **Configure Import Job**

    Configures existing import job by defining sets of significant parameters including import type, its mode, fields mapping. This step covers two configuration options: 

    - Documents Configuration 

    - Rdos Configuration.

  

3. **Add one or multiple DataSources**

    Creating data source entity or entities for particular import job. It represents the configuration that corresponds to dataset being imported. Data source is identified by its unique Id generated by user and provided in the request. Data source configuration includes path to “load file” and other significant parameters telling how data in load file will be read and interpreted by system.

    Many data sources can be added to the same import job. Data sources can be added both before job is started and after, so user can add additional sources to running importJob.

4. **Begin Job**  

  Starts Import Job which enables the process that schedules importing data to workspace based on the configuration assigned in previous steps.

  Started job does not mean that data are instantly imported. However DataSources are added to the queue and scheduled by background mechanism.

  The import Job state or data source state shows the current stage. 

5. **Cancel Job**

User can cancel running Import Job in every moment. All related data sources will not be imported except for those whose import has already started.

6. **End Import Job** 

  Ends import job that was already started. It is optional step but it is highly recommended in case when no more data source is plan to be added for particular job. All data sources added to the job before the end request was sent will be imported.

## Example of Simple Import Documents Flow

1.  **Create Import Job** 

        

    > curl

    

         curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/e694ad62-198d-4ecb-936d-1862ddfa4235'

        -H 'X-CSRF-Header: -' 

        -d '{

        "applicationName": "simpleImportDocuments",

        "correlationID": "c0r31ati0n_ID"

        }'




2. **Create Import Job Configuration** 

    > curl

        curl -X POST \'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/e694ad62-198d-4ecb-936d-1862ddfa4235/documents-configurations/' 

        -H 'X-CSRF-Header: -' 

        -d "$importSettingsPayloadJson"

    Import Configuration payload example:

    > JSON

        {

        "importSettings": {

            "Overlay":null,

            "Native":{

                "FilePathColumnIndex": "22",

                "FileNameColumnIndex": "13"

            },

            "Image": null,

            "Fields": {

                "FieldMappings": [

                    {

                    "ColumnIndex": 0,

                    "Field": "Control Number",

                    "ContainsID": false,

                    "ContainsFilePath": false

                    },

            ]

            },

            "Folder": {

                "RootFolderID": 1003663,

                "FolderPathColumnIndex": 2

            }

        }

        }'

    > C#  Builders

        ImportDocumentSettings importSettings = ImportDocumentSettingsBuilder.Create()

            .WithAppendMode()

            .WithNatives(x => x

                .WithFilePathDefinedInColumn(filePathColumnIndex)

                .WithFileNameDefinedInColumn(fileNameColumnIndex))

            .WithoutImages()

            .WithFieldsMapped(x => x

                .WithField(controlNumberColumnIndex, "Control Number")

            .WithFolders(f => f

                .WithRootFolderID(rootFolderId, r => r

                    .WithFolderPathDefinedInColumn(folderPathColumnIndex)));

    > C# 

        ImportDocumentSettings importSettings = new ImportDocumentSettings()

        {

            Overlay = null,

            Native = new NativeSettings

            {

                FileNameColumnIndex = fileNameColumnIndex,

                FilePathColumnIndex = filePathColumnIndex,

            },

            Fields = new FieldsSettings

            {

                FieldMappings = new[]

                {

                    new FieldMapping

                    {

                        Field = "Control Number",

                        ContainsID = false,

                        ColumnIndex = 0,

                        ContainsFilePath = false,

                    },

                },

            },

            Folder = new FolderSettings

            {

                FolderPathColumnIndex = folderPathColumnIndex,

                RootFolderID = 1003663,

            },

            Other = null,

        };




3. **Add DataSource** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/e694ad62-198d-4ecb-936d-1862ddfa4235/sources/0cb922a2-8df4-42fd-9429-c241410a0d1e'

        -H 'X-CSRF-Header: -' \

        -H 'Content-Type: application/json' 

        -d "$dataSourceSettingsJson"

        }'

    Data source configuration payload example:

    > JSON

        {

        "dataSourceSettings": {

            "Path": "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat",

            "FirstLineContainsColumnNames": true,

            "StartLine": 1,

            "ColumnDelimiter": "|",

            "QuoteDelimiter": "^",

            "NewLineDelimiter": "#",

            "MultiValueDelimiter": ";",

            "NestedValueDelimiter": "&",

            "EndOfLine" = 0

             Encoding" = null

            "CultureInfo" : "en-US",

            "Type": 2

            }

        }'

    > C# builders

   

			DataSourceSettings dataSourceSettings = DataSourceSettingsBuilder.Create()

				.ForLoadFile("\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat")

				.WithDelimiters(d => d

					.WithColumnDelimiters('|')

					.WithQuoteDelimiter('^')

					.WithNewLineDelimiter('#')

					.WithNestedValueDelimiter('&')

					.WithMultiValueDelimiter(';'))

				.WithFirstLineContainingHeaders()

				.WithEndOfLineForWindows()

				.WithStartFromBeginning()

				.WithDefaultEncoding()

				.WithDefaultCultureInfo();

    > C#

			DataSourceSettings dataSourceSettings = new DataSourceSettings

			{

				Type = DataSourceType.LoadFile,

				Path = "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat",

				NewLineDelimiter = '#',

				ColumnDelimiter = '|',

				QuoteDelimiter = '^',

				MultiValueDelimiter = ';',

				NestedValueDelimiter = '&',

				Encoding = null,

				CultureInfo = "en-us",

				EndOfLine = DataSourceEndOfLine.Windows,

				FirstLineContainsColumnNames = true,

				StartLine = 0,

			};




4. **Begin Job**

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/e694ad62-198d-4ecb-936d-1862ddfa4235/begin/' 

        -H 'X-CSRF-Header: -' 

        -d ''




5. **End Import Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/e694ad62-198d-4ecb-936d-1862ddfa4235/end/'

        -H 'X-CSRF-Header: -' 

        -d ''





## Example of Simple Import Images Flow

1. **Create Import Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/'

        -H 'X-CSRF-Header: -' 

        -d '{

        "applicationName": "simpleImportImages",

        "correlationID": "img0r22ati0n_ID"

        }'

2. **Create Import Job Configuration**

    > curl

        curl -X POST \'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/documents-configurations/' 

        -H 'X-CSRF-Header: -' 

        -d "$importSettings"

    Import Configuration payload example:

    >JSON

        {

        "importSettings": {

            "Overlay":null,

            "Native":null,

        "Image": 

            {

                "PageNumbering": 1,

                "ProductionID": null,

                "LoadExtractedText": false,

                "FileType": 0    

            }

            "Fields": null,

            "Folder": null

            }

        }'

    >  C# Builder

        ImportDocumentSettings importSettings = ImportDocumentSettingsBuilder.Create()

            .WithAppendMode()

            .WithoutNatives()

            .WithImages(i => i

                .WithAutoNumberImages()

                .WithoutProduction()

                .WithoutExtractedText()

                .WithFileTypeAutoDetection())

            .WithoutFieldsMapped()

            .WithoutFolders();

            

3. **Add DataSource** 

    > curl

            curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/sources/0cb922a2-8df4-42fd-9429-c241410a0002'

            -H 'X-CSRF-Header: -' \

            -H 'Content-Type: application/json' 

            -d "$dataSourceSettingsJson"

            }'

    > JSON

    

        {

        "dataSourceSettings": {

            "Path": "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\opticon_file.opt",

            "FirstLineContainsColumnNames": false,

            "StartLine": 0;

            "EndOfLine" = 0

             Encoding" = null

            "CultureInfo" : null,

            "Type": 1

            }

        }'

    > C# Builders

			DataSourceSettings dataSourceSettings = DataSourceSettingsBuilder.Create()

				.ForOpticonFile("\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\opticon_file.opt")

				.WithDefaultDelimitersForOpticonFile()

				.WithEndOfLineForWindows()

				.WithStartFromBeginning()

				.WithDefaultEncoding()

				.WithDefaultCultureInfo();

    > C#

			DataSourceSettings dataSourceSettings = new DataSourceSettings

			{

				Type = DataSourceType.Opticon,

				Path = "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\opticon_file.opt",

				NewLineDelimiter = default,

				ColumnDelimiter = default,

				QuoteDelimiter = default,

				MultiValueDelimiter = default,

				NestedValueDelimiter = default,

				Encoding = null,

				CultureInfo = null,

				EndOfLine = DataSourceEndOfLine.Windows,

				FirstLineContainsColumnNames = false,

				StartLine = 0,

			};




4. **Begin Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/begin/' 

        -H 'X-CSRF-Header: -' 

        -d ''

5. **End Import Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/end/'

        -H 'X-CSRF-Header: -' 

        -d ''





## Example of Simple Import Relativity Dynamic Objects (RDO) Flow

1. **Create Import Job** 

    > curl

    

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/77140fb9-f515-4b65-a2ce-c347492e2905/'

        -H 'X-CSRF-Header: -' 

        -d '{

        "applicationName": "simpleImportRdo",

        "correlationID": "rdor31ati0n_ID"

        }'

2. **Create Import Job Configuration**

    > curl

        curl -X POST \'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/77140fb9-f515-4b65-a2ce-c347492e2905/rdo-configurations/' 

        -H 'X-CSRF-Header: -' 

        -d $"importRdoSettings"'

    Import RDO Configuration payload example:

     > JSON

        {

            "importSettings": {

                "OverwriteMode": "Append",

                "Fields": {

                "FieldMappings": [

                    {

                    "ColumnIndex": 0,

                    "Field": "Name"

                    },

                    ]

                },

                "Rdo": {

                    "ArtifactTypeID": 1000066,

                    "ParentColumnIndex" :null

                },

            }

        }

    > C# Builder

        

        ImportRdoSettings importSettings = ImportRdoSettingsBuilder.Create()

            .WithAppendMode()

            .WithFieldsMapped(f => f

                .WithField(nameColumnIndex, "Name")

            .WithRdo(r => r

                .WithArtifactTypeId(domainArtifactTypeID)

                .WithoutParentColumnIndex());

    > C# 

        ImportRdoSettings importSettings = new ImportRdoSettings()

        {

            Overlay = null,

            Fields = new FieldsSettings

            {

                FieldMappings = new[]

                {

                    new FieldMapping

                    {

                        Field = "Name",

                        ContainsID = false,

                        ColumnIndex = nameColumnIndex,

                        ContainsFilePath = false,

                    },

                },

            },

            Rdo = new RdoSettings

            {

                ArtifactTypeID = rdoArtifactTypeID,

                ParentColumnIndex = null,

            },

        };

3. **Add DataSource** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/4c4215bf-d8a3-48d4-a3e0-3a40428415e7/sources/0cb922a2-8df4-42fd-9429-c241410a0002'

        -H 'X-CSRF-Header: -' \

        -H 'Content-Type: application/json' 

        -d "$dataSourceSettingsJson"

        }'

    Data source configuration payload example:

    > JSON

        {

        "dataSourceSettings": {

            "Path": "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat",

            "FirstLineContainsColumnNames": true,

            "StartLine": 1,

            "ColumnDelimiter": "|",

            "QuoteDelimiter": "^",

            "NewLineDelimiter": "#",

            "MultiValueDelimiter": ";",

            "NestedValueDelimiter": "&",

            "EndOfLine" = 0

            Encoding" = null

            "CultureInfo" : "en-US",

            "Type": 2

            }

        }'

    > C# builder

   

			DataSourceSettings dataSourceSettings = DataSourceSettingsBuilder.Create()

				.ForLoadFile("\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat)

				.WithDelimiters(d => d

					.WithColumnDelimiters('|')

					.WithQuoteDelimiter('^')

					.WithNewLineDelimiter('#')

					.WithNestedValueDelimiter('&')

					.WithMultiValueDelimiter(';'))

				.WithFirstLineContainingHeaders()

				.WithEndOfLineForWindows()

				.WithStartFromBeginning()

				.WithDefaultEncoding()

				.WithDefaultCultureInfo();

    > C#

			DataSourceSettings dataSourceSettings = new DataSourceSettings

			{

				Type = DataSourceType.LoadFile,

				Path = "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\load_file.dat",

				NewLineDelimiter = '#',

				ColumnDelimiter = '|',

				QuoteDelimiter = '^',

				MultiValueDelimiter = ';',

				NestedValueDelimiter = '&',

				Encoding = null,

				CultureInfo = "en-us",

				EndOfLine = DataSourceEndOfLine.Windows,

				FirstLineContainsColumnNames = true,

				StartLine = 0,

			};

4. **Begin Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/77140fb9-f515-4b65-a2ce-c347492e2905/begin/' 

        -H 'X-CSRF-Header: -' 

        -d ''

5. **End Import Job** 

    > curl

        curl -X POST 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/77140fb9-f515-4b65-a2ce-c347492e2905/end/'

        -H 'X-CSRF-Header: -' 

        -d ''

---

### REST API

 Review the open API spec for import service: [OpenAPI spec](https://github.com/relativitydev/relativity-import-samples/blob/main/OpenAPI/openapidoc.json)

## API Response

Each HTTP response to POST request has unified schema:

        {

        "IsSuccess": true,

        "ErrorMessage": "",

        "ErrorCode": "",

        "ImportJobID": "00000000-0000-0000-0000-000000000000"

        }

Each HTTP response to GET requests has unified schema:

        {

        "Value": {

            ....

            ....

        },

        "IsSuccess": true,

        "ErrorMessage": "",

        "ErrorCode": "",

        "ImportJobID": "00000000-0000-0000-0000-000000000000"

        }

---

## Import Job States

Import job state can be read from  GET Import Job details response

> curl 

    curl -X GET 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/ca04baf0-4a1a-4787-94d8-5bba89d2eb0f/details' 

    -H 'X-CSRF-Header: -'

 > response JSON

    {

        "Value": {

            "IsFinished": false,

            "State": "New",

            "ApplicationName": "import demo",

            "Errors": [],

            "CreatedBy": 9,

            "CreatedOn": "2023-01-11T13:47:45.513",

            "LastModifiedBy": 9,

            "LastModifiedOn": "2023-01-11T13:47:45.513"

        },

        "IsSuccess": true,

        "ErrorMessage": "",

        "ErrorCode": "",

        "ImportJobID": "ca04baf0-4a1a-4787-94d8-5bba89d2eb0f"

    }

| value | State                       | Description                                                                                       |

|-------|-----------------------------|---------------------------------------------------------------------------------------------------|

| 10    | New                         | Initial state, job created.                                                                       |

| 13    | Configured                  | Job has been configured and is ready to begin.                                                    |

| 16    | InvalidConfiguration        | Job has been configured but the configuration is invalid.                                         |

| 20    | Idle                        | Job is ready for running but is waiting on new data source or all data source has been processed. |

| 22    | Scheduled                   | Job is ready waiting on queue to begin the process of import.                                     |

| 25    | Inserting                   | Job is executing, import of data source is currently in progress.                                 |

| 26    | PendingCompletion_Scheduled | Job is ended but data source is still waiting on queue to begin the process of import.            |

| 27    | PendingCompletion_Inserting | Job is ended but the import of data source is currently in progress.                              |

| 29    | Paused                      | Job is paused and waiting.                                                                        |

| 30    | Canceled                    | Job canceled.                                                                                     |

| 40    | Failed                      | Job has failed to import data.                                                                    |

| 50    | Completed                   | Job has ended with success.                                                                      |

## Import Data Source States

Data source state can be read from GET Data source details response

> curl 

    curl -X GET 'https://relativity-host/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/ca04baf0-4a1a-4787-94d8-5bba89d2eb0f/sources/40ddb007-4330-41cc-b5aa-2ea6961073a5/details' 

    -H 'X-CSRF-Header: -'

 > response JSON

        {

            "Value": {

                "State": "New",

                "DataSourceSettings": {

                    ...

                    ...

                },

                "JobLevelErrors": []

            },

            "IsSuccess": true,

            "ErrorMessage": "",

            "ErrorCode": "",

            "ImportJobID": "ca04baf0-4a1a-4787-94d8-5bba89d2eb0f"

        }

| Value   |   State            |                   Description                                   |

|----|-------------------------|-----------------------------------------------------------------|

| 0  | Unknown                 | Invalid state for a data source.                                |

| 10 | New                     | Initial state, data source was created.                         |

| 22 | Scheduled               | Data source is waiting on queue to begin the process of import. |

| 24 | PendingInserting        | Data source has been sent to Worker to begin the import.        |

| 25 | Inserting               | Data source is currently in progress of processing.             |

| 30 | Canceled                | Data source canceled.                                           |

| 40 | Failed                  | Failed to import data from Data source.                         |

| 45 | CompletedWithItemErrors |  Data source processed, import finished with item errors.       |

| 50 | Completed               | Data source processed, import finished.                         |

|    |                         |                                                                 |

# Errors

## General remarks

Import service throws two kind of errors:

 - job level errors

 - item level errors.

 Job level errors are severe enough to cause the entire import job to fail. These errors can be found in the *GetDetailsAsync* endpoints for *IImportJobController* and *IImportSourceController*.

 Item level errors are specific to rows within the data source being imported. Unlike job level errors, item level errors do not cause the entire import job to fail. Instead, they are logged and the import process continues with the next row from the load file.

 Item level errors can result in whole record, meaning document or RDO, not being imported to the workspace or the record in the workspace can be incomplete.

 You can retrieve all item level errors that occurred during the import process from *GetItemErrorsAsync* endpoint for *IImportSourceController*.

## Retrieving errors

Error handling in Import Service returns Error Codes and Error Messages:

 - in every response for failed HTTP request

 - when requested by user for all job level errors that occurred during importing of particular data source e.g.:

 > curl

        curl -X GET 'https://relativity.roadie.so/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/00000000-0000-0000-0000-000000000000/sources/00000000-0000-0000-0000-000000000000/details'

        -H 'accept: application/json'

        -H 'X-CSRF-Header: -'

  > JSON response

    {

	"Value": {

		"State": "Failed",

		"DataSourceSettings": {

		    Path": "file.dat",

		    "EndOfLine": "Windows",

		    "Type": "LoadFile",

		    "FirstLineContainsColumnNames": true,

		    "StartLine": 0,

		    "Encoding": "utf-8"

		    "ColumnDelimiter": "a",

		    "QuoteDelimiter": "b",

		    "NewLineDelimiter": "c",

		    "MultiValueDelimiter": "d",

		    "NestedValueDelimiter": "e"

		},

		"JobLevelErrors": [

		{

		"LineNumber": -1,

		"ErrorDetails": [

		        {

		            "ColumnIndex": -1,

		            "ErrorCode": "S.RD.EXT.0217",

		            "ErrorMessage": "Cannot read Data Source. Could not open file for reading by RestartableStream.",

		            "ErrorMessageTemplate": "Cannot read Data Source. Could not open file for reading by RestartableStream.",

		            "ErrorProperties": {}

		        }

		    ]

		},

		"CreatedBy": 777,

		"CreatedOn": "2022-10-18T15:09:12.69",

		"LastModifiedBy": 777,

		"LastModifiedOn": "2022-10-18T15:10:00.497"

	},

	"IsSuccess": true,

	"ErrorMessage": "",

	"ErrorCode": "",

	"ImportJobID": "00000000-0000-0000-0000-000000000000"

	}

- when requested by user for all item errors that occurred during importing of particular data source e.g.:

 > curl

        curl -X GET 'https://relativity.roadie.so/Relativity.REST/api/import-service/v1/workspaces/10000/import-jobs/00000000-0000-0000-0000-000000000000/sources/00000000-0000-0000-0000-000000000000/itemerrors?start=0&length=10'

        -H 'accept: application/json'

        -H 'X-CSRF-Header: -'

  > JSON response

    {

    "Value": {

        "DataSourceID": "00000000-0000-0000-0000-000000000000",

        "Errors": [

        {

            "ErrorDetails": [

            {

                "ColumnIndex": 1,

                "ErrorCode": "S.LN.INT.0001",

                "ErrorMessage": "Error message.",

                "ErrorMessageTemplate": "Template error message.",

                "ErrorProperties": {

                "additionalProp1": "string",

                "additionalProp2": "string",

                "additionalProp3": "string"

                }

            }

            ],

            "LineNumber": 1

        }

        ],

        "TotalCount": 1,

        "NumberOfSkippedRecords": 0,

        "NumberOfRecords": 1,

        "HasMoreRecords": false

    },

    "IsSuccess": true,

    "ErrorMessage": "",

    "ErrorCode": "",

    "ImportJobID": "00000000-0000-0000-0000-000000000000"

    }

## Error code structure

Error code returned from the Import Service API endpoint has the following structure:

**[Resource].[Action].[ErrorType].[ErrorNumber]**

Examples:

|Error code      |Description                                      |

|----------------|-------------------------------------------------|

|J.CR.VLD.1501   |Cannot create job because validation has failed. |

Resources

---

|Resource code|Description  |

|-------------|-------------|

|J            |Job          |

|C            |Document Configuration|

|S            |Source       |

|E            |ItemErrors   |

|R            |RDO Configuration    |

Actions

---

|Action   code|Description         |

|-------------|--------------------|

|BEG          |Begin               |

|CR           |Create              |

|CNL          |Cancel              |

|END          |End                 |

|GET          |Get                 |

|GET_COL      |Get columns         |

|GET_CFG      |Get config          |

|GET_DAT      |Get data            |

|GET_DTLS     |Get details         |

|GET_PRG      |Get progress        |

|LN           |Line                |

|PS           |Pause               |

|RD           |Read                |

|RES          |Resume              |

|RUN          |Run                 |

Error Types

---

|Error type code|Description                         |

|---------------|------------------------------------|

|INT            |Internal service error              |

|EXT            |External dependency error           |

|VLD            |Validation error                    |

Error Numbers

---

Error number has 4 digits. Digits on the first and on the second position has the special meaning.

Meaning of the first digit is the same for all error types.

|Resource code|Description                                    |

|-------------|-----------------------------------------------|

|0XXX         |General error                                  |

|1XXX         |Job related error                              |

|2XXX         |Configuration related error                    |

|3XXX         |Source related error                           |

|4XXX         |ItemErrors related error                       |

Meaning of the second digit differs for each error type.

|Error Type |Resource code|Description                                    |

|-----------|-------------|-----------------------------------------------|

|INT        |X[0-9]XX     |Service errors                                  |

|EXT        |X[0-9]XX     |Runtime errors                                  |

|VLD        |X0XX         |Invalid input data                             |

|VLD        |X5XX         |System state does not allow to execute request |

|VLD        |X6XX         |Data in the system does not exist              |

|VLD        |X7XX         |Data in the system is incorrect                |

|VLD        |X9XX         |Data in the system is corrupted                |





---

# Samples

## Samples types and structure

There are three types of sample application that demonstrate the use of Import Service API features.

- DotNetClientConsole - .NET console application  (.NET 7, C#).

- KeplerClientConsole - .NET console application (.NET Framework 4.6.2, Kepler client, C#).

- REST client -  Powershell scripts.

Examples structure:

- Each code example for particular import flow is contained in separate file (e.g. [Sample04_AddDataSourceToRunningJob.cs](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample04_AddDataSourceToRunningJob.cs)).

- Each sample is numbered. Number of sample is included in file and class name.

- The individual samples for each application are consistent. For instance Sample_08 in Net7ConsoleClient presents the same import flow as in KeplerClientConsole and in PS scripts.

- Sample code contains accurate comments describing the flow. 

- Expected console result of demonstrated sample flow is included at the end of the file. 

- Repository with samples contains also structured data set used in all examples - load files, opticon files, folders structure with native files, images, text files.

List of samples:

| Sample name | .NET | .Net Framework & Kepler | PowerShell |

| :---- | :----: | :----: | :----: |

 | Sample01_ImportNativeFiles | [Sample01](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample01_ImportNativeFiles.cs) | [Sample01](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample01_ImportNativeFiles.cs) | [Sample01](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample01-import-native-files.ps1) | 

 | Sample02_ImportDocumentsInOverlayMode | [Sample02](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample02_ImportDocumentsInOverlayMode.cs) | [Sample02](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample02_ImportDocumentsInOverlayMode.cs) | [Sample02](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample02-import-documents-in-overlay-mode.ps1) | 

| Sample03_ImportFromTwoDataSources | [Sample03](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample03_ImportFromTwoDataSources.cs) | [Sample03](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample03_ImportFromTwoDataSources.cs) | [Sample03](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample03-import-from-two-data-sources.ps1) | 

| Sample04_AddDataSourceToRunningJob|  [Sample04](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample04_AddDataSourceToRunningJob.cs) | [Sample04](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample04_AddDataSourceToRunningJob.cs) | [Sample04](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample04-add-data-source-to-running-job.ps1) | 

| Sample05_ImportDocumentsWithExtractedText | [Sample05](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample05_ImportDocumentsWithExtractedText.cs) | [Sample05](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample05_ImportDocumentsWithExtractedText.cs) | [Sample05](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample05-import-documents-with-extracted-text.ps1) | 

| Sample06_ImportDocumentsToSelectedFolder | [Sample06](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample06_ImportDocumentsToSelectedFolder.cs) | [Sample06](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample06_ImportDocumentsToSelectedFolder.cs) | [Sample06](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample06-import-documents-to-selected-folder.ps1) | 

| Sample07_DirectImportSettingsForDocuments.cs | [Sample07](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample07_DirectImportSettingsForDocuments.cs) | [Sample07](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample07_DirectImportSettingsForDocuments.cs) | [Sample07](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample07-direct-import-settings-for-documents.ps1) | 

| Sample08_ImportImages | [Sample08](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample08_ImportImages.cs) | [Sample08](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample08_ImportImages.cs) | [Sample08](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample08-import-images.ps1) | 

 | Sample09_ImportProductionFiles | [Sample09](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample09_ImportProductionFiles.cs) | [Sample09](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample09_ImportProductionFiles.cs) | [Sample09](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample09-import-production-files.ps1) | 

 | Sample10_ImportImagesInAppendOverlayMode |[Sample10](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample10_ImportImagesInAppendOverlayMode.cs) | [Sample10](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample10_ImportImagesInAppendOverlayMode.cs) | [Sample10](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample10-import-images-in-append-overlay-mode.ps1) | 

 | Sample11_DirectImportSettingsForImages| [Sample11](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample11_DirectImportSettingsForImages.cs) | [Sample11](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample11_DirectImportSettingsForImages.cs) | [Sample11](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample11-direct-import-settings-for-images.ps1) | 

 | Sample12_ImportRelativityDynamicObject | [Sample12](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample12_ImportRelativityDynamicObject.cs) | [Sample12](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample12_ImportRelativityDynamicObject.cs) | [Sample12](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample12-import-relativity-dynamic-object.ps1) | 

 | Sample13_ImportRdoWithParent | [Sample13](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample13_ImportRdoWithParent.cs) | [Sample13](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample13_ImportRdoWithParent.cs) | [Sample13](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample13-import-rdo-with-parent.ps1) | 

 | Sample14_DirectImportSettingsForRdo | [Sample14](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample14_DirectImportSettingsForRdo.cs) | [Sample14](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample14_DirectImportSettingsForRdo.cs) | [Sample14](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample14-direct-import-settings-for-rdo.ps1) | 

 | Sample15_ReadImportRdoSettings|  [Sample15](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample15_ReadImportRdoSettings.cs) | [Sample15](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample15_ReadImportRdoSettings.cs) | [Sample15](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample15-read-import-rdo-settings.ps1) | 

 | Sample16_ReadImportDocumentSettings | [Sample16](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample16_ReadImportDocumentSettings.cs) | [Sample16](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample16_ReadImportDocumentSettings.cs) | [Sample16](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample16-read-import-document-settings.ps1) | 

 | Sample17_GetImportJobs | [Sample17](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample17_GetImportJobs.cs) | [Sample17](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample17_GetImportJobs.cs) | [Sample17](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample17-read-import-jobs.ps1) | 

 | Sample18_GetDataSource | [Sample18](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample18_GetDataSource.cs) | [Sample18](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample18_GetDataSource.cs) | [Sample18](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample18-get-data-source.ps1) | 

 | Sample19_GetImportJobDetailsAndProgress | [Sample19](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample19_GetImportJobDetailsAndProgress.cs) | [Sample19](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample19_GetImportJobDetailsAndProgress.cs) | [Sample19](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample19-get-import-job-details-and-progress.ps1) | 

 | Sample20_GetDataSourceDetailsAndProgress | [Sample20](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample20_GetDataSourceDetailsAndProgress.cs) | [Sample20](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample20_GetDataSourceDetailsAndProgress.cs) | [Sample20](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample20-get-data-source-details-and-progress.ps1) | 

 | Sample21_CancelStartedJob | [Sample21](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample21_CancelStartedJob.cs) | [Sample21](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample21_CancelStartedJob.cs) | [Sample21_](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample21-cancel-started-job.ps1) | 

 | Sample22_ReadResponse | [Sample22](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample22_ReadResponse.cs) | [Sample22](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample22_ReadResponse.cs) | [Sample22](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample22-read-response.ps1) | 

 | Sample23_GetDataSourceErrors | [Sample23](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/SampleCollection/Sample23_GetDataSourceErrors.cs) | [Sample23](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/SamplesCollection/Sample23_GetDataSourceErrors.cs) | [Sample23](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/SamplesCollection/sample23-get-data-source-errors.ps1) | 

## .NET Code Samples - How to

 

 To run a sample code:

 

 - Copy the content of [sample dataset](https://github.com/relativitydev/relativity-import-samples/tree/main/SampleDataSources) to your Relativity fileshare.

 - Uncomment line with sample invocation you want to run in Main method.  

            // await sampleCollection.Sample08_ImportImages();

            // await sampleCollection.Sample09_ImportProductionFiles();

            await sampleCollection.Sample10_ImportImagesInAppendOverlayMode();

 - Set the proper credentials and URI of your Relativity instance in [RelativityUserSettings](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/DotNetClientConsole/Helpers/RelativityUserSettings.cs) helper class.

        public class RelativityUserSettings

        {

            public const string BaseAddress = "https://host/Relativity.REST/";

            public const string UserName = "sample.user@test.com";

            public const string Password = "password!";

        }

 - Update workspaceId const to the proper value equals Id of the workspace where you intend to import data. It is required in each sample. 

 

        // destination workspace artifact Id.

        const int workspaceId = 1000000;

 - Update other Ids related to your workspace - productionSetsArtifactId , rootFolderId,rdoArtifactTypeID. They are required only by specific samples.

 - Update const which defines the path to the load file (e.g. const string  loadFile01Path) according to the location where you copied sample data.

 

			// Path to the file in opticon format used in data source settings.

			const string opticonFilePath = "\\\\files\\T001\\StructuredData\\Import\\SampleDataSources\\opticon_01.opt";

 - Run application 

## KeplerClient Code Samples - How to

 

 To run a sample code:

 

 - Copy the content of [sample dataset](https://github.com/relativitydev/relativity-import-samples/tree/main/SampleDataSources) to your Relativity fileshare.

 - Uncomment line with sample invocation you want to run in Main method.

 - Set the proper credentials and host address of your Relativity instance in [RelativityUserSettings](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/KeplerClientConsole/ImportSampleHelpers/RelativityUserSettings.cs) helper class.

 	    public static class RelativityUserSettings

	    {

		public const string HostAddress = "https://host-address";

		public const string UserName = "sample.user@test.com";

		public const string Password = "password!";

	    }

 - Update workspaceId const to the proper value equal Id of the workspace where you intend to import data. It is required in each sample. 

 - Update other Ids related to your workspace - productionSetsArtifactId , rootFolderId,rdoArtifactTypeID. They are required only by specific samples.

 - Update const which defines the path to the load file (e.g. const string  loadFile01Path) according to the location where you copied sample data.

 - Run application 

## Powershell Script Samples - How to

 

 To run a sample code:

 

 - Install Powershell "Pester" Module  (ver. >= 5.3.3)

 

        Find-Module -Name "Pester" | Install-Module -Force;

 - Copy the content of [sample dataset](https://github.com/relativitydev/relativity-import-samples/tree/main/SampleDataSources) to your Relativity fileshare.

- Uncomment line with sample invocation you want to run in [run-sample-import.ps1](https://github.com/relativitydev/relativity-import-samples/blob/main/Samples/RestSamples/run-sample-import.ps1).

        Describe "Sample import" {

        . "$global:rootDir\SamplesCollection\sample01-import-native-files.ps1"

        # . "$global:rootDir\SamplesCollection\sample02-import-documents-in-overlay-mode.ps1"

 - Set the proper credentials and host address of your Relativity instance in "run-sample-import.ps1".

        $hostAddress = "https://sample-host/"

        $userName = "sample@username"

        $password = "password!"

 - Update workspaceId  to the proper value equal Id of the workspace where you intend to import data. It is required in each sample. 

- Update variable which defines the path to the load file/opticon file (e.g. $opticonFilePath) according to the location where you copied sample data.

        $workspaceId = 1000000

        $loadFilePath = "\\files\T001\StructuredData\Import\SampleDataSources\load_file_01.dat"

 - Update other Ids related to your workspace - productionSetsArtifactId , rootFolderId,rdoArtifactTypeID. They are required only by specific samples.

 - Invoke run-sample-import.ps1





---

# Performance Best Practices

## Import Job Settings

### Encoding

For best performance, we highly recommend using UTF-16 encoding for any single long text field (including Extracted text). Other encodings are still supported, but will be converted to UTF-16 which will add delay to document or image import process.

For the document workflow, set **FieldMapping.Encoding** to UTF-16. Similarly, for the image workflow, configure **ImageSettings.ExtractedTextEncoding** as UTF-16. With these settings in place, the conversion overhead is eliminated, and your files will be copied directly in the unicode encoding, resulting in faster processing times.

			ImportDocumentSettings importDocuments = ImportDocumentSettingsBuilder.Create()

				.WithAppendMode()

				.WithNatives(x => x

					.WithFilePathDefinedInColumn(filePathColumnIndex)

					.WithFileNameDefinedInColumn(fileNameColumnIndex))

				.WithoutImages()

				.WithFieldsMapped(x => x

					.WithField(controlNumberColumnIndex, "Control Number")

					.WithExtractedTextField(extractedTextPathColumnIndex, e => e

						.WithExtractedTextInSeparateFiles(f => f

							.WithEncoding("UTF-16")

							.WithFileSizeDefinedInColumn(fileSizeColumnIndex))))

				.WithoutFolders();

    

			ImportDocumentSettings importImages = ImportDocumentSettingsBuilder.Create()

				.WithAppendMode()

				.WithoutNatives()

				.WithImages(i => i

					.WithAutoNumberImages()

					.WithoutProduction()

					.WithExtractedText(e => e.WithEncoding("UTF-16"))

					.WithFileTypeAutoDetection())

				.WithoutFieldsMapped()

				.WithoutFolders();

 

If your mapping contains more than one long text field, you should use UTF-16. No other encodings are supported in this case.

### FileSizeColumnIndex

Another valuable setting that can enhance performance is the **FieldMapping.FileSizeColumnIndex**. By configuring this setting, the need for additional file size calculations can be eliminated. The file sizes will be automatically extracted from the load file, streamlining the process and saving valuable processing time.

**Note:** The FileSizeColumnIndex setting will only take effect if FieldMapping.ContainsFilePath is set to true, and the FieldMapping.Encoding is set to UTF-16. This property applies only to long text fields stored in Data Grid, including Extracted Text.