https://github.com/letalys/convertfrom-robocoplog
A Powershell Robocopy Log Parser.
https://github.com/letalys/convertfrom-robocoplog
powershell powershell-script powershell-scripts robocopy windows
Last synced: 6 months ago
JSON representation
A Powershell Robocopy Log Parser.
- Host: GitHub
- URL: https://github.com/letalys/convertfrom-robocoplog
- Owner: Letalys
- License: mit
- Created: 2024-10-22T21:13:13.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-11-04T22:03:39.000Z (7 months ago)
- Last Synced: 2024-12-14T21:11:43.476Z (6 months ago)
- Topics: powershell, powershell-script, powershell-scripts, robocopy, windows
- Language: PowerShell
- Homepage:
- Size: 134 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ConvertFrom-RobocopLog : Universal Robocopy Log Parser
`ConvertFrom-RobocopLog.ps1` is a powerful PowerShell tool designed to analyze log files generated by Robocopy, providing a clear and structured overview of file copy operations.
It converts logs into information-rich PowerShell objects, facilitating automation, reporting, and analysis of data transfers.
This script supports various Robocopy log formats with different log options, including those with full or partial file paths.
- Detailed extraction of header information (source, destination, options).
- Complete analysis of copied, modified, error, or ignored files.
- Processing of overall statistics (number of files, bytes, errors) with the possibility of calculating the total processing time.
- Flexible filtering options through parameters to include or exclude file and directory classes for parsing.
- **Raw data extraction**: With the `-RawParsing` option, users can retrieve unprocessed log lines for further customization or external analysis.
- **Verbose output**: Enhanced progress tracking and diagnostic output with `Write-Verbose`, available by specifying the `-Verbose` flag.
This script is ideal for system administrators looking to better understand or automate the management of Robocopy operations in complex environments, while maintaining an accurate view of performance and results.
## Prerequisites
To use ConvertFrom-RobocopLog.ps1, you must ensure that your environment meets the following prerequisites:
- **Operating system:** Windows with PowerShell (version 5.1 or later).
- **Robocopy:** The log file must be generated by a Robocopy command with the log generation option enabled (e.g., `/LOG`, `/UNILOG`).
- **Permissions:** You must have the necessary permissions to read the specified log file.
- **Full file paths:** For analysis, Robocopy should be run with the `/fp` (Full Path) option to log full file paths.
- **Show Class:** For analysis, Robocopy should be run **without** the `/nc` (Class not logged) option.
For French logs, use the `/unilog` option instead of `/log`.
## How to use ConvertFrom-RobocopLog
### First Way: Use `ConvertFrom-RobocopLog.ps1` file directly
To use ConvertFrom-RobocopLog.ps1, specify the Robocopy log file to analyze using the required `-RoboLog` parameter. The script will parse the content and return a structured PowerShell object containing the header information, processed files, and summary statistics.
- `-RoboLog` : Path to the Robocopy log file to analyze (required).
- `-LogLanguage` : Log language (optional, possible values: "en-US" or "fr-FR", default "en-US").
- `-IncludeFileClass` : File classes to include in the parsing (optional, default "All").
- `-ExcludeFileClass` : File classes to exclude from the parsing (optional).
- `-IncludeDirClass` : Directory classes to include in the parsing (optional, default "All").
- `-ExcludeDirClass` : Directory classes to exclude from the parsing (optional).
- `-RawParsing` : Enables output of unprocessed log lines, bypassing structured parsing (optional).
### Second Way: Use the internal function and integrate it into your own scripts
You can copy/paste the function `ConvertFrom-RobocopLog { ...}` and either declare it in your script or create your own `.psm1` module to integrate it without polluting your script.
The settings are completely identical.
## Examples of Use
Complete analysis of a log file in English (default):
```
$LOG = .\ConvertFrom-RobocopLog.ps1 -RoboLog "C:\Logs\RobocopyLog.txt"
```
Analysis of a log file in French with specific directory classes excluded:
```
$LOG = .\ConvertFrom-RobocopLog.ps1 -RoboLog "C:\Logs\RobocopyLog.txt" -LogLanguage fr-FR -ExcludeDirClass "NewDirs"
```
Analysis excluding all file and directory classes (only header and summary will be parsed):
```
$LOG = .\ConvertFrom-RobocopLog.ps1 -RoboLog "C:\Logs\RobocopyLog.txt" -ExcludeFileClass "All" -ExcludeDirClass "All"
```
Raw data extraction mode for unprocessed log lines:
```
$LOG = .\ConvertFrom-RobocopLog.ps1 -RoboLog "C:\Logs\RobocopyLog.txt" -RawParsing
```
### Object Properties
1. `HeaderInfo`: Contains log header information.
- `Start` : Start date and time.
- `Source` : Transfer source.
- `Destination` : Destination of the transfer.
- `Files` : File criteria.
- `Options` : Options used with Robocopy
**Exemple**
```
$LOG.HeaderInfo.Start # 2024/09/25 08:14:10
$LOG.HeaderInfo.Source # C:\SourceFolder\
$LOG.HeaderInfo | Format-List
######
Start : 2024/09/25 08:14:10
Source : C:\SourceFolder\
Destination : C:\DestFolder\
FilesOptions : *.*
Options : *.* /V /X /TS /FP /S /E /COPYALL /PURGE /MIR /NP /R:1000000 /W:30
```
2. `Files`: Contains information about processed files according to their class
- `New`,`Same`,`Newer`,`Older`,`Modified`,`Lonely`,`Tweaked`
- `FilePath` : Full path of file
- `FileSize` : With `/byte` or with default Robocopy unit (m, g, t, ..)
- `TimeStamp`
- `Failed` : Contains the list of files with error.
- `FilePath` : Full path of file
- `ErrorInfo` : Code error
- `ErroAction` : Indicates what action was attempted (Copy, Delete, etc.)
- `Destination` : Destination of the transfer.
- `Files` : File criteria.
- `Options` : Options used with Robocopy
**Exemple**
```
$LOG.Files.Extra
######
FilePath FileSize Timestamp
-------- -------- ---------
C:\SourceFolder\LargeFile_1.dat 1.0 g 25/09/2024 14:11:56
C:\SourceFolder\LargeFile_2.dat 1.0 g 25/09/2024 14:12:00
$LOG.Files.New | Out-GridView
$LOG.Files.Failed
######
FilePath ErrorInfo ErrorAction
-------- --------- -----------
C:\SourceFolder\LargeFile_10.dat ERROR 32 0x00000020 Copying File
C:\SourceFolder\LargeFile_11.dat ERROR 32 0x00000020 Copying File
C:\SourceFolder\LargeFile_12.dat ERROR 32 0x00000020 Copying File
```
3. `Dirs`: Contains information about processed Directory according to their class
- `New`, `Extra`, `Lonely`
- `Dirpath`
- `ItemCount` : Count Inner Directory Object (-1 if Deleting)
**Exemple**
```
$log.Dirs.New[0].DirPath # "C:\SourceFolder\NewDirectory\"
```
4. `SummaryInfo`: Overall summary of operations.
- `Dirs` :
- `Total`, `Copied`, `Skipped`, `Mismatch`, `Failed`, `Extras`.
- `Files` :
- `Total`, `Copied`, `Skipped`, `Mismatch`, `Failed, `Extras`.
- `Ended` : End date and time.
- `TotalTimeFromInnerLog` : Total duration calculated from Header et Summary Data Date.
- `TotalTimeFromFile` : Total duration calculated from log file metadata
**Exemple**
```
$LOG.SummaryInfo
######
Dirs : @{Total=2; Copied=1; Skipped=1; Mismatch=0; Failed=0; Extras=1}
Files : @{Total=18; Copied=3; Skipped=15; Mismatch=0; Failed=0; Extras=2}
Ended : 2024/09/25 08:25:10
TotalTimeFromInnerLog : 00d:00h:00m:00s
TotalTimeFromFile : 00d:00h:53m:45s
$LOG.SummaryInfo.Files.Skipped # 15
```
## Supported Robocopy Log language, Options and FileClass
### Language
|Language | Comments |
|---------|:--------------------------------------------------------------|
|English | By Default, can use with `/log` option from robocopy command. |
|French | By Default, can use with `/unilog` option from robocopy command. Needed to handle accents |
### Add Your Language KeyWord
Just add Your Language CultureMap From the variable `$CultureMap = @{...}`.
Then remember to add your language option to the function and/or script parameter.
```
[Parameter(Mandatory = $false)]
[ValidateSet("en-US", "fr-FR")]
[String]$LogLanguage = "en-US",
```
### Options (Non-exhaustive list)
**This section covers the options that Robocopy must use to generate its log in order to be analyzed by RobocopLog**
*For more information about Robocopy options click on the link: [Microsoft Learn : Robocopy](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy)*
**JOB Options** : Not tested.
| Options | Supported | Microsoft Description |
|:------------:|:-----------:|:--------------------------------------------------------------|
|/xf | no | Excludes files that match the specified names or paths. Wildcard characters (* and ?) are supporte |
|/nc | no | Excludes directories that match the specified names and paths. |
|Log Options | Supported | Comments | Microsoft Description |
|:------------:|:-----------:|:----------------------------------------------------------------------------------------|:--------------------------------------------------------------|
|/fp | yes | **Mandatory** : FullPath need to be generated for bein captured by parsing |Includes the full path names of the files in the output. |
|/nc | no | **Forbidden** : Class need to be generated during log because they are used for parsing |Specifies that file classes aren't to be logged. |
|Log Options | Supported | Recommended* | Microsoft Description |
|:------------:|:-----------:|:------------:|:----------------------|
|/log+ | **no** | - |Writes the status output to the log file (appends the output to the existing log file).|
|/unilog+ | **no** | - |Writes the status output to the log file (appends the output to the existing log file).|
|/unicode | **no** | - |Displays the status output as unicode text. |
|/v | yes | yes |Produces verbose output, and shows all skipped files.|
|/ts | yes | yes |Includes source file time stamps in the output.|
|/byte | yes | yes |Prints sizes as bytes.|
|/ns | yes | no |Specifies that file sizes aren't to be logged.|
|/np | yes | yes |Specifies that the progress of the copying operation (the number of files or directories copied so far) won't be displayed. |
|/ndl | yes | no |Specifies that directory names aren't to be logged.|
|/nfl | yes | no |Specifies that file names aren't to be logged.|
|/njh | yes | no |Specifies that there's no job header.|
|/njs | yes | no |Specifies that there's no job summary.|
|/x | yes | yes |Reports all extra files, not just the ones that are selected.|
\* ***Note** : Recommended for Optimal Analysis*
### Robocopy Files Class Information
#### English
```
File Exists In Exists In Source/Dest Source/Dest Source/Dest
Class Source Destination File Times File Sizes Attributes
=========== =========== ================ =============== ============= ============
Lonely Yes No n/a n/a n/a
Tweaked Yes Yes Equal Equal Different
Same Yes Yes Equal Equal Equal
Changed Yes Yes Equal Different n/a
Newer Yes Yes Source > Dest n/a n/a
Older Yes Yes Source < Dest n/a n/a
Extra No Yes n/a n/a n/a
Mismatched Yes (file) Yes (directory) n/a n/a n/a
```
#### French
```
Classe de Existe dans Existe dans Source/Dest Source/Dest Source/Dest
Fichier la Source la Destination Date fichier Taille fichier Attributs
=========== =========== ================ =============== ============= ============
Solitaire Oui Non n/a n/a n/a
Tweaked * Oui Oui Identique Identique Différent
Identique Oui Oui Equal Identique Identique
Modifié Oui Oui Equal Different n/a
Plus récent Oui Oui Source > Dest n/a n/a
Plus ancien Oui Oui Source < Dest n/a n/a
Supplément. Non Oui n/a n/a n/a
Discordance Yes (Fic) Yes (Rép) n/a n/a n/a
```
\* ***Note** : Tweaked is KeyWord used for french too.*
## Releases
- Version `1.0` | 2024-09-26
- Version `1.01` | 2024-09-27
- Bug fix : When the $ParseType option was full, the end summary was not correctly parsed due to a bad condition
- Version `2.0` | 2024-10-27
- *Issue #2* : New Method to browse log file. This script now integrate the log file into an array and browse it.
- Version `2.1` | 2024-11-03
- *Issue #4* : Integration of parameters for selecting file and directory classes to return in the object.
- Removal of the ParseType option in favor of new options allowing you to exclude the recovery of file or directory classes to only manage the header and summary.
- Version `2.2` | 2024-11-03
- *Issue #7* : Add RawParsing Option to return the raw data of the file classes without structuring them.
- Add Write-Verbose to follow the progress of the script When the -Verbose parameter is used.
- Update the help section to include the new RawParsing parameter and the new behavior of the script.
## Links
- [https://github.com/Letalys/ConvertFrom-RobocopLog](https://github.com/Letalys/ConvertFrom-RobocopLog)
## Autor
- [@Letalys - Christophe GOEMAERE (GitHUb)](https://www.github.com/Letalys)