Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/visokio/omniscope-custom-blocks

Public repository for custom blocks for Omniscope
https://github.com/visokio/omniscope-custom-blocks

business-intelligence data-science dataanalytics datapreparation python rstats

Last synced: 26 days ago
JSON representation

Public repository for custom blocks for Omniscope

Awesome Lists containing this project

README 

          
# Omniscope Custom Blocks · [![Refresh index](https://github.com/visokio/omniscope-custom-blocks/actions/workflows/refresh_index.yml/badge.svg)](https://github.com/visokio/omniscope-custom-blocks/actions/workflows/refresh_index.yml)[![Refresh readme](https://github.com/visokio/omniscope-custom-blocks/actions/workflows/refresh_readme.yml/badge.svg)](https://github.com/visokio/omniscope-custom-blocks/actions/workflows/refresh_readme.yml)



Public repository for custom blocks for Omniscope Evo.



#### [Python / R API docs](https://help.visokio.com/support/solutions/articles/42000071109-custom-block-r-python-api-reference)



## How to add a block to this repository

### The simple way

1. Design your custom block in Omniscope Evo 2020.1 or later.

   The source code should be reasonably documented and potentially contain sections to describe input fields and parameters.

2. Export as a ZIP file from the block dialog.

3. Send the file to support@visokio.com and we will include it for you.



### The hard way

1. Follow points 1-2 from the simple way.

2. Fork the repository.

3. Create or use a directory in the forked repository under one of the main sections that specifies the general area of what the block does.

4. Extract the ZIP file into this directory.

5. Consider adding a README.md for convenience, and a thumbnail.png.

6. Run the python scripts create_index.py and create_readme.py located in the root of the repository.

7. Create a pull request.



## Table of blocks (Omniscope 2020+)

1. Analytics

   1. Clustering

      1. [DBScan](#AnalyticsClusteringDBScan)

      2. [KMeans](#AnalyticsClusteringKMeans)

      3. [Gaussian Mixture Model](#AnalyticsClusteringGMM)

   2. Network Analysis

      1. [Attribute Analysis](#AnalyticsNetwork-AnalysisAttribute-Analysis)

      2. [TSNE](#AnalyticsNetwork-AnalysisTSNE)

   3. Prediction

      1. [K-Nearest-Neighbours](#AnalyticsPredictionKNN)

      2. [Support Vector Machine](#AnalyticsPredictionSVM)

   4. Validation

      1. [Model Validation](#AnalyticsValidationModel-Validation)

   5. Website

      1. [Website Analysis](#AnalyticsWebsitesWebsite-Analysis)

   6. [Data Profiler](#AnalyticsData-Profiler)

   7. [Survival Analysis](#AnalyticsSurvival)

2. Preparation

   1. ForEach

      1. [ForEach multi stage](#PreparationForEachForEachMultiStage)

      2. [Project Parameters Batch Setting](#PreparationForEachProjectParameters)

   2. Geo

      1. [Gridsquare](#PreparationGeoGridsquare)

      2. [Shapefile](#PreparationGeoShapefile)

   3. Interfaces

      1. [Kedro](#PreparationInterfacesKedro)

   4. JSON

      1. [JSON Expand fields](#PreparationJSONExpand_Fields)

      2. [JSON Normalise](#PreparationJSONNormalise)

   5. Join

      1. [Interval Join](#PreparationJoinInterval-Join)

      2. [Fuzzy Terms Join](#PreparationJoinFuzzy-Join)

      3. [Inequality Join](#PreparationJoinInequality-Join)

   6. Partition

      1. [Partition](#PreparationPartition)

   7. Pivot

      1. [Melt De-pivot](#PreparationPivotMelt-De-pivot)

   8. Standardisation

      1. [Standardise](#PreparationStandardisationStandardise)

   9. Workflow

      1. [For Each (Separate Workflows)](#PreparationForEachForEach)

   10. [Time Duration Unit Converter](#PreparationTime-Duration-Unit-Converter)

   11. [Markdown to HTML](#PreparationMarkdown-to-HTML)

   12. [Smart Schema Normaliser](#PreparationSmart-Schema-Normaliser)

   13. [Split Address](#PreparationSplit-Address)

   14. [Expand Date Fields](#PreparationExpand-Date-Fields)

   15. [Data Quality Analyser](#PreparationData-Quality-Analyser)

   16. [Set Project Parameters](#PreparationSetProjectParameters)

   17. [Add row ID field](#PreparationAdd-row-ID-field)

   18. [Anonymise](#PreparationAnonymise)

   19. [Unstack Records](#PreparationUnstack-rows)

   20. [Conditional Execution](#PreparationConditionalExecution)

   21. [URL Encode](#PreparationURL-Encode)

   22. [Streaming Field Renamer](#PreparationStreaming-Field-Renamer)

   23. [Canonical Schema Mapper](#PreparationCanonical-Schema-Mapper)

   24. [Smart Date Parser](#PreparationSmart-Date-Parser)

   25. [Unescape HTML](#PreparationUnescape-HTML)

   26. [Field Renamer](#PreparationField-Renamer)

   27. [Sort fields](#PreparationSort-fields)

   28. [Centroids from GeoJSON](#PreparationGeoCentroids)

3. Connectors

   1. Azure

      1. [Azure Data Lake Storage Gen2 Blob](#ConnectorsAzure-Data-Lake-Blob)

   2. Flightstats

      1. [Flightstats Airports](#ConnectorsFlightstatsAirports)

      2. [Flightstats Flights](#ConnectorsFlightstatsFlights)

      3. [Flightstats Airlines](#ConnectorsFlightstatsAirlines)

   3. Overpass

      1. [Overpass Street Coordinates](#ConnectorsOverpassStreet-Coordinates)

   4. Slack

      1. [Slack API WebClient](#ConnectorsSlack-API-WebClient)

   5. Weather

      1. [OpenWeatherMap](#ConnectorsWeatherOpenWeatherMap)

   6. [Trello](#ConnectorsTrello)

   7. [Yahoo Finance](#ConnectorsYahooFinance)

   8. [Etherscan](#ConnectorsEtherscan)

   9. [Dune](#ConnectorsDune)

   10. [HubSpot](#ConnectorsHubSpot)

   11. [Jira](#ConnectorsJira)

   12. [Slim CD Transaction](#ConnectorsSlim-CD)

   13. [Google BigQuery Custom SQL](#ConnectorsBigQueryGoogle-BigQuery-Custom-SQL)

   14. [Google BigQuery Import Table](#ConnectorsBigQueryGoogle-BigQuery-Import-Table)

   15. [XPT Reader](#ConnectorsXPT-Reader)

   16. [AirTable](#ConnectorsAirtable)

   17. [Flipside](#ConnectorsFlipside)

4. Code & AI

   1. [AI Chat - Google Gemini](#ConnectorsAI-Chat-Google-Gemini)

   2. [AI Chat - OpenAI GPT](#ConnectorsOpenAI-GPT)

   3. [AI Chat - Local LLM](#ConnectorsAI-Chat-Local-LLM)

   4. [AI Chat - Claude](#ConnectorsAI-Chat-Anthropic-Claude)

   5. [AI Chat - DeepSeek](#ConnectorsAI-Chat-DeepSeek)

   6. [Execute Command](#Custom-scriptsExecuteCommand)

5. Outputs

   1. API

      1. [Multipart Form Upload](#OutputsMulti-Form-Upload)

   2. BigQuery

      1. [Google BigQuery Export](#OutputsGoogle-BigQuery-Writer)

   3. Github

      1. [GitHub](#OutputsGitHub)

   4. Messenger

      1. [Send to Telegram](#OutputsTelegram)

   5. PDF

      1. [Web Image-PDF output](#OutputsWeb-Image-PDF-output)

      2. [Append PDF files](#OutputsAppend-PDF-files)

      3. [Multi-tenant Report to PDF](#OutputsReport-to-PDF-batch-output)

      4. [Report tab to PDF](#OutputsReport-tab-to-PDF)

   6. PowerPoint

      1. [Report to PowerPoint](#OutputsReport-to-PowerPoint)

   7. Slack

      1. [Slack Bot](#OutputsSlack-Bot)

   8. [Send Report to Slack](#OutputsReport-to-Slack)

6. Inputs

   1. Databases

      1. [MongoDB](#InputsDatabasesMongoDB)

      2. [Snowflake Custom Query](#InputsDatabasesSnowflake-custom-query)

   2. R

      1. [Rds Batch Append](#InputsRds-Batch-Append)

      2. [R Data Reader](#InputsRdata)

   3. [SFTP Downloader](#InputsSFTP-Downloader)

   4. [Sharepoint Online Downloader](#InputsSharepoint-Online)

   5. [PDF Reader](#InputsPDF-Reader)

## Block Overview




### DBScan






Performs DBScan clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.



[Link to Github page](Analytics/Clustering/DBScan)






### KMeans






Performs KMeans clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.



[Link to Github page](Analytics/Clustering/KMeans)






### Gaussian Mixture Model






Performs GMM clustering on the first input data provided. The output consists of the original input with a Cluster field appended. If a second input is available, it will be used as output instead.



[Link to Github page](Analytics/Clustering/GMM)






### Attribute Analysis






Given a dataset in which each record represents an edge between two nodes of a network, and each node has an associated categorical attribute, the block analyses connections between attributes, based on connections between associated nodes. The result of the analysis is a list of records in which each record specifies a connection from one attribute to another. The connection contains a probability field, which gives an answer to the question that if a node has the specified categorical attribute, how probable it is that it has a connection to another node with the linked categorical attribute.



[Link to Github page](Analytics/Network-Analysis/Attribute-Analysis)






### TSNE






Given a dataset in which each record represents an edge between two nodes of a network, the block will project all the nodes onto a (e.g. 2)- dimensional plane in such a way that nodes which share many connections are close together, and nodes that do not share many connections are far apart.



[Link to Github page](Analytics/Network-Analysis/TSNE)






### K-Nearest-Neighbours






Performs k-nearest-neighbour prediction on the data. The prediction for a new point depends on the k-nearest-neighbours around the point. The majority class is used as the prediction.



[Link to Github page](Analytics/Prediction/KNN)






### Support Vector Machine






Predicts classes of new data from old data by drawing a boundary between two classes whereas the margin around the bondary is made as large as possible to avoid touching the points.



[Link to Github page](Analytics/Prediction/SVM)






### Model Validation



Computes a confusion matrix as well as model validation statistics



[Link to Github page](Analytics/Validation/Model-Validation)






### Website Analysis



Extracts the structure and content of a website and its pages.



[Link to Github page](Analytics/Websites/Website-Analysis)






### Data Profiler






Provides detailed statistics about a dataset



[Link to Github page](Analytics/Data-Profiler)






### Survival Analysis






Computes an estimate of a survival curve for truncated and/or censored data using the Kaplan-Meier or Fleming-Harrington method



[Link to Github page](Analytics/Survival)






### ForEach multi stage



The ForEach multi stage block allows to orchestrate the execution of another Omniscope project and running the workflow multiple times, each time with a different set of parameter values. Unlike the ForEach block allows multiple stages of execution, executing/refreshing from source a different set of blocks in each stage.



[Link to Github page](Preparation/ForEach/ForEachMultiStage)






### Project Parameters Batch Setting



None



[Link to Github page](Preparation/ForEach/ProjectParameters)






### Gridsquare






Converts gridsquare / Maidenhead 



[Link to Github page](Preparation/Geo/Gridsquare)






### Shapefile



Match regions in shapefile with geographical points having latitude and longitude



[Link to Github page](Preparation/Geo/Shapefile)






### Kedro



Intefaces with kedro workflows



[Link to Github page](Preparation/Interfaces/Kedro)






### JSON Expand fields



Expands JSON strings in a specified field into separate columns, optionally including the original input data



[Link to Github page](Preparation/JSON/Expand_Fields)






### JSON Normalise






Normalise semi-structured JSON strings into a flat table, appending data record by record.



[Link to Github page](Preparation/JSON/Normalise)






### Interval Join



Performs a join between values in the first input and intervals in the second input. Rows are joined if the value is contained in an interval.



[Link to Github page](Preparation/Join/Interval-Join)






### Fuzzy Terms Join






Performs a join between the first (left) and second (right) input. The field on which the join is performed must be text containing multiple terms. The result will contain joined records based on how many terms they share, weighted by inverse document frequency.



[Link to Github page](Preparation/Join/Fuzzy-Join)






### Inequality Join






Performs a join between the first (left) and second (right) input. The join can be performed using equality/inequality comparators ==, <=, >=, <, > , which means the result will be a constraint cartesian join including all records that match the inequalities.



[Link to Github page](Preparation/Join/Inequality-Join)






### Partition



Partitions the data into chunks of the desired size. There will be a new field called "Partition" which contains a number unique to each partition.



[Link to Github page](Preparation/Partition)






### Melt De-pivot



Keep all selected fixed fields in the output, de-pivot all other fields



[Link to Github page](Preparation/Pivot/Melt-De-pivot)






### Standardise



Standardises the values in the selected fields so that they are in the range between 0 and 1. I.e. The new value of the highest value in each field is going to be 1, and the lowest value 0. All other values are scaled proportionally.



[Link to Github page](Preparation/Standardisation/Standardise)






### For Each (Separate Workflows)






Executes another Omniscope project multiple times, each time with a different set of parameter values.



[Link to Github page](Preparation/ForEach/ForEach)






### Time Duration Unit Converter



Converts specified datetime, time duration, or time string fields into a chosen time unit



[Link to Github page](Preparation/Time-Duration-Unit-Converter)






### Markdown to HTML






None



[Link to Github page](Preparation/Markdown-to-HTML)






### Smart Schema Normaliser



Automatically cleans and stabilises incoming datasets by normalising field names, merging duplicates, and inferring data types, with zero configuration.



[Link to Github page](Preparation/Smart-Schema-Normaliser)






### Split Address






Splits an address field into streetname, streetnumber, and suffix.



[Link to Github page](Preparation/Split-Address)






### Expand Date Fields






Expands selected date fields into separate year, month, day, and other date-part fields for easier analysis



[Link to Github page](Preparation/Expand-Date-Fields)






### Data Quality Analyser



Checks datasets for common quality issues and outputs detailed issue logs and cleaned, annotated data.



[Link to Github page](Preparation/Data-Quality-Analyser)






### Set Project Parameters



The block updates project parameters using the input data



[Link to Github page](Preparation/Set%20Project%20Parameters)






### Add row ID field



Adds a Row ID field with a sequential number.



[Link to Github page](Preparation/Add-row-ID-field)






### Anonymise



Anonymise sensitive text data within the input data.



[Link to Github page](Preparation/Anonymise)






### Unstack Records



Unstack all records by splitting on text fields with stacked values, filling records with empty strings where needed.



[Link to Github page](Preparation/Unstack-rows)






### Conditional Execution



This block conditionally triggers the execution of specified workflow blocks via the Workflow API, running them only when the Conditional Parameter is set to true.



[Link to Github page](Preparation/Conditional%20Execution)






### URL Encode



URL encode strings in a field using the UTF-8 encoding scheme



[Link to Github page](Preparation/URL-Encode)






### Streaming Field Renamer



Renames the fields of a input data, optimised for streaming and big data, given a set of rules  defined in a CSV file



[Link to Github page](Preparation/Streaming-Field-Renamer)






### Canonical Schema Mapper



Enforces a stable, business-defined schema by mapping aliases to canonical fields, coalescing values, and applying consistent types and defaults.



[Link to Github page](Preparation/Canonical-Schema-Mapper)






### Smart Date Parser



Automatically parse date fields with mixed or unknown formats into ISO datetime



[Link to Github page](Preparation/Smart-Date-Parser)






### Unescape HTML



Convert all named and numeric character references to the corresponding Unicode characters



[Link to Github page](Preparation/Unescape-HTML)






### Field Renamer



Renames the fields of a data set given a list of current names and new names.



[Link to Github page](Preparation/Field-Renamer)






### Sort fields



Sort fields in the input data by name or type



[Link to Github page](Preparation/Sort-fields)






### Centroids from GeoJSON






Calculates the centroid points (lat,long) and output them together with the shape ID from a specified GeoJSON file.



[Link to Github page](Preparation/Geo/Centroids)






### Azure Data Lake Storage Gen2 Blob






Storage Gen2 Blob connector to load a CSV or Parquet blob/file in Omniscope.



[Link to Github page](Connectors/Azure-Data-Lake-Blob)






### Flightstats Airports






Downloads a list of airports as provided by flightstats (https://www.flightstats.com). The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.



[Link to Github page](Connectors/Flightstats/Airports)






### Flightstats Flights






Requests information about flights specified in the input data from flightstats (https://www.flightstats.com). If the flight exists the result will contain live information, otherwise it will not be part of it. The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.



[Link to Github page](Connectors/Flightstats/Flights)






### Flightstats Airlines






Downloads a list of airlines as provided by flightstats (https://www.flightstats.com). The script needs your flightstats app id and key which needs to be obtained either through buying their service or signing up for a test account.



[Link to Github page](Connectors/Flightstats/Airlines)






### Overpass Street Coordinates






Finds all matching streets given a street name and requests multiple coordinates along the street using data from Overpass API. It will create a row for each point found that is part of a street that matches the given street name. The resulting rows will include the street name, the street Id and the coordinates of the point. The script needs an input with a field with the street name.



[Link to Github page](Connectors/Overpass/Street-Coordinates)






### Slack API WebClient






Allows you to call public Slack endpoints.



[Link to Github page](Connectors/Slack-API-WebClient)






### OpenWeatherMap



Retrieves current weather and forecasts from OpenWeatherMap



[Link to Github page](Connectors/Weather/OpenWeatherMap)






### Trello






Retrieves boards, lists and cards, and allows you to search in Trello.



[Link to Github page](Connectors/Trello)






### Yahoo Finance






Fetches price data for tickers from Yahoo Finance



[Link to Github page](Connectors/YahooFinance)






### Etherscan






The Ethereum Blockchain Explorer.



[Link to Github page](Connectors/Etherscan)






### Dune






Execute queries and retrieve blockchain data from any public query on dune.com, as well as any personal private queries your Dune account has access to



[Link to Github page](Connectors/Dune)






### HubSpot






Retrieves contacts, companies, deals and lists



[Link to Github page](Connectors/HubSpot)






### Jira






Retrieves projects and issues from Jira



[Link to Github page](Connectors/Jira)






### Slim CD Transaction



Pull Slim CD gateway transactions



[Link to Github page](Connectors/Slim-CD)






### Google BigQuery Custom SQL



Executes a SQL query on Google BigQuery and imports the query results



[Link to Github page](Connectors/BigQuery/Google-BigQuery-Custom-SQL)






### Google BigQuery Import Table



Allows to import a table from Google BigQuery.



[Link to Github page](Connectors/BigQuery/Google-BigQuery-Import-Table)






### XPT Reader






Reads a SAS Transport *xpt* file, extracting a dataset.



[Link to Github page](Connectors/XPT-Reader)






### AirTable



Retrieve records from an AirTable table using the REST API



[Link to Github page](Connectors/Airtable)






### Flipside






Executes a SQL query on Flipside and retrieves the blockchain data



[Link to Github page](Connectors/Flipside)






### AI Chat - Google Gemini






Executes a one-off prompt to Google Gemini and returns the generated text result



[Link to Github page](Connectors/AI-Chat-Google-Gemini)






### AI Chat - OpenAI GPT






Executes a one-off prompt to OpenAI GPT and returns the generated text result



[Link to Github page](Connectors/OpenAI-GPT)






### AI Chat - Local LLM



Executes a one-off prompt to a local LLM and returns the generated text result



[Link to Github page](Connectors/AI-Chat-Local-LLM)






### AI Chat - Claude



Executes a one-off prompt to Anthropic Claude and returns the generated text result



[Link to Github page](Connectors/AI-Chat-Anthropic-Claude)






### AI Chat - DeepSeek



Executes a one-off prompt to DeepSeek and returns the generated text result



[Link to Github page](Connectors/AI-Chat-DeepSeek)






### Execute Command






Execute a system command.



[Link to Github page](Custom-scripts/ExecuteCommand)






### Multipart Form Upload






Uploads files from a path column to a remote HTTP endpoint using multipart/form-data.



[Link to Github page](Outputs/Multi-Form-Upload)






### Google BigQuery Export



Allows to write data to a Google BigQuery table. The table can be created/replaced, or records can be appended to an existing table



[Link to Github page](Outputs/Google-BigQuery-Writer)






### GitHub



Reads from and writes data to GitHub



[Link to Github page](Outputs/GitHub)






### Send to Telegram



Sends data to Telegram



[Link to Github page](Outputs/Telegram)






### Web Image-PDF output






Grabs screenshots of webpages, optionally producing a PDF document.



[Link to Github page](Outputs/Web-Image-PDF-output)






### Append PDF files






Append multiple PDF files combining them into one PDF file.



[Link to Github page](Outputs/Append-PDF-files)






### Multi-tenant Report to PDF






Prints Report tabs to PDF files for each record of the input data.



[Link to Github page](Outputs/Report-to-PDF-batch-output)






### Report tab to PDF






Prints Report tabs to PDF files for each record of the input data.



[Link to Github page](Outputs/Report-tab-to-PDF)






### Report to PowerPoint






Export a Report to a PowerPoint pptx file



[Link to Github page](Outputs/Report-to-PowerPoint)






### Slack Bot






Posts messages on a channel.



[Link to Github page](Outputs/Slack-Bot)






### Send Report to Slack






Send a Report link with screenshots and optional PDF to a Slack channel with a message



[Link to Github page](Outputs/Report-to-Slack)






### MongoDB






A connector for MongoDB



[Link to Github page](Inputs/Databases/MongoDB)






### Snowflake Custom Query



Executes a SQL query on a Snowflake database.



[Link to Github page](Inputs/Databases/Snowflake-custom-query)






### Rds Batch Append



Reads multiple rds files either from an upstream block, or a folder, and appends them



[Link to Github page](Inputs/Rds-Batch-Append)






### R Data Reader



Joins regions defined in a shapefile with points defined as latitudes and longitudes, and gives meta information about the content of the shapefile



[Link to Github page](Inputs/Rdata)






### SFTP Downloader






Download files from a SFTP server folder.



[Link to Github page](Inputs/SFTP-Downloader)






### Sharepoint Online Downloader






Download a file from a Sharepoint Online site.



[Link to Github page](Inputs/Sharepoint-Online)






### PDF Reader






Extract text from PDF files.



[Link to Github page](Inputs/PDF-Reader)