{"id":14958972,"url":"https://github.com/iondv/tensorflow-dataset","last_synced_at":"2025-10-24T16:30:38.240Z","repository":{"id":56152836,"uuid":"298442725","full_name":"iondv/tensorflow-dataset","owner":"iondv","description":"IONDV application for forming and marking up datasets and dataset based checking of the results of differently set tensorflow models training","archived":false,"fork":false,"pushed_at":"2021-04-13T01:08:47.000Z","size":6693,"stargazers_count":5,"open_issues_count":4,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-31T03:01:31.555Z","etag":null,"topics":["dataset","iondv","iondv-app","tensorflow","tensorflow-models"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iondv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-25T02:05:06.000Z","updated_at":"2024-12-22T12:35:53.000Z","dependencies_parsed_at":"2022-08-15T13:40:46.618Z","dependency_job_id":null,"html_url":"https://github.com/iondv/tensorflow-dataset","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iondv%2Ftensorflow-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iondv%2Ftensorflow-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iondv%2Ftensorflow-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iondv%2Ftensorflow-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iondv","download_url":"https://codeload.github.com/iondv/tensorflow-dataset/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237999504,"owners_count":19399892,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","iondv","iondv-app","tensorflow","tensorflow-models"],"created_at":"2024-09-24T13:18:36.787Z","updated_at":"2025-10-24T16:30:34.782Z","avatar_url":"https://github.com/iondv.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Эта страница на [Русском](./README_RU.md)\n\n# IONDV. Tensorflow dataset app \n**IONDV. Tensorflow dataset app** - is an application based on the [IONDV. Framework](https://iondv.com) ([code repository](https://github.com/iondv/framework)) to accumulate data, normalize and mark up images, create, train and compare [Tensor Flow](https://www.tensorflow.org/) models without programming for standard functionality. It’s also possible to fully customize the logic in the form of model modifications and dataset processing using the Node.js code development.\n\n\nThe application uses the [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) dataset as an example for data import. \n\n### IONDV. Framework in brief\n\n**IONDV. Framework** is a node.js open source application that implements the functionality of a digital tool platform for rapid development of web applications and micro-services based on metadata and can be extended with modules. \nThe main purpose of the complex of solutions is to speed up the development of accounting web applications (ERP) using low-code technology. \nThis platform consists of the following open-source components: the [IONDV. Framework](https://github.com/iondv/framework), \nthe [modules](https://github.com/topics/iondv-module) and the ready-made applications expanding its functionality, as well as the [Studio](https://studio.iondv.com) ([repository](https://github.com/iondv/studio)) \nopen source visual development environment to create metadata of an application. The UML-scheme modeled applications can be launched [in 80 seconds](https://youtu.be/s7q9_YXkeEo).\n\n* For more details, see [IONDV. Framework site](https://iondv.com). \n\n* Documentation is available at [Github repository](https://github.com/iondv/framework/blob/master/docs/en/index.md).\n\n## Demo\nWatch a brief video about creating and marking up a dataset, creating a neural network model, teaching a model and verifying the quality of recognition - all without a single line of code - https://www.youtube.com/watch?v=529TwrJoEKQ\n\nDemo access to the system without registration: https://tensorflow-dataset.iondv.com. User login demo, password ion-demo. Demo mode has learning capabilities restricted to 30 patterns per run and dataset imports limited to 50 objects per run. You can build the app locally at your computer and use it without restrictions. Read the instruction below.\n\n\n## Description of features\n\nDataset is formed as system objects that can be created, loaded, deleted, or edited.\n\nFor each object, it is possible to determine whether it is included in the training sample or in the sample for verification.\n\nThe system allows creating models and parameterizing them, as well as creating various trained models on the current dataset.\n\nTrained models can be downloaded in tensorflow format.\n\nFor trained models, reports can be generated to compare the recognition quality.\n\nTODO\n\n## How to use the demo\n\n### Fill the database with objects for training\n\nThe first thing to do is to fill the database with objects for training. The easiest way is to import ready-made objects. To do this, go to the datasets navigation tab and open the fashion-mnist set.\nThis set should contain all import settings and two files attached: \n- 2k_train. csv for network training (consisting of a sample of 2000 elements from fashion-mnist)\n- 300_test.csv for training verification (300 elements from fashion-mnist).\nTo import, click the Import data button in the object window. The import takes place in the background window. It takes about 30-40 seconds for all 2300 objects to be imported (you can check it at the Object navigation tab).\n\n### Train the model\nThe next step is to train the model. The Model tab should contain a ready-made model from the tensorflow review from IBM. You can make your own, but since the metadata for the model isn’t yet perfectly formed, it is better to use this one for the demo. \n\nHaving decided on the model, go to the Model snapshot tab. This tab is for snapshots of the model States and there should be one created in advance, you need to open it. The snapshot contains a link to the model and sets all 10 types of objects from fashion-mnist. If you only need to train for certain types, you can edit the list. \nNext, you need to compile the model in the snapshot editing window, click Compile. After that, the files of the compiled model should appear in the Model file and Weights file fields. \n\nAt this stage, you can start training by clicking the Teach button. The model is trained on data in parts, each part goes through 10 epochs. Part sizes and number of epochs are set in IMPORT_BATCH_SIZE, TRAINING_BATCH_SIZE and EPOCHS. The training and verification logs are soon also to be developed. \n\n### Object recognition\n\nWhen the model is finished training, it can be used for object recognition. For this, a new recognition object is created in the objectPrediction tab. You need to attach a snapshot of the model in the Model snapshot field. In the Object field, a new object is created (or an existing one is attached). Then fill out a in the Object field. It can be any name. Then you need to attach a picture in the image or normalized image fields. If the picture is not normalized before loading, then it should be in image. You can set crop settings in the crop settings field. After the name and picture have been set, the object must be saved (without closing).\n\nAfter saving, click the Verify button in the upper-right corner of the screen to normalize the image and check for its uniqueness. If the verification is passed, the object will be added to the set. After that it can be used it in training. For this you need to set the type in the Type field and specify Learn or Check in the State field depending on whether the model will be trained or verified with this image. \n\nWhen everything is ready, close the object, return to object Prediction and click save.\n\nAfter that, the \"Process with tensorflow\" action will become available. When click this button, the network will try to recognize the object based on what it has learned. In the \"Logs\" field it will be written for each type how likely it is that the object belongs to this type. \nThe \"Prediction\" field will display the probability with which the network is confident in the decision, and the \"recognized type\" field shows the type that the object was eventually assigned to.\n\n-------\n\nIf the results are unsatisfactory, the network needs more examples for training. You can download the entire fashion-mnist dataset, which contains 61,000 elements. You can find the instructions in prebuild/readme.txt, paragraph 1. The sets will be loaded to a subfolder. You need to attach them to the dataset object and import and retrain the model after that.\n\n\n## Loading and recognizing an object\n\nCreate an instance of the object class, upload the image to image field, or, if the image is already normalized, upload it to normalizedImage. After loading, follow the workflow using the Verify button. If necessary, the image will be normalized or checked for its uniqueness in the set. If everything is successful, the Verified field will be ticked. After that, the object can be recognized.\n\n## Importing data from a ready-made set\n\nIn the Dataset class specify the data source and click the object editing button at the top of the window. In demo mode (NODE_ENV = demo) 50 random patterns are loaded (or less, if the random selection fell on a pattern that is already in the set).\n- the amount is regulated in lib/util/importFromDataset.js by the DEMO_IMPORT_LIMIT constant.\n\n## Description of metadata\n\nSee metadata structure at the picture\n\n\u003cimg src=\"/data_model.png\"\u003e\n\nThe functions of the main classes:\n* object – dataset data object – contains an original image, a normalized image, the classification type specified by operator, and a link to the prediction results.\n* object prediction – a class that links the data object to the training result – contains a link with training result and the object, prediction percentage, type, and logs.\n* learning result – contains the date of creation and editing, trained model file, and logs.\n* models – contains information about the model type, compilation parameters, and a collection of related layers.\n* layers – model layer – contains information about the layer name, activation, content, and others, depending on the specified layer.\n* dataset – contains information about the name, type of source, source of training and testing, and type of marked label.\n\n## Building the application\n\nBuilding this application on linux may require the g++ to be installed (to build tfjs-node). Usually it can be found in the OS' package tree by itself or contained in a basic development bundle like build-essential on Ubuntu.\n\nUbuntu g++ installation example:\n```\napt install g++\n```\nor\n```\napt install build-essential\n```\n\n## Configuring the application\n\n**Maximum file size.**\n\nAllows to set the maximum size of uploaded files. The configuration is performed for the \"File\" type attribute on the view form.\n\n_Example:_\n```\n\"options\": {\n    \"maxSize\": 256000000\n} \n```\n\nThe size is indicated in KB.\n\n**Allowed file types.**\n\nAllows to specify valid extensions for uploaded files. The configuration is performed for the `\"allowedFileTypes\"` property of the \"File collection\" attribute on the class form.\n\n_Example:_\n```\n\"allowedFileTypes\":  [\"csv\", \"zip\"] \n```\n --------------------------------------------------------------------------  \n \n \n  #### [Licence](/LICENSE) \u0026ensp;  [Contact us](https://iondv.ru) \u0026ensp;  [Russian](./README_RU.md)   \u0026ensp;           \n \n \n --------------------------------------------------------------------------  \n \n Copyright (c) 2020 **LLC \"ION DV\"**.  \n All rights reserved. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiondv%2Ftensorflow-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiondv%2Ftensorflow-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiondv%2Ftensorflow-dataset/lists"}