Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/mcxxmc/simple-webpage-document-storage-sys

A simple implementation to simulate a document-based database system (instead of tables).
https://github.com/mcxxmc/simple-webpage-document-storage-sys

database golang react

Last synced: 3 months ago
JSON representation

A simple implementation to simulate a document-based database system (instead of tables).

Awesome Lists containing this project

README 

          
# simple-webpage-document-storage-sys



## General



Keyword: document-based database system, Golang, Gin framework, React.



A simple implementation to simulate a document-based database system (instead of tables).



Backend supported by go and includes a gin server. Frontend is currently developed in React.



Unlike the file explorer in most systems, the hierarchy of the files & directories are not reflected in the physical level.



All the files owned by the same user are put under the same "physical" folder and their relations are managed by the profile of that user. All the directories are logical and virtual as they do not physically exist. Each user profile is a JSON file containing the relative relationships of all the real files and virtual directories.



User profiles are maintained by a bigger index (also a JSON file). Adding or deleting users will modify that big index, but register and unregister will not.



The two important parts of the backend is manager and cache. "manager" maintains the user collections (the profiles) for login user only. "cache" keeps all the basic user info (name, id, password, etc.) regardless of logging in or not. I am considering that replacing "cache" with an outer SQL server or Redis may simplify the system.



## Frontend



Implemented in React and javascript. This is my first time using React and I am still learning. Currently, I have been familiar with the concepts of state, props and callback functions.



Basically, it is a single page application and some certain flags to control which information to show the users. 



It has login and logout (using tokens). It visualizes the hierarchy structure of users' data and allow creating, reading, renaming, moving and deleting. The interactions with the backend are achieved using `fetch`.



## Backend



This is the most important part of this project and I am mostly proud of and interested in it. 



As is mentioned in the "General" section, it simulates a document-based database system. This system consists of 3 parts: the index of all users (a single JSON file), the profile of each user (lots of JSON files, 1 for each user) and the collections of user documents (each user own his/her own collection, which is an independent folder containing all the documents). 



Though a user may sort and store his/her documents into different categories and put them under different folders, those "folders" are not real and do not exist on the disk physically. Those relationships are stored as JSON objects in the user profile and all the documents are stored in the same collection folder. So, if you directly open the collection folder, you will see all the documents in parallel and you won't see any hierarchy structure here.



Note that each user and each document he/she owns has a unique id (64-character long by default) to avoid collision. Ids are randomly generated by the backend server.



Users use ids as the names of their collection folders instead of their customized usernames; documents use ids as their filename on disk instead of their filenames given by user. The document owned by different users may have the same id as they are distinguished by different owners and are put under different collection folders.



To be honest, I am not sure whether such design will bring significant benefits compared to the traditional way of storing documents under nested folders. I think the main advantage will be the convenience provided by the abstraction between the hierarchy structure and real physical storages. 



It will be less costly to create, delete, rename directories and files and update the relationships between them, as they (called `Image` in my codes) are simply some loosely organized JSON objects. 



You do not need to move the file physically on your disk; the only thing you need to do is to modify the attributes of the corresponding JSON objects (e.g., `Parent` and `Children`). It is even more convenient for directories, as there are no real folders for them. Most modification is made and only made to the JSON profile. Exceptions are creating, deleting and modifying a file (not including renaming).



Mainly, the whole implementation consists of 4 packages. The controller, the file system (`filesys`), the logging, and the manager.



### controller



It has achieved Restful API using Golang Gin framework. It uses middleware to allow CORS, deal with Options and handle tokens.



### filesys



Note that this `filesys` is different from the `file-sys` folder. The later is the physical storage while the former is my golang used for interacting with the os.



It defined many fundamental structs for the whole project, and is a brigde between the os and the "manager". 



Note that the "controller" should never calls this package directly; it should call "manager" instead.



### logging



Basically a wrap-up of the Uber zap log. 



### manager



The most important part that achieves most functions of the database system. It defines a "manager" class to manager all the collections for login users. It also has a "cached" object to manager all the user information (e.g., username, password). 



The "manager" must be initialized at the beginning of each execution. It calls `filesys` to load user collections for users that have logged in. When the server is shutting down, the "manager" also needs to be called to save the modified user collections into the disk.



Before commiting any operation (e.g., create a new file), the "manager" will do necessary checkings to ensure that the parameters are valid (e.g., the user id exists, the parent id is valid, the file id is unique, ...). Then, it will update the `Image` and the collection before any real change is made to the disk. 



After several updates and fixes, the logic behind those operations should be error-free now.



In very rare cases (probably an accidential shutdown), the collection may contain a false `Image` that leads to a file that does not exist. Currently there is no mechanism to compensate for that.



### token



There is also a token package for JWT tokens.



## TODO



### 1. Most Important: React App Data Structure Reconstruction (frontend)



Reconstruct the way React App stores the data (`state.organized`) from array to a tree, which will make the computation much easier for collapsing (using `state.show` and `state.collapse`). A possible way is to rewrite the `recursivelyAppend()` method. Also, it will make it easier to update the view after creating, renaming or deleting a file or a directory, as there is no need to call `fetchAndSort()` each time, which is very expensive.



### 2. Collapse (frontend)



Need to achieve the visualization effects: click on an icon to collapse the children nodes of a parent node.



### 3. Hardcoding (frontend & backend)



Hardcoding some static strings as variables and use these variables instead of directly using those strings.



### 4. Better CSS (frontend)



Unluckily, the webpage is ugly now.



### 5. Middleware (backend)



Improve the middleware by applying router groups. Should not check token for the login request.



### 6. (Optional, backend) Changing `cached` to SQL  



### 7. Rare Cases for "manager" (backend)



Develop a mechanism to handle the real cases mentioned in the end of the "manager" section above. The simplest way is to enable the user to just delete that phantom file from the collection. 



I have included the corresponding codes in the comments in `manager/strcut.go deleteTxt`. However, it may lead to uncontrolled behaviours that are not known yet. Considering the rare cases should be really rare, please use it at you own risk.