Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zentrocdot/lazyimageupscaler
LazyUpscalerGui
https://github.com/zentrocdot/lazyimageupscaler
ai artificial-intelligence gradio image images opencv pil python3 skimage stable-diffusion upscale upscaler
Last synced: 13 days ago
JSON representation
LazyUpscalerGui
- Host: GitHub
- URL: https://github.com/zentrocdot/lazyimageupscaler
- Owner: zentrocdot
- License: mit
- Created: 2024-11-12T12:25:12.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2024-12-03T04:08:07.000Z (19 days ago)
- Last Synced: 2024-12-03T05:19:46.664Z (19 days ago)
- Topics: ai, artificial-intelligence, gradio, image, images, opencv, pil, python3, skimage, stable-diffusion, upscale, upscaler
- Language: Python
- Homepage:
- Size: 9.37 MB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Lazy Image Upscaler []()
[](https://www.python.org/downloads/release/python-31014/)
[](https://pypi.org/)
[](https://www.gradio.app/)
> [!IMPORTANT]
>
🚧 This documentation is still under construction. The
> documentation will developed up to the point the application works as expected.
> The last version of the application needs also some fine-tuning after publishing
> here.
If you like what I present here, or if it helps you, or if it is
useful, you are welcome to [donate](#Donation) a small contribution. It motivates me a lot
and speeds up my work a much.
## Preface
The Lazy Image Upscaler is a web user interface
for the upscaling of images written using the programming language Python.
A bunch of Internet browsers can be used to run the Lazy Image Upscaler.
At the moment the Lazy Image Upscaler offers six possibilities to upscale
an image.
One can use for the upscaling of images standard methods from
OpenCV, PIL and scikit-image whereas scikit-image is a little bit exotic. Some
references states that this is not working well. My experience is different to this
statement. From my point of view the results using these standard methods are
sufficient for most cases. The fourth and fifth method are using pretrained models
and Machine Learning approaches. To be able to work with the web user interface, at
least minimum one of such a pretrained model is required for the fourth and fifth
method. At OpenCV one can find the links to GitHub for downloading such pretrained
models for the fourth method. The fifth method is also using pretrained model which
can be found at Hugging Face. The last method is using Stable Diffusion. I implemented
the possibility to use a 2 times and 4 times latent upscaler for upscaling. And I added
a pipeline for upscaling which can use SD 1.5 together with the 2 time latent upscaler.
Both upscaler models as well as the SD 1.5 needs to be downloaded and installed in the
directory tree of the repository.
As I am still making small changes to the layout, it cannot be
said that the illustrations reflect the latest version of the application.
## Motivation
I used the web user interface Easy Diffusion and I am using
currently the web user interface AUTOMATIC1111. One integrated feature of both
web user interfaces is the possibility for upscaling of images. This is a function
that is in general required in nowadays AI image generation. Depending on the existing
hardware equipment, the generation of AI images with a higher resolution is limited.
The VRAM memory of the GPU on the graphic card in use is the limiting factor here.
Using a web user interface such as AUTOMATIC1111 to upscale
images is complex, time-consuming and energy-intensive. The time required for a
conventional computer system should not be underestimated, even if energy consumption
is not an issue. For example, parameters have to be selected appropriately to achieve
the desired result. These points raised the question of whether it is possible to scale
up images appropriately, quickly and reliably independently of user interfaces such as
AUTOMATIC1111. This application provides an answer to this question.
It is important to note that it is important to preserve the original
Exif metadata when upscaling. This is not possible by default with AUTOMATIC1111
.
> [!NOTE]
>
I assume in all descriptions that a computer system with
> Linux as operating system is used. Basic knowledge of installing software
> on such a computer system is a prerequisite, as is the use of terminal emulators.
## Introduction
The Lazy Image Upscaler offers the possibility to upscale images.
Base don the fact that the application is intendet for use with AI generated images
the upscaling factor is an integer. Floating point numbers cannot be used for upscaling.
The tabs of the web UI are arranged according to logic of the
implemented methods for upscaling (see Figure 1):
* Tab 1 to Tab 3
+ Standard methods (numerical interpolation methods) on base of OpenCV, PIL and scikit-image
* Tab 4 to Tab 5
+ Methods using pretrained models from Machine Learning on base of OpenCV and super-image
* Tab 6
+ Methods based on a Stable Diffusion model and Stable Diffusion upscaler models
## Brief Introduction of the State of the Art
Graphical software tools like RawTherapee and
GIMP can be used to upscale images. During some tests I carried out,
I noticed that the images become sometimes somewhat blurred when they are
upscaled. However, one can only recognise this if one make a direct comparison
with the original image.
Another way for the upscaling of image is using Artificial
Inteligence. A distinction can be made between two approaches. Direct use of
pretrained models from machine learning. It is also possible to use special
diffusion models such as the Stable Diffusion models for upscaling.
Every programming language has its own way to handle
downscaling or upscaling of images. In the context of this application, I will
limit myself to the possibilities offered by Bash in combination with
Python.
## Presumption
Use a computer system with a Linux version as operating system
which is state of the art. Install Python 3.10 or higher if you want to use the
Lazy Image Upscaler. Some third-party modules are also required. Prtrained models
as well as diffusion models can be installed later.
> [!Note]
>
The installed Python version must bei 3.10 or higher.
> Otherwise the application will not working. The in the code used Match
> Case statement was introduced in Python 3.10.
## Installation of The Web UI
Clone the repository to a local location of of your choice.
Therefore move to the locoation of your choice. Then run the following command:
```bash
git clone https://github.com/zentrocdot/LazyImageUpscaler.git
```
Afterwards you are ready to use the application.
As alternative you can also download the zip-file with the repository.
```bash
LazyImageUpscaler-main.zip
```
Extract the zip-file to a location of your choice. Preserve
the directory structure. Then it is propsed to rename the main directory which is
```bash
LazyImageUpscaler-main
```
to
```bash
LazyImageUpscaler
```
In both cases you are ready to use the application.
The base modules will work out of the box after installation.
The modules which are using pretrained models and the diffusion models will work
after download of these models. How to do this is described later.
## Installation of the Required Python Dependencies
Following Python requirements have to be fulfilled,
that the Lazy Image upscaler is working:
* gradio
* numpy
* PIL
* opencv
* skimage
* piexif
* SSIM_PIL
* super_image
* diffusers
* torch
Install the Python dependencies using pip (pip3). Do not
install the modules as sudo.
First update pip. It could be that you have to update
pip also in between.
```bash
pip3 install --upgrade pip
```
Afterwards you can install most of the dependencies.
```bash
pip3 install gradio
pip3 install opencv-contrib-python
pip3 install numpy
pip3 install pillow
pip3 install piexif
pip3 install SSIM_PIL
pip3 install diffusers
pip3 install torch
```
For scikit-image you have to perform following steps.
```bash
# Update pip
python -m pip install -U pip
# Install scikit-image
python -m pip install -U scikit-image
```
Everything will be installed to a local location,
which ma look like:
./.local/lib/python3.10/site-packages
The local installation can be found in the hidden
directory .local in the user's main directory. Changes
here may result in problems while running the application.
> [!TIP]
>
If you do not want to install torch or
> diffusers, disable super-image and Stable Diffusion in the
> config file.
## Start
Use start_webui.bash in the main directory
to start the local server. If there is a problem one can move into the
subdirectory scripts. From there lazy_image_upscaler.py can
be started.
Open a webbrowser and open localhost on
http://127.0.0.1:7865
If everything was okay so far, the web UI starts
in the browser window.
## Control of the Web User Interface
You can control the size of the visible area of
the window where the application is shown using keyboard shortcuts.
Zoom in can be done by CTRl++ and zoom out can be done by
CTRl+-.
## Configuration of the Interface
In the directory /configsthere is a
configuration file that can be used to make various settings
```bash
# Lazy Image Upscaler configuration file
# --------------------------------------
# Output related parameter.
isRawImage = False
isExifImage = True
isChopsImage = False
# Image related parameter
JpgQuality = 70
# Tab related parameter.
isOpencvTab = True
isPilTab = True
isScikitTab = True
isSuperResolutionTab = True
isSuperImageTab = True
isStableDiffusionTab = True
# Safetensor related parameter.
SafeTensor = False
Scikit related parameter.
AntiAliasing = False
```
By setting the corresponding value
to False, individual Tabs can be deactivated in the
user interface.
## Implemented Methods
### Numerical Interpolation Methods
#### OpenCV Interpolation Methods Related to the First Tab [](https://docs.opencv.org/4.10.0/da/d54/group__imgproc__transform.html)
OpenCV can be used for upscaling images. The in OpenCV
implemented interpolation methods for resizing are:
* INTER_NEAREST
* INTER_LINEAR
* INTER_AREA
* INTER_CUBIC
* INTER_LANCZOS4
* INTER_LINEAR_EXACT
* INTER_NEAREST_EXACT
The interpolation methods can by selected over a drop-down menu.
#### PIL Interpolation Methods Related to the Second Tab [](https://pillow.readthedocs.io/en/stable/reference/Image.html)
In addition to OpenCV, pillow or PIL can also be
used for upscaling of images. The in pillow implemented interpolation
methods are:
* NEAREST
* BOX
* BILINEAR
* HAMMING
* BICUBIC
* LANCZOS
The interpolation methods can by selected over a drop-down menu.
#### skimage Interpolation Methods Related to the Second Tab [](https://scikit-image.org/docs/stable/api/skimage.transform.html#skimage.transform.resize)
* NEAREST-NEIGHBOR
* BI-LINEAR
* BI-QUDRATIC
* BI-CUBIC
* BI-QUARTIC
* BI-QUINTIC
### Machine Learning Methods
#### Pretrained Models (OpenCV)
Pretrained models which can be used are:
* EDSR
* ESPCN
* FSRCNN
* LAPSRN
#### Pretrained Models (super-image)
Predefined in the directory structure are:
+ a2n
+ awsrn-bam
+ carn
+ carn-bam
+ drln
+ drln-bam
+ edsr
+ edsr-base
+ han
+ mdsr
+ mdsr-bam
+ msrn
+ msrn-bam
+ pan
+ pan-bam
+ rcan-bam
### Stable Diffusion Methods
Stable Diffusion model and Upscalers are:
+ stable-diffusion-v1-5
+ sd-x2-latent-upscaler
+ stable-diffusion-x4-upscaler
## Web UI
The web UI is simple to use. One can select the pretrained model from a dropdown list. Per drag and drop or per upload the image can be loaded. Using the Upscale Image button scales the image up. Download Image downloads the image to the local storage.
Figure 1: Web userinterface in a web browser without image
Figure 2: Web userinterface in a web browser with image and upscaled image
The web user interface is organised in several logical parts.
There is a header and a footer. Both are not of interest for the functioanlity of the web UI.
Directly below the header are buttons that provide basic functions. Below this
are buttons for the parameters that need to be set.
In the centre there is an area on the left-hand side for entering an image and on the right-hand side for outputting an image.
## Download Upscaled Images and Name Convention
When a download is done image names looks like:
2024-11-13_16:32:48.931703.jpg
To make sure that each image is unique I am using the date, the time and the remaining microseconds as filename. The formatstring is:
"%Y-%m-%d_%H:%M:%S.%f"
The downloaded images can be found in the folder outputs.
## Additional Implemented Features
### Implemented Features
In addition to upscaling images, rudimentary
image editing is also possible with the application. These features
are:
+ Horizontal Flip
+ Vertical Flip
+ Rotate Left
+ Rotate Right
+ Inversion
+ Grayscale
+ Brightness
+ Contrast
+ Sharpening
+ Smoothing
+ Gamma
+ Denoising
+ Sepia Filter
+ Color Filter
### Horizontal and Vertical Flip
Horizontal Flip and Vertical Flip are mirroring an image in horizontal and vertical direction.
### Rotate Left and Right
Rotate Left and Rotate Right are rotating an image in clockwise and anti-clockwise by an angle of 90°
### Inversion
Inversion is inverting an image.
### Grayscale
Grayscale creates an image in grayscale from the original.
### Brightness
to-do ...
### Contrast
to-do ...
### Sharpening
to-do ...
### Smoothing
to-do ...
### Gamma
Gamma is a classical Gamma correction.
### Denoising
to-do ...
### Sepia Filter
The sepia filter looks like:
sepia_matrix = [[0.393, 0.769, 0.189],
[0.349, 0.686, 0.168],
[0.272, 0.534, 0.131]]
correct_matrix = [[0.607, -0.769, -0.189],
[-0.349, 0.314, -0.168],
[-0.349, -0.534, 0.869]]
sepia_filter = sepia_matrix + scalar * correct_matrix
Valid values are in a range [0,1]
### Color Filter
The color filter is using a HSV color model.
HSV is the abbreviation for Hue-Saturation-Value. This color model
is different to the well known RGB or BGR color which is often used.
models. The HSV color model is using Hue, Saturation and Value for
the description of the colors as the abbreviation states.
To use the color filter one needs some parameters.
These parameters/values are:
- Hue
- Saturation
- Value
By default this vaules are set to:
(0.75,1.5,0.5)
The values have to be given as a tuple (triple).
If not no image is modified and a warning is shown.
Simply spoken, the Color filter is able, to modify or
shift the colors. The colors range from red (0) to magenta (1). To check this out use:
(0.5,1,1)
If Saturation is 1 and Value is also one, the color is shifted to green/cyan value.
The following values can be taken as a guide.
* Red 0.0 - 0.17
* Yellow 0.17 - 0.33
* Green 0.33 - 0.5
* Cyan 0.5 - 0.67
* blue 0.67 - 0.83
* Magenta 0.83 - 1.0
* 1.0 (image colors remains unchanged)
Values outside the interval [0,1] will lead to colour distortion and false colours
A value for the Saturation 0f 0.0 will give a grayscale image.
(1,0,1)
A value for the Value 0f 0.0 will give a pure black image.
(1,1,0)
Values outside the interval [0,1] will lead to colour distortion and false colours for Saturation and Value.
Other color models next to HSV e.g. are:
- RGB (red, green, blue)
- BGR (blue, green, red)
- CMYK (cyan, magenta, yellow, black)
- HSL (hue, saturation, lightness)
- YIQ (luminance, in-phase, quadrature)
### Comments on the Special Features
#### Horizontal and Vertical Flip
The horizontal and vertical flip of an image goes hand in hand with a loss of quality.
## Test Environment
I developed and tested the Python script with following specification:
* Linux Mint 21.3 (Virginia)
* Python 3.10.14
* OpenCV 4.10.0
* PIL 11.0.0
* Gradio 5.6.0
* Torch 2.4.1+cu121
* Numpy 2.1.3
* Chromium Browser, Firefox and Google Chrome
* Monitor with a resolution of 1366 x 768 pixel
## Time Consumption
The numerical approaches are the fastest. The AI approach is the one which takes the most time.
## Power consumption
The numerical approaches use the CPU and not the GPU, so this approach saves energy. Machine Learning
and AI use the GPU extensivly and have a high power consumption.
## Limitations
In the Machine Learning Tabs and in the AI Tab there are input images
larger than 512 x 512 pixel problematic.
## Known Problems
The critical parts of the software are the parts that use the GPU. In this
sense, three upscaling approaches are critical. Two approaches are using pretrained models from
Machine Learning and one is the well know AI approach.
Common errors if on talks about the GPU usages are.
* RuntimeError
* OutOfMemorError
## Super Image [](https://pypi.org/project/super-image/)
to-do ....
## Super Resolution [](https://learnopencv.com/super-resolution-in-opencv/)
to-do ....
## Stable Diffusion Upscaler Model
> [!IMPORTANT]
> Execute only one instance of Stable Diffusion on the GPU. Otherwise upscaling will fail.
### Implemented Approaches
Figure 2: Web userinterface in a web browser
Figure 3: Web userinterface in a web browser
## Pretrained Model Installation
#### stable-diffusion-x4-upscaler
Move to directory LazyImageUpscaler/stabilityai. Perform following steps.
```bash
# Make sure you have git-lfs installed (https://git-lfs.com).
git lfs install
```
```bash
# Clone the repository.
git clone https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler
```
```bash
# If you want to clone without large files - just their pointers.
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler
```
#### sd-x2-latent-upscaler
Move to directory LazyImageUpscaler/stabilityai. Perform following steps.
```bash
# Make sure you have git-lfs installed (https://git-lfs.com).
git lfs install
```
```bash
# Clone the repository.
git clone https://huggingface.co/stabilityai/sd-x2-latent-upscaler
```
```bash
# If you want to clone without large files - just their pointers.
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/stabilityai/sd-x2-latent-upscaler
```
#### stable-diffusion-v1-5
Move to directory LazyImageUpscaler/stabilityai. Perform following steps.
```bash
# Make sure you have git-lfs installed (https://git-lfs.com).
git lfs install
```
```bash
# Clone the repository.
git clone https://huggingface.co/stabilityai/stable-diffusion-v1-5
```
```bash
# If you want to clone without large files - just their pointers.
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5
```
## Repository & Directory Structure
The repository and directory structure of the Lazy Image Upscaler
is looking as follows:
```bash
└── LazyImageUpscaler
│ └── start_webui.bash
├── scripts
│ └── LazyImageUpscaler.py
├── configs
│ └── LazyImageUpscaler.config
├── tools
│ ├── get_edsr.py
│ ├── get_escpn.py
│ ├── get_fsrcnn.py
│ ├── get_lapsrn.py
│ └── remove-metadata.bash
├── test-images
│ ├── vulture.jpg
│ ⫶
│ └── shedevil.jpg
├── outputs
├── super-resolution
│ ├── EDSR_x2.pb
│ ├── EDSR_x4.pb
│ ├── ESPCN_x2.pb
│ ├── ESPCN_x4.pb
│ ├── FSRCNN_x2.pb
│ ├── FSRCNN_x4.pb
│ ├── LapSRN_x2.pb
│ └── LapSRN_x4.pb
├── super-image
│ ├── carn
│ ├── pytorch_model_2x.pt
│ └── pytorch_model_4x.pt
│ ├── edsr
│ ├── pytorch_model_2x.pt
│ └── pytorch_model_4x.pt
│ └── pan
│ ├── pytorch_model_2x.pt
│ └── pytorch_model_4x.pt
└── stabilityai
├── stable-diffusion-x4-upscaler
├── sd-x2-latent-upscaler
└── stable-diffusion-v1-5
```
In the folder scripts there are four Python scripts,
which can be used to download the models into the resources folder directly.
Under the main branch there are four directories. In scripts are the
Python scripts for the web user interface. In resources there can be the .pb models be
stored. After the installation this directory is empty. Created images are saved in outputs.
images is the directory where documentation related images are stored.
The folders super-resolution, super-image and stabilityai
are by default empty. There can be the pretrained models placed as well as the
upscaler model.
The directory tree shows how a tree looks, wenn some files
are downloaded.
## Troubleshooting
### Web UI
In the case of unexpected persistent problems, shut down
the Gradio server in the terminal window. After relaunch of the Gradio server,
refresh the browser window.
If there is a problem with the server and with the port, one can
change both values in the source code e.g. from
webui.launch(server_name="127.0.0.1", server_port=7865)
to
webui.launch()
in the script LazyImageUpscaler.py in the directory /scripts.
### Super Resolution
The error message
module cv2.cv2 has no attribute dnn_superres
or similiar error messages can be handeled as desribed bwlow. This error occured appeared from one moment to the next
without me being able to understand why the error message occurred.
Following solved this problem:
pip uninstall opencv-python
pip uninstall opencv-contrib-python
Then install latest version of OpenCV with pip3:
pip3 install opencv-contrib-python
### Super Image
Error:
```
ImportError: cannot import name cached_download from huggingface_hub
```
Possible Solution:
```
pip3 install huggingface_hub==0.25.00
pip3 install -U sentence-transformers
```
### Stable Diffusion
The more I go in the direction of AI image upscaling,
the greater the problems with setting up the computer to get the desired
result.
A big problem that I had before, independent of my
application, is a problem in the mismatch of version numbers of Python
modules. If I install one module, the other is uninstalled. So it is
tricky until the system runs with all required modules.
The error messages and solution approaches reflect
this problem.
Error:
```
cannot import name CpuOffload from accelerate.hooks
```
Possible Solution:
```
pip install git+https://github.com/huggingface/accelerate.git
```
Error:
```
AttributeError: StableDiffusionControlNetPipeline object has no attribute enable_vae_tiling
```
Possible Solution:
```
pip install git+https://github.com/huggingface/diffusers.git@main
```
## Spin-Off
The internal structure of the script is organised
in such a way that the user interface can be expanded to include
additional methods for upscaling. In principle, the basic structure
can always be adopted. Essentially, fields for input parameters must
be adapted and the upscaler integrated as a function.
## To-Do
So far I integrated the possibility to compare two
images with respect to there quality. I need a separate independent
Python tool, which is able to compare images from different sources
like Gimp and RawTherapee with the original image. This
is important to compare the results of my web user interface with the
state of the art software tools.
Improvement of this documentation. The web UI has
to checked that it is more fail safe. The current work was quick and
dirty programming. I need to sanitize and optimize the code.
## Outlook
I need to take a deepler look in how ESRGAN works. I tried to get it
run on the quickly, but this failed for the moment. An implementation
in Python will give me further possibilities for upscaling. Initial
tests show interesting results.
## Free for Contributions
Due to a lack of time, I can only develop my tools
in between. If anyone would like this tool to be developed further,
the best way is to support me with a donation as mentioned below. Then
I can concentrate on these things and don't have to do anything else.
## General Note
All company names, brands, brand names and trademarks
mentioned in this documentation belong to their respective owners. This
application is an independently developed software product by the author.
## References
[1] https://github.com/opencv/opencv_contrib/tree/master/modules/dnn_superres
[2] https://github.com/Saafke/EDSR_Tensorflow/tree/master/models
[3] https://github.com/fannymonori/TF-ESPCN/tree/master/export
[4] https://github.com/Saafke/FSRCNN_Tensorflow/tree/master/models
[5] https://github.com/fannymonori/TF-LapSRN/tree/master/export
[6] https://github.com/cyc0102/opencv_super_resolution/tree/master
[7] https://huggingface.co/
[8] https://pypi.org/project/super-image/
[9] https://huggingface.co/stabilityai/sd-x2-latent-upscaler
[10] https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler
[11] https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5
[12] https://pillow.readthedocs.io/en/stable/reference/Image.html
[13] https://pillow.readthedocs.io/en/stable/handbook/concepts.html
[14] https://scikit-image.org/
## Donation
If you like what I present here, or if it helps you,
or if it is useful, you are welcome to donate a small contribution. Or
as you might say: Every TRON counts! Many thanks in advance! :smiley:
${\textnormal{\color{navy}Tron}}$
```
TQamF8Q3z63sVFWiXgn2pzpWyhkQJhRtW7
```
${\textnormal{\color{navy}Doge}}$
```
DMh7EXf7XbibFFsqaAetdQQ77Zb5TVCXiX
```
${\textnormal{\color{navy}Bitcoin}}$
```
12JsKesep3yuDpmrcXCxXu7EQJkRaAvsc5
```
${\textnormal{\color{navy}Ethereum}}$
```
0x31042e2F3AE241093e0387b41C6910B11d94f7ec
```