https://github.com/hmknapp/lizardfs-docker
Decided to open source my personal lizardfs cluster setup. This works (with qemu) on aarch64 hosts.
https://github.com/hmknapp/lizardfs-docker
docker lizardfs moosefs
Last synced: about 2 months ago
JSON representation
Decided to open source my personal lizardfs cluster setup. This works (with qemu) on aarch64 hosts.
- Host: GitHub
- URL: https://github.com/hmknapp/lizardfs-docker
- Owner: HMKnapp
- Created: 2023-08-05T20:51:18.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2025-04-04T11:34:20.000Z (about 2 months ago)
- Last Synced: 2025-04-04T12:29:00.648Z (about 2 months ago)
- Topics: docker, lizardfs, moosefs
- Language: Shell
- Homepage:
- Size: 47.9 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.adoc
Awesome Lists containing this project
README
= Notes
== Services
- ``lizardfs-cgiserver``
- ``lizardfs-master``
- ``lizardfs-metalogger``
- ``lizardfs-chunkserver``
- ``lizardfs-mount``
=== Master
Use default ``.env`` from ``master`` branch.
=== Chunkserver
IMPORTANT: Give each chunkserver its own branch with its own ``.env`` file. This is because ``.env`` is used to configure ``LABEL`` and disk mount points.
== Build
.Always build ``lizardfs-master``
[NOTE]
====
``lizardfs-master`` in ``docker-compose.yml`` contains the image that all services share: ``hmknapp/lizardfs``
No other image needs to be built. Different roles are achieved by different service names and their ``ROLE`` environment variable.
====
=== Upgrade or rebuild
IMPORTANT: Before upgrading, make sure you have a backup of all ``./data/*`` folders of all nodes, especially master and metalogger. Also keep ``authorized_keys`` from ``./data/mount/ssh/``.
. Stop container
+
```
docker stop lizardfs-cgiserver
docker rm lizardfs-cgiserver
```
+
. Build and start container
+
```
docker-compose build lizardfs-master
docker-compose up lizardfs-cgiserver -d
```
+
. Renew SSH fingerprint for mount points by logging into ``lizardfs-mount`` once: ``ssh -p 22022 lizardfs@localhost``
== Tips and tricks
=== Disaster recovery using metaloggers
NOTE: These steps are recorded from memory, may not be 100% correct.
. Log into metalogger and run ``mfsmetarestore``
+
```
mfsmetarestore -m mfsmetadata_ml.mfs -o /tmp/mfsmetadata.mfs change*
cp session_ml.mfs /tmp/session.mfs
```
+
. Copy metadata ``/tmp/*mfs`` to master’s ``./data/master/``
. Add ``entrypoint: mfsmetarestore -a to master’s docker-compose.yml``
. Run ``docker-compose up lizardfs-master`` once
. Remove lock files from master dir, ``./data/master/*lock``
. Remove ``entrypoint: mfsmetarestore -a`` from lizardfs-master in ``docker-compose.yml``
. Start master again, ``docker-compose up lizardfs-master -d``
=== Common Errors
- ``[error] [1:1] : can't create lockfile /var/lib/lizardfs/.mfschunkserver.lock: Permission denied``
+
Make sure the ``./data/*`` volume directories have correct permissions for user running docker.
=== Startup
.Example: Crontab on chunkserver (192.168.111.3)
[source]
----
@reboot while ! timeout 2 ping -c 1 -n supernode.ntop.org; do sleep 5; done; sleep 5; sudo edge -c SWARM_NAME -k SECRET_KEY -a 192.168.111.3 -l supernode.ntop.org:7777
@reboot while ! [[ -d /mnt/wd-elements-16T/docker ]]; do sleep 3; done; while ! timeout 2 ping -c 1 -n 192.168.111.1; do sleep 0.5; done; sleep 10; sudo systemctl start docker
@reboot while ! timeout 2 ping -c 1 -n 192.168.111.1; do sleep 0.5; done; while ! netcat -z localhost 22022; do sleep 1; done; sleep 1m; sshfs -p 22022 -o allow_root,allow_other,reconnect,ServerAliveInterval=15,ServerAliveCountMax=999,max_conns=12 lizardfs@localhost:/lfs /mnt/lizardfs/
----
NOTE: I am aware you can use services to wait for a network connection, but I have had success with the above method.