Introduction
I made a small ceph cluster from 3 low power x86_64 machines last year just for testing. I kinda want to use it more and I found that adding more nodes is a good way to distribute load around to achieve higher capacity(in my case).
I have stuck with low power x86 for a long time because:
- Electricity is expensive where I live
- x86 means guaranteed software compatibility with whatever I throw at it
Considering how the Pi4 only consumes about 3-4 watts on idle (as opposed to 8-12W on a low power PC), it seemed tempting to run ceph daemons here, as running cost will not be a problem.
I am not worried about performance constraints though as my cluster runs on HDDs only (and I only have a gigabit network). The challenge I faced is that I am using Raspbian and it only ships ceph luminous, but I built my cluster around octopus.
Disclaimer
I am only sharing what worked for me. I am assuming you know what you are doing and that you have backups.
What you will need
- Raspberry Pi 4 (I have the 4GB variant)
- An existing cluster
- Have your ceph.conf config set up at /etc/ceph like you did for the other nodes
Step 1: Update the firmware and kernel on the Pi
Run:
$ sudo rpi-update
Step 2: Use the 64-bit kernel
Open /boot/config.txt and locate the [pi4] section. Add arm_64bit=1 at the end.
Mine looks like this:
[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
arm_64bit=1
After you reboot you should see aarch64 come up under uname -a
$ alex@rpi4:~ $ uname -a
Linux rpi4 5.10.23-v8+ #1406 SMP PREEMPT Mon Mar 15 15:49:01 GMT 2021 aarch64 GNU/Linux
Step 3: Installing Docker
I managed to make ceph 15.2 work via the docker builds for arm64. Following the instructions from the docker website did not work for me. It only worked for me with an older, specific version of docker. Running the ff. command as root was what worked for me:
$ wget -qO- https://get.docker.com/ | sed 's/docker-ce/docker-ce=5:19.03.13~3-0~raspbian-buster/' | sh
You may need to restart the pi after.
Step 4: Running the container
Running the container from the ceph docs did not work. This is because it needs the --privileged arg passed with the docker run command. I have read online that this is caused by Debian having outdated libraries that block certain calls.
Running the object gateway daemon without --privileged yields nothing in the logs so it took a while to figure this one out.
$ sudo docker run -dit --privileged --net=host -v /var/lib/ceph/:/var/lib/ceph/ -v /etc/ceph:/etc/ceph ceph/daemon:v5.0.8-stable-5.0-octopus-centos-8-aarch64 rgw
You may get an error after you run this. For me, I had to run this too:
$ sudo ceph auth get client.bootstrap-rgw -o /var/lib/ceph/bootstrap-rgw/ceph.keyring
Step 5: Verify
Connect to the mgr node to get the ceph cluster status:
$ sudo ceph -s
(...)
rgw: 3 daemons active (rpi4, node3, node4)
(...)
On the pi itself, you can check if the HTTP port is open:
$ root@pi4# curl localhost:7480
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
Hope this helps. I’ll try to see if this will work with an OSD if I get another disk to test.