skip to content
walterra.dev
Table of Contents

Back in February I wrote about running Elasticsearch on my Synology DS925+. DSM’s kernel did not provide the seccomp support required by Elasticsearch 9, so I ended up running Elasticsearch and Kibana with Docker inside an Ubuntu VM.

Seven months later the stack was still happily running Elasticsearch 9.3.0. Time for the less exciting, but rather important, follow-up: upgrading it.

The short version is that the update to 9.5.4 worked and all data survived. The longer version includes a forgotten Ubuntu password, booting from the installer ISO, a VMM snapshot, and a Docker bind mount that pointed at the wrong directory.

Is the VM still necessary?

I checked this again before touching the installation. Synology has released newer DSM versions since the original setup, but I could not find a documented change that enables CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER on this NAS. Updating Container Manager would not solve that on its own anyway: containers use the host kernel, so the container runtime cannot add a kernel feature that was not compiled in.

The practical test on DSM is:

Terminal window
if [ -r /proc/config.gz ]; then
  zgrep -E '^CONFIG_SECCOMP(_FILTER)?=' /proc/config.gz
elif [ -r "/boot/config-$(uname -r)" ]; then
  grep -E '^CONFIG_SECCOMP(_FILTER)?=' "/boot/config-$(uname -r)"
fi

For Elasticsearch 9.x I want to see both of these:

CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y

Until that is true on the NAS itself, the Ubuntu VM remains the cleanest option. A disposable Elasticsearch container would be a useful final test if Synology enables those flags in a future DSM release, but I would not point it at the existing data directory.

First problem: getting back into Ubuntu

I could reach Kibana and see the VM in Synology Virtual Machine Manager, but I had forgotten the Ubuntu login. The local account visible in VMM turned out to be a DSM permission: it allowed me to manage and open the VM, but it was not an account inside Ubuntu. DSM and the guest OS have separate user databases and passwords.

Getting GRUB’s recovery menu to appear through the browser console did not work reliably with either Shift or Escape. The Ubuntu Server installer ISO was the better rescue environment:

  1. Power off the VM.
  2. Attach the Ubuntu 22.04 ISO in VMM and boot from CD/DVD.
  3. Choose Try or Install Ubuntu Server.
  4. Open the installer shell with Ctrl+Alt+F2.
  5. Activate and mount the installed LVM root volume.
  6. chroot into it and reset the password.

lsblk -f showed the two logical volumes from the original setup:

ubuntu--vg-ubuntu--lv  ext4
ubuntu--vg-docker      ext4

The first one is the Ubuntu root filesystem. The second contains Docker and the Elasticsearch data, so it was important not to confuse them.

The commands that mattered were:

Terminal window
mkdir -p /mnt/root
mount /dev/mapper/ubuntu--vg-ubuntu--lv /mnt/root


awk -F: '$3 >= 1000 && $3 < 65534 {print $1}' /mnt/root/etc/passwd


mount --bind /dev  /mnt/root/dev
mount --bind /proc /mnt/root/proc
mount --bind /sys  /mnt/root/sys
mount --bind /run  /mnt/root/run
chroot /mnt/root /bin/bash


passwd <vm-user>
usermod -aG sudo <vm-user>
systemctl enable ssh

After shutting down the rescue environment, detaching the ISO, and booting from disk again, SSH worked:

Terminal window
ssh <vm-user>@<vm-ip>

This was a useful reminder that VM access needs the same basic housekeeping as any other server. The next bit of cleanup is adding my SSH public key so the password is no longer the only way in.

Take a snapshot before changing versions

Elasticsearch cannot be downgraded safely by changing the Docker image tag after a newer version has opened the data directory. I therefore stopped the Compose stack first:

Terminal window
cd ~/elasticsearch
docker compose down
sudo shutdown -h now

With the VM fully powered off, I created a VMM snapshot named pre-es-9.5.4. A native Elasticsearch snapshot repository would still be the better long-term data backup, but a powered-off VM snapshot gave me a consistent recovery point for this update.

The preflight checks had looked good: all 54 primary shards were active, there were no relocating or initializing shards, and the Docker volume had about 9.3 TB free. Cluster health was yellow because three replica shards cannot be allocated to the same single-node cluster. That is expected here; the important number was zero unassigned primary shards.

Updating both images together

Elasticsearch and Kibana should stay on the same version. I changed both image tags in docker-compose.yml:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.4


  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.4

Then validated the file and pulled the images before starting the stack:

Terminal window
docker compose config --quiet
docker compose pull
docker compose up -d

The images downloaded correctly, but Elasticsearch became unhealthy. Its logs contained the familiar-looking error:

failed to obtain node locks, tried [/usr/share/elasticsearch/data]
AccessDeniedException: /usr/share/elasticsearch/data/node.lock

At first this looked like another ownership problem. The real issue was more interesting.

The bind-mount trap

The Compose file in my home directory used a relative mount:

volumes:
  - ./elasticsearch/data:/usr/share/elasticsearch/data

That resolved to:

/home/<vm-user>/elasticsearch/elasticsearch/data

The directory was empty and owned by root. My actual Elasticsearch data lived on the large LVM volume at:

/srv/docker/elasticsearch/elasticsearch/data

The old container had originally been created with that /srv/docker path. Docker Compose could still identify and manage it later by its project labels and fixed container name, even when I ran commands from the similarly named directory in my home folder. Recreating the container exposed the mismatch: Compose evaluated the relative path from the current project directory and mounted the empty location instead.

That is exactly the sort of problem where blindly changing ownership would have been the wrong fix. Before touching permissions, I searched the Docker volume for the existing node.lock and Elasticsearch node metadata. The data was still there.

I changed the mount to the explicit path:

volumes:
  - /srv/docker/elasticsearch/elasticsearch/data:/usr/share/elasticsearch/data

After recreating the stack, Elasticsearch became healthy immediately and Kibana followed shortly afterward.

Verifying that this was the same cluster

Checking that two containers are running is not enough after an Elasticsearch upgrade. I loaded the password from .env and queried the cluster directly:

Terminal window
set -a
source .env
set +a


curl -u "elastic:$ELASTIC_PASSWORD" http://localhost:9200/
curl -u "elastic:$ELASTIC_PASSWORD" \
  http://localhost:9200/_cluster/health?pretty
curl -u "elastic:$ELASTIC_PASSWORD" \
  'http://localhost:9200/_cat/indices?v'

The final state was:

  • Elasticsearch 9.5.4
  • Kibana 9.5.4
  • the original cluster UUID, unchanged from the preflight check
  • all 54 primary shards active
  • zero unassigned primary shards
  • Kibana returning HTTP 200
  • 37,741,988 documents still present in bluesky-firehose-ner-0001

The preserved cluster UUID and index counts were the useful proof. The yellow health status remained because this is still a single-node cluster with replica settings, not because the upgrade lost anything.

What I learned this time

The version bump itself was straightforward. Most of the work was around the operational details that accumulate after a VM has quietly run for months:

  • A DSM user with VMM access is not an Ubuntu user.
  • The Ubuntu installer ISO makes a perfectly workable rescue environment when GRUB is hard to reach through VMM.
  • Stop Elasticsearch before taking a VM-level snapshot if that snapshot is the rollback plan.
  • Keep Elasticsearch and Kibana on the same version.
  • Treat the data mount as the most important line in the Compose file.
  • Prefer an absolute host path for important bind-mounted data, especially when the same Compose project might be invoked from different directories.
  • Verify the cluster UUID, primary shards, and index counts after an upgrade, not just container status.

I used pi-coding-agent over SSH for the preflight checks, shutdown, update, log inspection, path diagnosis, and final verification. Having it stop when the data directory looked wrong—and locate the original node metadata before changing anything—was the right kind of caution for this job.

The VM approach has now survived its first Elasticsearch minor-version upgrade. Next up is configuring a proper Elasticsearch snapshot repository so future upgrades do not rely primarily on a VMM snapshot.