Storage monitoring with Docker

─── ❖ ── ✦ ── ❖ ───

Introduction

I’m looking for a dockerized tool that can monitor and index my NAS systems (Unraid & Synology).

A list of solutions I’ll cover here:

• Diskover
• sist2

Only suitable for local files (or docker volumes):

• Doku
• Glances

Voidtool's Everything

Sadly Voidtool’s Everything doesn’t run on Linux (yet).
Alternatively Everything offers a WebUI → Idea: mount shares to a minimal Windows VM and expose port.
Or with this container an EFU index file can be generated, that Everything can then ingest.
And there are also alternative tools for Linux, like FSearch.

Diskover

Warning

Most features of Diskover are NOT available in the free Community Edition, noticeably Analytics functions (File Tree, Treemap, Heatmap, …), tagging files, export to JSON/CSV, checksums etc.
However even the Community Edition has a nice dashboard, a (regex) search, many filet options (size, date, file type) and helps identifying large folders.
Also Diskover doesn’t index the content of files or does OCR on them.

Docker-compose setup

[Click me] **docker-compose.yaml**

original docker-compose.yaml from linuxserver.io
text a

docker-compose.yaml

services:
  diskover:
    image: lscr.io/linuxserver/diskover
    container_name: diskover
    environment:
      - PUID=1068
      - PGID=100
      - TZ=Europe/Berlin
      - ES_HOST=elasticsearch
      - ES_PORT=9200
    volumes:
      - /volume1/docker/diskover/config:/config
      - /volume1/share1:/data/share1:ro
      - /volume1/share2:/data/share2:ro
    ports:
      - 9206:80
    mem_limit: 4096m
    restart: unless-stopped
    depends_on:
      - elasticsearch
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.22
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - cluster.routing.allocation.disk.watermark.low=600mb         # https://stackoverflow.com/a/78486057
      - cluster.routing.allocation.disk.watermark.high=500mb
      - cluster.routing.allocation.disk.watermark.flood_stage=400mb
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
#      - /volume1/docker/diskover/esdata:/usr/share/elasticsearch/data
#      - /volume1/docker/diskover/esdata:/var/lib/elasticsearch/data
      - diskover_esdata:/usr/share/elasticsearch/data
    ports:
      - 9205:9200
    depends_on:
      - elasticsearch-helper
    restart: unless-stopped
  elasticsearch-helper:
    image: alpine
    command: sh -c "sysctl -w vm.max_map_count=262144"
    privileged: true
    
volumes:
  diskover_esdata:
    driver: local
    name: diskover_esdata

First start

The default username is diskover with the password darkdata, a custom password must be set upon first login.

Manually creating indices

Warning

With the Community Edition of Diskover, only one index can be viewed/analyzed at given time. It might be good to mount each share/folder that should be watched in a separate subfolder in /data and create a single index over /data with

docker exec -u abc -d diskover python3 /app/diskover/diskover.py -i diskover-hostname /data

If the connection to the elasticsearch container works, the message No completed indices found in Elasticsearch. Run a crawl and after it finishes reload select indices page. should be displayed after login.

After starting the stack, at least one index has to be created manually as described in the Application Setup. In our case we create two indices by running the following commands from the host:

docker exec -u abc -it diskover python3 /app/diskover/diskover.py -i diskover-index_share1 /data/share1
docker exec -u abc -it diskover python3 /app/diskover/diskover.py -i diskover-index_share2 /data/share2

Update indices via cronjob

…

─── ❖ ── ✦ ── ❖ ───