The immich-ml service fails on reboot

[jcw]

As mentioned here: #6785

I'm not quite sure how to create a "service log", but this is what I see in dietpi-services:

┌────────────────────────────────┤ DietPi-Services ├────────────────────────────────┐
│ Please select an option for: immich-ml                                            │
│                                                                                   │
│                              ●─ Service control ---------------------------●      │
│      State                   : [failed]                                           │
│      Mode                    : [enabled]                                          │
│      Include/Exclude         : [included]                                         │
│      Status                  : Display systemd status log                         │
│      Edit                    : [/etc/systemd/system/immich-ml.service]            │
│                              ●─ Process tool ------------------------------●      │
│      CPU Affinity            : [0-3]                                              │
│      CPU Scheduling Policy   : [other]                                            │
│      CPU Nice                : [0]                                                │
│      I/O Scheduling Class    : [best-effort]                                      │
│      I/O Scheduling Priority : [4]                                                │
│                              ●─ Presets -----------------------------------●      │
│      Reset                   : Resets all CPU/IO settings to system defaults      │
│      Highest Priority        : Highest priority realtime CPU/IO preset            │
│      High Priority           : High priority CPU/IO preset                        │
│      Low Priority            : Low priority CPU/IO preset                         │
│      Lowest Priority         : Lowest priority background CPU/IO preset           │
│                                                                                   │
│                                                                                   │
│                      <Select>                      <Back>                         │
│                                                                                   │
└───────────────────────────────────────────────────────────────────────────────────┘

and

┌─────────────────────────┤ DietPi-Services | File viewer ├─────────────────────────┐
│                                                                                   │
│ × immich-ml.service - Immich Machine Learning (DietPi)                            │
│      Loaded: loaded (/etc/systemd/system/immich-ml.service; enabled; preset:      │
│ enabled)                                                                          │
│      Active: failed (Result: resources)                                           │
│  Invocation: 602dae32ba714a58a92faa3fe90879bb                                     │
│    Mem peak: 0B                                                                   │
│         CPU: 0                                                                    │
│                                                                                   │
│ Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed to load environment  │
│ files: No such file or directory                                                  │
│ Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed to spawn 'start'     │
│ task: No such file or directory                                                   │
│ Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed with result          │
│ 'resources'.                                                                      │
│ Apr 06 18:31:12 houtje systemd[1]: Failed to start immich-ml.service - Immich     │
│                                                                                   │
│                                      <Ok>                                         │
│                                                                                   │
└───────────────────────────────────────────────────────────────────────────────────┘

This is on DietPi v10.2.3, RPi 5 Model B, 1334 of 8063 MiB RAM used.

As mentioned, all photos are on an external SMB mount, but IIUC this would not be relevant, as it's immich-ml which is failing, not immich itself.

Please let me know what other info I can dig up. System still rebooted w/ immich-ml not running.

[Joulinar]

can you share following

journalctl -u immich-ml.service
lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid
df -h

[jcw]

Thanks for providing these cmds:

# journalctl -u immich-ml.service | cat
Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed to load environment files: No such file or directory
Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed to spawn 'start' task: No such file or directory
Apr 06 18:31:12 houtje systemd[1]: immich-ml.service: Failed with result 'resources'.
Apr 06 18:31:12 houtje systemd[1]: Failed to start immich-ml.service - Immich Machine Learning (DietPi).
# lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid
NAME FSTYPE LABEL   SIZE RO TYPE MOUNTPOINT PARTUUID                             UUID
nvme0n1
│                 476.9G  0 disk
├─nvme0n1p1
│    vfat   --      128M  0 part /boot/firm b814eb97-01                          12A1-1A9E
├─nvme0n1p2
│    ext4   --       10G  0 part /          b814eb97-02                          e1b563e2-1ffd-4657-883e-748a19617af2
└─nvme0n1p3
     btrfs        466.8G  0 part /mnt/extra b814eb97-03                          72927842-8f0b-460e-aac5-3c0ed1e74ea0
# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/root                     8.9G  6.6G  1.8G  80% /
devtmpfs                      4.0G     0  4.0G   0% /dev
tmpfs                         4.0G  7.9M  4.0G   1% /dev/shm
tmpfs                         1.6G   17M  1.6G   1% /run
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                         4.0G   32K  4.0G   1% /tmp
tmpfs                          50M   64K   50M   1% /var/log
/dev/nvme0n1p1                127M   43M   84M  34% /boot/firmware
/dev/nvme0n1p3                467G  289G  178G  62% /mnt/extra
tmpfs                         1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
//192.168.178.3/Media/Photos   11T  9.0T  1.9T  83% /mnt/photos
#

I also manually edited the SMB mount in fstab to get the proper file permissions:

//192.168.178.3/Media/Photos /mnt/photos cifs cred=/var/lib/dietpi/dietpi-drive_manager/mnt-photos.cred,iocharset=utf8,gid=immich,file_mode=0664,dir_mode=0775,vers=3.1.1,noauto,x-systemd.automount

Changed owner and modes.

Update: swap is not enabled.

Update 2: dietpi_userdata is on /mnt/extra/ (maybe not mounted soon enough?)

[Joulinar]

ok SMB should not be an issue. Can you download the entire system log after reboot? This way we can see timing for immich-ml and mount of the 3rd partition of your NVMe. I have a similar setup but Immich is busy indexing 72.000 images and 5.500 videos right now.

journalctl > /tmp/log.txt

This will create a log.txt file within /tmp. Pls download and attach as file

[jcw]

(sure, I'm fine with Unix/Linux cmds, just not too familiar with systemd ...)

log.txt

[Joulinar]

ok it's a timing issue.

Immich ML issue at 20:34:06

Apr 06 20:34:06 houtje systemd[1]: immich-ml.service: Failed to load environment files: No such file or directory

while the mount happen only at 20:34:07

Apr 06 20:34:07 houtje systemd[1]: Mounted mnt-extra.mount - /mnt/extra.

can you share following

cat /etc/fstab

[jcw]

Aha!

$ cat /etc/fstab
# You can use "dietpi-drive_manager" to setup mounts.
# NB: It overwrites and re-creates physical drive mount entries on use.
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------
//192.168.178.3/Media/Photos /mnt/photos cifs cred=/var/lib/dietpi/dietpi-drive_manager/mnt-photos.cred,iocharset=utf8,gid=immich,file_mode=0664,dir_mode=0775,vers=3.1.1,noauto,x-systemd.automount

#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs size=4031M,noatime,lazytime,nodev,nosuid,mode=1777
tmpfs /var/log tmpfs size=50M,noatime,lazytime,nodev,nosuid

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf, glusterfs, mergerfs, bind, Btrfs subvolume
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAP SPACE
#----------------------------------------------------------------


#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
PARTUUID=b814eb97-02 / ext4 noatime,lazytime,rw 0 1
PARTUUID=b814eb97-01 /boot/firmware vfat noatime,lazytime,rw 0 2
UUID=72927842-8f0b-460e-aac5-3c0ed1e74ea0 /mnt/extra btrfs noatime,lazytime,rw,noauto,x-systemd.automount

[Joulinar]

ok let's try to start ML after the mount happen. Can you share following as input

systemctl list-units --type=mount

[jcw]

Here it is (never used this myself ...):

# systemctl list-units --type=mount
  UNIT                          LOAD   ACTIVE SUB     DESCRIPTION
  -.mount                       loaded active mounted Root Mount
  boot-firmware.mount           loaded active mounted /boot/firmware
  dev-mqueue.mount              loaded active mounted POSIX Message Queue File System
  mnt-extra.mount               loaded active mounted /mnt/extra
  proc-sys-fs-binfmt_misc.mount loaded active mounted Arbitrary Executable File Formats File System
  run-lock.mount                loaded active mounted Legacy Locks Directory /run/lock
  sys-fs-fuse-connections.mount loaded active mounted FUSE Control File System
  sys-kernel-config.mount       loaded active mounted Kernel Configuration File System
  sys-kernel-debug.mount        loaded active mounted Kernel Debug File System
  sys-kernel-tracing.mount      loaded active mounted Kernel Trace File System
  tmp.mount                     loaded active mounted /tmp
  var-log.mount                 loaded active mounted /var/log

Legend: LOAD   → Reflects whether the unit definition was properly loaded.
        ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
        SUB    → The low-level unit activation state, values depend on unit type.

12 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

[Joulinar]

mnt-extra.mount               loaded active mounted /mnt/extra

ok you can try adding following to your ML service. Open your service file (usually located in /etc/systemd/system/immich-ml.service) and add the Requires= and After= directives under the [Unit] section.

nano /etc/systemd/system/immich-ml.service

add

Requires=mnt-extra.mount
After=mnt-extra.mount

There may already be other Requires= and After= directives. In that case, just add it there.

After saving the file, you must tell systemd to look at the new configuration

sudo systemctl daemon-reload
sudo systemctl restart immich-ml.service

or simply reboot 😄 Check system afterwards

[jcw]

Yes, that works: immich-ml comes up [active] after a reboot, and the big memory-eating python process is running \o/

So it looks like when dietpi_userdata is mounted, then immich-ml needs to start after that mount.

Fantastic, I think you've nailed it!

[Joulinar]

I will will close the issue for now. Feel free to reboot if needed.