Too many open files

If you need expert help to solve this or any other problem, explore our technical support terms and conditions.

Description

If you have a large number of cameras, around 128, you may encounter Inotify restrictions, then you may see a message in the server log when adding a camera:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0xdda663]

goroutine 1258 [running]:
gitlab.com/yuccastream/yucca/pkg/runner/latest.(*Runner).GetStats(...)
        /builds/yuccastream/yucca/pkg/runner/latest/runner.go:327
gitlab.com/yuccastream/yucca/pkg/server/executor/exec.(*Worker).GetStream(0xc002b74500)
        /builds/yuccastream/yucca/pkg/server/executor/exec/worker.go:188 +0xa3
gitlab.com/yuccastream/yucca/pkg/server/executor/executor/exec.(*Executor).StatusStream(0xc0003f01c0, 0xc00074b6c0)
        /builds/yuccastream/yucca/pkg/server/executor/executor/exec/executor.go:126 +0xb8
gitlab.com/yuccastream/yucca/pkg/server.(*Server).stateWatcher(0xc000c1e000)
        /builds/yuccastream/yucca/pkg/server/server.go:914 +0x233
created by gitlab.com/yuccastream/yucca/pkg/server.(*Server).Start in goroutine 1
        /builds/yuccastream/yucca/pkg/server/server.go:695 +0x4db

And the log of one of the cameras:

Error: too many open files

The default settings in many Linux distributions are as follows:

sysctl fs.inotify

fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288

ulimit -n
10240

grep NOFILE /etc/systemd/system.conf
#DefaultLimitNOFILE=1024:524288

Solution

To increase the limits, add the following parameters to the /etc/sysctl.conf file:

sudo su
cat << EOF >> /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 10240
fs.inotify.max_user_watches = 524288
EOF

In the /etc/systemd/system.conf file:

DefaultLimitNOFILE=10240:524288

and reboot the system.

Make sure the settings are applied:

ulimit -n
10240

sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 10240
fs.inotify.max_user_watches = 524288