Skip to content

GitLab

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • Sympl Sympl
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 61
    • Issues 61
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Sympl
  • SymplSympl
  • Issues
  • #179

Closed
Open
Created Apr 14, 2019 by Paul Cammish@kelduumOwner

Symbiosis: Symbiosis monit failure emails in Stretch

Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/129

The symbiosis-monit script will return an exit code of 75 for a few reasons: if it's been disabled, if the machine is still booting, if the load is higher than the number of CPU cores, or if dpkg is running:

root@jessie:~# grep -c processor /proc/cpuinfo     

root@jessie:~# cat /proc/loadavg
4.00 4.00 3.87 5/130 5696
root@jessie:~# /usr/sbin/symbiosis-monit -t email /etc/symbiosis/monit.d -a
root@jessie:~# echo $?
75

In Symbiosis Stretch, this will be printed to syslog:

upgrade2 systemd[1]: symbiosis-monit.service: Main process exited, code=exited, status=75/n/a

upgrade2 systemd[1]: symbiosis-monit.service: Unit entered failed state.

upgrade2 systemd[1]: symbiosis-monit.service: Failed with result 'exit-code'.

And also as an email:

Subject: Symbiosis monitor detected service failure
root : TTY=unknown ; PWD=/ ; USER=nobody ; COMMAND=/usr/bin/tee /var/tmp/symbiosis-monit.cursor
pam_unix(sudo:session): session opened for user nobody by (uid=0)
Started Symbiosis monitor.
symbiosis-monit.service: Main process exited, code=exited, status=75/n/a
symbiosis-monit.service: Unit entered failed state.
symbiosis-monit.service: Triggering OnFailure= dependencies.
symbiosis-monit.service: Failed with result 'exit-code'.

Server load will frequently rise above the number of CPU cores on busy servers, generating a large amount of emails. Printing to syslog is useful if there are problems with the symbiosis-monit service itself, but we should probably only send a failure email when an individual test has failed (e.g. apache2), rather than the entire service.

Edited Apr 14, 2019 by Paul Cammish
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking