Emergency-Script

...
 -DRECOVERY_SCRIPT=/opt/factfinder/e.sh\
 -XX:OnOutOfMemoryError=/opt/factfinder/e.sh\
...

If you took a closer look at the optional Java settings during installation, you might have wondered, why a script called e.sh appears in several parameters. This script collects data before closing Tomcat processes. You can find it and other scripts we are using on our GitHub-Account. Of course, you can also use your own script in its place.

e.sh

To make e.sh work, you need to make some customisations. The environment variable productiveTomcat is needed to identify the Tomcat process you want to close. This variable is already part of the optional Java settings.

During its data collection process, the script generates a heap and thread dump using jmap. This means, you need to have a JDK installed. For debian, this means installing the  openjdk-8-jdk package.

sudo apt-get install openjdk-8-jdk

In our example, we installed Tomcat as service, so the user tomcat8 tries to run the script. You need to make sure that he is actually allowed to do so.

Automatically restarting Tomcat

The e.sh shuts down the Tomcat process in case of an exception, but does not restart it. Tomcat is essential for FACT-Finder, so a script combination makes sure that it is always running. THe scripts are called  checkserver.sh and cron_checkserver.sh.

checkserver.sh

#!/bin/bash
VERBOSE=0
while /bin/true ; do
  if [ ! "$(ps aux|grep java|grep -o productiveTomcat)" == "productiveTomcat" ]; then
    sudo service tomcat8 restart
  fi
  sleep 5
done

This scripts checks in a loop, if a  productiveTomcat process is running. This makes setting the correct parameters for the script extremely important (see optional Java settings). If the process can't be found, the script launches it. After a 5 second wait, this is rechecked. This is due to cron jobs only performed every minute, but an essential process should not be offline for so long.

cron_checkserver_check.sh

#!/bin/bash
if test `ps -eaf | grep checkserver.sh | grep -v "grep" -c` = 0 ; then
  bash ~/checkserver.sh >> ~/checkserver.log 2>&1 &
  echo `date` checkserver.sh neu gestartet.
fi

The cron_checkserver_check.sh script is a safety net to ensure that checkserver.sh actually performs its checks. If the loop is broken for any reason, this script restarts checkserver.sh. The script is run once a minute as a cronjob, yo you need to add this entry to the cronfile:

* * * * * ~/cron_checkserver_check.sh