OpenSpecimen outage – Debugging Guide

1 Basic Service Checks
2 Is the Application Accessible?
3 Apache2 Logs
4 Tomcat (OpenSpecimen) Logs
5 MySQL Troubleshooting
- 5.1 1. Check MySQL status:
- 5.2 2. Login to MySQL:
- 5.3 3. Diagnostic Queries:
6 Disk & Memory Checks
- 6.1 Disk usage:
- 6.2 Memory usage:
- 6.3 Java memory issues:
7 Port & Network
8 Tips

This guide walks through troubleshooting an OpenSpecimen outage.

Basic Service Checks

Check if the required services are running:

# Tomcat / OpenSpecimen
sudo systemctl status openspecimen

# Apache Web Server
sudo systemctl status apache2

# MySQL 8
sudo systemctl status mysql

To restart a service:

sudo systemctl restart <service-name>

Is the Application Accessible?

Try accessing:

http://<your-server-ip>/openspecimen

If not reachable, proceed with the steps below.

Apache2 Logs

Location:

/var/log/apache2/error.log   
/var/log/apache2/access.log

Check for:

HTTP 502/503 errors
Reverse proxy failures
Port binding or permission issues

Tomcat (OpenSpecimen) Logs

Tomcat Log Path:

$(TOMCAT_HOME)/logs/catalina.out

CopyEdit

Application Log Path:

$(OS_HOME)/data/logs/openspecimen.log

Look for:

Java exceptions
OutOfMemoryError
Deployment failures

MySQL Troubleshooting

1. Check MySQL status:

sudo systemctl status mysql

2. Login to MySQL:

mysql -u <username> -p

3. Diagnostic Queries:

use <database_name>
--Check which queries are running
show full processlist;

--Check innodb status
show engine innodb status;

--Check locked tables
show open tables where in_use > 0;

--check isolation
show variables like '%isolation%';

Look for:

Long-running queries
Table locks or metadata lock waits
Deadlocks in the InnoDB status output

Disk & Memory Checks

Disk usage:

df -h

Memory usage:

free -m

Java memory issues:

Check catalina.out or GC logs for:

java.lang.OutOfMemoryError

Port & Network

Check if key ports are listening:

sudo netstat -tulnp | grep -E '8009|443'

Check firewall:

sudo ufw status

Tips

Always back up config files before making changes.
If unresolved, escalate with logs and above information to OpenSpecimen support.