2017-04-15 - Graphite, Grafana with gunicorn and NGINX
At work I have wanted to implement system monitoring using graphite and grafana. Eventually I will use this particularly to monitor our Lustre storage system with either collectl or collectd shipping Lustre and general host stats to graphite. To try things out I have installed the stack onto my Kimsufi server.
The graphite installation documentation uses apache mod_fcgi, but I'm running various other python projects with gunicorn and then an NGINX or Apache front-end proxy, so will use gunicorn to run the graphite-web app, and NGINX to proxy it and grafana over https.
Installation on CentOS 7
I run CentOS 7 on my Kimsufi 2c server, and the procedure to install graphite, grafana, gunicorn, nginx is as follows:
EPEL repository
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Graphite Dependencies & Graphite Packages
# Distribution packages
yum install -y httpd net-snmp perl python-devel git gcc-c++ pycairo mod_wsgi libffi-devel
yum install -y python-pip node npm
# Python package via pip
pip install django
pip install django-tagging
pip install pytz
pip install Twisted==16.4.1
# Installing graphite from master, as the stable release is quite old at the moment
export PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
pip install --no-binary=:all: https://github.com/graphite-project/whisper/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/carbon/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/graphite-web/tarball/master
Configure Graphite
Copy the example configuration into live locations
sudo cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
sudo cp /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
sudo cp /opt/graphite/conf/graphTemplates.conf.example /opt/graphite/conf/graphTemplates.conf
sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
sudo cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
sudo cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
Configure the storage schema:
vi /opt/graphite/conf/storage-schemas.conf
Add the default retention:
[default]
pattern = .*
retentions = 12s:4h, 2m:3d, 5m:8d, 13m:32d, 1h:1y
Add a user account for graphite:
useradd -d /opt/graphite graphite
chown graphite /opt/graphite -R
Configure the carbon daemon, and start it
vi /opt/graphite/conf/storage-schemas.conf
Make these edits:
USER = graphite
Create system.d unit file /etc/systemd/system/carbon.service
[Unit]
Description = Carbon Metrics store
[Service]
Type = forking
GuessMainPID = false
PIDFile = /opt/graphite/storage/carbon-cache-a.pid
ExecStart = /opt/graphite/bin/carbon-cache.py start
[Install]
WantedBy = multi-user.target
sytemctl daemon-reload
systemctl start carbon
systemctl status carbon
systemctl enable carbon
*** Configure and enable graphite-web
Edit /opt/graphite/webapp/graphite/local_settings.py
SECRET_KEY = '' # Get a random hash to put here
DEBUG = False
Install nginx, gunicorn:
yum install nginx
pip install gunicorn
Setup webapp:
cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/webapp/wsgi.py
cd /opt/graphite/webapp/graphite
sudo -u graphite python manage.py migrate
Create /etc/systemd/system/graphite-web.service:
[Unit]
[Service]
Environment=PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
WorkingDirectory=/opt/graphite/webapp
ExecStart=/usr/bin/gunicorn -u graphite -g graphite -b 127.0.0.1:8080 --log-file=/opt/graphite/storage/log/webapp/gunicorn.log wsgi:application
Restart=on-failure
User=graphite
Group=graphite
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=true
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl start graphite-web
systemctl status graphite-web
systemctl enable graphite-web
NGINX proxy for graphite-web
Setup a self-signed certificat for nginx right now
cd /etc/nginx
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt
Create /etc/nginx/conf.d/graphite.conf
server {
listen 80;
return 301 https://$host$request_uri;
}
server {
listen 443;
server_name graphite.example.com;
ssl_certificate /etc/nginx/nginx-selfsigned.crt;
ssl_certificate_key /etc/nginx/nginx-selfsigned.key;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location /graphite/static/ {
alias /opt/graphite/webapp/content/;
}
location /graphite {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the “It appears that your reverse proxy set up is broken" error.
proxy_pass http://127.0.0.1:8080/graphite;
proxy_read_timeout 90;
proxy_redirect http://127.0.0.1:8080/graphite https://graphite.example.com/graphite;
}
}
Start and open firewall
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --runtime-to-permanent
systemctl start nginx
systemctl enable nginx
Try collectl locally to sanity check the install
Will use collectl running locally on the same server to make sure that data is getting into graphite.
yum install collectl
# Test it
collectl --export graphite,127.0.0.1
# Configure in /etc/collectl.conf
DaemonCommands = -smcdn --export=graphite,127.0.0.1
# Start
systemctl start collectl
systemctl enable collectl
Now browse to ~https://localhost/graphite~ and check that data is making it into graphite, and the graphite-web interface is working OK.
** Install grafana
Grafana is a nicer dashboard front-end than the default graphite-web, so we will use that by default.
yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.2.0-1.x86_64.rpm
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
systemctl enable grafana-server
Add nginx proxy setup to /etc/nginx/conf.d/graphite.conf
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the “It appears that your reverse proxy set up is broken" error.
proxy_pass http://127.0.0.1:3000/;
proxy_read_timeout 90;
}
systemctl restart nginx
Configure Grafana
- Browse to the host URL and login with default user/password admin:admin
- Add data source
- Name graphite, type graphite, URL http://localhost:8080/graphite, access=proxy