Prometheus is a metric collection (scraping), alerting engine, time-series database, and visualisation all-in-one used by some of the largest companies in the world.
I like to use it for its main purpose Metric (data) collection
benifits of Prometheus is its plantory scale think just like DNS
The above is a over simplification but you get the point.
If you want reliance just use two nodes scraping the same device yes this gets double the number of data points but is how you interpret the data that’s important not the raw data collected, as a friend of mine keeps reminding me one = none, two = one, build reliable systems but don’t fall into the trap of NASA why build one when you can build two for twice the price.
Prometheus scrapes metrics on a time interval that you set in a simple configuration file, now you are thinking how to get the data into Prometheus you need to present the data in a format that is compatible to do this the company and community have created node exporters for the majority of enterprise solutions.
Installing
Prometheus on Docker is as simple as the below but sometimes we dont want to do it in docker !?
docker run -p 9090:9090 prom/prometheus. This starts Prometheus with a sample configuration and exposes it on port 9090.
Installing in Ubuntu
# Add the GPG key
wget https://s3-eu-west-1.amazonaws.com/deb.robustperception.io/41EFC99D.gpg | sudo apt-key add -
# Update and install Prometheus
sudo apt-get update -y
sudo apt-get install prometheus prometheus-node-exporter prometheus-pushgateway prometheus-alertmanager -y
# Start Prometheus service and enable it to start on turn on
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheusnow go to the website http://X.X.X.X:9090 this should load welcome to Prometheus.
Configuration
Like most things in Linux world its a simple config file, you need to update with parameters and restart the service.
# default configuration file location
cat /etc/prometheus/prometheus.yml
# edit with your favorite text editor
vi /etc/prometheus/prometheus.yml
# restart the service
sudo systemctl restart prometheus
# check the status of the service
sudo systemctl status prometheus
What Next with Prometheus
TLS need to secure some of these services or do we if you’re in the trusted network and the data is not being used for customer-facing why to add the extra overhead and hassle or are you just being lazy.
Add a node firewall there are a lot of ports we are opening which I am not sure I want to be open for everyone so we need to lock it down via configuration and \ or a firewall.
Switch up the local node exporter to Netdata as you then have two useful dashboards and will help you with how you want to start graphing
Graphing (Grafana) yes this is the easiest to use I find there will be follow up blogs posts on all this stuff just give me time,
External storage of the data Elastic search or OenTSDB
Warning if your scrape interval is to short the amount of data you are going to collect and put over the network gets large very large
I know you think 14KB for a metrics text page isn’t a lot well that is the default metric page and is just one device
14 Kilobyte = 0.112 Megabit
if we scrape that every second the network throughput would be
0.112 Mbps
times that by 100 devices = 11.2 Mbps this is with the default amount of metrics to be collected. just be careful.