I had to pour my Death Wish coffee:

Into my 150 Years of Canada Tim Horton’s mug.

Happy Coffee Day! Slurp!
I had to pour my Death Wish coffee:
Into my 150 Years of Canada Tim Horton’s mug.
Happy Coffee Day! Slurp!
Time: 60 minutes
Audience: Developers & DevOps Personnel
Purpose: Introduce basic monitoring using Prometheus so you may apply new skills.
Goals:
Before beginning these exercises, it is assumed that the student has a certain level of familiarity with Linux and the command-line, as well as Git commands.
For this exercise, we will use a version of Linux (CentOS) provided by Hashicorp called a Bento Box. If you have experience with other virtual environments, you can use them.
Regardless of which environment is selected, it is assumed that some Linux command line skills are possessed.
If the selected environment is a physical environment, skip ahead to download the correct version of Prometheus for your OS. If a virtual environment is selected, please follow the following steps.
A host operating system is the OS the virtual environment will run on. The guest is the virtual environment.
vagrant --version
vagrant init bento/centos-7.6
2:vagrant up
vagrant ssh
It is important to put vagrant in front of the ssh command so it finds the key correctly to log in. Feel free to tour around inside the guest.
~/Vagrantfile
on a Mac, or C:\Users\username. Look for the line: # config.vm.network "private_network", ip: "192.168.33.10"
. The comment #
can be removed, or the line retyped. Any valid IP address can be used. vagrant reload
.You will need to add the git application on the virtual machine.
vagrant ssh
$ sudo yum install git -y
To demonstrate the Java metrics and Grafana, we will install a Docker container with some prebaked examples, but to run them, we need to install Docker.
$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2
$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce
$ sudo usermod -aG docker $(whoami)
$ sudo systemctl enable docker.service
$ sudo systemctl start docker.service
For this exercise we will not need Docker Compose.
vagrant plugin install vagrant-scp
Then type: vagrant scp local_file_path_in_HostOS :remote_file_path_in_GuestOS
. For example: vagrant scp prometheus-2.7.2.linux-amd64.tar.gz :~/
(If your guestOS is not running you will get an error message. You can type vagrant up
to start it). tar -zxvf prometheus-2.7.2.linux-amd64.tar.gz
$ ./prometheus
which will spin up a number of messages like:And then type <CTRL>-C to stop it, for now.
Adding Java instrumentation to your code can be managed in a couple of ways. As we discussed in the class, we have the following examples.
import io.prometheus.client.Counter;
class YourClass {
static final Counter requests = Counter.build()
.name("requests_total").help("Total requests.").register();
void processRequest() {
requests.inc();
// Your code here.
}
}
class YourClass {
static final Gauge inprogressRequests = Gauge.build()
.name("inprogress_requests").help("Inprogress requests.").register();
void processRequest() {
inprogressRequests.inc();
// Your code here.
inprogressRequests.dec();
}
}
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import io.prometheus.client.exporter.HTTPServer;
import java.io.IOException;
import java.util.Random;
public class Main {
private static double rand(double min, double max) {
return min + (Math.random() * (max - min));
}
public static void main(String[] args) {
Counter counter = Counter.build().namespace("java").name("my_counter").help("This is my counter").register();
Gauge gauge = Gauge.build().namespace("java").name("my_gauge").help("This is my gauge").register();
Histogram histogram = Histogram.build().namespace("java").name("my_histogram").help("This is my histogram").register();
Summary summary = Summary.build().namespace("java").name("my_summary").help("This is my summary").register();
Thread bgThread = new Thread(() -> {
while (true) {
try {
counter.inc(rand(0, 5));
gauge.set(rand(-5, 10));
histogram.observe(rand(0, 5));
summary.observe(rand(0, 5));
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
bgThread.start();
try {
HTTPServer server = new HTTPServer(8080);
} catch (IOException e) {
e.printStackTrace();
}
}
}
You can also modify the build to include the appropriate repository. For Maven, you edit your POM.XML something like this:
<!-- The client -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Hotspot JVM metrics-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Exposition HTTPServer-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_httpserver</artifactId>
<version>0.6.0</version>
</dependency>
<!-- Pushgateway exposition-->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_pushgateway</artifactId>
<version>0.6.0</version>
</dependency>
For more examples, please consult the Prometheus Java Client page.
A Docker container with the examples above can be installed.
$ git clone https://github.com/sysdiglabs/custom-metrics-examples
$ sudo docker build custom-metrics-examples/prometheus/java -t prometheus-java
Depending on the state of your Docker container, it may rebuild itself while it downloads. This will take a couple of minutes.
$ sudo docker run -d --rm --name prometheus-java -p 8080:8080 -p 80:80 prometheus-java
$ curl localhost:8080
which will give you something like:These are now exporting successfully from the application. Now we need to configure Prometheus.
prometheus.yml
file. This is the core configuration file for Prometheus. This file is YAML (Yet Another Markup Language) and so spaces matter. Because our Java is being exported to port 8080, we will configure Prometheus to listen on 9090.
global:
scrape_interval: 10s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
This is a basic configuration.
global: Defines the global parameters for Prometheus operation. The scrape interval is the time, in seconds, between system scrapes. Prometheus defaults to one minute [1m] as the time between scrapes. For demonstration purposes we will use ten seconds. The actual time should be a balance between need for metrics and load on the system.
scrape_configs: Define job name, labels, and other things that define the scrape. It scrapes http by default, but can scrape other protocols as configured. There is also the ability to pass usernames and passwords if needed. The entire documented suite of variable is available in the Prometheus documents.
For this exercise, we will scrape the localhost, port 9090, every 10 seconds. The job name is prometheus. Save the YAML above as prometheus.yml
, then start Prometheus with $ ./prometheus
.
Open a web browser and point it at: 192.168.33.10:9090
(or the address you configured in Step 10 in setting up Vagrant). This is the main visualization page. From here you can examine a number of default metrics collected by Prometheus. You can verify the host status by clicking on Status | Targets, which will show you:
If you click on the Prometheus name, you will return to the PromQL Expression Browser main screen where you any type any number of PromQL (the Prometheus Query Language) queries. For example, by typing up
will result in a returned value of 1, for up. This is useful for debugging.
If you click on Insert Metric at Cursor, you can get an idea of the default metrics that can be viewed. You will notice there are no Java variables.
To add them:
global:
scrape_interval: 10s
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- localhost:8080
There are hundreds of configuration options for Prometheus, including hosts and other library metrics. For example, if you have number of hosts, you might see a configuration like this:
global:
scrape_interval: 10s
evaluation_interval: 10s
rule_files:
- rules.yml
scrape_configs:
- job_name: prometheus
static_configs:
- targets:
- localhost:9090
- 192.168.1.181:9090
- 192.168.1.183:9090
- 192.168.1.184:9090
- 192.168.1.181:8080
- job_name: node
static_configs:
- targets:
- localhost:9100
- 192.168.1.181:9100
- 192.168.1.183:9100
- 192.168.1.184:9100
This configuration comes from a network with four instances of prometheus, collecting data from the node_exporter module (which exports metrics related to the OS, network, and other node related information. You would get a report like this if we aggregate network interfaces:
The Prometheus documentation provides a number of configuration suggestions, and there is a prometheus channel in Stack Overflow with just as many suggestions. There is also a great book available from O’Reilly’s Safari subscription Prometheus Up and Running.
The Prometheus dashboard is handy for quick sanity checks, but it is not a great solution for long-term use. Enter Grafana, a popular tool for building just these sorts of dashboards.
The easiest way to plug into our existing configuration is to grab a copy of the Grafana Docker console. This will not use a volume mount so it is not a good long term solution.
$ sudo docker run -d --name=grafana --net=host grafana/grafana:5.0.0
Grafana will present itself on port 3000. Log in as admin/admin
java_my_guage
and you will get something like this being pulled from your Prometheus instance. Now that you have an overview of the pieces, take some existing code and modify it to include counters or gauges as appropriate, build it, and then run it where Prometheus can capture the metrics. Once you have done that, validate that they are being picked up by looking in the Prometheus web page and then in Grafana.
$ /home/vagrant/custom-metrics-examples/prometheus/java/src/main/java/Main.java
in the custom-metrics-examples
repository you clonedrate(node_network_receive_bytes_total[1m])
I recently discovered fountain, a markdown syntax specifically designed for script writing. i have done a couple of experiments with it, both quite successful. The real test though is how well it imports into Celtx, my script writing software of choice. And I must say it works pretty well. i had to make a couple of minor corrections with parentheticals. I made a couple of errors when I composed the base document, but that was easily remedied and when I corrected the base document, it imported cleanly.
I have been an avid user of Celtx for years. The product is solid and has a robust user community. Their updates are well thought out and their Studio application/eco-system that supports more than just writing scrips is great for those on a budget. The only negative is that it is web based (with some features available for iOS). In most cases this is not a big show stopper, unless, like me, you are disconnected when you do your primary writing. The iOS apps do allow for off-line editing, but what about when I am using my Linux desktop?
Celtx no longer supports their desktop client (and I never could get it to run on Linux properly), so for these situations, the fountain format is a great find. It has a robust ecosystem around it and is also good for those who are just starting out and looking for an entry into script writing.
Experienced script writers will like a number of power features that allow you to go from treatment to script in the same document, depending on how you post-process it. And because it is an open standard, it allows you to store and reprocess scripts over time. A major plus as the software landscape is always changing.
Another nice feature is the ability to embed script segments into a blog with a nice WordPress plug in. As you can see, it is quite a nice little feature, and no additional work on my part than wrapping the text.
INT. LIVING ROOM – DAY
The door crashes open and BILL TURNER crashes into the room, blood pouring from two gun shot wounds. He is holding his abdomen as he staggers and falls to his knees. ALICE GEORGE is sitting on the couch.
ALICE
(screams)
Who are you! What are you doing here?
BILL
Currently bleeding. Call 9-1-1!
Bill falls to the floor.
If you are not already familiar with it, I encourage you to look into it and see if it fits your needs. I am more than happy with what it provides.
Every now and then, I find a new tool to make my life easier, at least that is the theory. My first new tool was to ditch Microsoft Windows for the MacOS. At least as my primary day-to-day OS. Yes, I spend a large portion of my work day in Linux. At the moment the distribution is Ubuntu, but I spend most of my day staring at a terminal emulator. When I am not doing that, the OS should be something I do not have to think about, and Windows, especially Windows 8, was causing me too much thought. Then with the release of Windows 10 and all the things that are talking back to Microsoft, I decided it was time to try something else. So Mac won, despite the costs.
As many of you know, I have a certain loathing for the Mac. My primary arguement has (and still is) cost. It is just too bloody expensive. It has the advantage of being Unix like under the covers though, and it has a couple of other advantages in terms of photo work that Windows, even with all the RAM I could throw at it, just could not measure up to. So, I bit the bullet and went Mac.
With the conversion, came a couple of new tools, of which this is one – it is a piece of blogging software call Byword, and is a combination text editor, markdown support. It seamlessly connects with my blogging platform (which is good) and supports markdown which is good because more and more of my documents are being written in markdown than they are in anything else. It is more portable and just a better way of doing things. This is the first post with the Byword, and I am doing it more to test out the software than anything else. So here we go.
Hello world!