viewBox="0 0 51 48">Five Pointed Star Five Pointed Star

Docker Logging and Monitoring in a Development Environment

Concept

We are going to run a Laravel app in docker containers inside a Virtual Environment on Ubuntu 16.04 Xenial. So Ubuntu 16.04 "headless" in a Virtual Machine the host OS. The choice of Laravel is arbitrary for this proof of concept, any containerized application will work. Salient system logs and Docker metrics will be extracted for analysis. Complete separation of concerns will be maintained by each service and app component running in a separate container. Fluentd is the only package that will be installed on the virtualized host OS.


Monitoring Opinion

It's my decision to use Kibana/Elasticsearch combination (Kibana as a frontend to Elasticsearch data) to monitor system log messages, and the stellar Grafana for docker container/system monitoring (Utilizing InfluxDB with a cAdvisor database as the data source).

So to summarize, we will look for application/server errors in Kibana, and monitor the Docker ecosystem in Grafana/cAdvisor. Alright, let's rock it out, this is long road to travel, but worth it in the end.


Development Setup

I chose to run Ubuntu 16.04 in a Vagrant Box virtualized in Oracle's VirtualBox. We will be running Laravel 5.3 on an Apache web server with PHP7, MySQL and phpMyAdmin access.


Environment Setup Host OS
PHP7
sudo apt-get update  
sudo apt-get install php7.0 php7.0-mbstring php-xml php7.0-mysql  



Composer

Download Composer

or just install globally

cd ~/Downloads  
curl -sS https://getcomposer.org/installer | php  
mv composer.phar /usr/local/bin/composer  



Laravel App Setup

Change directory to where you want to store your Laravel app's source code.
You can check packagist … Laravel on Packagist
We will create an app called laravel5-project.

composer create-project laravel/laravel laravel5-project  
php artisan key:generate  

You should see something like this after generating the app key:

Application key [base64:jpE7*************************xfFA=] set successfully.  

Copy the key to the clipboard — the string inside the brackets.

Open the project in Atom, Sublime Text, PhpStorm, or whatever your preferred IDE/editor is.
Configure the .env file pasting your key, setting debugging to true, and editing the database connection for the MySQL settings we will be using.

APP_ENV=local  
APP_KEY=base64:jpE7*****************************xfFA=  
APP_DEBUG=true  
APP_LOG_LEVEL=debug  
APP_URL=http://192.168.33.10

DB_CONNECTION=mysql  
DB_HOST=mysql  
DB_PORT=3306  
DB_DATABASE=laradock  
DB_USERNAME=homestead  
DB_PASSWORD=secret

BROADCAST_DRIVER=log  
CACHE_DRIVER=file  
SESSION_DRIVER=file  
QUEUE_DRIVER=sync

REDIS_HOST=127.0.0.1  
REDIS_PASSWORD=null  
REDIS_PORT=6379

MAIL_DRIVER=smtp  
MAIL_HOST=mailtrap.io  
MAIL_PORT=2525  
MAIL_USERNAME=null  
MAIL_PASSWORD=null  
MAIL_ENCRYPTION=null

PUSHER_APP_ID=  
PUSHER_KEY=  
PUSHER_SECRET=

Now for some git source control. Change directory in the terminal to your project source directory. Go to github/bitbucket/wherever and initialize a new repository to store your code. Create an appropriate .gitignore file.

Example .gitignore for PhpStorm Laravel project:

# Created by .ignore support plugin (hsz.mobi)
# IntelliJ project files
.idea
*.iml
out  
gen  
# Laravel
vendor  
node_modules  
public/storage  
storage/*.key  
.env

Do not ever commit your .env file to source control!

git init  
git remote add origin <repository ssh or https address>  
git add --all  
git commit -am "Initial"  
git push origin master  
git submodule add https://github.com/LaraDock/laradock.git  
git add --all  
git commit -am "added Laradock"  
git push origin master  



VirtualBox & Vagrant Installation

Download VirtualBox
For Ubuntu 16.10 "Yakkety", I downloaded the 64 bit .deb package, and double clicked to install via dpkg.

Download Vagrant

At the terminal:

cd ~  
mkdir -p vagrant/ubuntu16.10  
cd vagrant/ubuntu16.10  
vagrant init ubuntu/xenial64; vagrant up --provider virtualbox  
vim Vagrantfile  

Uncomment/add/edit the following lines:

# Create a private network, which allows host-only access to the machine
# using a specific IP.
   config.vm.network "private_network", ip: "192.168.33.10"

. . . 

# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
   config.vm.synced_folder "<path to your Laravel project>", "/home/ubuntu/src/laravel5-project", type: "nfs", create: "true"

. . . 

# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
   config.vm.provider "virtualbox" do |vb|
#   # Display the VirtualBox GUI when booting the machine
#   vb.gui = true
#
#   # Customize the amount of memory on the VM:
   vb.memory = 4096
   vb.cpus = 2
 end

I chose to give 4gb of my 16gb of RAM, and 2 of 8 cores to the virtual machine (VM).
Load the Vagrant box after installing nfsd.

sudo apt-get install nfs-kernel-server  
cd <Vagrantfile location>  
vagrant up  



Docker Setup

Docker and Docker Compose installation on VM
cd <Vagrantfile location>  
vagrant ssh

sudo apt-key adv \  
               --keyserver hkp://ha.pool.sks-keyservers.net:80 \
               --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

# If the above keyserver is not available, try hkp://pgp.mit.edu:80 or 
# hkp://keyserver.ubuntu.com:80

echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" | sudo tee /etc/apt/sources.list.d/docker.list

sudo apt-get update  
apt-cache policy docker-engine  

You should get the following output to verify you are pulling from the correct repo.

 docker-engine:
    Installed: 1.12.2-0~trusty
    Candidate: 1.12.2-0~trusty
    Version table:
   *** 1.12.2-0~trusty 0
          500 https://apt.dockerproject.org/repo/ ubuntu-trusty/main amd64 Packages
          100 /var/lib/dpkg/status
       1.12.1-0~trusty 0
          500 https://apt.dockerproject.org/repo/ ubuntu-trusty/main amd64 Packages
       1.12.0-0~trusty 0
          500 https://apt.dockerproject.org/repo/ ubuntu-trusty/main amd64 Packages

Enable use of the aufs storage driver on Docker

sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual

#install Docker
sudo apt-get install docker-engine  
sudo service docker start

#add your user to the docker group so you can run without sudo privileges
sudo usermod -aG docker $USER

#start Docker on boot
sudo systemctl enable docker

#install docker-compose
curl -L "https://github.com/docker/compose/releases/download/1.9.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

#test installation
docker-compose --version  



Docker Containerization of Laravel App

We aren't going to do throw-away command line docker run commands. We want to this to be reproducible and deployable to a production environment at some point, so we will be creating docker-compose yaml files and using docker-compose to control the containers.

Change directory to your app in the virtual box, and then into the Laradock subdirectory.

vim docker-compose.yml  

Here's a file dump for docker-compose. Basically, the parts we are changing are the docker host address and the logging driver for docker containers for web servers apache, nginx, MySQL, and php-fpm (for now). We are also going to bump phpMyAdmin's port from 8080 to 8081. We could send the logs to fluentd from any container we want to, this is just what we are going to use right now (quick & dirty). You could setup logging with this by using Laradock's elasticsearch container, but we are not going to. There is more than one way to a logging solution, and this is a stellar boilerplate for app development in a Docker environment. I choose to just use this as an app setup template.

Watch your indentation levels in YAML files. Use lorry.io to validate your docker-compose and other YAML config files if necessary.

So, there are a ton of cool services in this file. For Laravel we only need a web server, a database, a database management interface (optional), and PHP. So making sure we are in the same directory as the Laradock docker-compose.yml file, we bring it up like this:

docker-compose up -d apache2 php-fpm mysql phpmyadmin  

Once the containers are loaded …

docker ps

#should show something like this
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                                NAMES

c41c6c95a119        laradock_apache2      "/opt/docker/bin/entr"   13 hours ago        Up 13 hours         0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                             laradock_apache2_1  
42b3ec837277        laradock_php-fpm      "php-fpm"                13 hours ago        Up 13 hours         9000/tcp                                                             laradock_php-fpm_1  
c99bd1b82d02        laradock_phpmyadmin   "/run.sh phpmyadmin"     13 hours ago        Up 13 hours         0.0.0.0:8081->80/tcp                                                 laradock_phpmyadmin_1  
0ac67d490d22        laradock_workspace    "/sbin/my_init"          13 hours ago        Up 13 hours         0.0.0.0:2222->22/tcp                                                 laradock_workspace_1  
2d9f1a685876        laradock_mysql        "docker-entrypoint.sh"   13 hours ago        Up 13 hours         0.0.0.0:3306->3306/tcp                                               laradock_mysql_1  



Monitoring and Logging
Fluentd install on VM

I like to make a directory tmp in my home directory for these kind of install scripts, wgets, and curl requests.

cd ~  
mkdir tmp  
cd tmp

\curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh -o install-td-agent.sh

# examine script because trust no one
vim install-td-agent.sh

# if comfortable with it, then run the shell script
sh install-td-agent.sh

# start service
sudo systemctl start td-agent

# sanity check
tail /var/log/td-agent/td-agent.log  

You should see output like this:

    port 8888
  </source>
  <source>
    @type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
</ROOT>  
2017-01-16 09:24:15 +0000 [info]: listening fluent socket on 0.0.0.0:24224  
2017-01-16 09:24:15 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"  

Now we install elasticsearch as a plugin to fluentd.

sudo td-agent-gem install fluent-plugin-elasticsearch  
Fluentd can also be installed as a ruby gem if you have a ruby environment configured. Also, elasticsearch goodness.
gem install fluentd --no-rdoc --no-ri  
gem install fluentd-plugin-elasticsearch --no-rdoc --no-ri  
Configure Fluentd
sudo vim /etc/td-agent/td-agent.conf  

There is a source section with @type forward. Make it like so by adding the port line:

<source>  
  @type forward
  port  24224
  bind  0.0.0.0
</source>  

Add this section:

<match docker.**>  
  @type elasticsearch
  logstash_format true
  host 127.0.0.1
  port 9200
  flush_interval 5s
</match>  

Back at the shell run:

sudo systemctl restart td-agent

# increase map count for elasticsearch
sudo sysctl -w vm.max_map_count=262144  

I'd recommend making a docker subdirectory in your home directory. So this is what I did …

cd ~  
mkdir -p docker/monitoring  
cd docker/monitoring  
vim docker-compose.yml  

Create the docker-compose.yml file for monitoring and logging services.

Finally bring up this monitoring/logging cluster of docker containers:

docker-compose up -d  



Viewing Results

Laravel app: http://192.168.33.10/

Laravel App

InfluxDB: http://192.168.33.10:8083

InfluxDB

Use dropdown to select the cAdvisor database and make sure the settings have your login (default for this article = root/root)

Kibana: http://192.168.33.10:5601/

Kibana

This is where Apache and PHP error/warning/access logs go. Nice interface with the search functionality.


Google cAdvisor: http://192.168.33.10:8080

cAdvisor

Grafana: http://192.168.33.10:3000/

Grafana

My Grafana Dashboard JSON for import

If you are short on time and want to see major metrics on the container system quickly, you can import this JSON via copy and paste in Grafana's interface.

Dashboard JSON

TODO

  • use LetsEncrypt to encrypt app and monitoring system
  • research best practices on securing or privatizing a network to access monitoring dashboards
  • define a practical deployment strategy
  • implement a test suite for application(s) in the docker "swarm" (bring all production apps into the mix) and therefore determine system parameters necessary for a cloud platform implementation in "production"
  • further enhance visualizations of metrics using visual tools (Grafana, Kibana)
  • keep modularity and plan an upgrade strategy
  • use docker container commits to a repository on Docker Hub (when production-ready)
  • use docker volumes over host OS directory injection

Sources

Laradock
DigitalOcean - Centralization of Docker Logs


L. Ball
L. Ball

Father. Developer. Coffee Connoisseur. Amateur Guitarist.