Deploying Kibana Using Nginx as an SSL Proxy
In my last post, I described how I use Packer and Terraform to deploy an ElasticSearch cluster. In order to make the logs stored in ElasticSearch searchable, I use Kibana. I follow the previous pattern and deploy Kibana using Packer to build an AMI and then create the infrastructure using Terraform. The Packer template has already taken into account that I want to use nginx as a proxy.
###Building Kibana AMIs with Packer and Ansible
The template looks as follows:
{% raw %}
{
"variables": {
"ami_id": "",
"private_subnet_id": "",
"security_group_id": "",
"packer_build_number": "",
},
"description": "Kibana Image",
"builders": [
{
"ami_name": "kibana-{{user `packer_build_number`}}",
"availability_zone": "eu-west-1a",
"iam_instance_profile": "app-server",
"instance_type": "t2.small",
"region": "eu-west-1",
"run_tags": {
"role": "packer"
},
"security_group_ids": [
"{{user `security_group_id`}}"
],
"source_ami": "{{user `ami_id`}}",
"ssh_timeout": "10m",
"ssh_username": "ubuntu",
"subnet_id": "{{user `private_subnet_id`}}",
"tags": {
"Name": "kibana-packer-image"
},
"type": "amazon-ebs"
}
],
"provisioners": [
{
"type": "shell",
"inline": [ "sleep 10" ]
},
{
"type": "shell",
"script": "install_dependencies.sh",
"execute_command": "echo '' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'"
},
{
"type": "ansible-local",
"playbook_file": "kibana.yml",
"extra_arguments": [
"--module-path=./modules"
],
"playbook_dir": "../../"
}
]
}
{% endraw %}
The install_dependencies.sh script is as described previously
The ansible playbook for Kibana looks as follows:
{% raw %}
- hosts: all
sudo: yes
pre_tasks:
- ec2_tags:
- ec2_facts:
roles:
- base
- kibana
- reverse_proxied
{% endraw %}
The playbook installs a base role for all the base pieces of my system (e.g. Logstash, Sensu-client, prometheus node_exporter) and then proceeds to install ElasticSearch.
The Kibana role looks as follows:
{% raw %}
- name: Download Kibana
get_url: url=https://download.elasticsearch.org/kibana/kibana/kibana-{{ kibana_version }}-linux-x64.tar.gz dest=/tmp/kibana-{{ kibana_version }}-linux-x64.tar.gz mode=0440
- name: Untar Kibana
command: tar xzf /tmp/kibana-{{ kibana_version }}-linux-x64.tar.gz -C /opt creates=/opt/kibana-{{ kibana_version }}-linux-x64.tar.gz
- name: Link to Kibana Directory
file: src=/opt/kibana-{{ kibana_version }}-linux-x64
dest=/opt/kibana
state=link
force=yes
- name: Link Kibana to ElasticSearch
lineinfile: >
dest=/opt/kibana/config/kibana.yml
regexp="^elasticsearch_url:"
line='elasticsearch_url: "{{ elasticsearch_url }}"'
- name: Create Kibana Init Script
copy: src=initd.conf dest=/etc/init.d/kibana mode=755 owner=root
- name: Ensure Kibana is running
service: name=kibana state=started
{% endraw %}
The reverse_proxied ansible role looks as follows:
{% raw %}
- name: download private key file
command: aws s3 cp {{ reverse_proxy_private_key_s3_path }} /etc/ssl/private/{{ reverse_proxy_private_key }}
- name: private key permissions
file: path=/etc/ssl/private/{{ reverse_proxy_private_key }} mode=600
- name: download certificate file
command: aws s3 cp {{ reverse_proxy_cert_s3_path }} /etc/ssl/certs/{{ reverse_proxy_cert }}
- name: download DH 2048bit encryption
command: aws s3 cp {{ reverse_proxy_dh_pem_s3_path }} /etc/ssl/{{ reverse_proxy_dh_pem }}
- name: certificate permissions
file: path=/etc/ssl/certs/{{ reverse_proxy_cert }} mode=644
- apt: pkg=nginx
- name: remove default nginx site from sites-emabled
file: path=/etc/nginx/sites-enabled/default state=absent
- template: src=nginx.conf.j2 dest=/etc/nginx/nginx.conf mode=644 owner=root group=root
- service: name=nginx state=restarted
- file: path=/var/log/nginx
mode=0755
state=directory
{% endraw %}
This role downloads a private SSL Key and a Certificate from a S3 bucket that is security controlled through IAM. This allows us to configure nginx to act as a proxy. The nginx proxy template is available to view.
We can then pass a number of variables to our role for use within ansible:
{% raw %}
reverse_proxy_private_key: mydomain.key
reverse_proxy_private_key_s3_path: s3://my-bucket/certs/mydomain/mydomain.key
reverse_proxy_cert: mydomain.crt
reverse_proxy_cert_s3_path: s3://my-bucket/certs/mydomain/mydomain.crt
reverse_proxy_dh_pem_s3_path: s3://my-bucket/certs/dhparams.pem
reverse_proxy_dh_pem: dhparams.pem
proxy_urls:
- reverse_proxy_url: /
reverse_proxy_upstream_port: 3000
kibana_version: 4.1.0
elasticsearch_url: http://myes.com:9200
{% endraw %}
This allows me to easily change the configuration of nginx to patch security vulnerabilities easily.
###Deploying Kibana with Terraform
The infrastructure of the Kibana cluster is now pretty easy. The Terraform script now looks as follows:
resource "aws_security_group" "kibana" {
name = "kibana-sg"
description = "Kibana Security Group"
vpc_id = "${aws_vpc.default.id}"
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = ["${aws_security_group.kibana_elb.id}"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = ["${aws_security_group.kibana_elb.id}"]
}
egress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags {
Name = "Kibana Node"
}
}
resource "aws_security_group" "kibana_elb" {
name = "kibana-elb-sg"
description = "Kibana Elastic Load Balancer Security Group"
vpc_id = "${aws_vpc.default.id}"
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = "0"
to_port = "0"
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags {
Name = "Kibana Load Balancer"
}
}
resource "aws_elb" "kibana_elb" {
name = "kibana-elb"
subnets = ["${aws_subnet.primary-private.id}","${aws_subnet.secondary-private.id}","${aws_subnet.tertiary-private.id}"]
security_groups = ["${aws_security_group.kibana_elb.id}"]
cross_zone_load_balancing = true
connection_draining = true
internal = true
listener {
instance_port = 443
instance_protocol = "tcp"
lb_port = 443
lb_protocol = "tcp"
}
listener {
instance_port = 80
instance_protocol = "tcp"
lb_port = 80
lb_protocol = "tcp"
}
health_check {
healthy_threshold = 2
unhealthy_threshold = 2
interval = 10
target = "TCP:443"
timeout = 5
}
}
resource "aws_launch_configuration" "kibana_launch_config" {
image_id = "${var.kibana_ami_id}"
instance_type = "${var.kibana_instance_type}"
iam_instance_profile = "app-server"
key_name = "${aws_key_pair.terraform.key_name}"
security_groups = ["${aws_security_group.kibana.id}","${aws_security_group.node.id}"]
enable_monitoring = false
root_block_device {
volume_size = "${var.kibana_volume_size}"
}
lifecycle {
create_before_destroy = true
}
}
resource "aws_autoscaling_group" "kibana_autoscale_group" {
name = "kibana-autoscale-group"
availability_zones = ["${aws_subnet.primary-private.availability_zone}","${aws_subnet.secondary-private.availability_zone}","${aws_subnet.tertiary-private.availability_zone}"]
vpc_zone_identifier = ["${aws_subnet.primary-private.id}","${aws_subnet.secondary-private.id}","${aws_subnet.tertiary-private.id}"]
launch_configuration = "${aws_launch_configuration.kibana_launch_config.id}"
min_size = 2
max_size = 100
health_check_type = "EC2"
load_balancers = ["${aws_elb.kibana_elb.name}"]
tag {
key = "Name"
value = "kibana"
propagate_at_launch = true
}
tag {
key = "role"
value = "kibana"
propagate_at_launch = true
}
tag {
key = "elb_name"
value = "${aws_elb.kibana_elb.name}"
propagate_at_launch = true
}
tag {
key = "elb_region"
value = "${var.aws_region}"
propagate_at_launch = true
}
}
This allows me to scale my system up or down just by changing the values in my Terraform configuration. When the instances are instantiated, the Kibana instances are added to the ELB and they are then available to serve traffic