�ݺ�ߣ

Hi, my name is Dave.
• DevOps Engineer at Bazaarvoice.
• Started working with puppet in 2008 while working at
Bioware. First version was 0.24.
• At Bioware, puppet configured over 14k nodes that
comprised of web servers, databases and game
servers.
• All 5 datacenters (in California, Virginia, Ireland,
Australia and Texas) housed puppet managed nodes
that all reported back to a centralized puppet
dashboard.
• My contact info:
– http://www.linkedin.com/in/jamesbarcelo

Bazaarvoice Plug
• We do embedded DevOps!
• Application teams are responsible for their
application’s operation success.
• 2.0 stack is 100% in Amazon!
• Conferences!
• Work on awesome projects with spiffy tech like
Cassandra or ElasticSearch.
• Send me your resume!
Dave.barcelo@bazzarvoice.com

Preview presentation
• Puppet in the legacy stack.
• Puppet in the Data Infrastructure Team.
– Focus on client/server.
• Puppet in the Data Services Team.
– Focus on masterless puppet.

Puppet in the Legacy Infrastructure
• Traditional puppet use with client/server.
• Multiple levels of inheritance.
• Puppet managed instances were configured
according to DNS naming convention.

node /my-hostname/ {
…
…
}

• Some issues encountered:
– Very hard to work with. Very complex.
– Large codebase. Adds to complexity.
– No confidence in making changes. Side effects
feared after code change. A jinga tower of puppet
code.
– Too many pivot points. Many places to configure.
Adds to complexity.
– Lots of code rot. Had not been refactored.

Puppet master/client in Data
Infrastructure teams

Architecture
• Each server type we care about will be referenced by
its role. We only care about roles, not hostnames.
• Centered around an uber IT tools server that runs
everything ops (including puppet) to do work in an
environment. The Mothership.
• Hiera and parameterized classes will be used to create
generic puppet modules that can be reused for
different roles.
• Development will be centered on using puppet
environments on the Mothership to protect devs from
stepping on each other.

Mothership
• Contains a cocktail of different application tools
for doing work in the environment. Tools
included:
– Mcollective/ActiveMQ
– Puppet server
– Puppet managed operation scripts.
• Motherships configured to be highly available in
regular AWS fashion(Autoscaling, cluster multiple
activeMQ, etc).
• Advertises multiple puppet environments that
clients can switch between via –environments.

Methods of passing in data
Getting environment data into puppet
configuration.
• Hiera datastore.
• Puppet stdlib/tags.txt
• Cloud formation parameters – Universe, VPC

Puppet Stdlib/facts.d
• Bootstrap process (EC2 user data) populates
/etc/facter/facts.d/tags.txt with mappings. These
mappings become facters.
• Example of data in tags.txt:
– Universe value.
– Ec2 metadata.

/etc/facter/facts.d/tags.txt:
universe=dev
Tag_region=us-east-1

Hiera datastore
• Hiera is used extensively where different data
needs to be passed into puppet according to
context. Different contexts would include:
– Amazon region.
– Instance role.
– Universe.
• Example usage:
$app_version = hiera(‘app_version’, nil)

/etc/hiera.yaml
--
:logger: console

:hierarchy:

- %{fqdn}
- 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/10-roles/%{role}
- 10-team/%{team}/10-region/%{tag_region}/10-universe/%{universe}/20-common
- 10-team/%{team}/10-region/%{tag_region}/20-roles/%{role}
- 10-team/%{team}/10-region/%{tag_region}/30-common
- 10-team/%{team}/20-universe/%{universe}/10-roles/%{role}
- 10-team/%{team}/20-universe/%{universe}/20-common
- 10-team/%{team}/30-roles/%{role}
- 10-team/%{team}/40-common

- 40-common

- environments

:backends: - yaml
- json

:yaml:
:datadir: /etc/puppet_env/%{environment}/manifests/hieradata

:json:
:datadir: /etc/puppet/env/global_hieradata

Puppet code on the Mothership
• The files that do the things:
– /etc/puppet/puppet.conf
– /etc/puppet/env/global_hieradata/environments.j
son
– /etc/puppet/puppetmaster.conf
– /etc/puppet_env/{puppet_env}/…
• manifests/…
• modules/…

/etc/puppet/puppet.conf
# File managed by Puppet. ssldir = $vardir/ssl

[main] [agent]
vardir = /var/lib/puppet runinterval = 1800
logdir = /var/log/puppet ca_server = <%= ca_srv %>
rundir = /var/run/puppet server = <%= logical_srv %>
ssldir = $vardir/ssl certificate_revocation = False
environment = <%= environment %>
usecacheonfailure = true report = true
pluginsync = true
factpath = $vardir/lib/facter
preferred_serialization_format = yaml

[user]
vardir = /var/lib/puppet
logdir = /var/log/puppet
rundir = /var/run/puppet

/etc/puppet/env/global_hieradata/env
ironments.json
{
"environments": [
{
"cert": [
{ "modulepath": "/etc/puppet_env/cert/modules" },
{
"manifestdir": "/etc/puppet_env/cert/manifests"
},
{ "manifest": "/etc/puppet_env/cert/manifests/site.pp"
}
]
}
}

/etc/puppet/puppetmaster.conf
[main] # Puppetdb.
vardir = /var/lib/puppet storeconfigs = true
logdir = /var/log/puppet storeconfigs_backend = puppetdb
rundir = /var/run/puppet <% end %>
ssldir = $vardir/ssl
[user]
usecacheonfailure = true vardir = /var/lib/puppet
pluginsync = true logdir = /var/log/puppet
factpath = $vardir/lib/facter rundir = /var/run/puppet
preferred_serialization_format = yaml ssldir = $vardir/ssl
syslogfacility = local1
# Environments
[master] <% environments.each do |env_val| -%>
certname=<%= certname %> <% env_val.keys.each do |env_key| -%>
ca = True [<%= env_key -%>]
certificate_revocation=False <% env_val[env_key].each do |env_data| -%>
dns_alt_names=<%= logical_srv %> <% env_data.each_pair do |k, v| -%>
ssl_client_header = SSL_CLIENT_S_DN <%= k %> = <%= v -%>
ssl_client_verify_header = SSL_CLIENT_VERIFY <% end %>
autosign = true <% end %>
<% end %>
# For puppet dashboard reporting. <% end %>
reports = store, datadog_reports

<% if store_configs == true %>

/etc/puppet_env/{env}/manifests/site.
pp
import 'roles/*.pp'

node default {
class { 'basenode_role': }

class { "$tag_role": }
}

/etc/puppet_env/{env}/manifests/role
/00_basenode.pp
class basenode_role {
class { ‘security’: }
class { ‘monitoring’: }
…..
}

/etc/puppet_env/{env}/manifests/role
/mothership_role.pp
class mothership_role {

class { 'puppet':
master => true,
ca_srv => $tag_caserver,
logical_srv => $tag_puppet_server,
}

}

Masterless Puppet in Data Services
Teams

Architecture
Still keeping bits of the Mothership project:
• Applications/Services scoped in zookeeper by
Universe.
• Emphasis is put on making things simpler.
Puppet code will not be monolithic. Individual
application teams will only need to maintain
there own modules/manifests.
• Changes to modules/manifests will not impact
other teams.

Methods of passing in data
• The usual suspects:
– Puppet stdlib/tags.txt.
– Hiera.
– Cloud formation parameters – Universe, VPC
• Some new ones:
– EC2 data/metadata -> facter.
– Zookeeper.
– Cloud formation parameters - DeployTag

getEC2data_cache.rb
• Script runs out of /etc/facts.d that converts
EC2 data/metadata into facts.

Zookeeper/Ostrich
• Custom functions to pull data from zookeeper
the same way applications do discovery.

Puppet code in Masterless
• No more Mothership. All work is done via
puppet apply.
– /etc/hiera.yaml
– /etc/puppet/manifests/{role}.pp
– /etc/puppet/manifests/00_common.pp
– /etc/puppet/manifests/01_users.pp
– /etc/puppet/modules/…

/etc/hiera.yaml
--
:logger: console

:hierarchy: - %{fqdn}

- 10-universe/%{universe}/10-roles/%{role}
- 10-universe/%{universe}/20-common
- 20-roles/%{role}
- 30-common

:backends: - yaml

:yaml:
:datadir: /etc/puppet/manifests/hieradata

/etc/puppet/manifests/{role}.pp
import '00_common'

node default {

# This class contains common modules that should be used by all roles.
class { 'common': }

class { 'activemq’: }
->
class { 'mcollective':
server => true,
client => true,
}

}

/etc/puppet/manifests/00_common.p
p
import '01_users'
#####################################
# Common
#####################################

class common {

class { 'stdlib': }

file { '/opt/bazaarvoice': ensure => directory, }

# Authorized keys for project developers.
class { 'user_setup': stage => setup, }

host { 'internal_ip':
ensure => 'present',
name => $fqdn,
ip => $ipaddress,
}

class { 'prompt': }
}

/etc/puppet/manifests/01_users.pp
class user_setup {

include users

users::user { 'dbarcelo':
groups => 'wheel',
sshKey => 'ssh-dss
AAAAB3NzaC1kc3MAAACBANL1zoZdYJp/6vQ4G5iNQXjdJ7NGmK0J2eqHbztvuD0CBPyqMuEtuYKRg14tFd4iwp5EpnT4UWpv8kXF/dkEN3b5xgN/R+1hYq7/3m
nRLchMFTl0tyryLuARC9zTI003mQrXd/W9jzXaNlCTpxh8Ihj2Ov3lvAAX65tN9nijxhCTAAAAFQCgMU0obmTLo5CRYtRwDCkj1mb2hQAAAIAiZF9axkCvMa9vwigD
iAf3rNMbut1gtqtwdzux8c9T1inApKV5sccjg5POKm+4WmWTBOtQfYR8cNot2Mn/mO+MRiKH8sYapYnU2es+KRBmhdARE+N7EqdD0WqoP7NrsNVbObHwDQB
NkODuc3ZPyTQuqv/w4poTXaS5u5M1XZbgZwAAAIEAjt4r7SN1I/m0V/TvedTVxJvKln4wZkFxyI5CAgpsAr435kwSLM08R9Hd0/5Vy9LfhYpH1aZTBaoTqmTCtnv3
mp1coXoscEp5nE0llfm+4DX3YvWnR80S/OeMUe71Ucm1ORwFpST/K4WKQoZ30TAVVsc8nYy2hyD7hyozjzsS09o= dave.barcelo@dbarcelo-mbpro'
}
users::user { 'lwadhwani':
groups => 'wheel',
sshKey => 'ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEAysFCPpffw9LIOqAEFZxOOb52m2FbHhumBFc07o8sm3c4cmdLq/bBtr5TyuQp89zVNEaTGRbw1nMpQCDno4i5ipTvCLoKk
OE1PRdtyJw6PGu6VV/0U1ghK+1xmveM2jDX/otj5hjnQiRm1+Fx/orYwNBkywDlDHZQCGxalWaFgXVyReCRUqq0jBwj3EKJfsQgoxuSrh7F6GjsQ6DUOsA3wBfew
S25hPmhulEqvga4/P58BMHemL9d4Ugu98Vg7fgaur/b1adX+LzbmE6C2T4Gn1kzAOEct6bFgLPRj3n5/EaspdOsZ/Nnik0LUvIwZNHgDCLgkS0D8aMIsiUrB4OqSw
== luveen@Pantalaimon'
}
}

/etc/puppet/modules/…
• Do stuff!
• Code is still generic but it does not have to be.

�ݺ�ߣ

Puppet atbazaarvoice

Recommended

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Puppet atbazaarvoice (20)

Puppet atbazaarvoice

Editor's Notes