Ansible Best Practices

CONFIDENTIAL DesignatorAnsible Best PracticesHow to write, how to execute, and how to use in real life1

GENERAL TIPS TO USE ANSIBLEHow to use2

AUTOMATION IS CODETreat your Ansible content like code3 Version control your Ansible content Iterate Start with a basic playbook and static inventory Refactor and modularize later

CODE NEEDS TO HAVE STYLE GUIDELINESDo it with style Create a style guide for consistency: Tagging Whitespace Naming of Tasks, Plays, Variables, and Roles Directory Layouts Enforce the style Nice example: openshift-ansible Style Guideexample: https://goo.gl/JfWBcW4

CODE MUST BEORGANIZEDUSE GIT!

Do it with stylesite.yml# master playbook, calling otherswebservers.yml# playbook for webserver tierdeployonce.yml# separate playbook for single-shot tasksinventories/production/hosts# different stages via inventory# inventory file for production serversgroup vars/host vars/london/# additional, alternative grouping if usefulroles/requirements.yml# includes roles from some other placecommon/# base line, company wide configurationwebtier/6

GIT - ONE OR MANY?Start with one Git repository - but when it grows,use multiple!At the beginning: put everything in one Git repositoryIn the long term: One Git repository per role Dedicated repositories for completely separated teams / tasksNew to git? Get your cheat sheet here: https://opensource.com/downloads/cheat-sheet-git7

SO, WHAT DOWE HAVE?

USE READABLE INVENTORY NAMESGive inventory nodes human-meaningful names rather thanIPs or DNS b2db3db4ansible host 10.1.2.75ansible host 10.1.5.45ansible host 10.1.4.5ansible host .comw19304.acme.comweb1web2web3web4ansible host w14301.acme.comansible host w17802.acme.comansible host w19203.acme.comansible host w19203.acme.com

TAKE ADVANTAGE OF GROUPINGGroup hosts for easier inventory selection and lessconditional tasks -- the more the 3[prod]db2web2db4web4

COMBINE ALL INVENTORY SOURCESUse dynamic sources where possible. Either as a singlesource of truth - or let Ansible unify multiple sources.11 Stay in sync automatically Reduce human error No lag when changes occur Let others manage the inventory

VARIABLESJUST WORDS,RIGHT?

DESCRIBE VARIABLES WITH THEIR NAMESProper variable names can make plays more readable andavoid variable name conflictsa: 25data: abdata2: abcid: 12313apache max keepalive: 25apache port: 80tomcat port: 8080

PREFIX ROLE VARIABLESAvoid collisions and confusion by adding the role name to avariable as a prefix.apache max keepalive: 25apache port: 80tomcat port: 808014

PLACE VARIABLES APPROPRIATELYKnow where your variables are Find the appropriate place for your variables based on what, where andwhen they are set or modified Separate logic (tasks) from variables and reduce repetitive patterns Do not use every possibility to store variables - settle to a defined schemeand as few places as possible15

MAKE YOUR PLAYBOOKREADABLE

USE NATIVE YAML SYNTAXNO!- name: install telegrafyum: name telegraf-{{ telegraf version }} state present update cache yesnotify: restart telegraf- name: start telegrafservice: name telegraf state started17

USE FOLDING ONLY IF REALLY REQUIREDBetter, but no- name: install telegrafyum: name telegraf-{{ telegraf version }}state presentupdate cache yesenablerepo telegrafnotify: restart telegraf- name: start telegrafservice: name telegraf state started18

USE KEY:VALUE PAIRSYes!- name: install telegrafyum:name: “telegraf-{{ telegraf version }}”state: presentupdate cache: yesenablerepo: telegrafnotify: restart telegraf- name: start telegrafservice:name: telegrafstate: started19

DO NOT OMIT THE TASK NAMEExhibit A- hosts: webtasks:- yum:name: httpdstate: latest- service:name: httpdstate: startedenabled: yesPLAY [web]********************************TASK [setup]********************************ok: [web1]TASK [yum]********************************ok: [web1]TASK [service]********************************ok: [web1]20

USE TASK NAMESExhibit B- hosts: webname: installs and starts apachetasks:- name: install apache packagesyum:name: httpdstate: latest- name: starts apache serviceservice:name: httpdstate: startedenabled: yes21PLAY [install and starts apache]********************************TASK [setup]********************************ok: [web1]TASK [install apache packages]********************************ok: [web1]TASK [starts apache service]********************************ok: [web1]

POWERFULBLOCKS

USE BLOCK SYNTAXBlocks can help in organizing code, but also enablerollbacks or output data for critical changes.- block:copy:src: critical.confdest: /etc/critical/crit.confservice:name: criticalstate: restartedrescue:command: shutdown -h now23

EXECUTING THE ANSIBLE COMMANDSHow to execute24

PROPERLAUNCHING

TROUBLESHOOT ON EXECUTIONAnsible provides multiple switches for command lineinteraction and -task26

ANALYZE WHAT YOUR ARE RUNNINGAnsible has switches to show you what will be doneUse the power of included x-check27

QUICKLY LAUNCH WITHOUT INVENTORYIf there is a need to launch something without an inventory- just do it! 28For single tasks - note the comma:ansible all -i neon.qxyz.de, -m service -a"name redhat state present"For playbooks - again, note the comma:ansible-playbook -i neon.qxyz.de, site.yml

THE RIGHTTOOLS

CHECK IMMEDIATELY WHAT WAS DONEDon’t just start services -- use smoke tests- name: check for proper responseuri:url: http://localhost/myappreturn content: yesregister: resultuntil: '"Hello World" in result.content'retries: 10delay: 130

USE NATIVE MODULES WHERE POSSIBLETry to avoid the command module - always seek out amodule first- name: add usercommand: useradd appuser- name: install apachecommand: yum install httpd- name: start apacheshell: service httpd start && chkconfighttpd on31- name: add useruser:name: appuserstate: present- name: install apacheyum:name: httpdstate: latest- name: start apacheservice:name: httpdstate: startedenabled: yes

MARK MANAGED FILESIf managed files are not marked, they might be overwrittenaccidentally Label template output files as being generated by AnsibleUse the ansible managed** variable with the comment filter{{ ansible managed comment }}32

ROLES ANDGALAXIES

USE ROLES WHERE POSSIBLERoles enable you to encapsulate your operations. 34Like playbooks -- keep roles purpose and function focusedStore roles each in a dedicated Git repositoryInclude roles via roles/requirements.yml file, import viaansible-galaxy toolLimit role dependencies

USE GALAXY - WITH CAREGet roles from Galaxy, but be careful and adopt them toyour needs 35Galaxy provides thousands of rolesQuality varies drasticallyTake them with a grain of saltPick trusted or well known authors

ACCESS RIGHTS

USE BECOME, DON’T BE A ROOTRoot access is harder to track than sudo - use sudowherever possible 37Ansible can be run as root onlyBut login and security reasons often request non-root accessUse become method - so Ansible scripts are executed via sudo(sudo is easy to track)Best: create an Ansible only userDon’t try to limit sudo rights to certain commands - Ansible doesnot work that way!

DEBUG YOURPROBLEM

HAVE A LOOK AT THE NODE LEVELCheck logging on target machineansible-node sshd[2395]: pam unix(sshd:session): sessionopened for user liquidat by (uid 0)ansible-node ansible-yum[2399]: Invoked with name ['httpd']list None install repoquery True conf file Nonedisable gpg check False state absent disablerepo Noneupdate cache False enablerepo None exclude None39

IN WORST CASE, DEBUG ACTUAL CODEHow to keep the code executed on the target machineLook into the logging of your target machine ANSIBLE KEEP REMOTE FILES 1 ansible target-node -m yum-a "name httpd state absent"Execute with: /bin/sh -c 'sudo -u SUDO USER /bin/sh -c"/usr/bin/python /home/liquidat/.ansible/tmp/."40

USE THE DEBUG MODULEDebugging tasks can clutter the output, apply somehousekeeping- name: Output debug messagedebug:msg: "This always displays"- name: Output debug messagedebug:msg: "This only displays with ansible-playbook -vv "verbosity: 241

GET TOWER TO ADOPT ANSIBLE IN YOUR DATA CENTERHow to use in reallife42

TOWER FUNCTIONSSimple: Use Tower. 43Tower was developed with Ansible in mindExtends the limits of Ansible to meet enterpriseneeds:Scalability, API, RBAC, aduits, etc.

TOWER FUNCTIONSTower has inbuilt help 44Tower provides in-program help viaquestionmark bubblesCan include examples or links to further docs

BRANCHES, ANYONE?

TAKE ADVANTAGE OF GIT BRANCHESTower can import a repository multiple times with differentbranches 46Use feature or staging branches in your GitImport them all separately, address them separatelyUseful for testing of new features but also to move changesthrough stages

MANY, MANY ROLES

TOWER & ROLESTower automatically imports Roles during Project update 48Do not copy roles into your playbook repository, just create aroles/requirements.ymlTower will automatically import the roles during ProjectinstallationMix roles from various sourcesFix version in roles/requirements.yml to have auditableenvironment!

WHAT ARE WETALKING TO?

TOWER FUNCTIONSUse dynamic & smart inventories QUICK TIPTry right clicking on the icon and using“Replace Image” to insert your own icons.50Combine multiple inventory typesLet Tower take care of syncing and cachingUse smart inventories to group nodes

DOING GOOD JOBS

USE THE POWER OF JOB TEMPLATESTower job templates provide multiple options - use themwisely 52Keep jobs simple, focussed - as playbooks or rolesAdd labels to them to better filterFor idempotent jobs, create “check” templates as well - and letthem run over nightCombine with notifications - and get feedback when a “check”failed

1 1 1 1

USE WORKFLOWS FOR COMPLEX TASKSMultiple playbooks can be combined into one workflow 54Simple jobs, complex workflowsReact to problems via workflowCombine playbooks of different teams, different repositoriesRe-sync inventories during the play

DO ASK PROPER QUESTIONS

TOWER FUNCTIONSUse surveys to get variable values QUICK TIPTry right clicking on the icon and using“Replace Image” to insert your own icons.56Use good, meaningful variable namesProvide a default choiceMultiple choice free textIf answer not required - do you really need it atall?

A POWERFULTEAM

USE TOWER TEAMS, SEPARATIONSTower provides tenants, teams, and users - use them forseparation 58Provide automation to others without exposing credentialsLet others only see what they really needUse personal view instead of full Tower interface

ONE KEY TO RULETHEM ALL .

USE TOWER SPECIFIC ACCESS CREDENTIALSTower credentials should only be used by Tower - not byothers 60Set up a separate user and password/key for TowerThat way, automation can easily be identified on target machinesThe key/password can be ridiculously complicated secureStore key/password in a safe for emergencies

NOTIFY YOURSELF!

LET TOWER SEND NOTIFICATIONS TO YOUTower can send notifications if a job succeeds, fails oralways - as mail, IRC, web hook, and so on 62Let Tower notify you and your team if something breaksSend mails/web-hooks automatically to a ticket systems andmonitoring if there is a serious problem

LOGS, ANYONE?

CONNECT TOWER TO CENTRAL LOGGINGSend all logs from Tower to central logging 64Splunk, Loggly, ELK, RESTSend results from Ansible runs - but also from Tower changes

ALWAYS KEEPTHE LIGHTS ON

USE HA, DEPLOY ISOLATED NODESTower can be easily set up HA - and for restricted networks,deploy isolated nodes 66Make Tower HA - it is easy! (Well, except the DB part maybe .)For distant or restricted networks, use isolated nodes

CONFIDENTIAL DesignatorThank youRed Hat is the world’s leading provider of user/RedHatVideosopen source software solutions. Award-winningsupport, training, and consulting services makefacebook.com/redhatincRed Hat a trusted adviser to the Fortune 500.twitter.com/RedHat67

Use become method - so Ansible scripts are executed via sudo (sudo is easy to track) Best: create an Ansible only user Don’t try to limit sudo rights to certain commands - Ansible does not work that way!