SUSE Template Jan 2014 V2 Ks

SUSE ManagerTUT17530 Patching / Troubleshooting / Best PracticesJack HodgeSascha WeberGTS Backline EngineerJack.Hodge@suse.comTechnical Support EngineerSaWeber@suse.comJeff PricePrincipal ArchitectJPrice@suse.com

Summary – Installation & Patching Evaluation Keys & ISO download Licensing overview Installation of SUSE Manager Appliance /var/log/susemanager setup.err/var/log/susemanager setup.log2 Requirements (RAM, disc space, etc.) Installation of patches Database schema upgrade Installation of PTF (Program Temporary Fix) How to run SQL files provided by R&D Adding products / repositories Custom channels / Clones / Clone-by-date Activation keys / bootstrap script

Summary – Troubleshooting Debuggins SUSE Manager (spacewalk-debug) Architecture overview / important log files Apache2 (access.log / error.log) vs. Tomcat (catalina.out) vs.rhn (rhn server xmlrpc.log)3 /etc/rhn/rhn.conf OSAD vs. SSH push jabberd Repo-sync Apache tuning (/etc/apache2/server-tuning.conf) Taskomatic Configuration Management cobbler

Summary – Best Practices4 PostgreSQL database backup (smdba backup-hot) How to tune PostgreSQL database with pgtune Lifecycle Management (patch archive, clone-by-date, etc.) SUSE Customer Center migration Using multiple mirror credentials Service Pack Migration Synching channels from SMT SUSE Manager Proxy SUSE Manager Inter-Server Synchronization

SUSE Manager - Patching

Evaluation Key6 https://www.suse.com/products/suse-manager/ SUSE Customer Center Account needed https://scc.suse.com/login

Download ISO Images 7https://download.suse.com/index.jsp

Licensing overview -buy/

How to register During installation in GUI: Command line:suse register -a email email -a regcode-sms reg-code 9

SUSE Manager - Requirements 10https://www.suse.com/documentation/suse manager/book susemanager install/data/ch-installation.html

SUSE Manager – Ports to open11

SUSE ManagerPatching & Database schema upgrade Stop services via spacewalk-service stop via YaST online update via zypper: run zypper patch twice If the update contained a postgreSQL update it is necessaryto restart postgreq before starting the schema update:rcpostgresql restart Perform database schema upgrade via:spacewalk-schema-upgrade Start services via spacewalk-service restartPlease note:If a Kernel update was part of the patching processa reboot is always neccessary12

SUSE ManagerPTF installation & SQL execution https://www.suse.com/support/kb/doc.php?id 7016640 PTF Program Temporary Fix Download the RPMs and install these via rpm -Fvh Restart Services via spacewalk-service restart Run mgr-sync refresh In case R&D has provided you with a SQL script pleaseexecute it via spacewalk-sql --select-mode If you got a file: spacewalk-sql --select-mode file.sql If you have a SQL statement:spacewalk-sql --select-mode - "select *frompg stat activity;"Important: do not miss the trailing “-”13

SUSE ManagerAdding products/channels14 mgr-sync add product or mgr-sync add channel Using the Web-Interface Admin Setup Wizard SUSE Products:

SUSE ManagerCustom channels / clones / clone-by-date15 spacewalk-manager-channel-lifecycle spacecmd softwarechannel clone spacecmd softwarechannel clonetree Web-Interface Channels Manage Software Channels spacewalk-clone-by-date (part of spacewalk-utilspackage)

SUSE ManagerActivation Keys16 https://wiki.microfocus.com/index.php/SUSE Manager/System Activation Keys Avoid using the “SUSE Manager Default” parent channels Create one Activation key per distribution & architecture Create bootstrap script/Activation Key "pairs" - designed towork together for proper channel, group, and configassignments Web Interface Systems Activation Keys:

SUSE ManagerActivation Keys 17Use something “human readable” for the Key:

SUSE ManagerActivation Keys continues Select the corresponding “Child Channels”, these shouldcontain (for SLES sSLES12-Updates 18Make sure to add additional packages as needed:

SUSE ManagerBootstrap mgr-create-bootstrap-repo Make sure to re-run this command from time to time (orcreate a cron job) SUSE Manager 3 will do this itselfwhen there is an update Check in path /srv/www/htdocs/pub/repositories/ mgr-bootstrap to create a new bootstrap script:mgr-bootstrap --activation-keys KEY1,KEY2 \--gpg-key /srv/www/htdocs/pub/MY CORPORATE PUBLIC KEY \--allow-config-actions –allow-remote-commands Make sure the bootstrap script contains the needed GPGKeys:ORG GPG KEY 9DB7C82.key 19The bootstrap scripts are stored in /srv/www/htdocs/pub/bootstrap

SUSE Manager - Troubleshooting

SUSE ManagerDebugging SUSE Manager spacewalk-debugThis tool collects several pieces of information and stores them in atarball at /tmp/spacewalk-debug.tar.bz2For SLES 11 SP3 and greater / SLES 12 the spacewalk-debug is partof the supportconfig utility Set debug level in /etc/rhn/rhn.conf:debug 5 To debug spacewalk-repo-sync problems: export URLGRABBER DEBUG DEBUG /usr/bin/spacewalk-repo-sync --channel channel-label --typeyum To disable the debug mode of spacewalk-repo-sync: unset URLGRABBER DEBUG21

SUSE ManagerArchitecture overview – important log files https://www.suse.com/documentation/suse manager/singlehtml/book susemanager install/book susemanager install.html#tb-log-files Apache config files: /etc/apache2/conf.d/*.conf Apache log ccess.log Tomcat config file: /etc/apache2/conf.d/zz-spacewalk-www.conf Tomcat log out Client-Server communication:/var/log/rhn/rhn server xmlrpc.log22

SUSE Managerrhn.conf Main SUSE Manager config file is /etc/rhn/rhn.conf Set debug levels (e.g. debug x / osa-dispatcher.debug 5) Adjust taskomatic memory (taskomatic.maxmemory 4096) If you used embedded or external database, rhn.conf will keep theconnection details:db backend postgresqldb user susemanagerdb password ***db name susemanagerdb host localhostdb port 5432 Forwarding registrations to SCC / NCC:server.susemanager.forward registration 023

SUSE ManagerOSAD OSAD requires all clients to have different credentials in/etc/sysconfig/rhn/osad-auth.conf On a cloned system, make sure to delete the NCCcredentials file first(/etc/zypp/credentials.d) Do not install the osad package on the SUSE Manager but only on theclient sideosad on the client sideosa-dispatcher on the server side Config files areClient: /etc/sysconfig/rhn/up2date & /etc/sysconfig/rhn/osad.confServer: osa-dispatcher is configured via /etc/rhn/rhn.conf24

SUSE Managerjabberd If jabberd services fail to start (during spacewalk-servicerestart):Starting spacewalk services.Initializing jabberd processes.Starting routerStarting sm startproc: exit status of parent of /usr/bin/sm: 2Terminating jabberd processes.Simply delete the jabberd database:spacewalk-service stoprm -Rf /var/lib/jabberd/db/*spacewalk-service start25donefailed

SUSE ManagerSSH Server Push For tunneling connections via SSH, two available high port numbers ( 1024) are needed: one is for tunneling HTTP and one for HTTPS(default 1232 and 1233) In order to overwrite these, edit values in /etc/rhn/rhn.conf:ssh push port http high port 1 ssh push port https high port 2 By default only 2 simultaneous SSH session are initiated. For largerenvironments the number of session can be changed with the followingoption in /etc/rhn/rhn.conf:taskomatic.ssh push workers 1026

SUSE ManagerRepo-sync spacewalk-repo-sync is scheduled by taskomatic Channel sync logs are in /var/log/rhn/reposync Taskomatic log is in /var/log/rhn/rhn taskomatic-daemon.log To troubleshoot repo-sync problems it is very helpful toview the HTTP protocol. To achieve this: export URLGRABBER DEBUG DEBUGTo manually start repo-sync: /usr/bin/spacewalk-repo-sync --channel channel-label --type yumTo disable the debug mode of spacewalk-repo-sync: unset URLGRABBER DEBUG27

SUSE ManagerPostgreSQL & Apache tuning Only do this if the host has enough memory. Can cause otherperformance issues! Check your # of connections:rhn-db-stats /tmp/db-stats.log cat /tmp/db-stats.log grepmax connections Increase number of connections with pgtune:/usr/bin/pgtune -T Mixed -c 200 -i postgresql.conf -opostgresql.conf.pgtuneRestart postgres with rcpostgresql restart 28In /etc/apache2/server-tuning.conf:ServerLimit200# max number of server processesMaxClients200# max number of requests a server process serves

SUSE ManagerPostgreSQL & Apache tuning By default taskomatic (the central scheduler for all actions withinSUSE Manager) reserves 2 GB of memory. In most environments this is simply not enough. As an easy rule: 50%of the available memory should be allocated. To allocate more RAM to the taskomatic scheduler edit/etc/rhn/rhn.conf and add:taskomatic.maxmemory 4096 Memory recommendations:Less than 100 systems connecting to SUSE Manager: 8GBBetween 100 and 200 systems: 12 GBAbove 200 systems: 16 GBWhen planning to use RedHat Expanded Support: 16 GB29

SUSE Manager – Best Practices

SUSE ManagerPostgreSQL database backup – restoring backup If the backup is running correctly you should now have the following filesin /mnt/backup:-rw------- 1 postgres postgres16777216 Sep 13 09:46 000000010000001000000035 -rw------- 1 postgres postgres16777216 Sep 14 01:36 000000010000001000000039 -rw------- 1 postgres postgres16777216 Sep 15 05:18 000000010000001000000040 -rw------- 1 postgres postgres16777216 Oct -rw-r--r-- 1 postgres postgres 1464526433 Jul 22 12:51 base.tar.gz drwxr-xr-x 2 postgres postgres4096 Jul 22 12:35 database drwx------ 2 postgres postgres4096 Jul 22 12:51 tmp5 02:23 000000010000001000000041 Use smdba backup-restore to restore to an earlier point in time. To restore the backup, proceed as follows: 31 Shutdown the database: smbda db-stop Start the restore process: smdba backup-restore start Restart the database: smbda db-startsmdba backup-restore force

SUSE ManagerPostgreSQL database backup (smdba backup-hot) Backup is not automatically active Allocate permanent space (on remove storage if needed), e.g.: /mnt/backup This directory should always be the same, do not change Create a directory with the correct permissions: sudo -u postgres mkdir /mnt/backup/database Alternatively as root: Or: mkdirinstall -d -o postgres /mnt/backup/database/mnt/backup/database & chown postgres:postgres/mnt/backup/database For the first time run: Perform hot backup: 32smdba backup-hot –enable on –backup-dir /mnt/backupsmdba backup-hot –backup-dir /mnt/backup/database

SUSE ManagerHow to tune PostgreSQL database with pgtune We have the tool pgtune installed on SUSE Manager.Usage: pgtune [options]Options:-M TOTALMEMORY, --memory TOTALMEMORYTotal system memory, will attempt to detect if unspecified-c CONNECTIONS, --connections CONNECTIONSMaximum number of expected connections, defaultdepends on database type So if you want to increated the number of connections: /usr/bin/pgtune -T Mixed -c 1500 -i postgresql.conf -o postgresql.conf.pgtune Recommended values for number of connections are either 200 or 400. Stop all spacewalk services and the database, then swap the configurationfiles and start everything again:33 spacewalk-service stop & rcpostgresql stop mv postgresql.conf postgresql.conf.bak mv postgresql.conf.pgtune postgresql.conf rcpostgresql start & spacewalk-service start

SUSE ManagerSUSE Customer Center migration To be able to get access to the SLES 12 repos you have to migrate from NCCto SCC ( SUSE Customer Center). There is two options available:34 mgr-ncc-sync –enable-scc Web Interface Admin SUSE Manager Configuration

SUSE ManagerMultiple Mirror Credentials With SUSE Manager 2.1 the handling of multiple mirror credentials does nothappy in /etc/rhn/rhn.conf anymore. There is two ways now:35 mgr-sync list credentials mgr-sync add credentials mgr-sync remove credentials Using the Web-Interface Admin Setup Wizar Mirror Credentials:

SUSE ManagerService Pack Migration Supported migration paths are: SLES 11 SP1 SLES 11 SP2 SLES 11 SP3 SLES 11 SP4 SLE 10 SP2 SLE 10 SP3 SLE 10 SP4Service Pack migration can be done via the Web-Interface: Systems select a system Software SP Migration:36