Open Source Tool Chains for Cloud Computing
This presentation was given at LinuxCon 2010. The proliferation of cloud computing is inevitable, hosted apps, software-as-as-service and now dynamic on-demand utility computing is becoming the norm. The session will be a fire-side chat style discussion of the types of challenges presented by IT management operations personnel and how they can manage cloud infrastructure using open source tools. The talk will discuss options for deploying and integrating tools that provision, configure, orchestrate and monitor cloud (and physical)infrastructure. The session will appeal to those IT professionals (syadmins, net-ops, developers) who develop and manage infrastructure that resides in hosted environments like Amazon EC2 without disregarding traditionally hosted internal infrastructure.
1.Open Source Toolchains for Managing Clouds Monday, August 9, 2010 2. Mark R. HinkleVP of Community Zenoss Inc.email@example.com@gmail.com Twitter: @mrhinkleJohn M. Willis VP of ServicesOpscode Inc. firstname.lastname@example.org@gmail.comABSENT Twitter: @botchagalupe Monday, August 9, 2010 3. %whoami Former Linux Desktop AdvocateFormer LinuxWorld Magazine Editor-in- ChiefOpen Management Consortium ConspiratorOpen Desktop Consortium Instigator Author - Windows to Linux Business Desktop Migration - ThomsonNetDirector Project - Open Source Conguration Management Project Monday, August 9, 2010 4. Definitions (Toolchains,Systems Management,Cloud Computing) Bad JokesTodaysOverview of OpenAgendaSource Management Tools Culture Changes Alien Autopsy Photos Example CloudComputing Toolchains Monday, August 9, 2010 5. Toolchain A set of programs where the output of one program formsthe input of another program. Monday, August 9, 2010 6. Open Source Management Tools Adoption 98%ofenterprisesuseopensourcesystems managementtools76%indicatetheyprefertouseopensource wheneverpossibleCompellingfactorsforusingopensourceis exibilityfollowedbycostsavings50%arealreadyusingsomeformofcloud technologyincludingbutnotlimitedtohosted applica@ons,AmazonWebservicesand/orhosted storageTopITmanagementpriori@esfor2010:monitoring, congura4onmanagement,patchingandSource: 2010 ZenossOpen Source Management Survey Monday, August 9, 2010 7. Cloud ComputingHype is Becoming a Reality UBS says Web Services will be a $15 billion+ market by 2014 IBM says Cloud Computing will be a $126billion by 2012 20% of Businesses wont have IT Assets by2012 - Gartner Bottom line a large portion ofour infrastructure no longerlives in our data center... but we gotta manage it. Monday, August 9, 2010 8. Systems ManagementDisciplinesProvisioning Installation of operating systems and other softwareConguration Management and AutomationSets the parameters for servers, starts and stops services,rotates logs and other menial task MonitoringMonitoring queries the servers for overall health andalerts administrators to problems Monday, August 9, 2010 9. How to Choose Open Source Cloud Tools Open Source (OSI Approved License) Manage Legacy Infrastructure and Cloud Extensible (Plugins, accept code contributions) Vibrant Communities (activity in Downloads, Forums, Extensions) Client/server (or at least network aware) Monday, August 9, 2010 10. Comparison of Provisioning ToolsYearInstallation LanguageLicense StartedTargetsRed Hat, Fedora, OpenSuSE,Cobbler 2007 PythonGPLDebian, UbuntuFully AutomaticInstallation (FAI)2000 PerlGPL Debian Most .deb and FedoraKickstart ?PythonGPLbased Linux OpenQRM 2005 PHP GPL Linux, Solaris, WindowsPerl, Python,Spacewalk 2008 GPL Fedora, CentOS JavaViper2008 PerlGPL Debian Monday, August 9, 2010 11. Comparison of CongurationManagement & Automation ToolsYear Language License Client/Server Backing Started AutomateIT2009 Ruby GPLNoNonebcfg2Argonne National2003 BSDBSD Yes Labs? Cfengine1993Apache Apache Yes Cfengine Inc. Chef Solo - Nochef2009 RubyApache OpscodeChef Server - Yes Puppet2004 Ruby GPL Yes PuppetLabs Monday, August 9, 2010 12. Comparisons of Open Source Monitoring Tools Year MonitoringLicense LanguageCollection MethodsStarted TypeCacti2001GPLPHPPerformance SNMP, Syslog SNMP, TCP, ICMP, IPMINagios 1999GPL C/PHPAvailabilitySyslog Availability,OpenNMS2000GPL JavaSNMP, JMX, HTTP Performance, Availability, SNMP, TCP, ICMP, IPMI,Zabbix 2001GPL C/PHP Performance, andSynthetic Transactions moreAvailability, SNMP, SSH, Syslog, EventZenoss 2005GPL PythonPerformance,Management, Synthetic Transactions Event Management Monday, August 9, 2010 13. CloudOps: Change in CultureGapingVoid - http://www.gapingvoidgallery.com/product_info.php?products_id=1643&osCsid=bc3tdqg6fuh8gato04m9obr0o1 Monday, August 9, 2010 14. Old SystemsManagement Practices Human Powered, Labor Intensive Repetitive Tasks Meat Cloud How many servers do you manage per admin? Monday, August 9, 2010 15. Cloud Changes Everything Hyperscalable Hardware Abstraction Dynamic Infrastructure Geography Independent Fast & Flexible How many cloud instances do you manage per admin? Monday, August 9, 2010 16. The Myth of the NinesAvailability % Downtime per YearDowntime per Month Downtime per Week99.9% (three nines)8.76 hours43.2 minutes10.1 minutes99.95% 4.38 hours 21.56 minutes5.04 minutes99.99% (four nines) 52.6 minutes4.32 minutes 1.01 minutes 99.999% (ve nines) 5.26 minutes 25.9 seconds6.05 minutes 99.9999% (six nines)31.5 seconds 2.59 seconds.0605 seconds Average polling interval for monitoring? 5 minutes? Even super human operations people cant be alerted and take action in under 5 minutes.One outage per year could drop service level to three nines or worse. Monday, August 9, 2010 17. P Automate or getrun over. Copyright 2010 Opscode, Inc - All Rights Reserved17 Monday, August 9, 2010 18. Cultural ChangesAgile IT and DevOps movements Operations and Developers shouldcollaborate with each other todeliver excellent products Systems Administrators need to become Systems Engineers buildingautomated, fault tolerant systemsnot just maintaining infrastructure More frequent changes, moreoutages (albeit short) to rapidlyimprove IT products and services Process, version control, andautomation are important Monday, August 9, 2010 19. Systems ManagementTools for Cloud ComputingCongurationProvisioning Management and Monitoring Automation KickstartGanglia AutomateITSpacewalkNagiosChefOpenNMS Cobbler Control Tier ZabbixOpenQRMPuppet Zenoss Core Monday, August 9, 2010 20. Off the shelf Open Source Toolchains OpenQRM and Nagios Cobbler and Puppet Zenoss & [Chef, Cfengine and Puppet] Spacewalk and Cobbler OpenNMS and Rancid OpenNMS and Puppet Monday, August 9, 2010 21. DevOps ToolChain ProjectProject centered around how to automate and improveinfrastructure management using Agile/DevOps methodologiesDiscussion Topics Open questions on unied pipe architecture Distribution methods: package vs le, rscyn/murder vs yum/rpm vs DFS Conguration management: RPMs vs puppet/cfengine/chef tool? Rollback methodologies for package and cong management tools Controlling and timing package release and cong management tools Log management (aggregating, crunching, charting)Change detection http://code.google.com/p/devops-toolchain/ Monday, August 9, 2010 22. CloudComputing ChangesEverything MeatCloud, Cant Keep up with Cloud ComputingDevops & Agile IT Philosophy Script Repetitive Tasks Automate, Automate, Automate Monday, August 9, 2010 23. Example Cloud ToolChain Multiple Cloud ProvidersMix and Match and match toolsPortability, Flexibility, and Monday, August 9, 2010 24. Example - GeeknetServers are automatically built using conguration management software Discovery tool nds infrastructure and populates a CMDB then spits out information to scripts that Hundreds of servers, serving translate information to BIND congurations for DNS web, databases, and other infrastructure for some of theMonitoring tool adds hosts to polling tool to check worlds most highly trafcked servers for availabilitywebsites over 40 million visitors per month.As infrastructure changes systems are updated automatically Servers can be spun up and managed in minutes, not hours automatically with little or no human interaction Monday, August 9, 2010 25. Summary Automate to improve service, apply leverage Rethink how your operations works Choose tools that can extend and adapt to new types of infrastructure (what does cloud look like in 2015?) Monday, August 9, 2010 26. Questions? Comments? Monday, August 9, 2010