A Deep Dive Into Trove

  • Published on

  • View

  • Download


PowerPoint PresentationA Deep Dive into TroveOpenStack MeetupMarch, 2015Amrith KumarWho Am I?Amrith Kumar, Founder & CTO of TesoraATC on several OpenStack projectsCore reviewer on TroveTesora: the Trove Company#1 contributor to Trove projectTesora DBaaS Platformpackaged, supported version of TroveNearly all work is upstream first Database & distribution certificationsDatabase Virtualization Engine (DVE)Open source, transparent sharding for MySQLTrove Contributors (Juno)Diverse and growing community128 contributors from 28 companies1723 commits, and 231829 lines of code2A Deep Dive into Trove OpenStack MeetupMarch 2015Traditional ITProvisioning by DBAsDatabase management by specialistsWaterfall developmentFew large machines / bare metalOracle enterprise licensesCaptive audienceCloud Self-service provisioningDevelopers manage their own databasesAgile development Many small machines / virtualizationMany data management technologiesCompetition with AWSTransformation of Cloud Data ManagementA Deep Dive into Trove OpenStack MeetupMarch 20153What is Trove?Database as a Service for OpenStackAPIs for both development and operationsSelf service database provisioningFull database lifecycle managementMulti-database supportBoth Relational and NoSQL4A Deep Dive into Trove OpenStack MeetupMarch 2015What is Trove? RDS for OpenStackand DynamoDB and RedShiftAmazon AWSOpenStackEC2SwiftS3DynamoDBRedShiftCinderEBSNovaDeveloper APIDeveloper APIOperations APIAmazon Internal APITroveRDBMSNoSQLDWRDSand APIs for DevOPs5A Deep Dive into Trove OpenStack MeetupMarch 2015What is Trove? The Trove MissionOpenStack Trove mission statement:To provide scalable and reliable Cloud Database as a Service provisioning functionality for both relational and non-relational database engines, and to continue to improve its fully-featured and extensible open source framework. 6A Deep Dive into Trove OpenStack MeetupMarch 2015TuningAutomatically tune my.cnfBuffer Pool SizeLog file sizemax_connectionsSane defaultsInnoDB onlyDisable load data infileDisable select into outfileAPI to programmatically set configuration groupsManagementCreate database / schemaCreate usersGrant permissions to a SchemaEnable root userResize flavorResize volumeFull and incremental backupsCreate replicaWhat is Trove? More than just provisioningSecuritySecurity groupsTurn off SSHRemove anonymous userRemove non-localhost usersRemove local file accessMangle root user passwordApply security patches automatically7A Deep Dive into Trove OpenStack MeetupMarch 2015Trove In Production TodayPublic CloudHP Cloud Relational DatabaseLaunched May 2012 Rackspace Cloud Databases Launched August 2012Private Cloud: eBayBegan mid 2013Multiple DatabasesMySQL, MongoDB, Redis Cassandra, CouchbaseMulti-region + HAWorking on ClusteringKey Use CasesDevelopment & testWeb application hostingOn-demand analyticsCritical CapabilitiesSelf-service provisioning & managementFleet wide configurationMulti-datastore architecture8A Deep Dive into Trove OpenStack MeetupMarch 2015What is Trove? OpenStack ArchitectureOpenStackTroveA Deep Dive into Trove OpenStack MeetupMarch 2015Datastore agnostic code in Trove Controller & DashboardDatastore specific code isolated to Guest AgentsWhat is Trove? Multi-Datastore ArchitectureTrove ControllerGuest AgentGuest AgentGuest AgentGuest AgentTrove Dashboard (Horizon)Guest AgentGuest AgentGuest AgentGuest AgentGuest AgentGuest AgentGuest AgentA Deep Dive into Trove OpenStack MeetupMarch 20151010trove/guestagent/strategiesbackup/[]replication/restore/storage/trove/guestagent/[]backup/common/datastore/[]strategies/[]trove/guestagent/strategies/backupbase.pybase.pyccouchbase_impl.pycouchbase_impl.pyc__init__.py__init__.pycmysql_impl.pymysql_impl.pycpostgresql_impl.pygrep class guestagent/strategies/backup/mysql_impl.pyclass MySQLDump(base.BackupRunner):class InnoBackupEx(base.BackupRunner):class InnoBackupExIncremental(InnoBackupEx):Trove Architecture: Code ModularityA Deep Dive into Trove OpenStack MeetupMarch 2015Lets shift gears for a couple of minutes and talk briefly about the architecture from a different perspective.Trove must be able to address many different kinds of datastores. Some are NoSQL, some are SQL (and Im playing with a graph database as well). Each of these implements a certain capability in a different way. Lets talk about backup and restore as an example.Once you get the software, in the trove/guestagent directory, youll find a directory called strategies. And under strategies, one called backup, and under backup youll see the actual implementation for the specific datastore. Now looking at the MySQL implementation, youll see that there are two backup capabilities (full backup) and one incremental backup method.At configuration you can specify which implementation you would like to use. And you could implement a new strategy for your datastore thereby making Trove extensible and modular.11Using OpenStack TroveGetting started with OpenStack TroveAs a Trove userOpenStack distribution that includes Trove (such as HP Helion Dev Platform)http://www8.hp.com/us/en/cloud/hphelion-openstack.htmlTesora DBaaS platform, a Trove packaging tailored for the enterprisehttp://tesora.com/download-tesora-dbaas-platform-community-editionAs a Trove developer$ git clone http://github.com/openstack/trove-integration$ cd trove-integration/scripts$ ./redstack install$ ./redstack kick-start mysqlOn top of DevStackAdd to localrc: ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-condSwift should also be enabled for Backup and Restore.ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-accountSWIFT_HASH=A Deep Dive into Trove OpenStack MeetupMarch 201513Provisioning a database instance with Trove$ trove create --size Support for flavorsSupport for volumes using CinderOptional parameters to createImage per datastore type and versionSupport AZs using --availability_zoneSupport for Neutron using --nicA Deep Dive into Trove OpenStack MeetupMarch 201514{ "instance": { "status": "BUILD", "updated": "2014-06-05T19:33:46", "name": "test", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/instances/ce629494-a64d-41ed-b73c-04cb28bb33bb", "rel": "bookmark" } ], "created": "2014-06-05T19:33:46", "id": "ce629494-a64d-41ed-b73c-04cb28bb33bb", "flavor": { "id": "1002", "links": [ { "href": "https://region-a.geo-1.database.hpcloudsvc.com/v1.0/39745227274204/flavors/1002", "rel": "self" }, { "href": "https://region-a.geo-1.database.hpcloudsvc.com/flavors/1002", "rel": "bookmark" } ] }, "datastore": { "version": "5.5", "type": "mysql" } }}14Managing a database with TroveResize flavorResize volumeDatastore specific extensions:Create Database / SchemaCreate UsersGrant Users Permissions on DatabasesEnable a Root UserA Deep Dive into Trove OpenStack MeetupMarch 201515$ trove backup-create Optional params:Description through --descriptionIncremental backups using parentTo Restore backup use create:$ trove create --size --backup Fully managedTriggered and tracked via APIStreamed to Swift (OpenStack Object Storage)Incremental & full backupsMultiple formats per datastore supported via strategies:XtraBackup (Percona)mysqldumpBackup and RestoreA Deep Dive into Trove OpenStack MeetupMarch 201516OpenStack Trove: Completed in JunoAsync MySQL replication (master-slave)Clusters for MongoDBNeutron SupportSupport for PostgreSQLConfig-groups enhancementsConfiguration groups per datastore / versionConfig-groups for MongoDBBackups for Cassandra and CouchbaseAdditional Tempest TestsA Deep Dive into Trove OpenStack MeetupMarch 201517New in Juno: Replication$ trove create --size --replica_of Support for async MySQL replication (MySQL slave instances)Manual detach using$ trove update --detach-replica-sourceA Deep Dive into Trove OpenStack MeetupMarch 201518New in Juno: Clusters$ trove cluster-create Optional parameters to cluster-create--instance Specify multiple times to create multiple instances for your clusterInitial support in Juno added for MongoDB ClustersSets up mongo config server, and mongo query routersTransparent and driven by configuration options Support for adding shards to existing cluster for horizontal scale out.A Deep Dive into Trove OpenStack MeetupMarch 201519Planned for KiloBuilding out clusters Semi Synchronous MySQL clusters (Galera)Async Replication v2 GTID based replication Manual failover supportAssociate flavors with datastoresAccess datastore logs via APIRemoving deprecated oslo-incubator codeUpgrade testing through grenadeA Deep Dive into Trove OpenStack MeetupMarch 20152020Where can I get OpenStack Trove?Get the sourceOpenStack Trove Projecthttps://git.openstack.org/cgit/openstack/trove / http://github.com/openstack/trove.gitTrove python client binding and command line clienthttps://git.openstack.org/cgit/openstack/python-troveclient / http://github.com/openstack/python-troveclient.gitTrove design specifications for blueprintshttps://git.openstack.org/cgit/openstack/trove-specs / http://github.com/openstack/trove-specs Trove scripts for installation and testing, and elements for building guest imageshttps://git.openstack.org/cgit/openstack/trove-integration / http://github.com/openstack/trove-integration.gitInstallable packages and guest images from TesoraCommunity Edition: http://tesora.com/download-tesora-dbaas-platform-community-editionEnterprise Edition trial: http://tesora.com/download-tesora-dbaas-platform-enterprise-editionA Deep Dive into Trove OpenStack MeetupMarch 2015More about TroveTrove Wikihttps://wiki.openstack.org/wiki/TroveTrove Sourcehttps://git.openstack.org/cgit/openstack/troveOn IRC#openstack-troveTrove DayTesora.com/trovedaySlideshare.net/TesoracorpLinkedInOpenStack Trove GroupContact informationContact Tesorainfo@tesora.comwww.tesora.com@tesoracorpContact Kenkrugg@tesora.comwww.tesora.com@kenruggA Deep Dive into Trove OpenStack MeetupMarch 201522Thank You!