#mesoscon14 has ended

Sign up or log in to bookmark your favorites and sync them to your phone or calendar.

Developer [clear filter]
Thursday, August 21

1:00pm CDT

Spark on Mesos: Handling Big Data in a Distributed, Multi-Tenant Environment - Paco Nathan
This presentation will include an introduction to Apache Spark, a general purpose engine for handling a range of batch, interactive, and streaming applications with large-scale data. Spark is built for mixed workloads, so we'll review how it operates and how to think about building Spark apps — particularly how those aspects fit well with Mesos. We will also show how to build and run Apache Spark on Apache Mesos. A demo based on using the free-tier service atop AWS https://elastic.mesosphere.io/ from Mesosphere will cover build, config, packaging, deployment, and then run sample apps using the Scala REPL for Spark. Based on these Spark jobs running, we will explore the Mesos and Spark consoles together, drilling down into details about system performance and troubleshooting.

This is intended primarily for a developer audience, but not with a lot of expertise required. Some familiarity with running Linux shell commands is needed. The expectation is that audience members will learn how to run Spark atop Mesos, along with some basic introduction to building Spark apps in general.

Many people using Spark in the field have heard more about YARN, but do not realize that Spark on Mesos is much simpler to get started running. Also, there are performance benefits, and a closer match for the mixed workloads intention of Mesos. We will show how the two blend together well atop Linux for a full-stack solution.  

Survey this Session


Paco Nathan

Evil Mad Scientist, Liber 118
Paco Nathan, is a "player/coach" who has led innovative Data teams building large-scale apps for several years. Paco is an O'Reilly author, Apache Spark open source evangelist with Databricks, and an advisor for Amplify Partners... Read More →

Thursday August 21, 2014 1:00pm - 1:20pm CDT
Sheraton Ballroom IV

1:30pm CDT

Run Applications Like a Boss: Fault-Tolerant and at Scale with Marathon and Mesos - Connor Doyle, Mesosphere
Gone are the days where you could build an app that only needs to run on a few servers. Now every app needs to be like Google: highly-distributed, fast, scalable and fault-tolerant. Devops everywhere have been wasting time building their own custom scale-out architectures for web apps. Marathon is a framework built on Apache Mesos that simplifies and automates operations, and provides a simple self-serve interface for developers to launch their apps on a shared cluster in a scalable and fault-tolerant way.

This talk is for application developers, IT organizations, and Ops teams who are looking to deploy and run Linux apps on large clusters. We will discuss ways to deliver a fault-tolerant and elastic scale-out architecture for any long-lived application like Ruby on Rails/node.js-based web applications or traditional J2EE servers like Tomcat.

This presentation will help organizations and developers understand how to build and deploy Linux apps at large scale, in highly-distributed systems.  

Survey this Session


Connor Doyle

Connor Doyle is a software engineer at Mesosphere, focused on developing tools around Apache Mesos. Before joining Mesosphere, Connor worked at the design firm Gensler where he worked on process improvement, internal libraries and distributed architecture in Scala. He completed his... Read More →

Thursday August 21, 2014 1:30pm - 1:50pm CDT
Sheraton Ballroom IV

2:00pm CDT

Improving Operational Insights with Reactive Stream Processing using Apache Mesos - Justin Becker & Sharma Podila, Netflix
Netflix customers stream more than one billion hours of content each month, accounting for over a third of downstream Internet traffic. At this scale, a lot can go wrong. Justin Becker and Sharma Podila will present how Netflix is building the next generation insight capabilities to help continuously improve the experience for streaming customers. Specifically, the talk will focus on a reactive stream processing system, built using Apache Mesos. The talk will provide the justification for building a new framework. Provide some insight into the scheduling algorithm; a brief overview of optimizations for “stream locality” and auto-scaling. The presentation will also include an overview of the executor; approaches to horizontal and vertical scaling. Then, finish with future directions and collaboration opportunities within the Mesos community.

The audience is anyone interested in building a custom Framework using Mesos. Specifically, those interested in building and deploying Mesos within the Amazon Web Service (AWS) Cloud Environment. The talk will also include detailed information about approaches to scheduling in a distributed, auto-scaling, environment.

The presentation will provide new framework developers with a real world Mesos case study, operating at web scale. Justin and Sharma also plan to discuss future collaboration opportunities within the Mesos community. Topics include: auto-scaling, fault tolerance, core scheduling behavior.  

Survey this Session


Justin Becker

Justin Becker and Sharma Podila are Senior Software Engineers working for the Edge Engineering team at Netflix. Justin Becker has experience building distributed systems for machine learning applications. Sharma Podila has a strong background in scheduling algorithms for distributed... Read More →
avatar for Sharma Podila

Sharma Podila

Senior Software Engineer, Netflix
Sharma Podila is a Senior Software Enginner in the Edge Engineering team at Netflix, Inc. His current work includes Fenzo, an open source generic scheduler with plugin based optimizations, developing cloud native Mesos frameworks, and evolving the microservices platform at Netfli... Read More →

Thursday August 21, 2014 2:00pm - 2:20pm CDT
Sheraton Ballroom IV

2:30pm CDT

Past, Present, and Future of the Aurora Scheduler - Bill Farner, Twitter
Apache Aurora has been managing Twitter's production services on Mesos for three years, and now powers the majority of compute at the company. In that time, our team has adapted Aurora's scheduler in many ways to be robust, usable, and performant. In this talk, Bill will discuss robustness features critical to Aurora's success, and explain the features we're interested in building in the scheduler.

This talk will be useful for anyone involved in writing or maintaining services, as well as Mesos framework developers. Audience members will learn about how the Aurora scheduler works, and gain an understanding of the features that make it resilient enough to power twitter.com.

This talk aims to share design decisions, mistakes made, and features developed over several years of writing and maintaining a business-critical Mesos scheduler. This will help promote our open-source software as well as share our experience with other Mesos framework developers.  

Survey this Session


Bill Farner

Bill Farner has been a Software Engineer at Twitter for the last five years, serving as the Technical Lead on Aurora since its inception four years ago. He previously worked at Google on Production Engineering and storage infrastructure. Bill holds a MS in Computer Engineering from... Read More →

Thursday August 21, 2014 2:30pm - 2:50pm CDT
Sheraton Ballroom IV