Big Media: Multimedia goes Big Data

  • Published on
    15-Jul-2015

  • View
    903

  • Download
    3

Transcript

  • Technische Universitt Darmstadt

    Prof. Dr. Max Mhlhuser

    BigMedia:Multimedia goes Big Data

  • Video as example & driver:

    YouTube: 100 h/min.

    Internet: >50%, > 40 XB/mo.

    LTE advanced!

    Cities: >500,000 surveillance cams

    e.g., London: 6 months screening effort

    Plus: zillions of sensors

    Crowd: cf. foto heatmaps

    BigMedia @ ISM 2014 M. Mhlhuser 2

    A Few BigMedia Facts

    www.statista.com/chart/624 ( Cisco Visual Networking Index)

    www.sightsmap.com

  • Multimedia & BigData: Do They Blend?

    BigMedia @ ISM 2014 M. Mhlhuser 3

    processinput output

    capture interacttransport store

    process

    multimedia

    sense analyze

    big data

    Volume Velocity

    Variety Veracity

    conventionaldata processing

    Unified BigMedia pipeline:1) capture/sense 2) transport 3) store 4) process 5) interact/analyze

    BigMedia

  • 1. CAPTURE/SENSE

    BigMedia @ ISM 2014 M. Mhlhuser 4

    capture interacttransport store

    processsense analyze

    BigMedia

  • Sensors (CaptureDevices):

    1. Hard:1a) scene capturing

    Camara, Mike1b) value sensing

    Accelerometer,

    2. Soft:2a) Documents

    browse, view, edit2b) Software

    contextualcommand-lvl. stroke-lvl. use

    BigMedia @ ISM 2014 M. Mhlhuser 5

    2x2 Categories of Sensors

  • Events in-situ

    w/ Smartphone

    Desire: link to

    people on site

    (+net)

    CoStream:

    BigMedia @ ISM 2014 M. Mhlhuser 6

    Crowd Media Example: CoStream

    Login Awareness Streaming

  • Portrait upright

    BigMedia @ ISM 2014 M. Mhlhuser 7

    CoStream Key Interaction Concepts

    Portrait parallelto the ground

    Landscape

    Rotate to watch Tap to stream

    Invitations, Sharing: Push & Pull

    (friend list video) Rating (Like/Dislike) Vibration to indicate

  • BigMedia is Multisensory Media AudioVideo + SocialMedia + Location/Motion/

    BigMedia @ ISM 2014 M. Mhlhuser 8

    Another Example 1st General Trend

    + conventional multimediamedia type: tweetsDomain:Small-ScaleIncident Detection

  • BigMedia is Blended Media:

    Generation (Sources): User Generated

    Authoriative

    Automatic

    Consumption (Targets):@Site

    @Net

    @Home / @Mobile

    BigMedia @ ISM 2014 M. Mhlhuser 9

    2nd General Trend

  • BigMedia is Mass Media: (user generated, automatic, authoritative) selection @ receiver?

    social & personal prefs

    automatic (BigData)

    or: summaries for massive reduction

    e.g., emotion metering

    e.g., hotspot / trend indicators

    e.g., event analytics

    BigMedia @ ISM 2014 M. Mhlhuser 10

    3rd General Trend

  • BigMedia means

    mass amounts of (streams of)

    blended (-source/-target)

    multi-sensory (hard + soft)

    media

    BigMedia @ ISM 2014 M. Mhlhuser 11

    Intermediate Summary

  • 2. TRANSPORT

    1. net/edge processing

    (2./3. basically skipped)

    BigMedia @ ISM 2014 M. Mhlhuser 12

    capture interacttransport store

    processsense analyze

    BigMedia

  • The Issue: Transport vs. Store&Process

    BigMedia @ ISM 2014 M. Mhlhuser 13

    Reality: Field Stakeholders: POIInfrastructure: Net

    capture interacttransport store

    processsense analyze

    ?!

    Cloud?

  • BigMedia @ ISM 2014 M. Mhlhuser 14

    Net/Edge Processing & Latency Needs

    real scenery encounteredby mobile user (here: outdoor!)

    real scenery captured by camera

    processed: 3D, semantics, location, orientation,

    virtual scenery overlayed3D, in real time, as user moves

    Dual Reality (DR)

  • Goal: dual reality DR (100% VR + 100% reality)

    + universal semantics, in- & outdoor, 3D overlay

    Grand challenge:

    universal semantics (my phone can see!)- geometry, location, domains

    3D robust real-time registration~today: popular buildings only

    BigMedia @ ISM 2014 M. Mhlhuser 15

    Net/Edge Processing & Latency Needs

    Junaio, Layar, Wikitude, TK

    indoorexample

    photo courtesy of TUD GRIS & FhG IGD

  • We are facing the age of latency DR (see above) + speech

    federated interaction

    realtime media stream analytics

    Cloud: not enough!

    add local cloud

    required: mobility, handover, replication, self-X

    driver: excess processing capacity @net

    cf. Microsoft Cloudlets, Cisco Fog Computing

    SWOT?

    - pro: low latency, cheap & ctx-aware processingownership@origin (see below)

    - con: distributed processing

    BigMedia @ ISM 2014 M. Mhlhuser 16

    Net/Edge Processing: Cloudlets

  • 2. Resilient Networks:

    3. Highly Adaptive i.e. Fluid Networks

    BigMedia @ ISM 2014 M. Mhlhuser 17

    Further Key Transport (=Net) Issues

    Smart GridIndustrial Facilities

    Smart CitiesSmart Transport

    Fluctuation:- Density

    - Intensity- Mobility

  • 3. STORE

    1. The Forgotten Forgetting

    2. Distributed Storage (&Processing) Privacy?

    BigMedia @ ISM 2014 M. Mhlhuser 18

    capture interacttransport store

    processsense analyze

    BigMedia

  • BigMedia: always-on recording LiveLogs, cf. Microsoft Sensecam

    Analogy: cam+mike eyes+ears

    brain intelligent processing

    Open Issue: data organization user swamped

    Google, FB surrender, timeline

    (cf. Gelernters lifestream)

    even less organization!

    remember brain: associative & forgetful

    BigMedia @ ISM 2014 M. Mhlhuser 19

    1. The Forgotten Forgetting

  • Eyes/ears Bionics:

    auto-organize + auto-consolidate (forget)

    idea: reinforce retrieved data + neighbors

    but: too many neighbors!

    idea: leverage user interaction

    1. Acquire: meta data

    2. Interact: N-dim. space

    3D visualization, user picks 3-of-N

    neighbors: cf. users selection!

    3. Consolidate:

    fade least reinforced data

    BigMedia @ ISM 2014 M. Mhlhuser 20

    1. The Forgotten Forgetting

    approach

  • BigMedia @ ISM 2014 M. Mhlhuser 21

    Ad 2: Interlude

    processing& storage

    @net/edge

    latency & bandwidth

    limits

    excesspower

    @net/edge

    AlwaysOn

    / LiveLogs

    Problems, e.g.:how to process,

    to archive,

    Further oppor-tunities, e.g.:privacy thruownership

  • BigMedia @ ISM 2014 M. Mhlhuser 22

    2. BigMedia Privacy

    Privacy rights, laws, or desiresw.r.t. PII: personally identifyable info.

    AnonymizationData Thrift

    big data

    collect more datadont throw

    any data away

    Prof. Narayanan, Princeton: essentially impossible in a foolproof way w/o losing utility of data1

    1: Privacy and Security: Myths and Fallacies of Personally Identifiable Information .CACM 53 (6), 2010, pp. 23-25

  • 1. Cyberphysical Spaces

    2. Cyberphysical Humans

    BigMedia @ ISM 2014 M. Mhlhuser 23

    2. BigMedia Privacy: Things are getting worse

    Consent?

    latent PII

    photos economist, siliconangle, ebiz-results, REX, Thinkstock/Ninell_art

    remote processingvs. local data

    quantified self& Assistence

  • anonymousstore?

    BigMedia @ ISM 2014 M. Mhlhuser 24

    Challenge

    tustedstore

    inte

    rfac

    e

    ?

  • 4. PROCESS

    Just a brief recap / overview, for the sake of time

    BigMedia @ ISM 2014 M. Mhlhuser 25

    capture interacttransport store

    processsense analyze

    BigMedia

  • Selected: 3rd Processing Category

    BigMedia @ ISM 2014 M. Mhlhuser 26

    observablesperceivables

    1. hard

    2. soft

    learning

    person group popul.whohowoffline

    realtime

    conceivables

    capture interacttransport store

    processsense analyze

    1 of 3 processcategories:

    machine learning

    trace movement intentionexample:

  • BigMedia pipeline unify&standardize 3 paradigms1. conventional & statistical processing

    2. machine learning

    3. crowd processing

    Remember: processing @net/edge (Cloudlets ) requiresmodularization, smooth mobility, appropriate algorithms & models,

    BigMedia @ ISM 2014 M. Mhlhuser 27

    Process Categories, @Net Processing

  • 5. INTERACT/ANALYZE

    technology proliferation new interaction concepts

    Mobile - Natural - Large Scale

    (many other trends ignored for the sake of time)

    BigMedia @ ISM 2014 M. Mhlhuser 28

    /capture interact

    transport storeprocesssense analyze

    BigMedia

  • 5A. MOBILE INTERACTION

    Resizable Displays

    AR DR Displays

    On-Body Interaction

    BigMedia @ ISM 2014 M. Mhlhuser 29

  • Lab equipment:

    targets user experiments& controlled user studies:UI concepts for devices-to-be!

    BigMedia @ ISM 2014 M. Mhlhuser 30

    Resizable Displays (1): Rollable

    display size dilemma: an innovation driver

  • Resizable Displays (1): Rollables, contd.

    (this was just a selection)further UI concepts concern,

    e.g.: semantic zoom,visual clipboard,

    horizontal scroll,

    BigMedia @ ISM 2014 M. Mhlhuser 31

  • OLEDs thin devices folding?

    The question, again: interaction concepts towards UX?

    BigMedia @ ISM 2014 M. Mhlhuser 32

    Resizable Displays (2): Foldables

  • BigMedia @ ISM 2014 M. Mhlhuser 33

    FoldMe Design Space

  • PicoProjectors? use environment as display

    daylight, power efficiency

    collaboration? privacy?

    empty space / wall

    hand-held hand jitter

    HOWEVER: Add depth cam tangible information space

    BigMedia @ ISM 2014 M. Mhlhuser 34

    Resizable Displays (3): Pico Projectors

    Samsung (Galaxy Beam)

  • Google glass (note: full computer) hype & $$$

    multimodal

    app ready in 2 sec. (vs. 22)

    Two sets of open issues:interaction concepts & experience1. degree of AR

    2. advancement of vision & speech

    3. direct manipulation!!

    4. sharing experience?- SeeWhatISee sufficient?

    5. acceptance as eyeware fashion

    6. privacy

    Note: DR-ready sophisticated glasses 2-6 remain!

    BigMedia @ ISM 2014 M. Mhlhuser 35

    ARDR displays: Google Glass lessons

    Google folks (today!): no line-of-sight, no AR

  • proliferation of technologies interaction concepts

    1. rollable! promising

    2. glass! heavy invest vs. open issues

    3. pico projector: information spaces?

    4. foldable: interaction concepts

    not promoted doomed to fail?

    BigMedia @ ISM 2014 M. Mhlhuser 36

    Future Mobile Displays: Summary

    3D displays? games as driver

  • Use Palm-of-Hand (plus fingers) for interactiona) buttons

    b) sliders

    c) numbers,

    d) ..

    On-Body Interaction: Hand

    a)

    b)

    8c)

    BigMedia @ ISM 2014 M. Mhlhuser 37

  • Wireless Sensor

    Arc-shaped Board (12 Touch Points)

    Interaction Techniques

    Evaluation

    here: control audio @ human audio device

    38

    On-Body Interaction: Ear

    BigMedia @ ISM 2014 M. Mhlhuser

  • 5B. MORE NATURAL INTERACTION

    implicit

    tabletop

    paper like

    spoken

    printed tangible

    BigMedia @ ISM 2014 M. Mhlhuser 39

  • Basis: user(s) pose, emotion, attitude,

    Interaction: appearance

    actions

    Example (1): CouchTV

    BigMedia @ ISM 2014 M. Mhlhuser 40

    Implicit Interaction: Idea, Example 1

    Implicit control

    Implicit suggest

    Implicit pause and record

  • Rollables again, but collaborative

    bridge phone tabletop

    Implicit interaction: auto-adapt UI to physical connectedness

    BigMedia @ ISM 2014 M. Mhlhuser 41

    Implicit Interaction (2), collaborative setting

  • BigMedia @ ISM 2014 M. Mhlhuser 42

    Table Based Interaction: Concern

    Desktop PC: isolated

    immersive tabletop:social

    table: social

    digitize

    tabletop: isolated

    augment

  • awareness/accessibility: interactive halo & icon, gradual & remote access, expos organization: teleporting, hybrid piling / hiding , hybrid binding

    BigMedia @ ISM 2014 M. Mhlhuser 43

    Table Based Interaction: ObjecTop

  • Table Based Interaction: PeriTop

    add top projection & depth camera

    BigMedia @ ISM 2014 M. Mhlhuser 44

    -digital info atop occluders-about object or else-low resolution OK here

  • Table Based Interaction: CoMAP

    BigMedia @ ISM 2014 M. Mhlhuser 45

  • 46

    Table Based Collaboration: Permulin

    personal

    shared

    personalized

    output

    personalized

    output

    personalized

    input

    personal

    shared

    personalized

    output

    personalized

    input

    personal

    sharedshared

    personalized

    output

    personalized

    input

    personal

    BigMedia @ ISM 2014 M. Mhlhuser

  • 47

    Table Based Collaboration: Permulin

    Divide View

    Merge Views

    Interaction Concepts Divide / Merge

    BigMedia @ ISM 2014 M. Mhlhuser

  • 48

    View of User A

    View of User B

  • 49

    Table Based Collaboration: Permulin

    SharePeek

    Sharing &Peeking

    Interaction Conepts Share / Peek

    BigMedia @ ISM 2014 M. Mhlhuser

  • 50

    View of User A

    View of User B

  • 51

    Table Based Collaboration: Permuli

    Better Parallel WorkLarger Interaction Area

    User A User B

    Mutual AwarenessTruly Fluid Transition

    Kinect for user recog-nition

    3D Display

    Multi-touchframe

    Kinect for hand recog-nition

    Modified3D shutterglasses

    hardware setup today

    and tomorrow?

    evaluation results

    BigMedia @ ISM 2014 M. Mhlhuser

  • Several projects atop Anoto ePen technology!

    just one pen for .

    BigMedia @ ISM 2014 M. Mhlhuser 52

    Paper Like (1): Pen&Paper Computing

    Paper table (& wall) hybrid: paper+table physical objects

    hand writtenannotations

    tagging usingmenue cards

    printed userinterfaces

    folders books

  • Paper Like (2): Paper like displays

    BigMedia @ ISM 2014 M. Mhlhuser 53

    Multiple Paper-like Displays

    ?

  • Custom printed objects

    if interactive(!): boost tangible interaction

    were demonstrated by U Saarbrcken

    and by TK (@TU Darmstadt)

    are still at an early stage

    BigMedia @ ISM 2014 M. Mhlhuser 54

    Printed Tangible Interaction

    a b c

    d e

  • Spoken Interaction?Example Smart-Space Dialogs

    Speak to smart spaces -homogeneous UI vs. heterogeneous devices

    requires:1.context awareness

    2.user awareness (multi-speaker!)

    3.mike awareness (array headsets)

    plus (not included here):

    federation w/ other modalities

    openness (cf. Siri++: for app/service developers)

    BigMedia @ ISM 2014 M. Mhlhuser 55

  • C. LARGE SCALE INTERACTION

    (just briefly touched here)

    In-door: e.g., walls

    Out-door: e.g., facades

    Global: e.g., social or virtual

    Overarching challenge:Collaboration of / with a large user base

    BigMedia @ ISM 2014 M. Mhlhuser 56

  • Still lots of passive walls

    Interactive walls still inappropriate

    Remember lesson learnt today:

    novel technology new interaction concepts

    Collaborative wall interaction:

    only few concepts known, rarely in use

    large walls: real estate reach

    quest for mobile device federation!

    BigMedia @ ISM 2014 M. Mhlhuser 57

    Wall size interaction

    Interactive Walls & Rooms

    QUT Brisbane CUBE

    negative example

  • City Scale Example: Media Facades

    2 approaches:

    1. Indirect interaction input abstract aggregation

    cf. emotion metering, hotspots,

    2.Temporal individual control often as competition

    user creativity framed by app

    BigMedia @ ISM 2014 M. Mhlhuser 58

  • Global Interaction

    Here, CoStream@Home: in-situ socialNet home

    BigMedia @ ISM 2014 M. Mhlhuser 59

    Tv Broadcast

    CoStream

    User Generated Videos

    Notifications

    Friend List

    by defaultcompressed to vibration

    Cheering Frustration Clapping

  • D. ORTHOGONAL ISSUES

    (one slide for brevity)

    BigMedia @ ISM 2014 M. Mhlhuser 60

  • Collaboration, Federation, Intelligence

    Collaboration: mentioned above local distributed, team social

    quest for research (cf. our ConCalls!)

    Federated Interaction leverage multimodality

    challenge: open ad-hoc federation, latency

    Intelligent UIs: Proactivity: adjust UI implicitly + in advance

    Intelligibility: UI explains itself & its reasoning

    BigMedia @ ISM 2014 M. Mhlhuser 61

    Web basedfederated UIs

  • SUMMARY

    BigMedia @ ISM 2014 M. Mhlhuser 62

    capture interacttransport store

    processsense analyze

    BigMedia

  • Capture / Sense:mass amounts of blended-source/-target multi-sensory media

    Transport: processing @edge/net (Cloudlets) + fluid networks + (if 24/7) resilience

    Store: (partial) user site storage: novel processing, forgetting & privacy opportunities

    Process: Unify three BigData/Media pipelines: conventional + ML + crowd processing selected challenge: processing @edge/net

    Interact/Analyze: many challenges, selected: proliferating technologies + interaction concepts mobile: resizable displays, AR DR, on-body interaction

    (more) natural: implicit, table based, paper like, spoken

    large scale: walls, facades, social networks

    collaboration, federation, intelligence as orthogonal aspects

    BigMedia @ ISM 2014 M. Mhlhuser 63

    For Your Long Term Memory

  • imagine consequences on industry sectors:

    Software industry (every app ready for 25 sets of interaction concepts?)

    Media industry (OSN convergence done right?)

    Telecom industry (Cloudlets embraced?)

    Critical infastructures (500k surveillance cameras plus mobile reports?)

    BigMedia @ ISM 2014 M. Mhlhuser 64

    Food for Smalltalk