Apple TV seen through the eyes of an iOS developer

I preordered the new Apple TV the very day it was announced.

The attractive $99 price point certainly helped but really the truth was I couldn’t wait to start writing apps for the Apple TV.

Apple’s engineers have done a phenomenal job with the iOS SDK, especially with UIKit, the API framework and visual language that gives all iPhone/iPad apps a consistent look and feel, and makes it so easy for developers to build sophisticated yet intuitive user interfaces.

I played with my Apple TV over the weekend, checked out the various apps, and jotted down my thoughts on what the UIKit equivalent for Apple TV might look like. It didn't take more than a few seconds playing with the Apple TV to realize that there is a common visual language that is applied consistently across all the apps on the device. It is a big part of what makes the Apple TV so easy to use, which is no small achievement given that it all works with a remote with just 7 buttons. (By contrast, my Comcast remote has 47.) TVKit would probably be the logical name for the framework responsible for the Apple TV's look and feel, but since all this is pure speculation, I'm going to call it ??Kit for now. Where I think there is a TV equivalent of a particular UIKit class, I am going to use the ?? prefix, e.g. ??Label for UILabel.

MVC for Apple TV apps

Cocoa, Apple's frameworks and tools for app development on Mac OS X and iOS, is all about the Model-View-Controller design pattern. In UIKit, you build an interface using UIView widgets and control them with a UIViewController. ??Kit is unlikely to be any different. You build your Apple TV as a series of controller-managed views and simply transition from one view to the next.

The split view layout is fairly common across the Apple TV experience. You could have an on-screen keyboard in the left half of the screen and search results in the right half. Or a list of song titles on the right and detailed metadata for each song title on the left. You may have seen this layout before on the iPad, and the UIKit controller class that is responsible there is the UISplitViewController. I'm going to say there is probably a ??SplitViewController class for Apple TV that makes it just as easy to set up these split screen layouts.

Typical split-view layout on Apple TV

The most important view controller class on Apple TV, however, is probably ??NavigationController. Like UINavigationController in UIKit, a navigation controller object manages a series of view controllers in a stack that you push and pop. This is the core view management model in the Apple TV UI. Think of every screen in Apple TV as a card. Whenever you press a button on the card to reveal another card, the new card is placed on top of the previous card. When you hit the Menu button on the remote (which really means "back"), the new card is removed, revealing the previous card. The navigation controller manages this stack for you and animates the transition between views. In UIKit, views typically slide in and out of the screen. In most cases this is too visually jarring on the Apple TV because of the large display area, which is probably why Apple opted for the more subtle fade-through-black transition effect.

User interaction via the Apple Remote

On a UIKit device, you have a multitouch screen and your app handles touch events (and gesture events in iOS 3.2+) to create rich user interfaces.

On the Apple TV, it's a bit like the 80s all over again. You get the d-pad: 4 arrow keys, the Select (or OK) button, and the button labeled "Menu." This is a huge consideration when designing an Apple TV app and pretty much means most existing iOS apps will simply not be usable on an Apple TV, even if they boot up just fine. Multitouch interfaces are fundamentally incompatible with a d-pad.

Apps on the Apple TV typically involve using the arrow key to highlight a particular interactive view element (e.g. a button) and pressing the Select button to trigger a specific action tied to the element. My guess is that the developer typically does not need to handle the arrow key events explicitly. The app framework does the hard work of figuring out the next element to highlight in response to the arrow key events, based on the visual layout of the elements on the screen. The framework even animates the selection hint (e.g. the blue border around the highlighted element) in the usual Core Animation style, tweening not just the position and size but also its appearance (different elements have different ways of indicating that they are highlighted). This subtle animation cue helps to guide the eyes to the appropriate part of the screen when the highlighted element is changed. All the developer needs to do is ensure that the selectable elements have their normal and highlighted states properly configured, and that they respond appropriately to the key-down and key-up events that they receive.

Interestingly, the long-press to activate the rearrange/edit layout mode that you use on the iPhone to move your app icons around has found its way to the Apple TV, with the same cutesy wobble animation (a combination of a side-to-side tilt and a small circular motion). Apple TV uses this gesture to enable you to rearrange your favorites. Because this gesture is non-obvious and very rarely used, there is an on-screen hint that tells the user what to do.

Make your favorites jiggle!

Labels

??Label is a simple way of displaying a text string with a uniform font and style. What's interesting to note here is that, unlike UILabel, ??Label supports an additional truncation style (possibly defined in the ??LineBreakMode enum) that provides ticker-style horizontal scrolling of one-line text strings that are longer than the width of the label. Scrolling begins and ends with the usual ease-in/ease-out animation timing, with a slight pause between iterations. This truncation mode is very useful for displaying the full title of the item that is shown as part of a table row or a button in the highlighted state. The user has the opportunity to review the additional information and confirm that they have indeed chosen the correct item before they hit Select.

Labels can scroll horizontally, ticker-style

It's not hard to understand why the horizontal scrolling effect, while arguably a good way of packing in a good amount of information given limited screen real estate, isn't part of UIKit. Generally you don't want a whole bunch of animated scrolling text on the text (think back to the aesthetically challenged web 1.0 days). You only want the effect to apply to the one element that is highlighted or focused on the screen. In the absence of mouse-based or keyboard-based navigation, there is no way to support a highlighted or "hover" state that is persistent. Control elements in UIKit do have a highlighted state -- that's when they have received a touch-down event and your finger is physically over the element and obscuring it. Not a great time to be showing more information.

Buttons

The conventional, multi-purpose button as we know it is used very sparingly on Apple TV. The vast majority of the time, you trigger an action by selecting a row from a scrollable vertical or horizontal list (more on this below), not by navigating to a button and pressing Select. This keeps the navigation model linear and one-dimensional: the user should only have to navigate up and down, or left and right to get to where they want to be. The UI must not require the user to march all over a 2D grid to get from point A to point B.

Where you do see the conventional button used -- let's call it ??Button after UIButton in UIKit -- is on the detail page of shows and movies, where the user has the option of pressing a button to watch a preview or rent the title. In the normal state, the ??Button is monochromatic against a transparent background. The image inside a ??Button appears to be an alpha mask image, not unlike the images you would use in a UITabBarItem to set up a tab bar in UIKit. In the highlighted state, the button gets a blue tint and a gloss treatment. The gloss effect is similar to the automatic treatment that iOS app icons get. Additionally, the image in the button is given a bloom filter (Core Image?) to give it nice soft glow, and text strings are given a drop shadow to stand out better.

The glossy conventional "Rent" button

Text fields and the on-screen keyboard

Ah free-form text entry, the bane of any d-pad-based UI.

Apple TV provides 2 on-screen keyboard (OSK) layouts to solicit text input.

The full-screen layout displays the entire OSK on a single page and contains both alphanumeric characters and symbols. In this layout, the only task presented to the user is filling out a single text field. This is used in situations where the user needs to enter non-dictionary words, e.g. their username and password. If you press and hold down a direction key, you can navigate across the OSK quickly. In this mode, the cursor stops the moment it hits the edge of the OSK. If you press and release the direction key normally, the cursor does wrap around to the opposite edge of the OSK. Nice touch.

The half-screen layout is designed to be used in a split view layout. The OSK appears in the left half, while the outcome of the text input (e.g. a list of search results) appears in the right half. The half-screen OSK supports multiple pages (currently 2 pages) of characters, with the basic alphanumeric characters (A to Z, 0 to 9, space, delete) on the first page, and commonly used symbols and letters with accents/diacritics on the second page. This OSK is typically used for entering dictionary words (e.g. search terms). The Apple TV seems to use the same OSK layout regardless of the UI language setting, which probably explains the alphabetical A-to-Z ordering of letters as opposed to the QWERTY layout. (Users in non-English speaking countries that use the Latin alphabet typically don't use QWERTY but some variation thereof.) Note that unlike in the full-screen OSK, the cursor never wraps around in the half-screen OSK. If you go past the right edge of the OSK, you will enter the right half of the split view layout.

Lists

The venerable list abstraction is the workhorse of virtually all Apple TV UIs. On the Apple TV, you are usually browsing lists of songs, TV shows, movies, etc. or selecting from among multiple options in a list. The list UI framework on Apple TV is most likely related to UITableView and its related classes (UITableViewCell, UITableViewController) in UIKit, which in turn are derived from NSTableView in AppKit for Mac OS X apps. (Unlike NSTableView, UITableView can only have 1 column, so it is effectively a list.)

There appears to be two distinct kinds of lists: the usual vertical list that contains multiple rows and enables the user to scroll vertically, and the Apple TV-exclusive horizontal list, where the user scrolls horizontally, typically to browse cover art and related metadata.

Let's first talk about the vertical list:

The vertical list is probably a ??TableView. Like its UIKit counterpart UITableView, ??TableView automatically manages cell-reuse and supports dividing the table into sections, each with its own header. There are no scroll bars in Apple TV. ??TableView applies a fade-out gradient to the top and bottom-most visible rows to indicate that there are more results in either direction. In UIKit, you can scroll very quickly by flicking across the table. The way you'd do the same on an Apple TV is by pressing and holding down the up or down key on the remote.

Each row of a ??TableView is a ??TableViewCell. Table view cells can be marked selectable and receive key-down events, but they should not have interactive objects in its subview that they cannot be highlighted or selected. In other words, you can't use the d-pad to navigate within a table view cell. You can only highlight and select the entire cell.

A ??TableViewCell can be given a disclosure indicator accessory type (similar to UITableViewCellAccessoryDisclosureIndicator), a chevron that appears against the right edge of the cell to indicate to the user that selecting the cell takes the user one level deeper into the browse hierarchy and presents the user with another set of options to choose from (the destination must not conceptually be a leaf node).

If there is no chevron, the expectation is that selecting the cell will either change the display state of the cell (e.g. flipping an option from On to Off) or launch a conceptually different experience (e.g. start playing a video in full screen).

The versatile vertical list

The horizontal list is a bit more of a mystery to me as there is no clear UIKit equivalent. Functionally it looks like a row of individually selectable buttons, each with the same visual template. Given that there can be a large number of buttons in the horizontal list, it makes sense for it to be implemented and managed the same way as a vertical list (managed scrolling, automatic cell reuse, a mediator object between the data model and the table, etc.). The difference between the two is really only in the presentation. It would not surprise me at all if the horizontal list is just another ??TableView display mode. Note that unlike the vertical list, the horizontal list indicates more elements at either end of the table by rotating the cell slightly about the vertical axis, a nod to the Cover Flow effect.

A list that scrolls horizontally instead of vertically

Is there a Grid view?

And then there is something that looks like a grid of thumbnails and labels when you first come across it. It is used on the Top Movies page and extensively in the Netflix app, where you are shown an entire screen of movie poster art. Each row corresponds to a particular set of selection criteria, e.g. Top Rentals, Just Added, Blockbusters. Unlike a true grid, however, each row can be scrolled horizontally, independently of the other rows. In other words, we seem to be looking at a vertical list where each row is a horizontal list. I know I said before that it should not be possible to highlight an element within a table row, but well this would be the only exception, and only because allowing so is immediately intuitive to the user and does not break the mental navigation model.

Grid-like layout: a vertical list of horizontal lists?

Other UI elements

There's 3 other UI elements that caught my attention. I'll share my notes here:

UIActionSheet in UIKit presents the user with a series of buttons laid out vertically, allowing the user to choose among a set of options for carrying out a given task. The equivalent on Apple TV seems to be a modal popover, displayed over the current screen. You can see an example of this when you do a long press on the Now Playing screen to reveal options like "Start Genius" and "Browse Album."

The UIAlertView class in UIKit is similar to UIActionSheet, except that the buttons are displayed in a lightweight modal dialog box. On Apple TV, there are no dialog boxes. Feedback to the user (e.g. in response to a text field that failed validation) is usually displayed as part of the UI where the error was committed so that the user is not interrupted with a dialog box that needs to be dismissed. (This is the norm on modern websites too.) In the rare case where the error is unrecoverable, there is a ??AlertView to show the user an important message and require them to hit the Dismiss button to continue. Unlike in UIKit, ??AlertView takes over the entire screen and isn't presented as an overlay. This design decision makes sense. If the user has truly run into an unrecoverable error, dismissing the alert view should take them to a different part of the app where they can do something useful. There is no need for the user to continue to see the UI that triggered the alert view; they won't be returning to it.

The last UI element that I am going to cover in this post seems a bit out of place for a TV UI  -- the Apple TV equivalent of UIKit's UISegmentedControl. A segmented control object looks like a horizontal capsule divided into multiple segments, where each segment behaves like a button. It is used where you would normally use a drop-down menu or a set of mutually exclusive radio buttons in the web world. A very similar control is used in the Apple TV podcast and YouTube apps to enable the user to filter the list of results. It feels odd to me because it's the only UI element I've seen thus far that made me feel like I have to use all 4 arrow keys to navigate to it. The reality of course is quite different. In the podcast app, the ??SegmentedControl "owns" the row of pixels that it occupies, so you can always get to it by pressing the up/down arrow from anywhere on the screen.

So what kind of apps will we see?

One can only speculate what Jobs really meant when Business Week quoted him as saying there could be an app store for the Apple TV when the time was right.

But when exactly is the right time? In the same Business Week article, when asked if the iPad could evolve into the TV of tomorrow, Jobs replied, "That's how I do most of my TV watching today." Will the TV continue to remain relevant if media consumption is moving to devices like the iPad?

There's always gaming, one might say. The abundance of iOS games is one of the reasons iOS devices are more popular than Android devices among "normal" folks like us. Would an app store turn the Apple TV into a mass market gaming console for the living room? Possibly. The Apple TV doesn't have the hardware to compete with the likes of the Sony PS3 and the XBOX 360, but the success of the Nintendo Wii shows that excellent game play and innovation in user interaction (the Wii Remote) can trump Hollywood-scale production budgets and GPU-fueled bells and whistles. All Apple needs is an affordable game controller accessory.

Finally, it is obvious that the d-pad is not the future of computing and human-computer interaction. Multitouch is. Displaying an interactive UI on a screen 10 feet away from you is a fundamentally flawed premise, and the reason we still do this today is because we are constrained by technology. When Apple launched the original iPhone, they showed us that the most compelling interaction comes from placing the UI right under the user's finger and creating the illusion that the user is directly manipulating the contents of the screen. The iPhone 4 went so far as to reduce the thickness of the glass separating the finger from the screen, which in subtle ways creates a stronger emotional bond between the user and the app they are interacting with (creepy I know).

So what is the future of Apple TV apps IMHO? I'd say look no further than AirPlay.

AirPlay turns your TV into a dumb display terminal (albeit a large fancy one), playing whatever your iPad streams to it. You interact with your TV via the iPad, never directly. The iPad is where you discover and manage content. It's where you decide to transfer playback to your TV -- no doubt the better form factor for passive couch potato-style viewing -- after you've found the video you want to watch. In other words, your TV is no longer your primary screen; your iPad is. The Apple TV is simply a $99 wireless "TV out" accessory that connects your iPad to your TV, and even if that's all the Apple TV is, the $99 price tag still seems surprisingly reasonable.

But let's not forget that the Apple TV can do much more than stream video. It's got enough CPU and GPU horsepower to do all sorts of fancy things to augment to video viewing experience. And here's where I think the 3rd party apps come in. Imagine a whole class of apps that are designed to run on an iPad (or iPhone) and on the Apple TV concurrently. On the iPad, the app presents a rich content discovery experience and acts as the "master" device. On the Apple TV, the app acts as the "slave", doing whatever the iPad tells it to. It could pull up a video for streaming (either from the iPad or from the "cloud") and augment the experience with dynamic content (e.g. a live twitter feed, stock tickers). These apps combine the intuitiveness and ease of use of a multitouch interface with the largest screen and best sound system in your house. These are exactly the apps I can't wait to write.