Photos Tech Brief Sept2019 - Apple

People Relationships infers closeness based on the number of photos of a person,set relationships in Contacts, and frequency of communication in Messages. Social groups identifies the people who often show up in photos together. Places Home and work leverages Contacts and Maps to understand where theuser has set “Home” and “Work” addresses.Smart photo previewsPhotos uses a new saliency engine thatidentifies what’s noticeable or importantin an image when cropping photopreviews. Frequent locations looks across patterns of time and location to determineif photos and videos were taken at a place where the user frequently takespictures. Special locations identifies photos and videos that were taken at new orunique locations. Trips infers when the user has taken photos and videos in a location that isfar away from their home. Dates Important dates uses Calendar and Contacts to understand importantpersonal dates like birthdays and anniversaries. Holidays understands which holidays the user has celebrated based on alist of popular holidays in the user’s country and the scenes of photos fromthose days.The knowledge graph helps Photos personalize the Photos tab by discoveringmeaningful patterns and predicting smart correlations. For example, Photosmay use the frequency of messages sent to a person as an indicator that theperson is important to the user. When this data is correlated with a namedperson in the People album, Photos will know to show more of that personthroughout the app, like in Memories or as a suggestion in Search.The user’s knowledge graph is used to identify the significant events thatare shown in the Months view. Significant events in Photos range from trips,weddings, and concerts to everyday events such as a day in the park. Everynight when the device is connected to power, Photos uses the on-deviceknowledge graph to analyze each event and select which events to presentin the Months view: The Months selection algorithm first checks whether there are anysignificant events in the month, selecting up to five. If there are more than five significant events in the month, the device willnarrow down the set, and Photos will attempt to spread out featured eventsacross the month. If there are less than five significant events, Months will include otherevents—like a series of photos taken at home—up to five, so there’salways something for the user to explore.Photos September 20199

Finding the perfect loopAutoplaying Live Photos and videosbrings the new Photos tab to life. Photosuses machine learning that examinessignals like video metadata, stabilization,and dynamism to detect great Live Photoloops and video clips so device knowswhich videos to present to the user.The Years viewRediscovering highlights in Years.The Years ViewThe Years view is an ever-changing view into the user’s library, letting the userquickly scan through significant events of the past. It’s designed to alwaysshow something new, prompting rediscovery throughout the entire library.Contextual Browsing in YearsThe Years view analyzes data from the user’s device to provide a look back intime that’s contextual to what’s happening today. On this dayFor the default Years view experience, the device will find the events in pastyears closest to today’s date and present them to the user, promptingrediscovery of past years. BirthdaysUsing the knowledge graph, Photos may determine that it’s the birthday ofsomeone that matters to the user. On these days, Photos will update eventswith photos of the birthday person to highlight a photo with them in it. Thiscreates an effect that transforms Months and Years to celebrate their birthdayso the user can relive past memories with the person. Annual conferences and festivalsThe Years view can even display past events that don’t happen on the exactsame date every year, like an annual conference (for example, WWDC) or afestival (for example, Burning Man). Photos identifies the event using thedevice’s public event classifier. If the event is happening around this time,Years will look for shots from the same event in past years and feature them.Photos September 201910

Photo and Video EditingThe Photos app provides powerful photo and video editing tools that help makeeveryone a creative pro. Using advanced technology, Photos helps users easilycreate the perfect look with their photos and videos. iOS 13 and iPadOS provideusers with more creative possibilities and control over their photos and videoswith easy-to-use tools that are powered by advanced technology and designedand optimized for iPhone and iPad.Editing in PhotosPhoto editing is more comprehensive and intuitive with new tools that are easier to apply, adjust, andreview at a glance.Media-specific editing toolsPowerful editing in Photos goes beyondphotos and videos. Media types, likeLive Photos, Slo-Mo videos, and Portraitmode, all have unique editing capabilitiesin Photos, like changing the key frame ofa Live Photo and adjusting the transitionfrom normal speed to slow motion in aSlo-Mo video.Editing in Photos is designed to be easy while also providing users with finegrain control over every aspect of their photos. There are over 17 adjustmenttools in the Photos app, ranging from one-tap controls like Auto Enhance topro-level adjustments like Noise Reduction. These effects include: Auto EnhanceAutomatically adjusts the color and contrast of a photo, bringing the importantfeatures, like deep blacks and properly exposed highlights, to the foreground. ExposureAdjusts the tones—the amount of light—throughout an entire photo. BrillianceApplies region-specific adjustments to brighten dark areas, pull in highlights,and add contrast to reveal hidden detail and make a photo look richer andmore vibrant. HighlightsChanges the highlight detail, the brightest areas of a photo. By decreasing theHighlights value, the brightest areas of the photo will become more dark.Photos September 201911

ShadowsAdjusts the detail that appears in shadows and other dark areas. By increasingShadows, the darkest areas of a photo will become more bright. ContrastControls contrast, the range between the lightest and darkest parts ofWhat is a photo histogram?A photo histogram represents thenumber of pixels in an image in acertain area of the tonal scale. In aphoto histogram, the graph shows thenumber of pixels for each brightnesslevel from black (0% brightness) towhite (100% brightness).A histogram helps a user understandthe balance of light within a photo. Forexample, the histogram above depicts aphotograph that’s clipping highlights, aspixels are stacked toward the right of thehistogram in the brightest tones.Photos uses image histograms, alongwith tonal curves and pixel statistics, toperform a number of intelligent editingadjustments that include Auto Enhance,Brilliance, and Shadows.a photo. BrightnessAdjusts the brightness—the overall lightness or darkness—by raising themidtones of a photo. Black PointSets the point at which the darkest parts of an image become completelyblack without any detail. Setting the black point can improve the contrast ina washed-out image. SaturationAdjusts a photo’s overall color intensity. VibranceBoosts muted colors to make a photo more rich without affecting skin tonesand saturated colors. WarmthBalances the warmth of an image by adjusting color temperature (blue toyellow). TintAdjusts the warmth of an image by adjusting color tint (green to magenta). SharpnessChanges a photo by making its edges crisper and better defined. DefinitionIncreases image clarity by adjusting the definition slider. Noise ReductionReduces or eliminates noise (such as graininess or speckles) in a photo. VignetteAdds shading to the edges of a photo to highlight a powerful moment.Auto EnhanceAuto Enhance is the best place for a user to start when creating the right lookfor their photo or video. With just a tap, Auto Enhance approaches the imagehow a pro photographer would, analyzing each and every pixel of the image andautomatically adjusting the tonal extremes of the photo, bringing the importantfeatures, like deep blacks and properly exposed highlights, to the foreground.And in iOS 13 and iPadOS, Auto Enhance now lets users control the intensityof their automatic adjustments. As users increase or decrease Auto Enhance,they’ll see other adjustments—including Exposure, Brilliance, Highlights,Shadows, Contrast, Brightness, Black Point, Saturation, and Vibrance—intelligently change with it.Photos September 201912

Because every photo is unique, Auto Enhance is designed to optimize each andevery photo in an intelligent way. Each editing adjustment relates to the othersto create a balanced, properly exposed image.When a user taps the Auto Enhance button, the following steps happen: Image analysisAnalyzes the tone curve of the full image using a histogram, and corrects forcommon lighting scenarios like a backlit or underexposed photo. Face detectionIntelligently determines if there’s a face in the photo and adjusts the whitebalance, Warmth and Tint, to account for skin tone. This ensures that theautomatic adjustments do not wash out the person’s skin tone. Sets initial adjustment valuesSets the initial values for each adjustment, letting the user view the results.Each individual adjustment, like Brilliance or Vibrance, can then beindependently controlled to fine-tune a single aspect of the user’s photo. Determines Auto Enhance rangeGenerates the relationship between each of the adjustments for a specificshot, letting the user turn up or turn down the Auto Enhance effect using amacro slider that controls many adjustments at once.Auto Enhance is designed to simplify the photo and video editing process,whether the user is a professional photographer or just getting started. As theslider changes, there’s visual indication that other adjustments are changingwith it. This enables the user to view and modify what’s happening to theirphotos, providing them with a unique look at how each adjustment relates tothe others. As the user slides to the right, Auto Enhance makes the imagebrighter without blowing out the highlights and affecting the image’s whitebalance. As the user slides to the left, the effect darkens the image withoutlosing shadow detail.Smart Adjustments with Brilliance and ShadowsBrilliance and perceived colorBrilliance is a color-neutral adjustment,which means that no saturation isapplied. However, there may be aperceived change in color becausebrighter images with more contrastoften appear more vibrant.While many tools, like Exposure, provide global adjustments, meaning that theyapply adjustment levels uniformly across the entire range of a photo, Brillianceand Shadows are designed with even more power for smart, spatially localizedimage editing. Both Shadows and Brilliance apply region-specific adjustmentsthat produce dramatic effects without impacting the overall image. For example,with Shadows, the user can adjust the detail that appears in shadows withoutimpacting highlights. With Brilliance, the user can brighten dark areas, pull inhighlights, and add contrast to reveal hidden detail and make their photo lookricher and more vibrant.To create these effects, Photos divides the image into a 31x31 grid in real time,taking a closer look into each region of the shot. The device identifies one ormore image characteristics for each tile, creating a region-specific histogramand tonal curve. This provides the device with an approximation of the exposurePhotos September 201913

of each area of the image, and the Photos app treats each region as if it werea tiny image itself, computing the proper Brilliance or Shadows adjustment forthat region.The device combines the results of these regions and preserves areas whereadjustments are different between regions. This helps ensure that spatiallyadjusted regions don’t create artifacts in the areas of an image that transitionfrom dark (underexposed) to bright (overexposed), such as where theforeground transitions to the background of a photo of a backlit subject. Finally,the device creates an adjustment curve for each pixel, determines the range forthe slider, and adjusts the original photo, completing the user’s edits.Photo editingAdjustments, filters, and crop support video editing, so users can rotate, increase exposure, or evenapply filters to their videos.Powerful Video EditingVideo editing supportVideo editing supports all formatscaptured on iPhone, including video in4K at 60 fps and slo-mo in 1080p at240 fps. It also works with videosimported in industry-standard formatslike M4V, MP4, AVC, AVCI, and MOV.iPhone doesn’t just capture still photos—it’s a professional-level video camerathat’s always with users, allowing them to capture high-quality 4K video withextended dynamic range. With iOS 13 and iPadOS, users can now access thepowerful photo editing tools in video. Adjustments, filters, and crop supportvideo editing, so users can rotate, increase exposure, or even apply filters totheir videos.Users can edit a video in the Photos app in 32 ways: Trim Highlights Black Point Tint Vignette Dramatic Silvertone RotatePhotos September 2019 Sound on/off Shadows Saturation Sharpness Vivid Dramatic Warm Noir Straighten Auto Enhance Contrast Vibrance Definition Vivid Warm Dramatic Cool Flip Vertical Exposure Brightness Warmth Noise Reduction Vivid Cool Mono Crop Horizontal14

When editing a video, the device renders an adjusted video in real time,letting the user add or remove multiple adjustments in a single session. Videoadjustments apply to each and every frame of the video, providing temporalconsistency throughout the user’s video. This ensures that each frame of theuser’s video has a consistent look. When adjustments are saved by the user, thedevice renders the changes frame-by-frame, across every pixel of each frame,throughout the video. This process is optimized to run efficiently on the device’sGPU, quickly rendering a beautiful video that’s ready to share.Video edits are nondestructive, so the user can remove any effect, like a filter,or undo a trim to return to their original video. When the rendering process iscomplete, the device creates an auxiliary video file that includes the adjustedvideo paired with the original. This allows the user—at any point—to update anadjustment value or revert all changes.Photos September 201915

Portrait LightingPortrait Lighting is inspired by the real techniques of professional studiophotographers, who move the position of light sources—like soft-boxes,reflectors, diffusers, absorbers, and spotlights—to change the look and feel ofa portrait photograph. Portrait Lighting brings this ability to iPhone users withadvanced on-device machine learning to virtually re-create these effects whilecapturing in Camera and while editing in Photos.Portrait LightingUsers can virtually adjust the position and intensity of studio lighting.Preserving Details with Semantic SegmentationTo achieve the level of precision needed for fine-grain Portrait Lightingadjustments, the Photos app uses an advanced segmentation technology thatlocates and separates the subject from the background of an image with greatdetail and clarity. With iOS 13, this technology has been updated to identifyspecific facial regions, semantically allowing the device to isolate hair, skin, andteeth in a photo. Semantic segmentation allows the device to understand whichregions of the face to light, like skin, while determining which areas, like beards,need to be preserved when simulating studio-quality lighting effects acrossdifferent parts of the face.What is segmentation?Segmentation is the process of locatingand separating the subject, or specificregions, from the background of animage with great detail and clarity.Segmentation mattesPerson, hair, skin, and teeth segmentation mattes.Photos September 201916

When a user takes a Portrait mode photo, the Camera app uses the AppleNeural Engine in the A12 Bionic chip (and later) and runs multiple on-devicemachine learning models. This process analyzes data from the camera sensor,depth data, segmentation, and the pixels of the image itself to determine if aperson is visible and locates the unique geometry of the subject’s face.If a face is visible in the photo, the device will generate a final output image,as well as multiple auxiliary images, like person, hair, skin, eyes, and teethsegmentation mattes. Each of these images is used when applying the followingPortrait Lighting effects, all of which are accessible by third-party developers: Primary imageDisplays the user’s Portrait mode shot with all effects. For example, thesubject on a blurred background with a depth-of-field effect. Depth mapIndicates distance from the camera to that part of the image (either inabsolute terms or relative to other pixels in the depth map). Person segmentation matteSeparates the subject from the background. Hair segmentation matteSeparates the subject’s hair region from the non-hair regions. It evenseparates hair details, like small strands of hair, from the background. Skin segmentation matteLocates the skin regions of the subject. Teeth segmentation matteLocates the subject’s teeth when they are visible.Using Machine Learning to Move Light VirtuallyWith the ability to analyze unique facial regions in real time, Portrait Lightingnow lets users virtually move the position of light sources right on their iPhone,helping them dial in their portraits—whether they’re shooting in Camer

The Months view understands if a set of photos is important—like a family outing, concert, birthday party, anniversary, or trip—and highlights these significant moments in a beautiful interface. The Photos Knowledge Graph Photos analyzes a user's photo library, deeply connecting and correlating data