What is a computer cluster?

A computer cluster is a group of connected computers which work together to solve complex or time-consuming tasks.

While most computer clusters are cloud-based, Apple provides the Compressor app on the Mac which can convert between video formats. To speed up video conversion, you can link multiple Compressor apps running on nearby Macs.

We provide the Photogrammetry Cluster and Photogrammetry Cluster Manager apps to speed up the generation, testing, and visualization of 3D models used in augmented reality and virtual reality (AR/VR).

What is an iPhone camera cluster?

An iPhone camera cluster allows you to remotely manage and synchronize the cameras from nearby iPhones. In addition to managing and synchronizing multiple iPhones, our iPhone cluster integrates an iPhone’s cameras with other capabilities of the iPhone across multiple iPhones, if available.

An advanced example of this integration is to use the iPhone cluster for multi-camera gait analysis– a new metric believed to be useful in the early diagnosis of Parkinson’s disease– whereby the iPhone’s health statistics for walking asymmetry and steadiness are displayed on the live video stream. A doctor can then watch a patient’s walk from multiple angles in addition to seeing the corresponding health metrics.

Another example is the integration of the iPhone’s cameras with its AI models of the human shape allowing our camera cluster to segment people in the video stream and render them in silhouette or invisible thus protecting their identities and actions. This is a much needed and requested feature of “in the field” reporters who want to conduct live interviews of migrants and trafficked individuals while still protecting their identities. Furthermore, with the integration of multiple iPhones our camera cluster provides interviewers with the ability to “Shoot for the Edit” and easily record a wide shot, medium shot, close up, and cutaway.

These and other cluster features are implemented by the Camera Cluster for iPhone app and Cluster Manager app for Mac and iPad. Together, these apps provide an AI clustering layer on top of Apple’s hardware and software linking macOS, iOS, iPadOS, and other devices via your local network.

In its initial release, the AI clustering layer provides distributed vision, processing, and dynamic cluster reconfiguration.

What are the major features of the iPhone camera cluster?

Broadcast and screen share any app from any iPhone in the cluster to the Cluster Manager app running on your Mac or iPad.
Stream live video from the cameras in the cluster to the Cluster Manager app and telestrate over the live video.
Capture better and new types of photos by synchronizing cameras across the cluster.

How do I create an iPhone camera cluster?

Download and install both the Cluster Manager app for Mac or iPad and Camera Cluster for iPhone app.

Once you launch the Cluster Manager, you will be presented with the cluster view:

The Cluster Manager on launch

After launching the Camera Cluster for iPhone on one or more nearby iPhones, the cluster view will display the state of each iPhone. Each cluster can include about 10 devices but can only have one cluster manager. iPhones or other devices will automatically join the cluster even though only iPhones can be remotely managed:

A 1-camera cluster with 1 connected iPhone

A 5-camera cluster with 5 connected iPhones

A 2-camera cluster with 1 connected iPhone and 1 connecting iPhone

A 2-camera cluster with 1 connected iPhone and 1 disconnected iPhone

A 4-camera cluster with 4 connected iPhones and 1 iPhone overheating.
Set the option “Migrate Video Processing” to have the cluster automatically reconfigure itself to have another device in the cluster process the video stream.

This option is available from the cluster configuration view:

The cluster view has options to configure the cluster and to configure synchronized photography

If supported on your version of the Cluster Manager, select the option “Migrate Video Processing”:

The Cluster Configuration window allows you to reorder the iPhones in the cluster, import a TLS certificate, and migrate video processing when an iPhone overheats

How do I broadcast and screen share any iPhone app?

Select the iPhone in the cluster view and then click the “Live Screen Broadcast” button.

Select an iPhone and then instruct it to Broadcast, Stream, or Capture

The Cluster Manager will open a new window and wait to receive the broadcast from the iPhone:

The live stream will be displayed in this window allowing you to telestrate over the live broadcast

On the selected iPhone, you will see a broadcast image informing you that it’s ready to screen share:

The broadcast image indicates that the cluster is ready for the broadcast

Launch the iPhone’s Control Panel and tap on the Screen Broadcast button:

Tap and hold (that is, long press) the Screen Recording button in the iPhone Control Panel

Select “Camera Cluster” to begin broadcasting to the Cluster Manager:

Broadcast a screen recording of your iPhone to the Cluster Manager

After a few moments, you will see the iPhone broadcast in the Camera Stream window on your Mac or iPad:

This window will display the iPhone’s broadcast

Resize the video stream window and telestrate. On the Mac use your mouse, trackpad, or a stylus such as a Wacom stylus. On the iPad, you can use your finger or Apple Pencil:

Telestrate the broadcast with a stylus or Apple Pencil

How do I live stream from an iPhone’s camera?

Select the iPhone in the cluster view and then click the “Live Camera Stream” button:

Select an iPhone and then instruct it to Broadcast, Stream, or Capture

The cluster manager will open a new window and wait to receive the live stream from the iPhone:

The live stream will be displayed in this window allowing you to telestrate over the live video stream

On the iPhone, the app will display a web cam icon in the top left to inform you that it’s live streaming:

The web cam icon informs you of live streaming

If you experience WiFi or other problems with the stream, tap the web cam icon to restart the live stream:

Tap the web cam icon to restart the live stream

Alternatively, if the iPhone is out of reach, you can restart the stream from the Cluster Manager.
Back on the Mac, resize the video stream window, add a background image or animated GIF and telestrate. On the Mac use your mouse, trackpad, or a stylus such as a Wacom stylus. On the iPad, you can draw with your finger or Apple Pencil:

Drop an image or animated gif to customize the background and telestrate

You can customize the live stream with the following menu options:

Customize the video stream

How do I remotely select different cameras, resolution, or use a hand or facial gesture to take a picture?

To speed up the creation of a thumbnail or poster image, you can have the cluster take a picture during a live stream. Click the “Camera” menu to select from the front and back cameras and video resolution such as 4K, 1080 HD, or 720 HD. You can select a physical gesture to trigger any iPhone in the cluster to take a picture:

Select from the front or back cameras, video resolution, and specific gesture to take a picture during the live stream

To remotely instruct the iPhone to take a picture when you smile, select the menu “Gesture to Take a Picture > Smile”:

Smile to a take a picture

The iPhone will display a smile to indicate that it will take pictures whenever it detects a smile during the live video stream:

A smiley face image indicates that a picture will be uploaded to the cluster manager whenever a smile is detected

Pictures are automatically uploaded from the iPhone to the Cluster Manager:

Pictures are automatically uploaded from the cluster to the cluster manager

The default upload directory is your Downloads/GroupThink-Camera-Cluster directory and you can choose a different directory from the Cluster Camera Configuration window which is accessible by clicking on the camera icon in the cluster view:

The cluster view has options to configure the cluster and to configure synchronized photography
The Cluster Camera Configuration window allows you to change the directory for photo uploads, select which camera or camera profile– such as Photogrammetry– to use by the iPhones in the cluster during synchronized photography, photo numbering, and turntable synchronization

How do I customize the video stream?

The initial focus of our iPhone camera cluster is centered around people, their privacy, and their empowerment by integrating the iPhone’s camera with other capabilities of the iPhone across multiple iPhones, if available. To enhance a video stream, you can add real-time filters, overlays, and backgrounds to any live stream from any camera on any iPhone in the cluster.

How do I add a real-time filter to a live video stream?

The cluster provides real-time segmentation filters so that you may:

  1. Hide your background from the video stream and only display people– whether you want to hide a messy environment or corporate secrets– more and more users want the option to remove distractions during video calls.
  2. Render people in silhouette and thereby obscure their identity or state of undress– with the former needed by journalists interviewing migrants and trafficked individuals and the later is very useful in home healthcare.
  3. Render people invisible and show a background image or animated GIF. In a break during a live media production, you can select this option until you are ready to start the live stream. In home healthcare, with Siri, the cluster will report how many people are seen and by which iPhone. This empowers parents and guardians with mobility or vision impairments to “watch over” their family without having to “watch” them.

To enable segmentation, select the “Segmentation” menu in iPhone Camera Stream window toolbar:

The segmentation filter allows you to only display people, display people in silhouette, and hide people from the live video stream

Once you have segmented people from their background, you can change the video’s background color. If you want to explore chroma key– a technique used in movies to simulate flying– select green for a real-time green screen. Select a black background to conduct an intimate interview much like on the Charlie Rose set– widely recognized as the gold standard for distraction-free interviews. If you want to see how the anonymity of a silhouette is able to represent every person, try to recreate the feel of the iconic iPod commercials using silhouette segmentation and different background colors. The iPhone camera cluster supports the following segmentation background colors:

From the “Segmentation” menu, choose a background color

Here is an example of a recording of a 2-camera presentation using real-time person segmentation with a black background:

A 2-camera cluster live streaming a presentation using person segmentation and a black background

How do I add a real-time overlay to a live video stream?

In support of healthcare and exercise, the cluster supports adding the iPhone’s health data to the bottom of the video stream. To simplify informal home healthcare, you can add heart rate and blood oxygen levels to the video stream which are the most commonly requested vitals to track by caregivers of elderly family members. In support of complex telehealth video calls– or if a cluster is deployed at a doctor’s office– the cluster can be configured with more iPhones to support multi-camera gait analysis by adding a user’s walking asymmetry and stability statistics from the iPhone’s Health data to the bottom of the video stream.

To add a health overlay, select from the “Overlay Health Data” menu:

You can add heart rate and blood oxygen levels in an overlay
You can add walking asymmetry and steadiness levels in an overlay

If you have not yet authorized access to health data on the iPhone, the iPhone will prompt you. Once authorized to access health data, the video streams will be updated with live health data:

A 2-camera iPhone cluster live stream with person segmentation and health data overlays

How do I add a background to a live video stream?

In addition to selecting a background color when segmenting people from their background, you can instead use a picture or an animated GIF. Just drag and drop an image file onto the live stream window. Make sure that you select a segmentation menu option such as silhouette.

Silhouette segmentation has many use cases: caregivers and receivers find that rendering an elder in silhouette protects their privacy and dignity during different states of undress more so than a baby-cam. We refer to this as “watching over” a person rather than “watching” a person. Journalists and “in the field” reporters are able to guard the identities of interviewees during live video streams. And aspiring filmmakers can explore different scene compositions and create storyboards with limited cost:

Silhouette segmentation has many use cases


How do I synchronize photo capture?

Whether you’re capturing 200 photos for a 3D photogrammetry model or 8 photos for a 360º view of a person, our iPhone camera cluster provides speed and accuracy. The cluster supports remote photo capture from a single iPhone and synchronized photo capture from multiple iPhones. iPhones in the cluster can also synchronize their photo capture with a turntable.

iPhones in the cluster coordinate to ensure photographs are correctly captured, numbered, and uploaded to the Cluster Manager.

How do I remotely capture photos from a single iPhone?

Select the iPhone in the cluster view and then click the “Photo Capture” button:

Select an iPhone and then instruct it to Broadcast, Stream, or Capture

Configure the remote iPhone settings and the cluster view will display a 20 second countdown ensuring that the iPhone or Phones have synchronized their capture configurations and providing you with sufficient time to synchronize with a turntable or otherwise prepare your subject:

Photo capture settings for a specific iPhone
The cluster view displays a photo capture countdown whether it’s for 1 iPhone as seen above, or all the iPhones in the cluster. In this example, a 2 second pause was set between shots and consequently, the target icon shows a 2 second delay from the selected iPhone informing you when to start the turntable rotation

How do I synchronize photo capture across all the iPhones in the cluster?

To configure all the iPhones in the cluster for synchronized photo capture, use the Cluster Camera Configuration window which is accessible by clicking on the camera icon in the cluster view:

The cluster view has options to configure the cluster and to configure synchronized photography
The Cluster Camera Configuration window allows you to change the directory for photo uploads, select which camera or camera profile– such as Photogrammetry– to use by the iPhones in the cluster during synchronized photography, photo numbering, and turntable synchronization
The cluster view displays a photo capture countdown whether it’s for 1 iPhone or all the iPhones in the cluster as seen above. In this example, a 1 second pause was set between shots and consequently, the target icon shows a 1 second delay from the iPhones in the cluster informing you when to start the turntable rotation

How do I synchronize photo capture with a turntable?

Both the photography settings for the cluster and an individual iPhone allow you to define how long to pause between shots. Use that value to synchronize your turntable’s intermittent rotation interval:

How do I know when all of the photos are uploaded from the cluster and how do I stop a photo capture?

The File Transfer window displays the upload status of each photo from every iPhone showing which photo has been uploaded and which photos have yet to be uploaded. You can also cancel a long photo capture session which has gone wrong:

The File Transfer window allows you to stop photo capture

Recipes

How do I configure the iPhone camera cluster for healthcare or fitness?

For home healthcare, you can have the cluster render an ill family member in silhouette or invisible and display their vitals in a video overlay. This protects their dignity while they are in a state of undress while allowing caregivers to “watch over” them without “watching” them.

For family members with mobility or vision impairments, use Siri to ask the cluster how many people does each camera observe and thereby empowering parents to know the status of the house and their children or caregivers.

For telehealth or if the cluster is deployed at a doctor’s office, you can segment people and set their background to black. This allows everyone to focus on the conversation rather than be distracted by everyone’s background.

You can use the multiple cameras for gait analysis and display walking asymmetry and steadiness in a video overlay.

For exercise where the cluster is deployed at the gym or at home for remote exercise over a video call, you can segment people in each video stream and use black for the background color to focus on the person and not their environment. Display their vitals in a video overlay.

Related Links:

How do I configure the iPhone camera cluster for an entertainment or media production?

You can use the multiple cameras in pre-production to explore different scene compositions and create storyboards with limited cost. You can explore chroma key and real-time green screens by segmenting people in the video stream and use green for the background color.

Related Links:

How do I configure the iPhone camera cluster for journalism/reporter’s interview?

To recreate the intimate set used on the Charlie Rose show, segment people in the video stream and use black for the background color. For “in the field” reporters who want to conduct live interviews of migrants and trafficked individuals while still protecting their identities, segment people in the video stream and render them in silhouette thus protecting their identities and actions. Furthermore, with the integration of multiple iPhones our camera cluster provides interviewers with the ability to “Shoot for the Edit” and easily record a wide shot, medium shot, close up, and cutaway.

Related Links:

How do I configure the iPhone camera cluster for 3D photogrammetry?

If using a turntable, set the turntable’s rotation mode to ‘intermittent rotation’ or ‘step by step’ mode. Nonetheless, ensure that you capture all sides of the object with at least a 70% overlap between photos. Set appropriate values in the Cluster Camera Configuration window and ensure that you select the camera profile “Photogrammetry” rather than selecting specific cameras. Enable the option “Capture order is preserved” in the section “Cluster-Based Photo Numbering” so that files will be numbered sequentially despite being captured by different iPhones. On the Cluster Manager app for Mac or iPad ensure that you choose the photography upload directory.

Related Links: