Can AI Fix the Portrait vs. Landscape Problem?

Since the first commercial camera came along in 1839, we’ve taken several trillion photos. About 80 percent have been horizontal—landscape orientation.

After all: We have two eyes, mounted horizontally on our faces. Our view of the world is a wide horizontal slice. Movie cameras, movie screens, TV screens, and computer screens are horizontal for the same reason.

But then the iPhone happened. It became the #1 most used camera on earth, and everything changed.

The smartphone wasn’t designed to accommodate our eyes; it was designed to accommodate our hands. Most people use two hands to hold the phone horizontally. That’s a huge psychological and physical barrier. A vertical (V) picture is a one-handed snap; a horizontal (H) one is a bother.

For people who work in TV (I’m a correspondent for “CBS Sunday Morning”), this development is an aesthetic disaster. What happens when we want to incorporate a phone video—a bystander video, for example—into one of our TV stories? What are we supposed to do with the huge empty “pillarbox” areas on either side?

Usually, we insert a huge, blurry slice of the V video on each side. It’s weird and distracting.

Similarly, if you post a horizontal video on social media, it gets squished the other way. Now it’s stubby and short.

Or they stack two videos. But you can’t watch both. Your eyes have to jump from one to the other.

And if a post (V) incorporates a TV clip (H) that itself contains a phone video (V)—forget it. You’ve now added letterbox bars on all four sides of the image. The result is a microscopic video surrounded by a sea of blur.

Now, if you do shoot your stuff vertically, it’ll also get seen by more people on Insta/TikTok. “Your 9-by-16s, your 4-by-5s, and your 1-by-1s are almost always gonna do significantly better algorithmically than your wides,” says New York-based advertising art director Charlotte Simons.

(Similarly, she says, the algorithms prefer videos to photos. “So if you want the most algorithmic love, you should actually turn your photo into a one-second video. It’ll do significantly better.”)

For 15 years, that’s where the H vs. V situation stood: a disaster with no solution. Then, in 2020, someone spent $1 billion trying to fix the problem.

It was a new streaming service called Quibi. It shut down after six months, but the short-form episodes brought a radical solution to the phone-shape problem: a technology called Turnstyle.

You, the viewer, could rotate your phone, even in mid-scene—and the screen would widen out from portrait to landscape or vice versa. Crazy!

Directors and editors had to shoot in both formats simultaneously. Most shot in 8K resolution—many more pixels than either format would need—and then, later, chopped out the two separate versions.

As it turns out, that option is available to you, too. You could be shooting both H and V simultaneously. Apps like DualShot Recorder capture video from the wide and ultra-wide rear cameras simultaneously.

Unfortunately, every video now takes up twice as much space on your phone (both formats are full-resolution). Furthermore, paying an annual fee just to capture duplicate videos will never have mainstream appeal. Not many people get as riled up about the H vs. V problem as I do.

There is, however, one final option: AI tools.

You’ve always been able to convert H to V, or vice versa, by cropping. Of course, you lose important elements of the image in the process.

But AI presents an option we’ve never had before: Cropping outward, generating material to fill in the letterbox bars.

In software like Photoshop and ChatGPT, you can now ask AI to turn a portrait shot into a landscape one by “outpainting.”

As a society, we’re going to have to decide how we feel about outpainting.

• Viewpoint 1: This is no big deal. We’ve always touched up photos and videos. We crop, brighten, boost their colors.

• Viewpoint 2: This is deceit. It’s manipulating viewers into misunderstanding the original scene. It’s like putting Oprah’s head on Ann-Margaret’s body.

And we’ll have to make our decision soon. In September, iOS 27 will arrive on 1.5 billion iPhones. The Photos app will include an Extend button. (A similar feature, called Uncrop, is already available on some Android phones.)

On one hand, Apple’s Extend button won’t let you generate more than 25 percent of the original photo on each edge.

On the other hand, at least in the beta-test version, Extend sometimes creates new objects in the scene, or even entirely fabricated people.

To me, that’s Viewpoint 2. It’s fakery, and it could be dangerous.

Outpainting might be useful for Instagram/TikTok, making it easy to turn casual H shots into V ones for more algorithmic love and better aesthetics.

But for everyone else, using AI to adjust your aspect ratio isn’t so much a technical problem as a moral one. At this point, the limiting organ isn’t our eyes or our hands. It’s our brains.