How smartphone reviews really work

As a certified Guy Who Writes About Android™, I get asked plenty of questions. Whether it’s device recommendations, app suggestions, or , the stack of incoming inquiries never seems to run short. 

What’s interesting, though, is that for every question I see about what I cover, there’s a question about how I cover it. Part of what I do is reviewing new devices — and as I’ve learned over the years, that’s a process surprisingly shrouded in mystery and misconception.

Since we’re on the brink of a busy few weeks for smartphone reviews, I thought now would be a prime time to pull back the curtains and offer a frank walk-through of how the phone-reviewing process actually works. After all, transparency’s a good thing — and there’s no reason you shouldn’t understand what goes on behind the scenes of the reviews you read.

I should preface this by saying this is how I do things. Other writers and publications may or may not always follow the same practices; I obviously can’t speak for everyone. But as far as my reviews are concerned, this is how things work:

Step 1: Getting a review unit

How do I actually get my hands on a smartphone review unit — and how far in advance of its release do I get it? The simple answer is: It depends.

With some products, I (or more often, Barbara, my New York-based reviews editor) have what’s called a pre-briefing in advance of a device’s launch. It might take place in person (often in New York, as you might have guessed) or sometimes just via email or over the phone. We’ll usually have to agree to an embargo beforehand, which merely means we won’t publish anything or talk about having the device until a certain date and time. We’ll then either get a review unit right then and there or be sent one soon thereafter.

Sometimes, there’s no formal pre-briefing and I’ll just get notified of review unit availability directly via email. (There’s frequently still an embargo involved.)

And sometimes, I’ll see reviews pop up at other publications before I’ve heard or received anything myself (always a lovely surprise!). Some of the time, that’s because a company has opted to give early access to one particular outlet or a select group of sites. Other times, it’s because the European or international arm of a company has launched a device and/or provided global review units before the company’s U.S. division has handed out U.S.-specific models — a strange and slightly confusing practice, but one that happens more often than you’d think (including once this very week!).

In any of those cases, I might get a phone a few days or a few weeks before it’s released. Or I might get no advance notice and scramble to snag one the day it’s announced. There really is no universal standard from one manufacturer or even product to the next; every device is a new adventure.

Step 2: Living with the device

Call me crazy, but I like to actually live with devices for a while before reaching conclusions and making recommendations. There’s certainly a place for quick-turn reviews and lab-driven analyses; they’re just not for me. I’m far more interested in what a gadget is actually like to use in the real world, the way a normal person would use it — and to get a meaningful and well-rounded sense of that, I need to spend a fair amount of time using the thing around the clock in my day-to-day life.

That often means I’ll end up publishing a review well after the aforementioned embargo time has passed — which is absolutely fine. An embargo just means you agree not to publish anything before that time, but there’s nothing preventing you from publishing later. Every writer and publication has different priorities, but for me, being “FIRST!” generally isn’t high on the list.

Once I receive a review unit, I put my own personal phone aside and prepare to move into the new device. It really only takes a matter of minutes to make any Android phone feel like home these days: I sign in, install a handful of essential apps, and set up the few things that still require manual configuration.

Thanks to Google Voice, calls and texts to my personal number come through to any device I’m using — even when it’s a review unit with a temporary SIM card provided by the manufacturer. And since I use cloud-based services for almost everything in my life — contacts, calendar, email, music, even document editing — anything I might need is automatically synced and available on any phone I use, without any real effort on my behalf.

So once I’m settled in, I simply go about my life, using the review unit as I would any other phone (albeit sometimes a bit more intensely — gotta be sure to push the limits a little to get a feel for things like stamina). How long this process lasts varies from one device to the next, but at a bare minimum, I like to live with a phone for at least several days before coming to conclusions and publishing a review. I find my first impressions often evolve and I notice different nuances as I spend more time with a product, so without that handful of days, I feel like I’m rushing the process and not really getting to know the device in any meaningful manner.

Step 3: Writing the review

This part’s pretty obvious, right? Once I’ve gotten a stable feel for what a phone’s like to use, I start working on a review. (I do make myself notes along the way, too, and those help guide me once I sit down to put my thoughts together.) After all the t’s are crossed and the i’s are dotted, the review goes through a few layers of editing and production and then makes its way to this lovely website, where you read it and leave amusingly derogatory comments about my conclusions and/or wrist circumference.

Hey, it’s all part of the job.

Step 4: Returning the device

Sorry to crush your phone reviewer fantasy, but I don’t keep any smartphones that I review. They’re all loaner review units that are returned to the manufacturers after the reviews are finished.

As for how long I have the devices, once again, it really varies from one situation to the next — but anywhere from a few weeks to a month, on average. When possible, I like to hold onto products for a little while after a review is finished in case any questions or opportunities for follow-ups arise. I love to do longer-term “revisits” of devices, too, but they’re usually feasible only when they involve a device I decide to purchase myself or when I borrow a review unit for a second time down the line.

(And yes, everything I use in my own personal life is something I bought on my own, at the same prices and from the same venues you or any other consumer could access.)

So that’s it — some straight-forward answers about how smartphone reviews really work. Now you know. And you don’t need any embargoed information to realize this discussion will be highly relevant before long.

Keep your eyes peeled and your ears open, gang: Busy times are ahead.