In a previous life, my job was to pick the best release dates for movies. Now if you’re releasing Marvel or Pixar movies it’s pretty easy: Christmas for this one, Thanksgiving for that one, first week of May for the other, and I’m going on break. However, I did not work on Marvel or Pixar movies. For the others, there’s a whole lot more research and analysis that goes into it than people think. Being that there’s plenty of data available and tremendous amounts of money at stake, I would often hear the sales pitch from various start-ups who thought they had the magic algorithm that could replace the human element. What few ever seemed to understand (or they did understand but would happily sell their product to those who didn’t) is that it’s not a problem that is completely solvable with data, and an algorithm can only be part of the equation. Data can be the tip of the spear, but not the whole spear.
In the housing market, Zillow learned this the hard way. Among other pretty standard big tech mistakes, reliance on their home valuation algorithm- what we all know as a Zestimate– led to the rapid demise of Zillow Offers, their glorified home flipping business. Zestimates are a fun tool that are ostensibly there for people to get an idea of their home’s value- or their neighbor’s home, their childhood home, or the one they want to buy. Zillow Offers used these Zestimates to value the homes they were buying, often inflating them to gain market share quickly. Turning a parlor game into the backbone of a multi-billion dollar business clearly wasn’t the wisest idea. Simply put, Zestimates were writing checks that Zillow Offers couldn’t cash.
Redfin has an algorithm too, which they call a Redfin Estimate. Not quite as Seussical as “Zestimate”, but I don’t know what else you’d call it. Realtor.com uses third-party companies, of which there are a handful, to provide their home valuations. All of these companies load up their estimates with disclaimers and caveats about how they’re not appraisals (which can only be done by licensed appraisers) and are mainly just a jumping off point for a discussion with a real estate agent. Preferably one you find on their site, of course.
So is anyone getting it right? If you’re thinking about selling somewhere down the road and just want a quick and easy don’t-get-a-pushy-Realtor-involved idea about what your house is worth, where do you look? Is a Zestimate more accurate than a Redfin Estimate, or vice versa?
Both Zillow and Redfin publish general stats about their own accuracy. Zillow says that for active listings (those that are currently on the market), their median error rate is 1.9%, meaning that half of the houses will sell for within 1.9% of the Zestimate (as measured just prior to the house going on the market) and half will not. For off-market listings, the error rate is 6.9%. In the Portland market, their median error rate for active listings is the same 1.9% as it is nationally, while the off-market error rate is a little better at 5.2%. Their data is a year old. Redfin claims a 2.58% on-market error rate and 7.67% off-market, and they update those numbers daily (for on-market) and weekly (off-market). They don’t measure the Portland market by itself but for the state of Oregon they’re- at the time of this writing- claiming a 6.22% median error rate for off-market homes.
I don’t care about how accurate they are when a house is already on the market, that part’s easy. Those houses already have a price tag, which is a pretty big data point. It’s the off-market error rate I’m interested in, and I’m only interested in right now: a year ago tells me nothing about the current market. So, as the kids like to say, I did my own research.
If this all sounds familiar to you, a regular reader of this blog, it’s because I sort of did this a while ago. This is the reboot, now with more data. This time around I found 120 houses in the Portland metro and grabbed their Zestimates and Redfin Estimates just prior (within a month) to their launch on the market. Then I waited and waited for those houses to sell, so I could compare the sale price to those estimates. Because that’s what this is all about- how accurate are the estimates when they matter the most.
The answer: not very.
So half of all Zestimates in my study came in within 9.7% of the sale price (plus or minus) while Zillow reported a much smaller error rate for Portland. For Redfin, the numbers are closer to each other and slightly better than Zillow’s but my own research still did report a higher error rate than theirs. There’s a small caveat that my numbers are for the Portland metro exclusively while theirs are for the whole state, but they also report county-wide estimates which indicate that their statewide estimate is going to be pretty close to Portland’s estimate.
Both services also report what percentage of sales fall within 5% of their estimates. Again, the numbers they reported from their own studies didn’t line up with mine:
Five percent is pretty meaningful when you’re trying to figure out what your house is worth- there’s a meaningful difference between pricing your house at $570k versus $630k- and from the results above, just getting it into that five percent range happens less often than not.
Lastly, Zillow and Redfin don’t report what the average difference is between their estimates and sold prices, just what the median error rates are. I wanted to know what that average difference was. In Zillow’s case, the average difference among the 120 homes I surveyed was -9.6%, while for Redfin it was -9%. The negatives matter here- both companies’ estimates were more likely to come in under the sale price than over. In fact, out of the 120 homes, the Zestimate came in under the sold price 106 times while the Redfin Estimate was under the sold price 102 times. This makes some sense, as the period in which I studied the market was perhaps the hottest housing market we’ve ever seen, and the estimates weren’t keeping up. Still, isn’t it the algorithm’s job to account for these kinds of variables?
There is no exact science to estimating home values, no matter how good your data is. The human element is always going to play a major part, as home buying is as much an emotional experience as a practical or financial one. An algorithm might know that the house is on a busy street and account for that, but does it know what buyers will think when they see the kitchen, or when they see the neighbor’s junkyard? Does it know that the school district is about to rezone their boundaries, or that there’s a new Trader Joe’s being built within walking distance? How well will it predict how buyers react to macroeconomic changes, like rapidly increasing interest rates or recession fears? Cloud-based algorithms might make for fun parlor games or at least give you a place to start, but they’re no substitute for the analysis of a real estate agent that knows the market well. Preferably one you find on this site, of course.
