Case Study: Impact of Valuation Accuracy on Mortgage Lending

Introduction

Industries across the board have been adapting to emerging technologies in order to improve operating efficiency and lower cost. The mortgage lending sector has been no exception. In the UK, Hometrack alone claims that 17 out of top 20 lenders use its Automated Valuation Model (AVM) and that it performs 50 million automated valuations each year. Hometrack also claims that its AVM is adopted by over 50 Residential Mortgage-backed Securities (RMBS).

Reliance on technology took a wholly new level during the pandemic, which has forced people to work from home or hybrid. Although the pandemic has largely ended, many industries appear to consider hybrid working and automation as strategic, long-term options. Again, mortgage lending is no exception.

As the industry’s reliance on automated valuation increases, an enormous emphasis is placed on automated valuation accuracy.

This case study aims to quantify the impact of increased automated valuation accuracy on mortgage lending in the UK.

We compare the valuation accuracy of Hometrack, representing the most established and perhaps the most accurate conventional AVM in the UK market against AccuVal. In contrast to AVMs, which rely on hand-crafted algorithms, AccuVal uses AI inferencing to value properties.

While AI is hardly a new technology, using AI to value properties is a novelty.

To avoid confusion, by “AI” we refer to models trained through a Deep Machine Learning algorithm rather than hand-crafted by experts.

About AccuVal

AccuVal is a novel, AI based property Valuation as a Service (VaaS) platform, available as a web app as well as via an API. AccuVal is part of REXSMART, the first self-service property marketplace in the UK. AccuVal is a registered trademark in the United Kingdom, number UK00003724155.

About Hometrack

Hometrack is part of Zoopla Limited. Hometrack is also a founding member of the European AVM Alliance.

Data and Methodology

AccuVal

In order to benchmark AccuVal, we valued 222K residential properties sold in Britain between July 2021 and January 2022. That’s almost all the transactions recorded with Land Registry as of early March 2022. We excluded a few outliers and properties that are missing the number of rooms in their EPC certificate.

The code snippet bellow illustrates how the benchmark works. A pre-trained AI model (per region) is loaded at a time. Properties from Land Registry for that particular region are then loaded. Some pre-processing steps take place (not shown in the code snippet) to add other property specific data, such as floor area and rooms as well as location data such as transport, schools and income level. The processed data are then scored using the corresponding AI model, which infers the Fair Market Value for each property in the region.

for(i in 1:length(RGNs)){
  # Load pre-trained models. A model per UK region at a time
  Price_Estimate.dl <-  h2o.upload_model(paste0(ModelPath, paste0(RGNs[i], "_Valuation")))
  
  # Get sold properties in the region
  RGN_Properties <- Properties[Properties$rgn == RGNs[i],]
  
  # Get valuation
  Price <- as.data.table(h2o.predict(Price_Estimate.dl, as.h2o(RGN_Properties, use_datatable = FALSE)))
  
  # Round to nearest integer
  Price <- as.numeric(Price$predict)
  
  # Merge valuations with properties
  Score <- rbind(Score, cbind(Price, RGN_Properties))
}

Model’s Accuracy

Validating Machine Learning is fundamentally different from conventional statistical modelling since the “learning” is done based on the chosen algorithm, the data and hyper-parameters (such as number of hidden layers, activation function and how missing values are treated). An increasingly popular validation technique is called “residual analysis”. It’s one of the methods we use to visually inspect accuracy.

In layman’s terms, “residual” is the difference between the model’s predicted value and the known value (aka the ground truth). Zero residual indicates perfect prediction. The blue line in the middle of the plot (see below) reveals positive or negative tendencies. Here, our model is perfectly balanced.

We holdout a small portion of the training data to use for scoring. Since the model has not “seen” that data, it has no way of “remembering” the prices in the holdout data. This ensures that the scoring process resembles real-life scenarios and that the model “generalises” well. As can be observed from the plot, the residuals are “squashed” around the middle line, indicating good overall prediction accuracy across the price spectrum. Dots plotted far from the middle line correspond to properties that the model failed to value. They are mostly outliers and properties with incorrect data. A tiny fraction (less than 1%) of errors are due to niche situations that the model failed to “learn” during training.

We also include what we call “Accuracy Plot”. It’s similar to the residual plot but instead of plotting the residuals, it plots the actual scores. A score of 100 indicates perfect prediction, whereas a score of 95 is similar to 105, both indicate prediction within 5% of the actual price. Here, we can see that the dense area is around the 100 line. The Accuracy Plot reveals that properties at the very low-end (sub £125K) are more challenging to value because even a small delta can translate into high valuation error. This situation is equally challenging to human valuers as well as automated valuations.

Hometrack

Hometrack benchmarks its AVM against “surveyor’s recommended value”, which can be as far as 10% off the final price paid. Also, we suspect that Hometrack may cherry-pick properties that would boost the overall accuracy score (i.e. over-fitting). Hence, Hometrack’s accuracy would be significantly worse if it applies the same strict methodology as ours.

Unfortunately, Hometrack only publishes one accuracy figure (within 10%), which as explained above, can be further off from the final price paid. AccuVal publishes valuation accuracies within 5%, 10%, 15% and 20%.

We reached out to Hometrack for comments. However, as of the time of writing, Hometrack has not responded.

[UPDATE: as of May 2023, AccuVal's accuracy has increased to 72%, 95%, 99.5% and 99.7% respectively]

Random Samples

We picked 10 random properties sold across Britain from Land Registry data, eight of which were sold 5 years ago or prior. We obtained estimated prices from AccuVal (Beta 1.6) and Hometrack for each.

The data suggest that Hometrack’s AVM, which tracks property prices is at its best when the property to value is sold very recently at a fair market value. For instance, Hometrack values the flat sold in April 2021 at 205K (CF10 5LQ), which is about 5% increase. On the other hand, AccuVal values the same flat at 180.5K, that’s 8% decrease in value. Interestingly though, the most recent transaction in that area (mid December 2021) was for a much larger 986 sqft flat for £190K only.

However, the tracking approach struggles with properties sold not so recently, or when the data about the property is incomplete. For example, if a property was sold at a lower price due to poor condition but renovated to a good standard afterwards, Hometrack would continue to de-value such a property. The opposite is also true. A property that was sold at an inflated value would continue to be overvalued.

AccuVal does not track prices. Instead, it relies on AI inferencing. Hence, instead of estimating the current price based on the property’s historical transactions, AccuVal is trained to learn the current prices purely from data. In fact, AccuVal is trained with millions of sale transactions combined with location data. As such, it doesn’t suffer from the tracking caveat by design.

AccuVal was able to value the detached house in the Isles of Scilly (TR21 0JL) with what appears to be a very convincing accuracy, albeit with low confidence due to the lack of comparable properties nearby. Hometrack fails to provide a valuation for that house.

The superiority of the AI approach can be seen in other cases too. For example, the 5 room flat in the Barbican, City of London (EC2Y 8DR) is significantly under-valued by Hometrack despite the presence of comparable properties. AccuVal appears to get it right. Here, Hometrack’s poor accuracy is likely attributed to its price-tracking as this flat was last sold in 2010 for a relatively low price.

It’s important to note that according to This is Money “unlike other types of property survey that might be carried out during the sale process, a lenders’ survey is not based on the condition of the property or any material, structural or planning factors. Instead, the surveyor will seek to ensure that the price being asked for the home is fair, based on other, similar transactions in the local area.”

Property Condition: A New Automated Valuation Metric

AccuVal’s AI models are trained to take the condition of the property into account when estimating prices. In this context, the condition is relative to the comparable properties in the neighbourhood.

In the table above, we assumed “average” condition for all the 10 samples to match Hometrack. For more accurate valuations, the user can change the condition to reflect the property and AccuVal will update the valuation accordingly.

To demonstrate this aspect, we looked at a semi detached house in Brent. This house was sold in February 2021 for £1.135M. By visually inspecting the listing’s photographs (see below), its condition appears to be slightly above the average in that area, though not by much.

We obtained two valuations for this house from AccuVal (beta 1.6) with “Average” and “Above average” conditions and got £1,023,000 and £1,272,000 respectively. Subjectively, this house’s condition falls in between. In which case, the price would be around £1.149M, about 1.2% above the sold price a year ago.

As stated earlier, lenders rely on valuation based on similar transactions in the local area. However, with AccuVal, lenders will have the opportunity to make more realistic and low risk assessments by including the condition of the property in the valuation. Our data shows that the difference between poor and excellent conditions can be as high as 30% for comparable properties. This is illustrated in the Partial Dependence Plot below for a popular area in London.

Defining the Property to Value: A New Host of Possibilities

More Accuracy, Less Frustration

In 2021 alone, about 400K mortgage applications were hit by down-valuation, a staggering 50%.

We traced the issue back to its root and discovered that initial valuations take place at a very early stage in the selling process.

In the vast majority of cases, landlords and home owners wishing to sell their properties contact multiple local estate agents for “free” valuations. Agents tend to deliberately overvalue properties in order to win sellers instruction (as sellers naturally want to sell high!).

Also, most agents don’t actually hold professional qualifications. Instead, they rely on their “local knowledge and expertise”.

On the other hand, lenders valuation is based on the prices of similar properties in the area. If a seller has priced their home too highly, they could get a down valuation.

While it’s not clear how many deals fail due to rejected or down-valued mortgages, we believe that the figures are large enough to be a major concern to all parties involved.

Failing to get a mortgage can cause serious emotional damage to buyers who fell in love with the property. It can also cause frustration and negative emotions towards the lender for declining the application.

Having said that, one of our fundamental goals is to help streamline the selling process by providing objective and impartial valuations that have a higher chance of securing “Yes” from lenders. For this reason, AccuVal’s web portal is freely and openly available for individuals.

Lenders that appreciate the benefit of increased automated valuation accuracy can access AccuVal’s secure API via paid subscription.

What Does 8% Higher Accuracy Mean to the Lending Market?

Based on data provided by Statista and the FCA, the value of gross mortgage lending (pre pandemic) was around £276 billion. Assuming positive outlook, the figure will likely hit £300 billion by 2023.

As demonstrated earlier, AccuVal achieves 8% or higher accuracy compared to Hometrack. Applying 8% to £300 billion yields £24 billion. With an average mortgage of £200K, 8% better accuracy translates into 120K more approved mortgages. That would also save lenders around 21M in valuation cost (£175 average saving per application).

AI in Valuation: It’s Just the Beginning

AccuVal is the first “instant” property valuation in the UK that’s based on Machine Learning AI. At the time of writing this case study, AccuVal is still in Beta and had been developed with very limited resources. It was also limited to freely available datasets.

Despite being in its infancy, AccuVal demonstrated for the first time that property valuation can be automated with agent-like accuracy while also being instantaneous, transparent and objective. It goes even further to explain the rationale for the valuation with facts – no more agents sticking a finger in the air.

Our initial objective was to achieve 90% valuation accuracy for 90% of all residential properties. Based on the results achieved so far, we are now pursuing a higher target and will continue to do so. With AI, the sky is the limit!

About the author

Jaafar Almusaad is a co-founder and CTO of REXSMART, the first self-service real estate marketplace in the UK. Jaafar has over two decades’ experience in computer engineering, software development and data science. He holds a Master’s degree in Information Technology Management from the University of Sunderland. With such rich and diverse background, Jaafar both conceptualised and executed his vision to democratise the real estate marketplace, starting the with huge success of AccuVal. Husband, father, blogger, enthusiast photographer and coffee lover.

https://www.linkedin.com/in/jalmusaad