Logo of jester cap with thought bubble.

Image source: The Motley Fool.

Independent Bank Group Inc (IBTX 3.16%)
Q4 2020 Earnings Call
Feb 2, 2021, 8:30 a.m. ET

Contents:

  • Prepared Remarks
  • Questions and Answers
  • Call Participants

Prepared Remarks:

Operator

Greetings, and welcome to the Independent Bank Group Fourth Quarter 2020 Earnings Call. [Operator Instructions] As a reminder, this conference is being recorded. It is now my pleasure to introduce Paul Langdale, Senior Vice President and Director of Corporate Development. Thank you. You may begin.

10 stocks we like better than Independent Bank Group
When investing geniuses David and Tom Gardner have a stock tip, it can pay to listen. After all, the newsletter they have run for over a decade, Motley Fool Stock Advisor, has tripled the market.* 

David and Tom just revealed what they believe are the ten best stocks for investors to buy right now... and Independent Bank Group wasn't one of them! That's right -- they think these 10 stocks are even better buys.

See the 10 stocks

*Stock Advisor returns as of November 20, 2020

Paul Langdale -- Senior Vice President, Head of Investor Relations and Director of Corporate Development

Good morning, everyone. I am Paul Langdale, Senior Vice President and Director of Corporate Development for Independent Bank Group, and I would like to welcome you to the Independent Bank Group Fourth Quarter 2020 Earnings Call. We appreciate you joining us. The related earnings press release and the slide presentation can be accessed on our website at ibtx.com. I would like to remind you that remarks made today may include forward-looking statements. Those statements are subject to risks and uncertainties that could cause actual and expected results to differ. We intend such statements to be covered by safe harbor provisions for forward-looking statements. Please see Page six of the text in the release or Page two of the slide presentation for our safe harbor statement. All comments made during today's call are subject to that statement. Please note that if we give guidance about future results, that guidance is a statement of management's beliefs at the time statement made, and we assume no obligation to publicly update guidance. In this call, we will discuss a number of financial measures considered to be non-GAAP under the SEC's rules. Reconciliations of these financial measures to the most directly comparable GAAP financial measures are included in our release. I'm joined this morning by David Brooks, our Chairman, CEO and President; Dan Brooks, our Vice Chairman and Chief Risk Officer; and Michelle Hickox, Executive Vice President and CFO. At the end of their remarks, David will open the call to questions. With that, I will turn it over to David.

David R. Brooks -- Chairman, President and Chief Executive Officer

Thanks, Paul. Good morning, everyone, and thank you for joining us on the call today. Our company's fourth quarter results represent a strong finish to a difficult year with healthy fourth quarter adjusted net income of $1.34 per share and adjusted return on tangible equity of 16.33%. Full year 2020 net income of $201.2 million reflects the value of our granular community banking model and our ability to adapt to a changing economic environment. Reflecting on the past year, I'm especially proud of how our employees across Texas and Colorado rose to the occasion during 2020 to serve our customers and communities under trying circumstances. As Dan will discuss further, our credit quality metrics remain strong and reflect our conservative credit culture and disciplined underwriting standards. We continue to be encouraged by how well our portfolio is holding up though as always, we are vigilantly monitoring for any emerging risks as general and macroeconomic uncertainty persists. Our capital ratios are at historically strong levels with a total capital ratio of 13.32%. And our robust liquidity position continues to be augmented by organic deposit growth of 21.54% annualized in the fourth quarter. During the quarter, we also took advantage of our recently reauthorized share repurchase program and repurchased 109,548 shares of our stock for an aggregate cost of $5.7 million. With that overview, I'll turn the call to Michelle for more detail on the operating results for the quarter.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Thank you, David. Good morning, everyone. Selected financial data for the quarter is on slide six. Fourth quarter adjusted net income was $58 million or $1.34 per diluted share compared with $56.8 million or $1.32 per diluted share for the fourth quarter last year, and $59.6 million or $1.38 per diluted share for the linked quarter. Net interest income was $132.8 million in the fourth quarter, up from $128.1 million in the fourth quarter last year, and up from $132 million in the linked quarter. While net interest income was negatively impacted by year-over-year reduction of $4 million in purchase accounting accretion, this was more than offset by a continued reduction in funding costs during the quarter and earning asset growth that was primarily driven by increased mortgage warehouse loans as well as cash held at other banks due to continued deposit growth. The adjusted NIM, excluding all loan accretion, was 3.24% for the fourth quarter compared with 3.49% from the fourth quarter last year and 3.32% in the linked quarter. The core margin decreased by eight basis points from the linked quarter due primarily to lower asset yields as well as increased balance sheet liquidity. Total noninterest income was $19.9 million for the fourth quarter compared to $25.2 million in the linked quarter. While mortgage production remains strong and gain on sale margins compressed only slightly mortgage banking revenue was impacted by fair value adjustments on loans and derivative hedging instruments of $4.3 million due to treasury rate increases during the fourth quarter. Noninterest expense totaled $75.2 million for the fourth quarter, an increase from $73.4 million in the linked quarter.

Professional fees increased $493,000 and charitable contributions were up $300,000 from the linked quarter. Fourth quarter noninterest expense also includes $1.3 million of unusual items due to PTO paid out because of the pandemic and two accrued contract terminations. slide 22 shows our deposit mix and cost. Total deposits were $14.4 billion as of quarter end, an increase driven by organic deposit growth of $747 million, or 21.54% annualized for the quarter. We estimate approximately $395 million of deposits related to PPP borrowers remains on the balance sheet as of December 31. Capital ratios are presented on slide 24. In the fourth quarter, the company's consolidated capital ratios continue to grow, with the common equity Tier one ratio increasing by nine basis points to 10.33% and the total capital ratio increasing by three basis points to 13.32% for the quarter. As David mentioned, we did utilize our stock repurchase plan during the fourth quarter, acquiring $5.7 million in shares. That concludes my comments. I will turn it over to Dan to discuss the loan portfolio.

Daniel W. Brooks -- Vice Chairman and Chief Risk Officer

Thanks, Michelle. Overall, loans held for investment, excluding mortgage warehouse purchase loans, were $11.6 billion at quarter end, down slightly from the linked quarter. Excluding PPP loans of $804.4 million, loans held for investment decreased year-over-year by $87.2 million, primarily as a result of the economic dislocation caused by the pandemic. While new loan originations continued to show recovery from earlier in the year, commercial real estate loan payoffs remained elevated in the fourth quarter. Mortgage warehouse purchase loans averaged $1.2 billion for the quarter, up from $894.9 million from the linked quarter. Our mortgage warehouse continues to see robust demand due to the low interest rate environment. slide 17 provides additional detail on our pandemic loan modifications. Of the $2.6 billion of loan balances that received temporary payment relief during the pandemic, only $205.7 million remain an active deferral as of January 15, 2021. Loans with active payment deferrals represent just 1.7% of overall loans held for investment. This number includes loans that have been restructured with payment deferral mechanisms under Section 4013 of the Cares Act. The largest group of loans remaining on deferral are hotel credits, and we remain confident in the strength of this conservatively underwritten book. Our credit quality metrics continue to reflect the overall strength of the portfolio with total nonperforming assets of $52 million or 0.29% of total assets at December 31, 2020. Nonperforming assets increased over the linked quarter due to the addition of a $12.6 million energy loan that has been discussed in prior quarters as well as two commercial real estate loans totaling $15.9 million.

These additions were offset by a $3.5 million energy charge-off and the renewal of the $15.7 million commercial real estate loan that was discussed last quarter. Net charge-offs remained low at 11 basis points annualized for the fourth quarter and were primarily driven by the previously mentioned energy charge-off that had been fully reserved against in prior quarters. As you know, we elected to defer the adoption of CECL last March with the expectation of adopting it as of 12/31/2020. On December 27, 2020, new legislation extended the adoption date to January 1, 2022. The SEC has indicated they will not object to an early adoption date of January 1, 2021. And therefore, we have elected to adopt CECL and record our day one returned earnings adjustment as of that date. Our 2020 provision was calculated using the incurred loss model, and we will use the CECL model going forward for 2021 provisions. These are all the comments I have related to the loan portfolio this morning. So with that, I'll turn it back over to David.

David R. Brooks -- Chairman, President and Chief Executive Officer

Thanks, Dan. Looking back at 2020. Our community banking model enabled us to effectively manage risk and deliver another year of consistent financial performance, despite an unprecedented operating environment. Most importantly, this allowed us to support our customers and serve our communities when they need us most. We are also proud to have continued to enhance shareholder value by growing tangible book value per share, increasing our dividend and reauthorizing our share repurchase program. Looking ahead, our company is well positioned to win new business and participate in the economic recovery across our footprint. Though uncertainty remains, we are encouraged by what we're seeing on the ground in Texas and Colorado and we look forward to seizing opportunities for continued growth and shareholder value creation as the new year unfolds. Thanks again for taking time to join in the call today. We'll now open the line to questions. Operator?

Questions and Answers:

Operator

Thank you. We'll now be conducting a question-and-answer session. [Operator Instructions] Our first questions come from the line of Matt Olney with Stephens. Please proceed with your questions.

Matthew Covington Olney -- Stephens Inc. -- Analyst

Hi, thanks. Good morning, guys.

David R. Brooks -- Chairman, President and Chief Executive Officer

Good morning, Matt.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Hey, Matt.

Matthew Covington Olney -- Stephens Inc. -- Analyst

I want to start with core loan growth, ex PPP and the warehouse, look pretty immaterial in the fourth quarter. Would love to hear about the pushes and pulls on this in the fourth quarter with respect to paydowns, originations and then expectations for 2021.

David R. Brooks -- Chairman, President and Chief Executive Officer

Matt, we had a continuing large amount of headwind from paydowns in the fourth quarter. Our loan -- at the end of the year, we went back and tracked. We booked approximately $3 billion of new funding's in 2020. We had about $3 billion of payoffs in 2020. And so we ended up being relatively flat or virtually flat for the year. And that compares to the year before where we actually had positive -- in 2019, positive loan growth. I believe our total funding's were around $3.2 billion. So the falloff in funding's was less than 10%, 2020 over 2019. But the payoffs accelerated and covered up the loan growth that we had. That said, we're seeing some positive early signs here early in the year, January, the payoffs seem to be trending down. We had positive loan growth, ex PPP and ex warehouse here early in the year. So, we're encouraged to us -- what we've indicated before, we did our planning around this year, which is a mid-single-digit loan growth, ex PPP, ex warehouse for 2021 and then accelerating to an upper-single-digit for 2022 and beyond. We really believe in a normalized environment, Matt, that weakened in the markets we're in. And also, we've hired a head of middle market commercial for Texas, very talented executive, who's been in this market for her entire career and somebody that we can really -- that we are going to build a significant team around here in 2021. So we think that a combination of the markets that we're in, Matt, as well as this team, we're adding building in middle market commercial as well as the investment on the retail side and all that will yield us the ability long-term to continue to grow at rates that we were growing at prior to the pandemic, which is upper-single-digit.

Matthew Covington Olney -- Stephens Inc. -- Analyst

Okay. I appreciate that. And then as a follow-up, I wanted to ask about the mortgage warehouse. I think you said average balances were around $1.2 billion, up again, just a strong year for the warehouse. Would love to hear what the crystal ball says about the outlook here. I think we're all trying to figure out what -- where the volumes will settle out once we get beyond this current surge? Thanks.

David R. Brooks -- Chairman, President and Chief Executive Officer

Yes, as we're thinking about that, Matt. The volumes were -- continued, as you said, to be really strong in the fourth quarter. We feel good about the quality of the mortgage companies we're doing business with, I feel like we've really been able to, by building the team that we've built, the leadership in that area, been able to build strong and important relationships with these companies. And in essence, Matt, the goal with these quality relationships is to move up in the funding stack, if you will, so that as their volume tapers off, they take that away from other participants before they take it away from you. And so in that regard, we think we've made a lot of progress, and we're seeing it here early in the first quarter that our average outstanding's have not come down as much as we would have expected. That said our planning for this year is that, on average, about $1 billion, we think, is a good landing place as the market calms down for us. So, around $1 billion in average outstanding's quarter-by-quarter. Right now, it looks like it'll be a little higher than that maybe in the first quarter, but for planning purposes; we're thinking $1 billion. Michelle, is that fair?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Yes. I think that's right.

Matthew Covington Olney -- Stephens Inc. -- Analyst

If I could sneak in one more with the warehouse, any color on pricing, whether it's settled down or if it's stabilized or improved at all over the last few months?

Daniel W. Brooks -- Vice Chairman and Chief Risk Officer

Matt, this is Dan. The pricing in mortgage warehouse is, I'd say, stable. There's always been pressure on it. And we expect, as we look at this year, that, that will continue to be there. But I think, as David said, the relationships we've built puts us in a good position, I think, to hold pretty well on the pricing for now.

Matthew Covington Olney -- Stephens Inc. -- Analyst

Okay. Great. Thank you, guys.

Daniel W. Brooks -- Vice Chairman and Chief Risk Officer

Thanks, Matt.

Operator

Thank you. Our next question has come from the line of Brad Milsaps with Piper Sandler. Please proceed with your question.

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

Hi, good morning.

Daniel W. Brooks -- Vice Chairman and Chief Risk Officer

Hey good morning, Brad.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Hey Brad.

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

Maybe just wanted to start with asset quality. I think previously, you guys thought there'd be around an $80 million adjustment whenever you -- when you did adopt CECL. I guess, first, do you think that would be about the same? And kind of how does that impact your decisions around or how you're thinking about provisioning in 2021?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

So -- yes, Brad. So we are going to record our day one adjustment now as of 1/1/21, which is a different day one. So, we're in the process of finalizing that number, primarily because of PCI loans. We're looking at the status of those and making sure that we're accounting for those correctly as of day one. But based on what we know today, I expect that last year, we disclosed, that would be about an $80 million adjustment. It shouldn't be much different than that. It could be a little less because we did have some larger PCI loans that paid off during 2020 that we don't have in that number anymore. So, my best guess would be a $70 million to $80 million adjustment at this point on top of what we have in the reserve.

David R. Brooks -- Chairman, President and Chief Executive Officer

For loan loss provision this year.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Yes. I think -- I mean -- and I think Dan would concur, I think we believe that we are fully reserved and wouldn't expect to have significant provisioning in 2021.

David R. Brooks -- Chairman, President and Chief Executive Officer

No, I'd agree with that.

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

Okay. Great. That's helpful. And then Michelle, writing quickly and around kind of some of your expense commentary, it sounds like expenses might have been roughly flat, absent, maybe a few things that kind of went against you this quarter. First, I guess, is that kind of a fair assessment? And then secondly, kind of how would you feel about expense growth going into -- in this year?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Yes. I think that's accurate, Brad. If you look at our core schedule, we reported $74.8 million in non-interest expense for the quarter four, but that does still include that $1.3 million of the PTO and the contract terminations that I mentioned earlier. And then if you go with my guidance that we figure we'll have 3% expense growth in '21, I think that puts us at about a $75 million run rate in non-interest expense going forward.

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

Great. That's helpful. And then maybe just a final question for David. I noticed -- and you mentioned that you bought back a small amount of stock in the fourth quarter. Just kind of curious, with the shares trading in the 60s, would you guys consider buying the stock back? Or do you think saving capital here for potential M&A is maybe a better use? Just kind of curious your thoughts around capital management.

David R. Brooks -- Chairman, President and Chief Executive Officer

Yes. Thanks Brad. Our view is that with the volatility in the stock that our stock trading below two times tangible, knowing on the other side, you mentioned an alternative use is M&A, which we're very keen on and have been in our history as a public company, as you know, that said, price expectations from sellers -- from high-quality sellers in the best markets are still pretty robust. And so when we balance it up, it feels like our stock at below two times tangible, we should be more active -- north of two times tangible, we should be more active -- north of two times tangible than I think we begin to lean more toward hang on to it with an eye toward M&A and strategic M&A. So that's an outline. So yes, we will be active this year when our stock is trading below two times tangible. There's just a lot more clarity now about risk. The elections behind us, the pandemic, the vaccines are rolling out. So regardless of your view about specifically when we'll get to herd immunity and everything, I think we have a better handle on risk now than we had early in the fourth quarter as an example, when our stock was trading in the 50s, and we were a little more hesitant to be super aggressive there, pending what was going to happen in the election and all that. So hopefully, that gives you a little clarity.

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

No, thanks. I appreciate all the color. I'll hop back in queue. Thank you.

David R. Brooks -- Chairman, President and Chief Executive Officer

Okay. Hey, thanks.

Operator

Thank you. Our next question has come from the line of Brady Gailey with KBW. Please proceed with your question.

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

Hey, thank you. Good morning guys.

David R. Brooks -- Chairman, President and Chief Executive Officer

Hey, good morning, Brady.

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

I wanted to ask about the other side of mortgage, not the mortgage warehouse, but your traditional gain on sale mortgage fees down in fee income. And if you back out the hedging adjustments, and it looks like the last couple of quarters that's been running a little on top of $13 million a quarter. So a very nice level. I know it's tough to forecast that into 2021, but how much the downside do you think we could see in mortgage fees just as that market calms down a little bit in 2021 versus 2020?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Yes. As you said, Brady, that mortgage volume is really hard to predict, because it's so based on rates really. I mean, I can tell you, through January, their volumes were still really good, maybe get down a bit from where they were in December, which is sort of seasonally unusual, right? Usually, it takes a bit for them to get going again. I think is what we've guided to before is that we thought they wouldn't go back to where they had been previously because we've really built out that team and that group and added lenders. So we're thinking though, they could go back down to 25% less volumes, but that's not really what we're calling for this year. I expect that they will continue to have good volume for the first half of the year anyway.

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

All right. That's fair. And then, Michelle, on the topic of the margin, I know you've been talking about the margin compressing down modestly, which is what has happened. Yes, I mean, off the 344 base this quarter, how should we think about the margin, excluding any sort of noise from PPP Forgiveness, kind of that core margin, is compression still the right way to think about it into 2021?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

It is. It is, and when I -- when we talk about core margin, I am excluding all of purchase accounting accretion, which we'll continue to have in PPP fees, which we anticipate we'll get most of our PPP income on the first round in this probably first and second quarter. Based on our modeling and our outlook for this year, we think our margin is probably going to compress a couple of basis points a quarter, so could compress eight basis points over the year. That's my early outlook anyway. Just given our loan yields currently are coming on 10 to 12 basis points lower than our overall average yield. We are trying to invest some of our excess liquidity in our securities book, but those yields currently are about 1.35. So while that's not great, it is better than what we're earning at FID, so just trying to boost our net interest income a bit there.

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

Okay. All right. And then finally for me, I just wanted to ask about bank M&A. It seems like everybody is expecting 2021 to be a fairly robust year for bank M&A, especially in Texas. I mean, David, I just heard your comments about sellers' expectations are still pretty robust. And with your stock under two times, then maybe you just look to continue to buy it back. But I think the market may be anticipating you guys kind of reengaging in your traditional bank M&A game. So maybe just a little bit of color on how you think bank M&A will unfold in Texas? And do you think IBTX will be active?

David R. Brooks -- Chairman, President and Chief Executive Officer

Yes. Our desire is to be active, Brady, as we have been in the past. There are a lot of discussions going on across the market and I'm encouraged by that. On the other hand, our list of targets are -- call it, $2 billion to $10 billion in asset, high-quality companies in the major markets in Texas and Denver. That's not a really long list. It's probably eight or 10 companies. And given that we are involved, I'm encouraged. I do think there's going to be M&A in Texas here in the first quarter, second quarter and through the year. I think it's just early to tell so much of it, as you alluded to Brady, depends on how the market is trading generally and how stock prices are doing and how sellers -- what seller's expectations are. But we're looking for a high-quality company that wants to be a part of what we're doing, and we want to do a smart deal with the right company. I am optimistic that that's going to happen for us this year. But that said, we're not -- kind of the joke around here is, we don't need the practice. So we're going to do a smart deal with the right company. We don't need to do a deal just to say, oh, geez, we can still do M&A. We're going to grow our company.

We're continuing to build out our infrastructure and building our platform to be a $25 billion, $30 billion company, which is still where we believe we'll be in three, four years and, kind of, really focus as well on building out that middle market, executing on our retail strategy and getting back to a more robust organic growth rate as the core of our company, that's always been the core of independent different than some acquirers. We're an organic growth company first, and then we layer in really high-quality material M&A on top of that, when we get the opportunity. So I'm optimistic about it, but also realistic that there aren't just three or four great companies out there going, oh, we're ready, we're ready. It just takes work and continuing to build relationships and then the timing has to be right. So...

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

All right. That makes sense. Thanks, David.

David R. Brooks -- Chairman, President and Chief Executive Officer

Thanks, Brady.

Operator

Thank you. Our next questions come from the line of Brett Rabatin of the Hovde Group. Please proceed with your questions.

Brett D. Rabatin -- Hovde Group -- Analyst

Hey. Good morning, David, Michelle.

David R. Brooks -- Chairman, President and Chief Executive Officer

Good morning, Brett.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Hi, Brett.

Brett D. Rabatin -- Hovde Group -- Analyst

Most of my questions have been asked, but wanted to circle back around on loan growth and the organic growth story. Can you talk about maybe hires -- maybe late in the year and then what the prospects you think might be for 2021? And I'm thinking about that core mid-single-digit loan growth for 2021, is that going to be more in specialty lines of business? Or maybe give us some color, if you could, just on how you think the organic growth or what buckets ends up coming from?

David R. Brooks -- Chairman, President and Chief Executive Officer

Sure. Happy to give more color on that, Brett. We did continue to build-out, and we're a little further along in building out the retail team so we've hired a leader there, Kenyon Warren and then a strong team of leaders with Kenyon. We had a strong presence already in Colorado on the retail side. So that's going very well. The biggest hire we made recently was the addition of as I mentioned earlier, Tiffany Cason, who is a long time, Texas Executive and middle market commercial lending, and she joined us, I believe, beginning of the year and is steady building out her team now. We're talking with lenders across Texas, particularly in Houston, as an example, where we expect to be active there building out the team as well. And so it's the early days, but I expect by the second half of this year when we kind of look out Brett, by the second half of this year, we should have some tailwind particularly in the middle market side on the loan growth, organic loan growth. So we think our loan growth, we expect here in the second half of 2021 and into 2022, it's going to be more balanced than it historically has, although we're still seeing a lot of great real estate opportunities and transactions. But we expect and are seeing a little more floating rate flavor to what we're adding. As Michelle said, we're adding a little below what -- adding loans a little below what our current portfolio rate is eight, 10 basis points or so. So it's not a year ago or, I guess, early in the pandemic. With the competition last summer, we were adding loans at 25 and 30 basis points below our portfolio rate that's narrowed now at eight or 10 basis points. So I think that's a contributing factor to what Michelle said about the NIM. But when we have an $11.5 billion book, and we're adding $50 million or $100 million a month to that, then it takes a while for -- at eight or 10 basis points, the math doesn't jerk your loan yields overall down quickly as long as you can add loans in and around five, 10 basis points below the total portfolio yield.

Brett D. Rabatin -- Hovde Group -- Analyst

Okay. That's helpful. And then just back on the margin. Maybe I appreciate the color on the core margin commentary, Michelle. Any -- I know it's difficult to predict to some degree, but any color around the stated margin in the first half of the year, how you expect that maybe the PPP Forgiveness to play out in the first half of the year? And then any thoughts around the, say, discount accretion, at least for the early quarters?

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

That PPP number, Brett, right now, we think that most of it's going to come this quarter. We have had a good amount of payoffs that have came in, in January, and I think that will continue through the second quarter. I really can't predict the impact. It will be a few basis points increase on our margin. It will be a bit lumpy. Purchase accounting accretion should just continue at a similar rate. It will just continue to go down. I think it was a little less than $7 million in the fourth quarter. That's going to continue to trail, just look at it as going down a couple of hundred thousand dollars a quarter is what I would expect at this point, that number has become pretty stable.

Brett D. Rabatin -- Hovde Group -- Analyst

Okay. Great. Appreciate the color.

David R. Brooks -- Chairman, President and Chief Executive Officer

Hey, thanks, Brett.

Operator

[Operator Instructions] Our next questions come from the line of Michael Rose with Raymond James. Please proceed with your questions.

Michael Edward Rose -- Raymond James and Associates, Inc. -- Analyst

Hey, good morning everyone.

David R. Brooks -- Chairman, President and Chief Executive Officer

Hey, good morning, Michael.

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Good morning, Michael.

Michael Edward Rose -- Raymond James and Associates, Inc. -- Analyst

Just had a question on the increase in classified balances this quarter. They're still really low as a percentage of loans, but just wanted to see what drove that increase? Was it some of the kind of COVID impacted sectors just classifying them?

David R. Brooks -- Chairman, President and Chief Executive Officer

Hey, Michael, this is Dan. Yes, as you noted, it was only up slightly really from the third quarter, a little bit of C&I in there, some hotel, as you might expect, would be part of that. But those would be the two primary categories.

Michael Edward Rose -- Raymond James and Associates, Inc. -- Analyst

Okay. And then maybe just one last one for me. So the -- just back to the mortgage warehouse. So it's grown essentially every quarter since you guys kind of got in the business a couple of years ago. If we're assuming growth is going to be on the health for investment side, kind of lower than it has been historically, at least during this period. Is there any reason that you might want to get a little bit more aggressive in the warehouse? And maybe can you just give us an update on number of customers, et cetera? Thanks.

David R. Brooks -- Chairman, President and Chief Executive Officer

I don't know the number of customers right off hand, Mike, we can get that to you -- Paul can get that to you later today for sure. Our core belief is that mortgage warehouse is a terrific business when it's in the right proportion to your balance sheet and to your income and because of the volatility, it's still a business we're cautious around. We like it a lot. We have great customers in it, but we don't want it to be $2 billion or $3 billion. So our core view is average outstanding's around $1 billion, give or take, is a good place to where we want to be. And look, I do think there's an opportunity to be aggressive in that business right now, if I wanted to be, we just are choosing not to be.

Michael Edward Rose -- Raymond James and Associates, Inc. -- Analyst

Okay. That's all I have. Thanks everyone.

David R. Brooks -- Chairman, President and Chief Executive Officer

Thanks, Mike.

Operator

Thank you. There are no further questions at this time. I would like to hand the call back over to management for any closing comments.

David R. Brooks -- Chairman, President and Chief Executive Officer

I really appreciate everyone joining in today. And if there are no further questions, we'll conclude the fourth quarter earnings call. Appreciate everyone in your interest in Independent Bank Group and hope the world allows us to be out on the road, seeing you in person sometime later this year. So in the meantime, be well. Thank you.

Operator

[Operator Closing Remarks]

Duration: 37 minutes

Call participants:

Paul Langdale -- Senior Vice President, Head of Investor Relations and Director of Corporate Development

David R. Brooks -- Chairman, President and Chief Executive Officer

Michelle S. Hickox -- Executive Vice President and Chief Financial Officer

Daniel W. Brooks -- Vice Chairman and Chief Risk Officer

Matthew Covington Olney -- Stephens Inc. -- Analyst

Bradley Jason Milsaps -- Piper Sandler & Co. -- Analyst

Brady Matthew Gailey -- Keefe, Bruyette, and Woods, Inc. -- Analyst

Brett D. Rabatin -- Hovde Group -- Analyst

Michael Edward Rose -- Raymond James and Associates, Inc. -- Analyst

More IBTX analysis

All earnings call transcripts

AlphaStreet Logo