Good afternoon. Welcome to the Origin Bancorp, Inc. Q3 2020 earnings conference call. All participants will be in listen-only mode. Since you need assistance, please signal a conference call specialist by pressing star key followed by zero. After today's presentation, there will be an opportunity to ask questions. To ask a question, you may press star then one on your touch tone phone. To withdraw your question, please press star then two. Please note this event is being recorded. I would now like to turn the conference over to Chris Reigelman, Head of Investor Relations. Please go ahead.
Good morning, and thank you for being with us. We issued our earnings press release yesterday afternoon, a copy of which is available on our website, along with the slide presentation that we will refer to during this presentation. Please refer to Slide 2 of our slide presentation, which includes our safe harbor statements regarding forward-looking statements and the use of non-GAAP financial measures. For those joining by phone, please note the slide presentation is available on our website at www.origin.bank. Please also note that our safe harbor statements are available on page five of our earnings release filed yesterday with the SEC. All comments made during today's call are subject to the safe harbor statements in our slide presentation and earnings release.
I'm joined this morning by Origin Bancorp's Chairman, President, and CEO, Drake Mills, our Chief Financial Officer, Steve Brolly, President of Origin Bank, Lance Hall, our Chief Risk Officer, Jim Crotwell, and our Chief Credit and Banking Officer, Preston Moore. After the presentation, we'll be happy to address any questions you may have. At this time, I'll turn the call over to you, Drake.
Thanks, Chris. Good morning. Our Q3 results continue to solidify our confidence throughout our organization that we have the right people, the right relationships in the right markets, supported by great infrastructure, strong capital and credit quality to not only work through this pandemic, but strengthen relationships while remaining in a position to capitalize on opportunities. This is detailed through our unique and defined culture shown on Slide 4. We'll get into more detail later in the presentation. I want to provide a high-level overview of our strong results for the quarter, starting on Slide 5. Total assets increased to $7.1 billion, while net income came in at over $13 million, or $0.56 of diluted EPS.
Our pre-tax, pre-provision earnings increased, finishing the quarter at $29.9 million. Our net interest income, also a high for our company, is up by approximately $4.3 million from the prior quarter to $50.6 million for the quarter. Even through the uncertainty of 2020, there are many bright spots throughout our markets and the company. Our bankers remain extremely focused on strengthening relationships, and our mortgage and mortgage warehouse teams have done an incredible job of taking advantage of the current rate environment and capitalizing on disruption in the marketplace. As we have stated since the beginning of the pandemic, we remain committed to the health and safety of our employees and customers, to being a partner within our communities, to protecting our balance sheet and taking steps to manage expenses. I'm proud of the results of the quarter and optimistic about the many opportunities before us.
I'll turn it over to Lance to go through how we support our customers, as well as discussion on loans and deposits. Lance?
Thanks, Drake. I'll start on Slide 7. We think we've smartly managed our clients' financial and psychological needs through the use of forbearances during this pandemic. As we've communicated over the past two quarters, our bankers were extremely proactive in reaching out to our customers when the pandemic first hit. For the Q2, we first reported our COVID forbearances at $1 billion, or approximately 21% of loans held for investment, less PPP. 60 days later, loans under forbearance were reduced almost 60% to 8% of loans held for investment, excluding PPP. As you can see, we've reduced those forbearances even more to less than 6% of loans held for investments, again, backing out PPP loans. Looking forward into Q4 , we believe those forbearances could decrease to as little as 2.5% of loans held for investment.
I'm extremely proud of how our bankers have responded over the first nine months of the year. Relationships are at the foundation of who we are as a company, and our teams throughout all of our markets have been hyper-focused on consistent dialogue and delivery with our clients. Slide 8 highlights our deposit trends, deposit cost reductions, and continued opportunity to further drive down deposit costs as time deposits mature. While growing deposits $1.17 billion year to date, including $400 million in Q3, we have lowered our cost of total deposits from 104 basis points at the beginning of the year to 42 basis points at the end of the Q3. Our average rate on time deposit renewals over the past 30 days has been 26 basis points, which indicates continued opportunity to reduce costs as the weighted average yield of our time deposits maturing in Q4 is 1.57%.
Effectively managing our funding costs will continue to be a huge priority through the rest of 2020 and into next year. We also continue to have success in driving non-interest-bearing deposits, as we have grown NIBs approximately $484 million year to date, bringing our NIB mix to 30% of the portfolio. Moving to Slide 9 shows where we have seen loan growth by category. Mortgage warehouse continues to be a significant win for us. We ended the quarter with over a billion dollars in warehouse outstanding balances, as we have effectively delivered on both needs derived from market rates and new client acquisition driven by dislocation in the market. This slide also shows we've been able to grow our loan portfolio 4.5% year-to-date outside of warehouse and PPP.
From a strategic planning perspective, we remain focused on making smart investments in technology that should drive a heightened customer experience and create efficiency in the company through partnerships focused on data management, the expansion of commercial and consumer lending platforms, and the integration of robotics to drive process automation and improvement. Along with better use of technology, we're also working diligently to manage expenses in areas such as leased real estate, staffing models, as well as vendor contracts. Our focus on relationship growth, technology, expenses, and margin management should continue to serve us well to position ourselves for the future. I'll now turn it over to Jim Crotwell to discuss our credit culture and loan portfolio.
Thanks, Lance. As we have previously reported, we continue to closely monitor those industry sectors that could experience a more protracted recovery from the current economic downturn, specifically the sectors of hotels, energy, non-essential retail, restaurants, and assisted living. As you can see on Slide 10, these sectors totaled $551 million, which is approximately 11% of total loans held for investment. Since the beginning of the pandemic, we have conducted focused reviews of these particular sectors to monitor the impact of the economic downturn and to ensure the appropriateness of assigned risk ratings. I'd like to point out that we have recently completed our annual external loan review, which validated our initial internal ratings. As Lance mentioned, we are pleased with the reductions in COVID-related modifications across our portfolio during the last quarter and are pleased with the resiliency of our portfolio, which we believe is driven by its diversity.
On Slide 11, we provide information on our hotel sector, which totals $64 million, or 1.3% of total loans held for investment net of PPP. We continue to feel optimistic about this sector given the low average pre-COVID loan-to-value of 41% and the liquidity in excess of $200 million from the guarantors of these loans. The energy sector, as outlined on Slide 12, continues to perform well with outstandings reducing to $55.5 million, which is 1.1% of total loans held for investment net of PPP. As we have indicated in the past, we have no direct exploration and production exposure. Our non-performing loans in this segment continue to be comprised of one relationship totaling $2.2 million, which has been a long-term workout. On Slide 13, we have provided information on our non-essential retail sector, which totaled $151 million, representing 3% of our portfolio.
Overall, this segment increased by $ 4.6 million during the quarter, driven by a $6 million opportunity in our Houston market, which included strong anchor tenants, strong debt service coverage, a low loan-to-value ratio, and significant guarantor support. Overall, pre-pandemic loan-to-value and debt service coverage ratios are strong at 56% and 1.47 times respectively. Lastly, as reflected in the previous quarter, non-performing loans and past dues in this sector are primarily driven by a single credit that was placed on non-accrual in the Q1 of this year. Our restaurant portfolio, as reflected on Slide 14, totaled $136 million or 2.7% of total loans. I've been especially pleased with the resiliency and performance of this sector, as evidenced by no past dues and no non-performing loans. Our assisted living portfolios, as reflected on Slide 15, totaled $145 million as of quarter end, representing 2.9% of total loans net of PPP.
While this portfolio segment increased $4.5 million for the quarter due to funding on projects under construction, we continue to focus on reducing our exposure in this segment. Subsequent to quarter end, we were successful in exiting one of our assisted living relationships, which will result in a reduction of $9.2 million in this segment and in classified assets. While we will have a moderate write-down during Q4 related to this credit, the amount of write-down was fully reserved for as of 9/30. Slide 16 provides a recap of our asset quality trends. As you can see, past due loans 30 days or more continue to be positive, coming in at 0.58% net of PPP loans and are primarily driven by our past due non-accruals.
As to non-performing loans, non-performing loans ended the quarter at 0.60% of total loans net of PPP, down from 0.63% as of June 30th and down from 0.74% from the beginning of the year. As anticipated, net charge-offs for the quarter reduced to an annualized rate of 0.15%, which resulted in year-to-date annualized net charge-offs of 0.28%. Total levels of criticized and classified loans remained stable during the quarter. Lastly, we did increase our allowance for credit losses to 1.45% from 1.33% of loans held for investment in the prior quarter, which results in a reserve of 2% of loans held for investment net of PPP and mortgage warehouse loans. The increase in reserve was primarily due to the estimated impact of COVID-19 on the company's loan portfolio, combined with an extension of the reversion period from 18 to 24 months within the CECL economic forecast.
This assumption contemplates that we will return to historical loss rates in the latter part of 2023, with reversion at this level beginning in the latter part of 2021. Our current forecast status suggests the decline in economic conditions may not be as severe as originally predicted. The duration of the recovery remains uncertain. While we do think it is prudent to build our allowance, we continue to be pleased with the performance of our portfolio. I'll turn it over to Steve now.
Thanks, Jim. Starting on S lide 17, you can see our trends related to net interest income and margin. Our net interest income was $50.6 million for the quarter, an increase of 9.3% from Q2. The main driver of this increase was an increase in mortgage warehouse lines of credit along with deposit cost savings. In the top right waterfall, you can see the warehouse loans contributed nearly $2.7 million on an increase in net interest income quarter-over-quarter. Our continued focus on lowering deposit costs, as Lance mentioned earlier, was the next largest contributor to increasing net interest income on a linked-quarter basis, adding just over $900,000 to net interest income. From a margin perspective, we did a better job this quarter on efficient deployment of cash, which increased our margin by six basis points. Our margin in Q3 ended at 3.18%, up nine basis points from Q2.
At the end of Q1 , we mentioned the increase in Federal Home Loan Bank advances and other non-deposit funding due to the uncertainty around the economic shutdown. NIM expansion was driven by our lower cash balances redeployed into higher earning asset classes, deposit cost reductions, and mortgage warehouse lines of credit volume increases. We received a five basis point lift due to a deposit mix and rate reductions quarter-over-quarter. The last thing to point out on the NIM trends is excluding PPP, we had a 13 basis points increase in NIM compared to the prior quarter. On S lide 18, we show trends of yields, costs, and a profile of loans held for investment.
You can see the impact on the falling rates had on the loan yields in Q2, and how in Q3 the loan portfolio did not see significant continued yield reductions, with loan yields reducing only 7 basis points or 2 basis points excluding PPP. With our continued focus on lowering deposit costs, we saw a 22% decline in the cost of deposits from the prior quarter, coming in at 42 basis points. The bottom right graph shows our mix of fixed and floating rate loans at quarter end, which has not significantly changed from prior quarter. On Slide 19, I want to touch on a couple points about non-interest income. Pre-pandemic, we typically had about 20% of our net revenue from non-interest income. While the past two quarters, we have performed really well in mortgage, which drove the non-interest income percentage to greater than 25% this quarter.
Total non-interest income was over $18 million, down approximately $1 million from Q2 due to a reduction of mortgage originations and sales from a historically high Q2. Other significant changes in non-interest income category included a decline in swap fee income due to fewer deals, an increase in $1.2 million in other miscellaneous income this quarter. Other miscellaneous income includes income from security sales, which increased $301,000, and a $661,000 decrease in losses on other asset sales. Slide 20 covers our non-interest expenses. As we look at the trend over the past five quarters, we can see progress in the operating leverage, especially in the most current quarter. While we see improvement in these metrics, our expenses did increase in the current quarter based on a few factors. Advertising and marketing increased $671,000, driven by $550,000 in contribution within our communities.
As you'll recall from our Q2 earnings call, we launched a strategic initiative with a portion of our proceeds from PPP fees, which strengthen our investment in the communities that we serve. Another expense that grew this quarter was regulatory assessment expense, which was up $544,000 in Q3, due primarily to asset growth in the first two quarters. These two items, coupled with the $475,000 in litigation settlement charges in the quarter, account for $1.7 million of the nearly $2 million increase in other non-interest expense seen in the chart on the left. We believe on a go-forward basis, we will be able to hold our expenses between $37 and $38 million per quarter on a normalized basis. I'll now turn it back over to Drake.
Thank you, Steve. As we look at our capital ratios, we remained well capitalized at the end of the quarter, and we took action early this month to further strengthen our capital position. We successfully completed an issuance of $80 million in holding company subordinated notes this month. The timing was right, the market was attractive, and this was a great opportunity for us to bolster our total capital levels as well as support future growth opportunities. You can see on Slide 21 the impact of the issuance has to total risk-based capital at the holding company and the bank, assuming a portion of the proceeds are downstreamed to the bank. In our 8-K release yesterday, we announced the upcoming retirement of Executive Risk Officer Cary Davis.
Cary's been an awesome partner of mine for over 20 years and been instrumental in the success of building our institution through multiple credit cycles. Cary has also been critical in building a risk and credit function that can continue to own for years to come with the same diligence and excellence he has shown for the past two decades at Origin. Now, due to our growth and increasing complexity of the environment in which we operate, early this year, we separated the roles of Chief Risk Officer and Chief Credit Officer. Cary has been instrumental in guiding us through this transition, and we thank Cary for his service. Jim Crotwell will assume the role of Chief Risk Officer in 2019, and Preston Moore assumed the role of Chief Credit Officer earlier this year.
Both Preston and Jim bring decades of banking experience to their roles and will continue to guide these functions of the company. I'm extremely proud of the success of our company throughout the year and where we finished the quarter. We crossed $7 billion in total assets, $1 billion in warehouse loans, posted record net interest income and record pre-tax, pre-provision income, expanded our margin, had another strong performance in mortgage banking, and experienced stabilization in credit metrics across the loan portfolio. A special thanks to all our employees who work tirelessly on a daily basis to deliver for our customers, shareholders, and community. I'll open the call for questions.
We will now begin the question and answer session. Our first question today comes from Brady Gailey with KBW.
Hey, thanks. Good morning, guys.
Good morning, Brady.
Good morning, Brady.
I wanted to start with growth. You all had great loan growth, both on a core basis and in the warehouse. How do you think about growth going forward, looking at those two segments, your core loans, ex warehouse, and then the warehouse has just done so well? Do you think it's stable from here? Do you think that you could see some downside from here in the warehouse?
Yes, Brady, thanks. A couple of things here. Let me answer your first question. I think going into 2021, we certainly see loan growth in the low to mid-single digit area. A couple of things that's impacted growth at this point, line utilization, especially on a C&I, which I think creates confidence, is down $167 million from the quarter. Just the migration in energy, where we started out the year at $128 million, now it's down to $55.5 million, is a natural progression in the down market. Certainly has worked like it was supposed to. Mortgage warehouse has just been a big win. If you look at, I've talked about this, I think, last quarter, if you look at our numbers based in a normalized environment, when the market slows down, we have outstandings of $200 million-$300 million.
We've taken, I think, strong advantage of the dislocation in the marketplace. Our team's done extremely well. We have picked up significant relationships in the last two quarters to bolster our numbers from running about 20 clients to close to 40 clients. We see that number normalized once things slow down, potentially in the $500 million-$600 million range, and probably see mortgage warehouse finishing the year in the $750 million-$800 million range. It's holding up well for us. We have good pipeline activity right now. Certainly, our C&I companies are starting to show signs of confidence. We just think we'll get back on track, certainly not to the higher end of the single digits and our 10%, but certainly in that low to medium single-digit range.
All right. That's helpful. That was good to see the reserve go a little higher in the quarter. I know you've kind of been talking about this two to two and a quarter range, where you're basically there now. Do you think the reserve build is over, or do you think you continue to add a little bit to that over time?
It's going to be interesting this quarter. We've got certainly mixed signals, and we're past credit deterioration and certainly any type of migration based on industry deterioration or anything. The numbers, the increases are going to have to come from our model and the metrics around that model. I think you have to look at the fact that we do see some renewed confidence, let's say, in the market. On the other hand, you have potential change. Let's say there's a change in administration, COVID heats back up, and some lockdowns and those type of things certainly kind of fog your thoughts about what that looks like.
I'm still saying that we could potentially see a slight build in the Q4, which at that point I would say would certainly wrap us up, and I don't see that number going to 2.5 in any way, but certainly could split that number between where we are today and 2.5 if the model supports it.
All right. Finally for me, in other conference calls this earnings season, we've heard some CEOs talk about a little uptick in the M&A conversations. I was just wondering if that was the same for you guys and just kind of what your take on M&A was at this point.
I wouldn't say an uptick. Certainly, M&A is a focus of ours today. We think for us to really perform the way we desire to perform, it would be extremely helpful to have some partners in the process, especially around strong deposit franchises. We continue to have those conversations, and certainly, that is a desire of ours. I don't see at this point, I could say that there is an uptick in it, but we're dealing with multiple opportunities, and it's a strong desire of ours to stay active.
Great. Thanks for the color, guys.
Our next question will come from Matt Olney with Stephens.
Thanks. Good morning, guys.
Good morning, Matt.
I want to circle back on the mortgage warehouse commentary, and Drake, you mentioned onboarding some new customers. Any more color you can add as far as how many customers you've added? I think you mentioned in normal times, the range was 200 to 300, and now you're saying normal times would be, call it 500 to 600. Are you saying that you've doubled the number of customers in the warehouse more recently?
I'm just going to use round numbers here. Probably this time last year, we were dealing with 20, 21 customers. Today, we have 36. We're about to onboard four additional customers. What's been interesting about the top, this next 20 that we're adding, is that they seem to be larger clients, a lot more stable, certainly stronger and better core deposit from the standpoint of non-interest-bearing deposit. That's why I'm up in this based on our belief of what utilization's going to do.
Okay. Drake, sticking on that same topic on the warehouse, what about the associated cost with adding these new customers? Are you adding new people, new systems? Just trying to figure out the operating leverage on these wins?
Yes, Matt, that's a big win for us. We are extremely efficient there. Ken and Jason Johnson drives that team, our backroom team that's driven by a lady that is completely awesome. I think we've added one additional person through this process. We don't see a buildup in expense at all. That's why it's a huge win for us.
Okay, great. Circling back on loan growth ex warehouse and ex PPP, it sounds like the levels you achieved in the Q3 may be the high water mark, or it may be tough to achieve that again. Can you just talk more about what we saw in the Q3, the strength there? Was it from slower pay downs than we've seen that could accelerate, or were originations higher than expectation? Just any more color on the 3Q growth.
Yes, no, it was originations. We saw strong growth in residential and CRE. I really think looking at pipelines, we could see a similar outcome for the Q4 based on what we're seeing and strong activity in the markets. It's just obviously a little more difficult to underwrite. There's some areas that we're not willing to expand the portfolio in. We're staying true to our strategy since COVID. I do see somewhat of a repeat of the third and the fourth.
Okay, just lastly on capital. You mentioned in your remarks that you raised some capital a few weeks ago. Love to hear more thoughts about deploying that capital and specifically thoughts about the buyback program. Just remind us what the current amount of authorization is and what's the appetite to activate that buyback program.
Yes, I'd like to back up a little bit because this goes back to a strategy, let's say, from 2019 when we really created a dividend strategy that would make our stock more attractive from a yield perspective. Each year, we'll continue to analyze where we are from a dividend perspective as we did this quarter. We'll do that next year and felt that we needed to continue that strategy, which really what that did was allow us to, I think, with the strong capital position we have and the confidence we have in the portfolio where we see 2021 performance come, we just felt like that dividend needed support, which runs into a buyback. In July of 2019, we established a $40 million buyback program. Certainly slowed that or suspended it in March.
That has been on the table, and through this process, it was just basically a communication board. Hey, we need to start focusing on buyback opportunity. What I would say is that we got $29.2 million remaining in that. It is a tool that's on the table for us to use. We certainly, with the capital position and why the sub-debt came to be a focus was when we looked at total capital, certainly we had strong Tier 1 total capital. We didn't want to slow down our growth opportunities. We see some opportunities in 2021. The market was perfect, I think, for us to take advantage of adding some additional Tier 2 capital, which what that did was really allowed us to focus on where we think those opportunities are continuing.
We had planned on, at one time, seeing a steep pullback in mortgage warehouse, which would have refreshed that capital ratio, especially on the Tier 2 side. With that opportunity, we just felt sub-debt could bridge that gap, give us the opportunity to stay in the game from a loan growth standpoint, and keep us focused on M&A. The unfortunate aspect is with our multiples the way they are, it's tough to think that we can be significantly successful with high-quality M&A targets. This just puts us in a position to recognize that our stock's the best buy out there for us right now. As we move in through the Q4 and continue to look at a couple of opportunities, I think it's going to be prudent for us to really get active, let's say, potentially in buyback if the market continues like it is.
Okay, great. Thank you.
Our next question comes from Brad Milsaps with Piper Sandler.
Hey, good morning, guys.
How are you doing?
Good. Flying a little blind today. No power or internet in Atlanta, so bear with me.
We know what that's like.
You guys are probably used to that, though, so Yes. I'm preaching to the choir. Just wanted to maybe ask quickly about the net interest margin. It looks, from best I can tell, that you guys did a pretty good job of holding loans, sort of, ex warehouse, ex PPP, whatever you want to throw in there, pretty stable or maybe down slightly linked quarter. Can you talk about your ability to continue to do that and how that relates to what you're seeing in terms of the net interest margin as you look out into 2021?
Yes, I think our team's doing an awesome job, especially on the deposit side. We continue to feel like we have some room to go reducing deposits in an extremely competitive environment on the loan pricing side that takes away some of that strength. We do feel we had some benefits of, I think, pricing strategy, especially on the deposit side, that drove our NIM expanse in this quarter. I do want to remind everyone that with the sub-debt, we will probably see a five basis point impact on the margin in the Q4 and feel that with that impact and where we see things at this point, that our margin for the Q4 could look a lot like our Q2 margin, if I could say that, which we feel very comfortable with.
At this point, a pullback because of sub-debt, maybe a basis point here or there, but very similar to what we saw in the Q2. Hopefully we'll start seeing some wins in 2021.
Great. That's very helpful. Just as a second follow-up, maybe for Steve, just on the retail mortgage banking business. I know the Q2 was impacted by a big positive swing in the MSR. Just kind of curious what that impact was in the Q3. I know that'll be in the 10-Q, but just kind of curious what that was, and then how your gain on loan sale margin compared in the third versus the Q2.
The gain on sale was higher this quarter, and we had a little bit less on volume on originations and sales. The MSR did not have a large impact at all. It was pretty much neutral, maybe $200,000 down. That's really why we went from 10 down to nine. If you were expecting MSR to go a little lower, that nine would've been maybe seven or eight, but the MSR really kept pretty much in line. After Q1, after March 31st, that really MSR came down with the market, and since then, it hasn't changed dramatically. Going forward in the Q4, we expect about $150 million-$170 million in production and sales, and we had about $220 million this quarter. The Q4 historically is not as strong as the second and third, and that's what we're expecting next quarter also.
Okay, great. No, that's really helpful. Just to follow up to that, Drake, I know you guys have spent the last couple of years making a lot of changes there, trying to get more efficient. Obviously a great year this year with refis and rates going lower. How much more market share do you think you can gain in 2021 relative to what you did this year? Just trying to get a sense of what the impact will be as eventually mortgage does start to contract a little bit. Just kind of curious what you're thinking around that business.
Yes, I think we can continue on the pace from a market share standpoint like we are because of really lift-out teams in Mississippi and Louisiana, especially in our Monroe market, that's done extremely well. We have Our top two producers are in Ruston and Monroe that are running in at eight to $10 million a month. We still have opportunities. Our Houston team is looking at expanding the mortgage origination side. I think we can continue to win like we have. Obviously, with volumes down, I think this quarter, our origination really showed strong at coming in about 50% of total volume, and we still see pipeline strong. At this point, I just think market share will continue to come our way.
Great. Thank you, guys.
Thank you.
Our next question comes from Kevin Fitzsimmons with D.A. Davidson.
Hey, good morning, everyone.
Good morning, Kevin.
I'm just curious on credit, drilling down a little further in terms of what I noted the impressive decline in deferrals and what you expect that to continue. I'm curious about loans, any migration you're seeing to the watch list, to special mentions, to substandard, whether any of those former deferrals or remaining deferrals or any other loans, for that matter, notable migration that you're starting to see? Thanks.
As we report, I think we're seeing a stabilization, and I would say at this point, it's not deferrals that are migrating to large degree. Thought it was kind of interesting that I heard a CEO kind of take a slap at banks that were aggressive with deferrals and saying that it must have been a feel-good moment, but I can tell you for us as a company that focuses on small businesses and C&I firms, that this is real. It's not a feel-good moment at all. It's supporting these companies. The way we approach this is, if they were in good standing and were a company that could move forward, we certainly are going to continue to support them. We don't play games.
We addressed some of these companies. We said that last quarter that if we had problems, we were going to purge ourselves of these things. We're very aggressive with that. It is a focus of mine, like I said, not a feel-good moment, to do everything I can to keep these companies focused on being viable and continuing to go on once we return to some type of normal environment. I am going to let Jim Crotwell for just a minute kind of walk through some of the migration and what components are. Jim?
Thank you, Drake. Good morning. Really, we've seen very strong resiliency in our portfolio. As you can see on Slide 16, that our overall level of classifieds and special mention loans have been very stable over the last two quarters and really no change. As we look forward, we continue to anticipate that stabilization in our total criticized asset base. Obviously, during this pandemic, we've been proactive in the pass watch category and in just the pass category as we look through internal risk ratings perspective. As mentioned in the presentation, all of our internal risk ratings were validated by our annual external loan review. Right now we're seeing a good bit of stability within our portfolio, and we do not anticipate any significant shifts.
Okay, great. That's helpful. One thing just as a follow-up I want to go back to, I just want to make sure I'm understanding it correctly. The mortgage warehouse lines of credit, which were a little over $1 billion on an end of period basis and about $724 million on an average basis in the quarter. Drake, your commentary was that in a normal environment that runs from $200 million-$300 million per quarter, and you're saying that because of the pickup in new customers, that's going to normalize out at more like $500 million-$600 million. You commented that by the end of the year, you could see that number being $750 million-$800 million.
With the slowdown, does that mean we're going basically from the $1 billion down to $500 million, $600 million over the next few quarters, but then it ramps up from there and ends the year where you said? I just want to make sure I'm understanding that correctly.
Sorry, Kevin, when I created confusion there. I said a lot in a very short period of time there that in a normalized environment, we would run $200-$300 with the customer base that we have, with the dislocation market and our ability to take advantage of that in a normalized environment with our current customer base on an ongoing, on a forward-going slow market, that would look like $500 million-$600 million. My comment about $750-$800 is where we think we'll end up at the end of the Q4. As we move forward in 2021, we'll start to see a slowdown. Like I said, comparing that to a normalized environment pre-2020, that now looks like $500 million-$600 million outstanding.
That's great. That cleared that up for me. Thanks, Drake.
Well, I apologize. I don't think I did a good job on that.
No, that was on me. One last one from me. Just the swap fees, that went from quite a high number to a low number this quarter, and just where you guys see that going forward?
Kevin, last quarter was one of the highest we've had. We normally, excuse me, we normally look at it and say maybe 100,000 a month, sometimes 200,000 a month. Last quarter, it was just the second highest we ever had. This quarter's a little bit lower, but I think going forward, it would be if you took both quarters and averaged them, that would be a good place to at least guesstimate where it would be going forward.
Okay. Thank you, everyone.
Thanks.
Thank you.
If you have any further questions, please press star and then one to join our question queue. Our next question comes from William Wallace with Raymond James.
Morning, guys.
How you doing?
Very good. Thank you, Drake. First question on the expense line. Steve, your prepared remarks, you said that you think you could hold the line at $37 million-$38 million? Did you mean that you think that you could hold it there for 2021 or for the Q4, then we would expect kind of normal seasonal pressures i n 2021?
That would be for both.
Okay.
That would be for where the budget is right now for next year.
Okay. How does that contemplate mortgage activity and all the variable comp associated with that?
The mortgage activity would be a little bit less than last year on an average, just because of all the refi we had in Q2 and Q3. As I said, about 150 to 160 million next quarter, and we did about $220. We think the Q1 may be somewhere in the low $100s, and we'll be back up to closer to $180, $200, Q3 next year. That does take into effect the compensation for the mortgage originators. We also had a lot of compensation this year also.
Hey, Wallace, if I could, I'd like to add to that just for a moment, because we have, as we've discussed, Lance Hall is extremely active with his team in looking at expenses across all lines and have made some real strong moves, and we'll continue that through 2021. The offset to that is, as we still have confidence in our markets, we've had, as I noted in the Q2, a great opportunity in Dallas and another office to really, what we think is, create a core deposit growth strategy that's significant for us in that market. We'll have some added expense there. We also, in The Woodlands in Houston, as we've communicated, will be moving offices and picking up and ramping up, from a lending standpoint, some opportunities there.
We're baking all that in and really trying to offset that with some strong management on the expense side. We still see opportunity for growth in this organization, but we feel like it's necessary to be very active on the expense side to be able to cover that.
Okay. Thank you. That's helpful. Drake, on net interest margin, you said you think Q4, we probably see margin back down to roughly where it was in the Q2, then that you would get some wins, I believe, is what you said from there. Does that mean that you expect that your margin could start to expand next year? If so, is that just a deployment of liquidity, or do you feel like you're bottoming out on the loan yields and deposit cost moves will help drive expansion?
Certainly see opportunity there. Bottoming out of deposit costs is certainly going to be a headwind for us, because we have what I think are some significant pushes through incentives to grow core deposits next year, with that being more of a focus than on the loan side. We're taking those teams to be able to really focus that. I think where the concern is for us, at this point, is where the competitive side of loan yields come in, because we're still seeing some pricing that's tough to handle at this point. We're going to stay focused on our customers. Really, I think the win is, if we can potentially focus on that, is the utilization of our cash and being able to put liquidity to work, which is some strategy that we'll be working on through this quarter.
Okay, thanks. Then last question, just kind of bigger picture, Drake. Gosh, a year ago, there was a very myopic focus on core deposit growth at the bank. Obviously, stimulus and PPP funds have muddied the picture, then it looks like this year or this quarter, you put some brokered deposits on to replace some other funding. Can you just talk maybe what's going on at the bank from a core deposit perspective? Do you feel like you're having successes? I just look at that loan-to-deposit ratio. I don't know what it would be if we took the brokered deposits out, do you feel like that focus a year or so ago is reaping rewards if you kind of peel back some of the impact of all the stimulus checks and PPP dollars that have come onto the balance sheet?
No, absolutely. If you look at our markets and the growth in our markets year-to-date from core deposit growth is pretty phenomenal. We're seeing some unbelievable wins. As I said earlier, it drove us to take advantage of an opportunity in Dallas that we think can add a couple hundred million dollars of core deposit growth over the next 24 months. That is significant for us. What we're doing is really shifting our teams from an incentive compensation standpoint, more to core deposit growth, in a period of time where we see the opportunity to take advantage of the market, as we see a slowdown on the loan side. If you normalize our loan deposit ratio, taking out all the noise, I would say we're running around 88%-89% and feel pretty good about where we are.
Understand that as we have this slowdown in loan activity, we see strong opportunity in all markets to win on the core deposit side. Our focus in 2021 is going to be core deposit- driven.
Great. Thank you for that, Drake. I appreciate it. I'll stick back.
You're welcome.
Again, if you do have a question, you can press star then one to join our queue. Our next question is a follow-up from Matt Olney with Stephens.
Yes, thanks for taking the follow-up. Just want to circle back on this discussion on the margin and specifically on loan yields. Loan yields came down a bit in the quarter, but it was pretty modest. Any commentary on how you see the loan yields playing out over the next few quarters? I guess I'm looking for any color on recent loan growth. If you have the new and renewed loan yields more recently, just trying to figure out how much more pressure we could see there. Thanks.
Yes. I think in an environment, and I know why you're asking, where you're coming from, that we see some headwinds on the loan yield from a loan yield aspect, and continue to think that it's going to take balance sheet management and a couple of other things to be able to fend that off because of where we see deposit costs starting to slow down from a decline. Moving forward, and I will say this, we're going to create, and I keep talking about incentives, and look, incentives drive everything we do. We're today working on a method to be able to incent and reward our relationship managers based on yield and their ability to get yield, very active and going to almost require floors in any situation.
I think that that's going to be the real opportunity for us to slow down or kill some of the headwinds coming from a loan yield perspective. You're going to see strong incentives that's going to, I think, drive better pricing. You're going to see floors that certainly are going to slow down this situation. I think we can do that in a very competitive environment and still see the loan growth I'm talking about.
Okay.
Matt, I would say this. The big win for us on the deposit side is really going to be where CD pricing is right now and where we see CD pricing heading. That's the next trigger to pull and where we see a win at this point.
Okay. Okay, great. That's all for me. Thank you, guys.
Thank you.
This concludes our question and answer session. I would like to turn the call back over to Drake Mills for any closing remarks.
Okay. First off, thank you everyone for the opportunity and support. It's pretty incredible how comfortable I am at this point in what we're doing, the confidence that we have in this organization, as I said in my closing remarks earlier. I will say that thanks to the support and thanks to you, the people inside this institution, that our culture is stronger, our attitudes are incredible. That was rewarded here in the last couple of days when American Banker released their top banks to work for, and it was such a pleasure to be voted in the top 20 of the best banks to work for. That shows that we're winning in a very difficult environment. Our culture is stronger. Our people feel supported, they feel safe, and all those things are going to end up and be rewarded back to the shareholders in return.
I do appreciate in a huge way, your confidence and your support. Thank you.
The conference has now concluded. Thank you for attending today's presentation. You may now disconnect.