Welcome everyone. I'd like to welcome you to the annual meeting of shareholders of Waterstone Financial, Inc., which I will refer to as the company. Given the unique circumstances presented by the COVID-19 virus, the annual meeting is being conducted virtually by webcast and audio conference. The annual meeting will please come to order. My name is Doug Gordon, President and Chief Executive Officer of the company and WaterStone Bank. Participating with me at the meeting are Mark Gerke, Chief Financial Officer and Executive Vice President of the company and WaterStone Bank, and Bill Bruss, Chief Operating Officer and Secretary of the company and WaterStone Bank, who will also act as secretary of the annual meeting. I'd like to welcome the other directors of the company in attendance at the annual meeting, Patrick Lawton, Chairman, Ellen Bartel, Thomas Dalum, Michael Hansen, Kristine Rappé , Stephen Schmidt, and Derek Tyus.
We have made available on the virtual meeting site the agenda and the rules of conduct of the meeting. If you would like to discuss a matter not on the agenda, I encourage you to contact an officer or director of the company after the meeting. We provided an opportunity to shareholders to submit questions in advance of the annual meeting, and questions will be addressed by management prior to adjournment. Although we will take a vote on the matters to be considered at the annual meeting in a few moments, any registered shareholder wishing to vote their proxy may do so electronically during the meeting. If you've already voted your proxy or proxies, you need not vote again unless you wish to make a change.
The board of directors has previously appointed Denise Mihalovic to act as the inspector at the annual meeting and any adjournments, and to count and examine all voting. The inspector's report will be attached to the minutes of the annual meeting. Secretary has delivered to the inspector a list of the shareholders of the company entitled to vote at the annual meeting, arranged in alphabetical order as of the close of business on March 24th, 2021, for record date of voting. The Secretary informs me that the records of the company show that there are 25,230,284 outstanding votes entitled to be cast at this annual meeting, of which 12,615,143 represent a majority.
We have previously received confirmation that the notice regarding the availability of proxy materials for the shareholder meeting was mailed on or about April 8th, 2021, to each shareholder of record as of the close of business on the record date. Copies of the affidavit of distribution with documents attached will be attached to the minutes of this annual meeting. The secretary has previously delivered to the inspector the list of shareholders and all proxies which have been received. The secretary informs me that substantially more than a majority of the total outstanding votes entitled to be cast at the annual meeting are present in person or by proxy. The inspector is making an exact count and will submit a formal report on the number of shares present or represented during the course of this annual meeting.
A quorum is declared present, subject to the confirmation of that fact by the inspector in her report. The business to be acted on at the annual meeting, as stated in the notice of annual meeting, is as follows. One, the election of two directors of the company. Two, ratification of the company's selection of RSM US LLP as an independent registered public accounting firm. Three, an advisory, non-binding resolution to ratify the approval of the executive compensation described in the company's proxy statement. In order to save time at this meeting, we propose to arrange the proceedings so that a vote will not be taken until all the items have been moved and seconded. Again, registered shareholders who are attending the annual meeting by webcast do have the opportunity to vote their shares during the meeting.
If you've already voted by proxy, you need not vote during the meeting. The first item of business to be voted upon is the election of two directors of the company. The directors to be elected are to serve for a three-year term, and until their respective successors have been elected and qualified. The board of directors has nominated to serve as directors Douglas Gordon and Patrick Lawton, each of whom are currently members of the board of directors. The nominees are prepared to serve if elected. The chair will entertain a motion that the proposal to elect the directors be adopted.
I so move. I second the motion.
The second item to be acted on is the ratification of the company's selection of RSM US LLP as its independent registered public accounting firm for 2021. The chair will now entertain a motion to ratify RSM US LLP as the company's registered independent public accounting firm.
I so move. I second the motion.
The third item to be acted on is the ratification of our executive compensation as described in the proxy statement. The chair will now entertain a motion to ratify our executive compensation.
I so move. I second the motion.
The vote will now be taken on proposals 1, 2, and 3. Will anyone who wishes to vote electronically do so now. If you've already voted by proxy or proxies, you do not need to vote now unless you wish to make a change. I will now pause for two minutes to allow an opportunity for electronic voting. Two minutes have passed, and I declare the voting closed on the proposals 1 through 3. I'll now present our analysis of 2020 and the status of Waterstone Financial at this point in time. Starting with Slide 7, we have the holding company's consolidated annual performance. You can see we've grown earnings per share the last three years, and 2020 was outstanding, being three times our 2018 earnings per share.
On the next slide, we break it down by operating segment, our pre-tax income between the community banking and the mortgage banking, with community banking in blue. Community banking earnings have been consistently increasing. We're down in 2020, but that's only because we had $7 million more in loan provision than in 2019 as a precaution for what may have happened to the portfolio as a result of COVID-19. As far as the mortgage company, 2018 was when we saw our rising interest rates. When we see that, we see a couple of things happen that you'll see is that our volumes decrease and our margins also decreases. There's just too much supply out there for demand without any refinance business. We remain profitable, and there were 85%-90% of the mortgage companies were not profitable in 2018. Our model continues to work.
We bounced back in 2019, and then 2020, you can see, is just off the charts, and you'll see the breakdown as to why that is. On the next slide, we have our return on average assets, where we compare ourselves to Wisconsin-based institutions, the median of all Wisconsin-based institutions. In the five years we present, we've outperformed Wisconsin banks all five years. We're 60%-75% higher in return on assets in the years 2016-2019, and we're 230% higher than in 2020. On the next slide, we compare our community bank segment's efficiency ratio to those of the Wisconsin-based institutions. You can see that part of our profitability is because we're a very efficient banking segment. We have $2.2 billion in assets and roughly 180-185 employees. We have just a very strong culture throughout the whole bank in terms of our cost controls.
Under 50% is where every bank would like to get and very seldom get there. We're happy with the culture that we've created, and it's certainly helped in our profitability. In the next slide, it's our net interest margin. You can see we had success of increasing that from 2016- 2018. That was predominantly because we had $400 million in Federal Home Loan Bank debt that averaged about 4%, and during that time, we refinanced it to about 2%. We got a significant increase in our net interest margin at that point in time. Again, in 2018, as we get into 2019 and 2020, we start moving to the zero interest rate environment that we've seen for almost for two years at this point in time, and it's compressed margins throughout the industry, and it certainly had an impact on our margin as a result.
We feel like we may have bottomed out as the first quarter. We jumped from last year, 2.67%. The first quarter, we're at 2.80%, as our deposit rates seem to be repricing down faster than our loan rates at this point in time. It'll be good to get that stabilization in the net interest margin, and I anticipate we will not see that increase until we start to see an ascending slope in the yield curve. On the next slide, this is how we've rewarded shareholders through those good earnings that we've talked about. Basically, in the last four years, with special dividends and quarterly dividends, we've paid between $0.98 and $1.36 in 2020, as we had two special dividends of $0.50 and $0.30 in that year.
That provided shareholders roughly a 6%-7% dividend yield during those times, which is very attractive when, again, we're talking about a zero interest rate environment. It's been also a good use of our capital as the earnings just continue to increase our capital ratio. With the capital ratio that we have almost twice the industry average, it's important for us to really look at stock buybacks as well as dividends and organic growth to leverage that capital to provide a better return on equity. We'd have to get that correlation between that return on assets that we talk about to get it to actually be return on equity, and that comes with leverage.
We raised our quarterly dividend at the end of 2020 from $0.12 a quarter to $0.20, which roughly is a 67% increase. On the next slide, we show the five years of what our loan and deposit growth has been. You can see from 2016- 2018, we experienced some good loan growth. As we've gone again into that zero interest rate environment and flat yield curve, it's led to a lot of refinancing of our portfolio into the secondary market or refinancing of a lot of our commercial real estate into longer-term fixed rate product that we don't hold on our balance sheet. As a result, originations have been pretty strong, but they haven't been enough or just been enough to offset what our payoffs are. We've been running like crazy to stand still, and that's not good considering the capital that we have.
Of all of these slides, this is going to be the one negative that we have and the biggest challenge that we have going forward. As far as deposit growth, you can see we had decent growth in 2015 and 2016. We tailed off in 2017. That was kind of unfortunate because you can see we had good loan growth during that timeframe, so we didn't match it very well. In 2018, we did match that loan growth with deposit growth. In 2019, we curtailed deposit growth a little just again, because the loan growth wasn't there. Like many in the industry, we sit with a lot of cash and a lot of liquidity. In today's interest rate environment, that liquidity is earning between 5 basis points and 15 basis points.
It certainly is a drain on that net interest margin that we talked about earlier. 2020, we just continue to have good deposit growth, and we continue to encourage it despite the fact that we don't have loan growth, because most of our deposit growth has been in our core deposits, those being our checking, savings, and money market accounts. When I started here 15 years ago, 16 years ago, we had 93% of our deposits were in CDs, and only 7% were in the stable core deposits and lower cost deposits. In 2020, we've come up and exceeded 40% in our stable deposits and core deposits. We've done a real good job of turning that ship around to get more valuable deposits and really increase shareholder value as a result.
Our next slide shows our net charge-offs and recoveries to our average loans versus the Wisconsin institutions. You can see in 2016 and 2017, we're slightly above. We're still coming out of our issues that we had in 2011, 2012. Still, the numbers are extremely low at 0.05% and 0.06%. We get into 2018, 2019, and 2020, and we actually have net recoveries. During that timeframe, there was a period of time, I believe, of 10 months that we did not have a loan charge-off. That leaves us, we have good metrics in our loan portfolio, and again, that really enables us to put up those profitability numbers that we talked about. On the next page, this reinforces the metrics that we have in our portfolio.
If you remember, I mentioned that we put $7 million extra in our loan loss provision in 2019 instead of 2018 because of COVID. It turned out that our metrics did not go up. Our portfolio continues to improve, in fact. You can see, even during this five years, we've outperformed the Wisconsin banks and now have non-performing assets to total assets of 0.27%. In our difficult times back in the day, we were at roughly about 10%. We've made significant improvements, not only in cleaning up the balance sheet, but also in our credit culture going forward. At this point in time, we only have one property that we own in real estate, totaling $150,000. We had hit a high at one point of $66 million back in the day. On the next slide, we get a little bit into the mortgage segment.
You can see we had steady growth from 2016- 2019, as we did about $2.9 billion in 2019 in loan originations. The best thing of this, we had loan officers in 2018. We had 263 loan officers. In 2019, we had 226 loan officers and actually increased our volume. We moved to 2020, we declined by another six loan officers. Our average loan officers was 220 in 2020, and our volume went from $2.9 billion- $4.5 billion. It was exceptional growth. A lot of it's attributable to refinance business that was there. If you know us, we concentrate very heavily on purchase business. We did maintain those metrics. In fact, we increased our purchase business by 14% in 2020.
The additional volume was all refinance business, and our mortgage segment did a wonderful job in being able to close that many loans, despite many of them working from home and all the disruptions that were in the industry in terms of appraisers not being able to get into homes, to title companies being closed. A number of different challenges that they faced and yet still had that fantastic year. As we go to the next slide, this reinforces that our model has always been to concentrate on purchase business. The theory there is purchase business is not as cyclical as the refi business. When we're reporting publicly quarterly earnings, we like to keep them as relatively flat as we can, rather than having the gyrations you would have if we were strictly a refinance shop.
You see historically, we've always been above what the Mortgage Bankers Association's average is. In fact, we've always been 80%-90% historically. Again, this was such a unique year with interest rates declining so rapidly and the number of refinances. We did drop to 61%, but that's still well above what the industry average was of 46%. On the next slide. Again, as we saw interest rates rising in the timeframes from 2016-2018, refinance business slows up, and the people that concentrate on refinance, the way they can compete with us that concentrate on purchase business is by cutting rates. We end up having to cut some of our margins to maintain volumes. You saw that phenomenon actually happening in 2016-2018, probably midway through 2019, it was still continuing to decline.
As we get later on in 2019 and 2020, with the volumes that we were doing, we were increasing our pricing, increasing our margins, because we could only handle so much business. Basically, as long as we're going to do $4.5 billion, we're going to do it at the most profitable level that we could. You can see that it jumps off the charts. Not only did our volumes jump off the charts, but so did our margins. That's how you end up with $82 million in pre-tax out of that mortgage segment. In 2020, we anticipate as these refis start to slow down and interest rates tick up a little, that those margins will compress to more normalized numbers. We move to the next slide. As a community bank, we feel it's extremely important to give back to the communities we serve.
We donate over $750,000 to over 200 local nonprofits and schools. We also had our employees, like I mentioned, we have 180 employees. They donated 572+ hours in volunteer time to these nonprofits and schools. On the next slide, gives examples of some of the organizations that we support. We really have three pillars that we look at. One is families in need, the other is children and education, and lastly, military and veterans. We obviously do more than just that, but those are our pillars of our giving and our donation. On the next slide, shows our branch network. In the last 16 months, we've added three branches. These are branches we bought from Associated Bank and the Bank Mutual consolidation.
We had to have them closed for a year as part of the sale contingency or what they put on as stopping us from being able to open. In the last 16 months, we opened three of them. This year, in 2020, we opened our first office in the city of Milwaukee on 60th and Oklahoma. I encourage everybody to go look at these branches. They're totally remodeled. They're in excellent locations. We're excited about growing them to the levels that we've grown all of our branches. As I mentioned, we only have 14 branches for a $2.2 billion institution. We're very efficient in growing those branches, and we look forward to growing these three to those levels and continuing that efficiency. On the next slide. On June 16th of 2020, we introduced our new digital banking platform.
It's a very robust and state-of-the-art digital banking platform for businesses and consumers. It took us a year to implement, but the timing turned out to be very good because this was at a time where our branches were closing due to COVID, and a lot of people were either using drive-ups or going to digital banking. I can say that our employees did an unbelievable job to introduce this. Again, many of them working from home under a lot of stress during the pandemic. We started it pre-pandemic, so we had to get it done. They did an unbelievable job, and I encourage you to look at the digital banking platform that we have because we feel that it's state-of-the-art and as good or better than our competitors. There is not any questions that have been submitted.
We are going to now turn to the results of the voting of the items of business. The inspector has completed her count, and the secretary will now read the certificate and report of the inspector of election.
Good morning. The inspector of election has reported as follows. I hereby certify the following. One, Douglas S. Gordon and Patrick S. Lawton, the nominees to serve on the board of directors of Waterstone Financial, Inc., have each received a plurality of the votes cast at the annual meeting and are hereby elected as directors to Waterstone Financial, Inc. Two, the appointment of RSM US LLP as Waterstone Financial, Inc.'s independent registered accounting firm has been ratified by a majority of the votes cast at the annual meeting. Three, the non-binding advisory vote regarding ratification of the company's executive compensation as set forth in the April 8, 2021 Proxy Statement received a majority of votes in favor of ratification.
Four, that at all times during this meeting, more than a majority of the shares outstanding and entitled to vote at the annual meeting were represented in person or by proxy, and consequently, a quorum has been in attendance for the entirety of the annual meeting. Signed, Denise Mihalovic, Inspector of Elections.
Thank you. The report confirms that a quorum is and has been in attendance at the annual meeting for all purposes. The report also shows that with respect to the first item of business, the majority of the votes have been cast in favor of the election of Douglas Gordon and Patrick Lawton as directors. With respect to the second item of business, the majority of the votes have been cast in favor of the company's selection of RSM US LLP as the company's registered independent public accounting firm. With respect to the third item of business, the majority of the votes have been cast in favor of the ratification of our executive compensation. The certificate and report of inspector of election has been accepted and approved and will be attached to the minutes of the annual meeting.
There being no further business to come before the annual meeting, a motion to adjourn is in order.
I move that the annual meeting be adjourned.
I second the motion.
Those in favor signify by saying aye.
Aye.
Those opposed say no. The motion's carried, the annual meeting is adjourned. I want to thank you for your attendance today and for your continued support. Have a wonderful day.