Ladies and gentlemen, thank you for standing by, and welcome to the 2021 annual meeting of the shareholders of Unity Bancorp. At this time, all participants are in listen-only mode. As a reminder, today's conference is being recorded. I would now like to hand the conference over to your speaker today, Mr. Dallas, Chairman of Unity Bancorp. Thank you. You may begin.
Thank you. Good morning, everyone. My name is David Dallas. I am honored to be the Chairman of the Board of Directors of Unity Bancorp. On behalf of the Board of Directors, management, and staff, I would like to welcome all those listening to our 2021 annual meeting of the shareholders of Unity Bancorp. As you all are aware, due to the ongoing health impact of the coronavirus and to support the health and well-being of our stakeholders, we are conducting this shareholder meeting in virtual meeting format. For those listening, we would like to thank you for your continued support as shareholders of our company. Despite the financial impact of the COVID-19 pandemic, earnings remained strong in 2020. Unity reported earnings of $23.6 million.
Another accomplishment was that Unity Bank has been recognized by the American Banker Magazine as the 15th highest-performing publicly traded community bank in the nation for banks less than $2 billion in assets. Unity was the top-ranked New Jersey community bank on this respected industry list, which Unity has been on for five consecutive years. In addition, Unity Bank was named one of the best places to work in New Jersey for the fourth year in a row by New Jersey Biz Magazine. Mr. Hughes will further elaborate on more of Unity's accomplishments, the year's highlights, and Unity's strategic goals and future growth of the organization later in this morning's meeting. I would now like to call the meeting to order and proceed according to today's agenda. The first item on the agenda is the recap of the proposals voted for.
As you may know, this year, our shareholders of record have been asked to cast their votes on the four proposals. First, for the election of Unity's three nominees for Director. Second, to ratify the selection of RSM US LLP as the company's independent external auditors for the year ending December 31, 2021. Third, an advisory vote for the executive compensation plan. Lastly, an advisory vote for how often the company will conduct an advisory vote on executive compensation. The next item on the agenda is the appointment of judges and the Inspector of Election and the announcement of a quorum present. Today's judges of election are Ms. Laureen Cook, Unity Bank's Vice President and Chief Accounting Officer, and [Ms. Linda MacDermid], Unity Bank's Corporate Secretary. Would you both please acknowledge your presence by stating your names?
Laureen Cook.
Linda MacDermid .
Thank you. Today's Inspector of Election is Ms. Marleen Grandeson-Mills , a Representative from the company's stock transfer agent, Computershare. Ms. Mills, would you please acknowledge your presence by stating your name?
Marleen Grandeson-Mills , present.
Thank you. Ms. Mills, may I please have the current tally of the total votes received and a percentage of the total outstanding shares that this tally represents?
Mr. Chairman, based on the total votes received of 9,542,093, representing 91.73% of the total outstanding shares, a quorum is determined to exist for the continuation of today's meeting.
Thank you.
Thank you.
Next on the agenda will be the opening of the polls for vote for the four proposals. I hereby open the polls for further voting. If you have not yet voted or wish to change your vote, you may do so now by clicking on the link provided online. Any shareholder who has already voted and does not want to change their vote need not take any further action. Now, as we wait for the final votes to be submitted, I would like to encourage any shareholder present to submit questions online by clicking on the D ialogue icon in the upper right-hand corner of the meeting screen. Please try to limit your questions to one or two each. We will try to address as many questions as possible towards the end of the meeting.
If your question is not answered, please feel free to reach out through our investor relations link on the Unity Bank website. Also within the virtual meeting portal, you will see a 2020 annual report and 2021 proxy report. Within our proxy report, you will find a full list of our directors and executive management team. With us today representing the shareholders of the company are Director Mark Brody and Director Aaron Tucker. Gentlemen, please say hello.
Good morning.
Hello.
At this time, I would like to entertain a motion to close the polls for the voting on the 2021 annual shareholders meeting of Unity Bancorp. May I have a motion, please?
So moved.
Second.
Second.
Thank you. The next item on the agenda is the company's report to the shareholders. I would now like to turn the meeting over to our President and Chief Executive Officer, James Hughes.
Thank you, David. Good morning, everyone. Again, thank you for attending today's meeting. I think we can all say that we're glad that 2020 is behind us. It was a challenging year helping our customers with deferrals, the PPP program. All things being equal, despite the pandemic, it was a record year for Unity Bank. You can see on the first slide, our net income was $23.6 million. Slightly down from the prior year, that includes a $5 million additional provision for loan losses. If it wasn't for the additional provision, our earnings would have been up another $3.7 million. On an EPS level, we did have a record earnings of $2.19, our high watermark. The impact of the provision there was an additional $0.35. Loan growth was significant in 2020.
We had a 14% increase, 9% increase in commercial loans, 7% increase in consumer, we had about $120 million of PPP loans. We'll get into a more detailed slide on loans coming up. Total deposits increased close to 25%, 64% increase in non-interest bearing, 28% increase in interest bearing. A lot of this increase is due to the funding of the PPP program. It probably won't stay, we are enjoying the growth in deposits nonetheless. On the next slide, you see our total loans by year. Again, $1.6 billion. Again, another high watermark. You can see the growth of 10% in the commercial loans growing from $765-$840. Residential loans was completely flat.
Although we had a record year of residential loan volume, we did about $450 million, 2/3 of which are sold into the secondary market, a third of which is going back into our portfolio. Due to the drop in interest rates, a lot of our non-performing residential loan customers were able to find alternative financing elsewhere. That's the reason that the portfolio is flat. We had some growth in consumer lending. SBA lending is flat, in terms of absolute volume, SBA lending has almost doubled from the prior year. You can see the new category of PPP 118. I think we did about $140 on round one, we'll do about $100 million in round two. The next slide shows total deposits. $1.5 billion, up from $1.2 billion.
Non-interest bearing at $460 million represents 30% of our total deposit base, which is up from 22%. We had significant increase in time deposits as well. This is intentional as we tried to lock in some longer-term brokerage CDs, five- and four-year CDs to take advantage of the low rate environment. The next page just shows a summary income statement. Again, our earnings per share were $2.19 versus $2.14. Our return on average assets and average equity at 1.35%, 14.20%, again, impacted by the additional loan loss provision. All those ratios have returned back to more normalized levels, which we'll report when I get into the last slide on the first quarter results. Our margin went down slightly. Let's turn to the next page.
When they adjusted interest rates for the pandemic, dropping prime quickly down to three and a quarter, it takes a while for us to reprice our deposits. Although it shows a decline in our margin to 3.85, it has now rebounded north of four, and we're optimistic that we'll be able to maintain stability in our margin at or around that level. In absolute dollars, net interest income has grown to $64 million, up from $57 million the prior year. Non-interest income up significantly from the prior year, up from $9.5 million to close to $13 million. Branch fee income and service fee income were down, and that was directly tied to the pandemic. As many businesses we bank, check cashers, they were shut down for several months.
We've been waiving a lot of late fees on loans, and so we're hopeful that those categories will return back to more normal levels. SBA gains, you can see, have increased significantly from the prior year. We're hopeful that trend will continue. Again, getting back to the mortgage business, we did about $450 million of mortgages, of which about $300 million were sold for gains of $6.3 million. That's the gross gain. Non-interest expenses. Again, we've done a good job in controlling our expenses. Our expense ratio has come down from $2.44-$ 2.24 over the last five years. Although it's an increase in actual dollars, you'll see on the next slide.
Most of the increase in non-interest expense is due to increase in comp and benefits, and the lion's share of that category is commissions that were paid to the loan officers who get paid solely on commissions. I'd say 3/4 of that increase is tied to mortgage commissions. We also have a separate line item in our non-interest expense called BSA expenses. For those of you who are not aware, we are under a BSA consent order. We believe that we're significantly through at least 75% through the consent order. We expect BSA expenses to be significantly lower in 2021. All other categories of non-interest expenses are virtually consistent with prior periods. The next slide just shows the profitability over the last five years. Our dividends, for 2020, was $0.32, a very low dividend payout ratio. We like to grow our capital.
We took advantage of the stock price being low in 2020, and we purchased about 550,000 shares. Our capital does remain extremely strong. Excluding PPP, our Community Bank Leverage Ratio is close to 11%, when the regulatory requirement is 8.5%, and our average equity to asset ratio, excluding PPP, is north of 10%. The last slide shows our first quarter results, which I would say, fully normalized because we are getting the revenue of PPP. That is helping. You can see, the income for the first quarter, again, another record quarter, $8.5 million, $0.80 per diluted share. A return on average assets at 1.85%. Return on average equity, close to 20%. Our efficiency ratio dropped below 50%-4 6%. Our net interest margin was slightly over 4%.
It was a great year, 2020. 2021, I'm very optimistic we're going to have another record year. I think we have a lot of tailwinds. This rate environment is helpful for our platform because we're such a strong loan generator. The mortgage division continues to be extremely strong. The SBA portfolio is benefiting from the 90% guarantee, the waiving of the fees. We've seen a significant increase in SBA volume. I think we're getting close to the end of the BSA remediation. I think 2021 should be an excellent year for Unity Bank. Once again, I want to thank everybody for attending. Do we have any questions? I see no questions. Okay. With no questions, I will hand it back to Chairman Dallas.
Thank you, Jim. Now I'd like to proceed with the remainder of today's meeting by requesting the report of the Inspector of Election. Ms. Mills, may I please have the final tally on the votes voting for the four proposals presented at today's meeting?
Mr. Chairman, for the proposal of election of Directors, Dr. Mark S. Brody received 7,802,765 or 95.78%. Raj Patel, 7,098,165 or 97.96%. Donald E. Souders, Jr., 7,179,113 or 88.13%.
Thank you.
Proposal-
Go ahead. I'm sorry.
For proposal number two, to ratify the selection of auditors for year ending December 31st, 2021. Voted for, 9,473,664 or 99.28%. Against, 9,558 or 0.10%. For proposal number three, to approve on an advisory basis the executive compensation plan. Voted for, 6,427,488 or 78.90%. Against, 1,490,610 or 18.30%. The final proposal number four, the frequency of the compensation one to three years. For year one, 3,791,358 or 46.54%. For year three, 4,028,071 or 2.40%.
Thank you.
Let me repeat that. Let me clarify that. For year number three, 4,028,071 or 49.45%. Thank you.
Thank you. Finally, on behalf of the Directors, management, and staff, thank you again for listening to today's meeting, and I look forward to addressing you all next year. There being no further business to come before today's annual meeting, I would entertain a motion to adjourn.
Motion.
Motion by Mr. Tucker. Second by?
Second.
Dr. Brody. The 2021 annual meeting of the shareholders of Unity Bancorp is adjourned. Thank you all.
Thank you for participating in today's conference call. You may now disconnect your lines at this time.