Right. Well, hello and welcome everyone. Very excited to be hosting this webinar today. This is due to the significant demand we've had from investors to see the Research Assistant and see a demo. So I'm very excited to be joined by Steve Tulenko, the President of MA-
Hello
- and, members of his team. The agenda for today will be that, Steve will say a few words, providing an update on our strategy GenAI. We'll then have a product demo of the Research Assistant, and then we will open it up to Q&A. If you could please put any of your questions into the channel, the chat channel, we will make sure that we do our best to cover as many as possible. But with that, Steve, happy to hand over to you.
Good morning, everyone in New York, good afternoon in Europe. If you're in Asia, good evening. Hope everybody's doing well. It's the holiday season, so what better way to spend enjoy our festivities than talk about Research Assistant? Shivani's right, we've had a lot of excitement, and in our meetings with investors over the last several weeks, we are getting many, many questions and inquiries as to how does this thing work, how did you do this, and for that matter, can you show it to us? So we thought we would do exactly that. So you've got an informal session here.
This is really a demonstration, very similar to what we might do for customers, to give you a feel for how we, what we've created here, maybe a little bit of a sense for how we've built the product here, what kind of work we do in terms of product development, and where this all came from. I thought it would be helpful just to frame this with a little bit of perspective. You know, we've talked a lot about innovation as it relates to generative AI, and machine learning, and AI generally.
I thought it'd be good just to sort of talk about the journey over the last year, but maybe start off with an acknowledgement that the Transformer model, which is sort of this famous model that started the GPT and large language model excitement in 2023, really got its start in 2017. The famous paper was released in 2017. We started doing work and experiments with that technology right around that time. We actually created a tool that analyzed the sentiment of text, especially as it related to news stories, to understand whether there were credit implications, and whether they were positive or negative in those news stories.
And, so we've got, you know, we've got a few years of experience in terms of, working through Natural Language Processing and deriving sentiment from text. We actually were experimenting with these Transformer models, a couple of years ago. And that set us up, I think, in good stead for, us to, embrace what I think has been a, a generational opportunity for us, and for us knowledge workers, and for all of you knowledge workers out there, to leverage this new generative AI capability. And really, in, in February or March, we started experimenting in earnest with the, advent of... You all are probably aware that ChatGPT was released, I think it was November 30th of 2022.
The GPT-3 version 3 was really impressive, and, you know, we certainly had a strong conviction that this was gonna be very helpful for us going forward. We already see tremendous improvements with 3.5 and 4.0, and then, all of the other models that many of you have probably experimented with as well. We are excited to engage with many of the other tools that are out there from an experimental purpose and explore the different benefits that are available through the other forms of large language models that are out there. We created a group in March. We called it the Generative Intelligence Group. Might have been late March.
That was the team where we consolidated a lot of our experiments and our, I'll call it innovation with purpose, initiatives, and tried to create a center of excellence that we could then share best practice throughout the company. They created a Copilot capability with us, or for us, so that we could work with this technology internally, not just in Moody's Analytics, but also in the rating agency. It was through our partnership with Microsoft that we were able to roll out that Copilot, literally firm-wide, through the Teams application. So literally every person in Moody's, all 14,000 of us, can now think of themselves as an innovator, leveraging GenAI capabilities.
You may remember, in an earnings call in the summertime, we announced that we were gonna be building a product and releasing a product. We moved through the preview stage, in the third quarter, gathering feedback. We worked with scores of customers, and many, many demonstrations. We engaged with a couple of dozen, actually 25 or 50, in a much more, I think, serious way, gathering feedback, learning where, where they thought we would be able to add value, and identifying those areas where we really needed to make tweaks in order for this, sorry, minimum viable product, to be ready for a corporate or commercial release, which we have done as of the first week of December.
So we've gone through our preview stage, made the changes, iterated basically constantly for six months or so, and are now at a point where we're making this available to our customer base as a part of our commercial launch here in the first week of December. So maybe I should add. There are several of these projects going on. We're finding GenAI capability to be tremendously valuable for everything from our lending solutions to our regulatory capital solutions to our commercial real estate efforts and beyond. We have several projects underway to release additional assistant GenAI capabilities to support our customer base already. Some of those are in preview now, and some of those will be released in the first quarter as we move through 2024.
So I will not take any more time, but instead hand over the call to Cristina Pieretti and Sylvia Baek. Cristina leads our effort related to our website and to our service that you undoubtedly have heard of called CreditView, and Sylvia is a member of the product team. And they will, they'll walk us through a demonstration to give you a sense for how this works, and what we can do with this initial launch of our gen AI capabilities here at Moody's. So, Cristina, take it away.
Thank you, Steve. I just realized my background. I think it's the other way, so I'll fix that during the call. But so it's a pleasure to be with all of you today. As Steve mentioned, we are very excited with the launch of Research Assistant. We've had great, great feedback both from the investor community and from the customers we've showed. I've been very engaged in customer meetings over the last month, and I've been 16 years in Moody's, and I've never seen the reactions I've seen from the customers, and the engagement when we show the product. So I hope that you are as excited as the... or I'm sure that you'll be as excited when Sylvia goes through the product. This is part of the CreditView franchise. It's an add-on.
It works. It's a great way of leveraging the Moody's content because it puts together, you GenAI technology that, as you all know, is extremely powerful, with the proprietary content from Moody's. Basically, research, data, and analytics that come from the rating agency. Before I pass it to Sylvia, I wanna highlight a couple of things. It is real time because of the way we've developed it, so we're not fine-tuning a model, but using RAG technology, and Sylvia will talk a little bit more about that. So that allows us, since we don't have to train a model, we can pull... The moment the rating agency publishes a piece of research, it's immediately available in the Research Assistant.
The second thing I would like to highlight is the trust that we've been able to create among the prospective user community. Because, as you'll see in the demo, we always provide a citation on the answers, and we've made a lot of work making sure that the Research Assistant only answers the questions for which we have Moody's information available. So we spent a lot of time making sure that if it doesn't know the answer, because Moody's doesn't have information on that topic, it will say, "I don't know," as opposed to going to the outer world and looking for an answer, and that has been incredibly valued by our customers.
I think with that, I'll pass it to Sylvia, who is gonna ask some prompts to the Research Assistant, and then we will make more comments about how it works.
Thank you for that intro, Steve and Cristina. Hello, everyone. My name is Sylvia Baek, and I work in the product team within Moody's. I'm very happy to kind of present to you guys today, Moody's Research Assistant, which, as we've all been mentioning, it's an AI-powered assistant that enables users to quickly access information from across all of CreditView to assist you with identifying opportunities, mitigating risk, or even something as simple as helping you find relevant research, all using the power of generative AI. The assistant offers insights at micro, macro, institutional levels, and it can also help you very quickly synthesize vast amounts of information, all for your analysis.
As we've kind of mentioned a little bit before, just for some background, it is built on OpenAI ChatGPT-4 model, of which it is in a controlled environment inside of Microsoft's back-end, which nobody else has access to. And what that really means is that our usage and our clients' usage is very tightly protected in our infrastructure. So the questions and the answers provided through Research Assistant are tightly permissioned and governed within our systems. What we have done on top of ChatGPT is we have implemented our own retrieval augmented generation, you've kind of heard it referenced as RAG for short, of our own logic on top of the large language model. And RAGs are really used for information retrieval.
So what it does, in essence, is it helps guide the Research Assistant look across the content of data that we have given it access to, and helps it determine where should it look to find the information to the user's prompts. So at the end of the day, what all that stuff really means is that Moody's Research Assistant, unlike other assistants or chatbots that you see out in the market today, is the only one that has access to Moody's proprietary data, as Cristina mentioned, and that's ratings, research, and financial data. And through the tool, we kind of enable you to utilize that content in your analysis. So with that preamble, let me move on to the demo, because I'm sure you guys are very interested in seeing the tool in real time. All right.
The Moody's Research Assistant start screen is laid out very simply. It's probably very similar to what you saw for ChatGPT, where we have a list of sample questions that the users can select from to get started, a list of the capabilities, but also what are the limitations, so users know exactly how to ground their questions, and what can they really engage with the assistant about. I'm sure many of you are very curious, so what does the Research Assistant actually do? So I will start off with really just a common workflow for a lot of our users. I will ask it to write me an investment report on Apple. Include the company description, provide key ratings drivers, top three peers with their ratings drivers, sector outlook, and key indicators.
You can be very specific about what kind of sections you might want it to include in your prompt. So now you can see here the Research Assistant working in real time. When you submit your prompt, what it's first trying to do is understand your intent. So what are you really asking me a question about? Is it financial information? Do you want me to look through our research, et cetera? So as it determines your intent, it'll then go and call multiple different APIs to pull in the information for what you're asking about.
So something like this, which many of our customers are already doing when they come to CreditView today, which normally takes them anywhere between, like, 2-4 hours, 'cause you have to find the relevant research, read the research, synthesize it, look up financial data, determine who are the peers, et cetera. So something that normally took you anywhere between 2-4 hours, you can see it being streamed here in real time in a matter of seconds or a few minutes, and that's really the huge benefit, I think, of Research Assistant. I think as Steve was kind of mentioning, the time savings is a really big quality that our users are very happy with.
On average, what they've been telling us is that somewhere between 25%-30% of the time that they normally spend trying to do this is being saved. So if you think about for the end user, what you can do with that time that you're saving, it's pretty much a, a goldmine for them. Another thing I wanna point out here, as Cristina was mentioning, when it comes to gen AI, most people are very concerned that things are just being made up. So how are we kind of solving for this? What we did, and as you can see, when I scroll up and show you the answer, there are many, kind of footnotes here. What those footnotes kind of denote are the citations, and the citations are links to things in CreditView. It's our research.
It's where we're actually going for the source information to then consolidate and provide as part of Research Assistant. So you can actually open up any of these links and see exactly where that source information is, whether it's a landing page, such as this one, or the latest credit opinion. And it's actually a really great thing for giving you that visibility as to where the information is actually coming from. The second quality that I do wanna point out here is that it'll also provide you some additional questions that it can... you can ask it. So it knows that you asked a question about topic A, so it'll also provide you some other questions about the same topic that you can ask it, and this helps you with your, you know, analytical journey, and it keeps the conversation going within Research Assistant.
The last thing here, which I think is probably the best feature in my opinion, is the feedback loop. For every response that the Research Assistant will give you, you can actually give feedback back to us, letting us know whether or not the response was good or bad, and for what reason. I'll give you an example here where I could say it was very good, I could say it was very helpful, or maybe it was accurate, but not very helpful. If it was bad, here are a couple of other options that you could choose from. Maybe the citation wasn't very good, et cetera. But you also have the option of entering additional feedback here, and we regularly will review this, and it helps us to better understand how the Research Assistant is actually performing.
And through the feedback, we will look for ways to improve Research Assistant and also the RAG model that we have developed. Oh, actually, I forgot, 'cause we were developing at such a quick speed. You can also now very easily copy your response. So say this is like a rough draft of the investment report that you are writing, and you wanna add some supplemental information. You can easily copy this, go into something like a Word document, paste it. You can see it's very nicely formatted. It has all the hyperlinks, the citations, et cetera. And you can just very easily use this draft or credit... I'm sorry, investment report, and then add any additional content as you see fit, which I think is very impressive.
But, the tool can do more than just summarize and put together research documents for your perusal. You can also ask it to create content for you, such as a table or a chart. So let me show you an example for something like that. Create a chart of debt to for Ford and GM over the last eight years.
Sylvia, this might even be interesting to equity investors, not just-
Yes
... the classic credit type that might work with CreditView, but you can imagine-
Right
... leveraging Moody's content to address questions, and additional use cases.
Yeah, very much so. So again, here, RA is determining intent and is trying to stream the response back to you in real time. It's gonna look a little weird at first, 'cause it's trying to create a JSON structure for the large language model to then read and interpret, but I promise when it's finished streaming the responses back to you, it will be a chart. So-
Yeah, and in fact, that was a deliberate decision we made because, well, it's crafting the response, there's a wait time, right? So we wanna convey to the user that we are working on the answer, as opposed to having, you know, a wheel that it's to have them waiting with no interaction.
Yes. So you can see here, as I promised, it did render a nice graphic. But if you wanted this in a different format, such as a column chart or a pie chart or even a table, the Research Assistant can do that as well. You just have to be very explicit about that in your prompt. Let's try a different type of question. I'll say, "What are the most recent rating actions in the banking sector? Answer me in French.
You can imagine, we have many CreditView customers that are in large financial centers. English is often very common, but as we expand into other geographies, language and local language can be more important. You know, think Japan, think Brazil, think parts of France, for example.
Yeah.
Where-
Spain, Latin America, you know, a lot of customers with banking customers that are either based on the territories or have subsidiaries, where the first language is Spanish. A couple of things while this is working that I wanted to note. You might have—We can show it again after we finish with the questions, but the Research Assistant is placed side by side from the search box, right? So it's very, very easy to find and very easy to interact. You don't have to go to somewhere else. And you might say: Well, why is that important? And I'm gonna tell you a little bit of an anecdote from a customer meeting I had last week. So it was... Basically, we were talking about CreditView, right?
You know, it was one of the big global banks. And the user, which is a core user of our product, said... We talked about the new enhancement, about Research Assistant, and the first reaction was: "Well, I don't really use that technology. My kids use more of that, but I'm not sure I would use it at work," right? And we said, "Well, wait until you see a demo." So first, you know, she didn't have to go anywhere, that it was right there by the search box. And then it's completely conversational, right? You don't really have to learn about anything to use it.
By the end of the meeting, you know, she was like, "This could be a huge, huge efficiency play for our analyst." And I wanna use that opportunity to recap a little bit the stats that Sylvia mentioned. When you think about the workflow of something that uses CreditView, but as Steve mentioned, it could not only be a CreditView user, there's this expands the scope of the profile of users significantly. You do data collection, right? You wanna answer a question around an investment, a lending decision, or entering some sort of commercial relationship with a third party. You're gonna first, you have to collect the data, and this collects the data, as Sylvia mentioning, in seconds, right?
You don't have to go to 3 different pages or 3 different places or more to collect information on the company, the sector, the peers, but it's right there. We've observed, and when I say we've observed, it's from the trials that we had during the preview phase with customers, from the usage of the tool from MIS analysts, and also from industry publications, that the savings on that collection phase are around 80%. Then you do analysis. We've observed that in the analysis, the saving, because of things like Sylvia said, because it does the peer comparison, because it does charts, because it renders it in a language you can, it can be around 50%. Overall, in the whole decision process, it saves between 25% and 30%.
I hope that gives you an idea of the value that, you know, we see in the tool and that the market is seeing in the tool.
Yeah, I think those are really great points, Cristina, and I just wanted to kind of show you an example here that, you know, our Research Assistant can accommodate more than just the English language. You could ask it to answer you in your native language. You could also ask the question itself in your native language. I would have written this in French, but unfortunately, I have not taken French since high school, so it's very rusty. So maybe I'll just close out with just showcasing, because we've kind of said the Research Assistant is a very trusted source. It only looks at the scope of content that we have given it access to, to be able to answer questions on.
What would happen if I actually asked it a question that we don't maybe mention in our research or have data on? I think it's a topic we are all probably very interested in. It's: What's the current stock price of Moody's?
So maybe while this is happening, let's just remind people, the RAG, the Retrieval-Augmented Generation, the prompting and the engineering we've done here is to try and determine the intent of the question, and then collect the information that might be relevant to help address that question and generate text associated with that question. So here you've got a good case of, something where we, we actually don't have information on stock prices in this data set or in this corpus. We're not able to present that to the, to the LLM. Now, there may be information in the LLM's training that would potentially create a hallucination, but we've engineered this to try and prevent that from happening.
So in addition to this being up-to-date with real-time content from Moody's, it's also designed to prevent or at least mitigate the risk associated with a hallucination. So you've got a response here, as Sylvia has indicated, to give you a sense for we really don't know that one. Maybe you wanna take a look at these links, but this is not my strong suit.
Yes, exactly.
Yeah, go ahead.
Well, I'm sorry. I just meant, like, it'll very explicitly state to you that it cannot answer this question, because it doesn't have, it mentioned anywhere in our research.
Yeah.
So I just wanted to kind of maybe make a few, like, closing remarks before I hand it back. The power of generative AI really stretches beyond my imagination, I think your imagination as well. We're really just on that cusp of discovering how much this can really retool the way that people are using our information today, and how we can become more sticky in our users' journeys. So I just want to thank you for your time and letting me demo the Research Assistant, and I hope this makes you as excited as I am about the possibilities that this could unlock. And with that, I'll pass it back to Stephen and Cristina.
Thanks, Sylvia. That's great.
Thank you.
Uh-
I just wanted to... Sorry, Steve, I wanted to make one more comment. Sylvia, can you share your screen again?
Yes
... for a second? So I wanted to go over the citations again, because I think the citations, we talked about the trust that they bring to the user experience, but I think there's another dynamic going on, and it's the ability to highlight content that you might not be aware of. Or when you look at if you scroll down from the citations and you look at the additional questions, that is gonna suggest... Do we have them here?
Yes.
Yes, additional questions. It allows the user to get, you know, to explore topics that might be relevant, and it weren't in their radar. And the other thing, it also has with it a cross-sell dynamic, right? Because we might be highlighting content that are not part of the current user subscription. So you see a lot of things at play with the product.
So, Cristina, what is next? Just maybe, you could probably pull this off the screen, Sylvia, but maybe you can talk, Cristina, about what's next for the product in the, in the coming months.
Yes.
So-
So what we think and when we think about what's next, you know, we have a million ideas, right? And we have a million ideas, first, because, you know, the technology's extremely powerful, as Sylvia mentioned, and also because we're getting a lot of feedback, both internally from, you know, the rating analysts, and also from customers that have been using it. We like to think about what's next in two dimensions, right? One, it's a content dimension. So what other content could we be surfacing through this? And this can be some of the rating sectors that are not included. So for example, it has research for structured finance, but it doesn't have all the ratings for structured finance. So that's an area that we're definitely going to explore.
We could also think about other content sets across Moody's Analytics, right? You know, our economics data, our default data, our commercial real estate data, to give you some examples. And then the other dimension we like to think about is in terms of features. So what other things could it do? You know, what other functionality could it-could we refine the way it writes an investment memo? Could we allow the users to customize more that investment memo? The ability to export, for example. Sylvia, this was just launched, the ability to copy, but could we allow the ability to export or save some sort of query?
Mm.
That's basically the two dimensions we're exploring when we talk about what's next. But what I can tell you, it's we're doing releases on a daily basis. As a user of the tool, you can see changes almost on a daily basis, which is extremely exciting, of course.
Two other comments I'll make there, Cristina. One is, you hit the point about other content. If you think about the vast data estate at Moody's, right, we can obviously talk a lot about credit. We can talk a lot about climate and physical risk associated with weather and climate, wildfire, et cetera. We can talk a lot about financial crime. We have tools that enable us to process news literally all over the world, and this is a number that we've been-
Yeah
... talking about a lot lately. You know, we process almost 1 million, some days, 1 million stories a day, understanding the sentiment that's in those news stories, and then giving us a chance or an understanding of whether that's, that sentiment is positive or negative. We can start to weave some of these other capabilities available in the data estate here at Moody's, sometimes proprietary, sometimes they're just historical databases that we have uniquely compiled, and make them available here. The other thing I think that's very exciting is, as we roll out capabilities like this in our product array, you can imagine some great cross-selling opportunities with things like our CreditLens product, where we have literally hundreds of thousands of users across thousands of banks, where they are interested in doing credit work.
And with the Research Assistant, and also with some of the work we've to expand our coverage across unrated names, you can imagine how that will really help the lenders of this world make themselves more productive and be more efficient, and maybe even create content that is more insightful. So we're really excited about the cross-selling opportunities as well. So Shivani-
Yeah
... should we see how, how we're doing with questions?
Yes. Yes, we've got some questions come through. So I think the first one is just for folks trying to understand the... This is an add-on to the CreditView product, so I think people are trying to understand the scale of CreditView, and just how much revenue does it generate?
Cristina, do you wanna take that?
Yeah. Yeah, yeah. So yes, this is being sold as an add-on to our CreditView product. I think it's safe to say the CreditView product generates north of $500 million.
Yeah.
Thank you. Then we have a question on the impact on your cost structure. What is the cost of compute?
Yeah. Cristina, another one for you guys.
Yeah. Okay. So, I would preface the answer with, it's early stages, right? And it's early stages. We just rolled out the product. I think Steve mentioned this in his remarks, but this is the fastest product that we've ever developed here. And so we have to be careful. I think we're very early on the journey. That said, I would say for the prompts you saw, you know, they are around $0.10 in terms of cost, something like that.
Yeah. So the tokens per second metric is what I think people in this space will be talking about. What you're seeing here is the cost of tokens per second in a prompt like you saw in this demonstration might be on the order of pennies to a dime, something like that. As things get a little bit more complicated, you can imagine creating that chart was a little bit more expensive than creating a summary of the rating changes. As things get a little bit more complicated and we introduce more content, and the engineering we do to make sense of the intention behind the prompt, right?
That may become a little bit more expensive, because we may be grabbing more data, more content, and more capabilities that we bring to the table, and then trying to synthesize those in a way that is more effective. It's one of the great benefits. It will probably be a little bit more costly. I don't think the costs are material to us at this point. If you were to think of this as a cost of goods sold item, we are, I would say it is not dramatically affecting our decision-making at the moment.
That leads on, Steve, there have been quite a few requests for more information on the economics of the Research Assistant, and how we're pricing this product. So I don't know if I hand it over to you or Cristina for this one.
Either one. Well, I mean, maybe the first thing to do is just talk about the return, the ROI thought here. You know, we talk about the surveys we've gotten from the customers. I think Cristina said, you know, between 25% and 30% of their time saved, or maybe more importantly, they can be more productive to the tune of 30% more productive. That is, I think, an estimate that we have heard echoed in our rating agency test group, you know, the group of people we work with internally to validate the results we're generating and give us a sense for, you know, the quality of the work that the Research Assistant is producing. And we're also, you know, hearing similar estimates from third parties.
You know, there's consulting firms, and published studies, and universities that have published studies that indicate that 30% number feels about right right now. So you can do the math, right? Depending on how many people you have doing investment work, or credit work, or research in your operation. If you have a hundred, and you're talking about, I don't know, $250,000 or $300,000 a year all in, including their office, and including their salary, and their bonus, and so on, you can imagine 30% savings or 30% more productive adds up pretty quickly. So on an organization that might be a hundred people, we might save or, or contribute $5 million or $6 million, maybe even more in terms of value.
The commensurate number for us, in terms of pricing, is probably about 20% of that number. An organization of 100 in size, you know, we might seek an incremental fee that might be about 20% of the return that they'd be seeing in terms of making the investment in us here. It amounts to, at the end of the day, a 30% premium for the module that you add on to CreditView.
Thank you. So we've got some questions about how do you handle security and compliance concerns?
Mm. Cristina, you wanna talk about the environment, how we set it up, or shall I?
You can do it, or I can do it, or we can pass it to Sylvia if you want.
Sylvia, how... can you just talk about the environment that's set up here? What, what protections do we have in place, and why can customers have confidence that, you know, this is an environment befitting of Moody's?
Oh, so as I mentioned before, It's built on ChatGPT-4, and it's built in Microsoft's backend. And so you could think about as confidential as, like, our company emails are, that's how safe this data is. The people that have access to the questions and answers that Research Assistant is providing, and also the feedback logs, are very tightly maintained within our team, and so we can assure you that the restrictions that we're placing around the information and visibility into it is quite strong.
I would add a couple of things from looking at the chat as well. You know, the data sits with us. So the data is Moody's data, and it's in our environments. And what we send to the... What we use the LLM for is we try to determine intent, and we then craft the response using the LLM. But all the data and all the processes are inside the current moodys.com/CreditView infrastructure, which has very high levels of security.
So we have all the benefits of an Azure environment that is basically our private instance of OpenAI model. We've constructed and architected this, by the way, so that we could use other LLMs. We find that OpenAI LLM today is the one that's most effective for us, but we are experimenting with several others and are ready to adjust if it makes sense. And I think over the long term, there will be developments with other LLMs, where they have special skills and special capabilities that we'll find interesting, and our customers will find interesting. So we're architecting this to be able to leverage any of these.
When you're in Azure, you've got all the benefits of working with Microsoft, and you've got all of the Moody's controls and processes that we normally would have for any of our products. So this is protected and secure, just like you would expect, from a company like Moody's.
Yeah. A couple of comments that I would make there, Steve. One is, we spent... Basically, during the last two months, we spent a lot of time in making sure that this was as robust and as protected as possible. So we could have kept adding more content, we could have kept adding more features, but we made the deliberate decision to make sure that this was a trusted, robust, secure product. So, yeah, so wanted to emphasize that. The other comment I wanted to...
Steve mentioned how we're kind of, I would call it, or you could think about it as LLM-agnostic, and this is something that I'm extremely proud that, in general, when you think about the moodys.com technology platform, the way we've built it, and we've been in a journey over the last two years rebuilding the whole platform, is to make sure that it's kind of tech-agnostic. So whenever new technology comes to the market, be it an LLM, be it anything else, we can always switch to it without a major revamp.
... Steve, we've had a question of will Research Assistant be rolled out to other kind of non-CreditView franchises?
Yeah. Yeah, something like it I think we'll make available, especially to people that are doing research on companies. You can imagine the lending community that I mentioned before is a good example of that. And we may... You know, the architecture here is to leverage these capabilities as APIs. They may be packaged in the form of Research Assistant, or they may be packaged in a way that might be more commensurate with the way a lender operates their workflow. But the components would be reused, and made available in that context. You can imagine doing this also with insurance underwriting, for example.
Think if you're underwriting a policy related to a specific company or a property owned by that company, you might wanna do some homework on that name or their peer group, maybe get an idea in terms of relative pricing. These same APIs, we can leverage with some of our workflow tools in the insurance sector. There's many examples of those kinds of applications that I think will follow here. The reason we're starting here is because we know the research business very well. Cristina, Sylvia, me, you know, I mean, literally I've been selling research since 1990. Cristina's been here for probably longer than a long time, and been very valuable all along. Fantastic contributor, especially when it comes to research in the last few years, right?
And we're all very familiar with the customer sentiment dynamics, maybe the shape of the demand curve here. And we wanted to start with something that we all know, and kind of can intuit as well as analyze. So this is why we started with this franchise.
We've had a question, and I know we've been in the market since the 1st of December, but we're being asked: "Has the rollout impacted the contract renewal period within 4Q?
Yeah.
Is it just too early to say?
Yeah. It is, it's probably too early to say what impact we have, right?
Yeah.
What we can say is we did it very intentionally with a December date because we have a lot of customers that renew in December. We have a lot of... December and January are both big months for us. We wanted to force or create a vector of force in the customer conversations to introduce it to many and as quickly as we could. So all of our sales reps are trained up and aware of and able to talk about the benefits here. All of them can demo. And you know, we've practiced talking to customers to make sure they can see how valuable this can be. So we're looking forward to evaluating, I'll call it the impact on the business over the coming weeks.
But it's a little early to expect that buying behavior would change on the spot.
Yeah.
And-
I would add to that maybe that you can definitely see the impact in the quality of the conversations, right? You know, every time we engage with a customer, I would say we run out of time, literally.
Yeah.
Because there's so much interest in asking questions, engaging with the product, and understanding the value. So definitely see the potential, but as Steve mentioned, too early to tell.
And we've had a kind of tangential question, which is about: "How does this initiative," and I'm reading it out, so, "articulate with the AI partnership you announced with Google Cloud?
Uh.
So...
Yeah. Yeah, sure. So, you know, we have a tremendous relationship with Microsoft. We have, I think, made some nice contributions for them. They've made some fantastic contributions for us. They've given us some, you know, an environment to work with out of the box that we didn't have to really worry about. That was fantastic. Their platform, the Azure platform is great. The engineering help that they've offered along the way has also been very, very useful, and I would say the people, the worker that we work with there have been great. With respect to Google, we've had some really good experience also.
The project we're doing there that I think is most remarkable is the idea of leveraging some of Moody's expertise and understanding financial statements, and applying that in the context of a large language model to see if we, together, can. I'll use these words a little loosely, but fine-tune and potentially train smaller LLMs in order to read financial statements and glean insights from them. So we're doing some experiments together with an eye toward understanding financial statements. You can imagine, just like many of you on this call would do, what do these ratios imply, and what does that footnote mean in light of those ratios, and how should I interpret that in terms of the implication? So we're trying...
You can imagine, there's lots of people at Moody's that do financial statement analysis around here, and we're trying to leverage that expertise to help deploy LLMs in a context where we can actually read financial statements and gain insight from them very quickly.
Steve, this goes back to comments that you gave at the start of the call. And just to clarify, we're being asked, you know: "What other Moody's Analytics solutions present would most naturally benefit from GenAI augmentation?" And then a linked question that we've had as well is, you know: "Do we expect to launch one or more additional products in 2024?
Yeah.
So they're tied together, I think.
Yeah, I kind of hit that a little bit at the beginning. I would say the current capabilities most effectively leverage text and natural language, and benefit from understanding the intent behind natural language, and then pulling the relevant content to generate or augment the generation of text to help you do analysis. You know, the LLM capability is fantastic in this way. You know, the idea of doing math with an LLM is something that is at least at early stages, and it's not quite the same, and not quite as powerful as what we're seeing with language at the moment.
But reading data tables, and leveraging models that we have around here, and we have hundreds of models that are used, industrial strength models, to do calculations on risk and to evaluate opportunities, these are things that we're very excited to experiment with and develop, in order to make these tools, like Research Assistant, even more capable and more powerful. We'll be rolling out things especially that benefit from the use of large language models with respect to text sooner, and then data and engaging other calculation engines a little later in the product life cycle. At the beginning of this year, we'll be talking certainly with our banking customers.
We've got a couple of different projects there, where we think we can, we can add a lot of value in lending, and in managing and understanding regulatory capital adequacy, and, and capital calculations for regulators, as well as some work in commercial real estate. There's some other areas where we've got investments made. What Cristina and Sylvia are doing is creating a franchise for us in many respects, earning us a right to sit at the table to support people in a way that is, I think, profoundly impactful. And adding more content to this capability is also something that we expect we'll be doing through the course of 2024.
And Steve, we're coming up to time, so just wanted to open it up and ask any final comments or remarks you'd like to make just to close this out?
Yeah. Okay. So that's for me, right? So I mean, I'll, I'll say... I, I've said this maybe before on a call, I really feel, we, we are very, very excited here, as knowledge workers, and as, people who help knowledge workers do their jobs even better. We believe we've got a fantastic opportunity to leverage these new technologies, and maybe add value in a way we've never been able to do before. The opportunity is tremendous. We've got a fantastic data estate, some of which we are uniquely able to provide, much of which we are one of the best providers of, and, we're really looking forward to leveraging that to add value for people in this generational opportunity.
This feels like the beginning of a very, very interesting ride, and we're very excited to see what comes of this, and how we can help people in the future.
Well, thank you, Steve, Sylvia, and Cristina. Really appreciate you making the time for our investor and analyst community. For those of you listening in, there will be a recording available, and it will be posted on the IR site shortly. So thank you everyone for your time and your participation, and all the best for the holidays. Thank you.
Yeah. Happy holidays, everybody. Thanks very much.
Thanks, all.
See you later. Thanks, Sylvia. Thanks, Cristina. Thanks, Shivani.