Hello. Good morning, good afternoon, good evening, everybody. Welcome to Appen's Investor Technology Day. Thank you very much for joining us. Our agenda today covers a number of topics. I'll provide the briefest of introductions. Then I'll hand it over to my colleagues to give you, first of all, an update on the AI market. Secondly, an update on our technology, including some demonstrations. We'll have a period for Q&A at the end of it. You'll be able to answer or, sorry, enter your questions via the app. Then we'll close. We'll be all done prior to 2:00 P.M. Sydney time today. I'm joined today by Wilson Pang, our CTO, who's coming live from California. Ryan Kolln, our head of corporate development.
Both Wilson and Ryan are far more interesting than me to talk to, and they'll be doing the bulk of the talk today. Wilson is an engineer with an extensive background in search and artificial intelligence. He worked for IBM and for eBay for many years. At eBay, specifically, he worked on search, and that gives him a lot of expertise in artificial intelligence. He was chief data officer for Ctrip, a travel company, where he built many models to help that business use data more effectively and grow and thrive. Ryan Kolln, also an engineer, has worked for telcos here in Australia and in the U.S. and has had a stint with the Boston Consulting Group, advising technology companies on growth strategy. Today, our theme is all about our transformation into being an AI power provider of AI data and solutions.
Our talk today will tell you why this is important and how we are going about it. First of all, though, just to recap on some of the things we covered yesterday, how we got to this point and the evolution of our business. When I joined Appen six years ago, we were a leading provider of language data. We've evolved over that time to be a provider not only of language data, but also of training data for all AI use cases, including all AI data types, so speech and natural language, text, relevance, image, video, three-dimensional data, including lidar, for example. We've moved quite a lot from our initial position as a provider of language data. We're also evolving our service model. Sorry, our delivery model. Essentially service led now to being more product led, and you'll see a lot about our products today.
From a revenue perspective, we are moving over time to do more committed revenue rather than just all project revenue, and that obviously goes to revenue visibility and earnings quality. From a customer perspective, we are still very concentrated to our largest customers, as many of you know. We're working to win many new customers to diversify our customer base over time. Yesterday, we announced a change to our organizational structure from one that is functional to one that is more aligned to our customer cohorts. We now have four P&L customer-facing business units. Our global business unit, which serves our five largest tech companies, the U.S. technology giants, our enterprise business unit, and then our business units in China and the government sector. Finally, yesterday, we announced some changes to our reporting. We were reporting by data modality, relevance in speech and image in Australian dollars.
Now we are reporting more by our customer segments and other strategic areas of interest, such as our new markets, which includes the enterprise China and government segment, but also the revenue that flows through our products from our major customers as well. To look back at the passage of time, when I joined the business again, the majority of our revenue came from the global customers, and we provided services to them essentially on their platforms. We acquired the Butler Hill Group in 2011 and then Leapforce in 2017. With Leapforce, we also gained Appen Connect, our crowd management program that helps us manage the crowd resources at scale for our customers. In 2019, we made an important acquisition with Figure Eight, which gave us our own annotation platform, and this provides a number of opportunities for us.
First, to sell to customers that don't have their own annotation and data preparation technologies. Second, it gives us the opportunity to do more types of work, so the platform covers all data modalities. Finally, we use it ourselves to improve the efficiency of our own operations. Most recently, we invested in the expansion of our business beyond our global customers with the addition of the business units I just mentioned, enterprise, government, and China, all of which is fueled by our platform and all of which requires the technology that we've invested, that we've acquired and invested in over the last few years. Increasingly, we'll be a product-led organization. Our products give us the opportunity for scale, for quality, for productivity, and underpin the growth of the business going forward. We are also, as discussed yesterday, increasingly customer-centric with our four customer-facing business units.
That's not the topic for today. It's more the topic around product. On that, I'll hand it over to Ryan to take it from here and provide the AI market update. Thank you, Ryan.
Thanks, Mark. I'm going to talk about the AI market today, but I'm going to start by talking about the AI application lifecycle. It is a useful grounding for us to think about, and particularly the role that we play with our customers to help them build AI-enabled applications. On the left-hand side, we see a very typical view of how a customer might build an AI application. It all starts with the business needs, the hypothesis around what the application is going to deliver. Typically, the first step is to collect and bring together the available data to build the model based off. That could be in-house data or it could be data that is collected bespoke for the application. The second part is the preparation of the data.
It's one thing to have the data, it needs to be in the right format with the right labels that are going to be able for the AI models to be trained on the data. That for our side, typically involves a lot of data labeling, and is a big role of what we play. Once the engineers have data that is ready for the model build, the next step is to build the model. This step typically involves the selection of modeling techniques. There is a wide variety of different approaches that can be used to train models. Once that's selected, they apply the training data and build the model. Next is testing. Once the model's been built, does the outcome of the model meet the requirements and support the business need?
In the case that it does, then the model will be put through to deployment. That's actually when it's put into the application, deployed in the real world. There's an interesting side loop here for some applications that may be of high criticality, and the confidence of the model is not where it needs to be. There'll be a human in the loop. That is effectively where low confidence predictions are routed to humans who will make the decision, and then that closes the loop. Monitoring is very important in AI models. There's an adage that it's not a question of if a model will degrade over time, it's when. We'll talk about this a bit more later. Once models get to that point that it's degraded below acceptable performance, it goes back to acquiring more data. This is a bit stylistic.
In reality, it's a lot messier, to be honest. There are many iteration loops that can occur. The most common ones are around the testing phase. An engineer will build a model, they'll test the application. It may or may not work. If it doesn't work or if it doesn't get the performance that they're looking for, they'll either acquire more data, prepare more and label more data, and try different model building techniques. We'll talk through some of the differences across those approaches later today. To simplify the AI lifecycle and the AI model development approach, an AI model consists of two main parts: the model instructions and training data, where the model instructions are the architecture for the model to learn.
It's not saying, "Here's the output." You're saying, "Here's a guideline for how the model should learn once training data is applied." This can be as little as 10 lines of code in some instances. The next important part for model development is the training data. These are the examples that the model learns from. It's typically the more and the higher quality training data, the better. I think it's helpful to contrast AI development to normal software or traditional software development. In a traditional software development sense, you'll have an idea of the outcome that you're looking to get to, and you write code and it's deterministic. By deterministic, it means every time that application is run with the same set of inputs, it's going to deliver the same outputs.
Test the code that you deploy, monitor, and if there's any changes, you rewrite or you edit the code. In AI model development, it's different. The labeling of the data is the really important part. That training data composition. The provision of the instructions, i.e., the architecture that we spoke about, is used to write the code, and then the model is tested. You can see the difference. In traditional software development, writing the code is the most important step. In AI model development, it's the gathering and the labeling of high-quality training data. It's interesting to think about what is an AI model. I've put up an example here, because AI models, they're really this indecipherable set of nodes, weights, and biases, that when you look at it from an outside-in standpoint, it makes absolutely no sense.
That's why when you hear about AI explainability and model debugging being really difficult, it's because the actual code that has been written as part of the modeling process is this highly complex system that's very difficult to debug. What's the important part, and the bit that is able to be debugged and improve quality is the training data. We've spoken about training data in past presentations, and obviously, it's core to our business. Breaking it down to really simple components, training data consists of three things. Firstly, it's the file. You can think about that as the example. It could be an image or a text file or a snippet of audio. There are attributes to the file. It's really important as part of the training process to assign meaning to the file.
Let's say the file was used for autonomous vehicles. The box surrounding a car and saying, "Within these pixels, there is a car." You can think about that as the attribute of the file. The next is the attributes of the labels. This is the metadata, what time it was labeled, who it was labeled by, under what conditions. We'll go through and step through a few examples of what training data actually looks like. On the left-hand side here, this is an example of our lidar annotation tool. Lidar is used in autonomous vehicles, and you can think about it similar to radar, where a pulse is sent out and received. What it does, it allows the sensor to measure distance and different rough shapes in a 3D point cloud environment.
You can see on the top left of that image what a standard camera sees, and in the dark blue points, that is the lidar frame. In this instance, the task for our annotator has been, can you draw a 3D cuboid, so a cube, around this picked car in the frame? On the right-hand side is the label, and this is what's called a JSON file, which is the actual meaning, and I've highlighted a few sections here. That first section I highlighted in light red is the center of the cuboid. It's saying, in this dimension in space, there's a cuboid. The next part is the height, width, and depth of the cuboid in meters. In this space, there's a cuboid, and it's roughly 1.9 m high by 1.9 m wide and four and a bit meters long.
That's the cuboid. Then it's saying within that space, there's a car. All right. When we think about how this trains the system in the model, when there's a representation of these types of cuboids in a real-world environment, because it's been trained to look for cuboids, it will be able to say, okay, I know that there's a car in this space. I know dimensions of the car, and I know how far it is away from me. Obviously, super important for autonomous driving. The third highlighted section in this space is interesting, and this is saying that in a 2D image, so in the image in the top left-hand corner, there is also a car. It's actually in this lidar frame, it's blending together two different types of sensors into one set of training data, and this is called sensor fusion.
A simple example, but you get the idea that this can be exceptionally complex, particularly when you're looking at hundreds of different objects in a frame. It could be vehicles, pedestrians, bicycles, other stationary objects. It gets quite difficult to label, and the JSON file or the annotation is very difficult in itself. Another example of training data here is speech, so spoken audio. On the left-hand side is an example of our speech annotation tool. Here you've got just one speaker, two speakers, sorry. A quite simple example. Speaker B saying a few things, and then you've got speaker A. On the right-hand side, you see the JSON file and a little bit simpler than the lidar frame. Effectively, at this start time in the file and this end time, here are the words which are being spoken. All right.
What's a little bit complex here and really important for the training of voice recognition system are the noise associated with it. You'll see that there's an insertion of certain types of noise. Here we've kept it quite simple just to say noise, but it can include quite specific things like a cough or a sneeze or. It's very important for training speech recognition systems. Training data quality is really important, and I think it's intuitive as we think through the development of an AI model. It's providing a lot of examples. If those examples are wrong or not representative of the real-world state, the model's not going to perform that well. Low-quality training data leads to low-performing models. The thing is, though, that poor-quality data is not always obvious, and there's many different types of quality issues with training data.
I'm going to step through these a little bit. We see three main buckets of problems. The first is where there's been an error in the labeling process. This is on a specific label, there's something wrong with it. The next is more about the composition of the training data, and unbalanced training data is a really big issue, and that's where you may have over-represented in some areas and underrepresented in others. Again, that leads to a suboptimal performance in the models. The third is bias in the labeling process. It may be that there's no errors in the labeling and it's balanced, but the individuals who have performed the labeling may have certain bias that leads, again, to suboptimal outcomes. We'll go through some examples of all of these. A really simple example here.
Let's say the task for the contributor is to draw a box around the cows, and that was the instructions provided. What we might be looking for, or in this instance, is a bounding box, so a tight bounding box means that there's not a lot of space between the image of the cow and the box around these three cows. The left is pretty clear. The cow in the middle is pretty clear. The one on the right is a little bit trickier because it's occluded. You can't see all of the cow, the intent is that we just see the box is drawn around just the part of the cow that we can see. First type of labeling error that may occur is just for whatever reason, the contributor missed the cow on the right, and obviously that's not great and pretty clear that that's an error.
The next could be the accuracy around the bounding box fit. Here the contributor's been a little bit generous in the space that's provided outside around the cow. While it seems somewhat trivial, it's actually really important because the way that models are trained, it's on pixel by pixel, so it needs to be as accurate as possible to get that best level of prediction. The next problem might be a misinterpretation of the instructions or bad instructions. We said that the task was to draw a box around the cows. This is not necessarily an incorrect interpretation by the crowd worker. If you've got a few thousand labels where there's a box around each cow and then a few thousand where it's a box around all the cows, you can quickly see how this could lead to problems.
We spoke about the cow on the right-hand side being occluded. Another potential error, it would be that the crowd worker could assume the length of the cow and draw the box around what it thinks the cow would look like, the actual size of the cow. Again, this isn't the desired outcome that we'd be looking for. A bunch of errors in the labeling process, errors in misinterpretation. Another big issue in training data is what's called class imbalance. Class imbalance, you can think of it of, we don't have a representative set of examples in the training data. We'll just talk through this a little bit more.
On the top, let's assume that we were building a model that was going to recognize cows, and it would come back with the breed of cow. You put in a photo, and it returns the breed of the cow. If the training data was limited to the top row, it would be probably quite good at recognizing dairy cows on green grass with a blue background. As soon as you put in a different set of cows, on the bottom left, you've got some white cows on pretty brown grass, some dairy cows on snow. I think the third one's a yak, the right is a Texas Longhorn. I think that's a bull rather than a cow. You quickly see that by limiting the training data size, how that would have a significant impact on performance for this particular type of model.
Class imbalance, this is a very simple and straightforward example, but this is a really big issue for the performance of high-quality models. Another type of Class imbalance is around data recency. We mentioned before that all models degrade over time, that's because the real-world environment continues to evolve. Training data, unless you refresh it's static, and it represents a point in time. I've got an example here around a search result, or the search result returns for Corona. Obviously, in May 2021, Corona, there's a lot of news articles and statistics around cases. If you did that same search result in April 2019, the top return is Liquorland. You start to get an idea around how important recency is here.
This is an extreme version, but it is a problem for a lot of training data, particularly where the real-world environment is continuing to evolve and continuing to change. The next example talks about bias. Another stylistic example here, let's assume that one was trying to build a model to identify breakfast foods. You asked a set of workers who are based in the U.S., "Can you look at each of these photos and tag which one is a breakfast food versus not?" On the left, you've got black pudding, which is from what I hear, quite acceptable in the U.K. for breakfast. In the middle is hagelslag, which is sprinkles on toast from the Netherlands. On the right-hand side, you've got our Vegemite. Someone in the U.S., probably unlikely that they would get these right.
It's a form of bias, and it's not intentional bias. It's just bias because the crowd worker is not representative of all of humanity to represent all of the different types of breakfast food that we see. A lot of datasets require specific knowledge and/or context for accurate labeling. We spoke about this equation earlier that an AI model is model instructions plus training data. What's really important is that a good AI model requires the model instructions plus high-quality training data. Our role in this AI lifecycle is to deliver that high-quality training data. We'll talk a lot more, particularly in Wilson's section, around how we're leveraging technology to do that. The training data market is continuing to evolve very quickly in some circumstances.
What we want to do now is talk through some of the trends that we see more specific to training data, and then we'll move on to some observations on the model development market overall. There are five major things that we want to talk through today. The first is that high-quality data remains a major roadblock for the development of AI. The second is that AI use case is becoming narrower, and by narrower we mean more specific, and this has implications on training data and then how training data is being used. We'll talk about the shift from model-centric to data-centric AI, which is a focus more on how to improve the quality of data, less on different modeling techniques. Fourth, as AI models become more mainstream and more in the production systems for a lot of companies, there's an emerging need for training data operations.
Finally, and this is something we've spoke about before, using AI in the training data preparation space is increasing, and we'll talk a bit about this now and also in Wilson's section. In terms of the first trend, data remains a major obstacle for AI. There was a survey completed recently by O'Reilly, and talking to people who actually built AI models and AI in production systems. If you look at the first largest bottleneck, it's skilled people and hiring the right people. The second is the lack of data or data quality issues. Third, it's identifying the right use case. Fourth is culture. You can see about those four major segments, which represent roughly about 60% of the total bottlenecks. The one that's actually related to the development of the model is training data quality.
That remains a huge issue and something. This has been fairly consistent over the last few years. It's a big issue. A lot of AI practitioners expend a lot of time preparing data. On the right-hand side is a quote from Airbnb who, one of the, I would say, more advanced players in having AI production models at scale. They did some research and discovered that nearly 70% of the time that a data scientist spends on developing the model is not the modeling piece, it's actually collecting data and feature engineering. Extracting the features, we call labeling the data. There's a huge amount of time being spent on data collection and data preparation for AI. The next trend we see is that AI is becoming narrower. Right?
We'll talk quickly through a few examples that we have supported at Appen, as illustrative areas, but we see this across the board. The first example, we've supported a bizspeak model. The challenge is that someone wanted to build a model to suggest improvements to common bizspeak. You can think about when you've written something in an email, there's a suggestion, "Hey, this looks a bit bizspeak-like. Here's an alternative." You think about the challenge with this. Bizspeak is highly nuanced. There's regional differences. There's context. It's a very difficult linguistic task to solve. Our task was go out, collect a lot of bizspeak and understand the intent and provide suggested alternatives, and having to do this is very large scale and with a lot of context involved in it. The next example of narrower AI is related to personal training.
There is a big push now to use computer vision as a way to suggest training regimes and to monitor the performance of the person that's doing the actual exercises. The challenge is that a person's movement changes with age. Particularly as people get older, they might be limited in their movements, et cetera. One of the tasks that we were asked to do is capture and annotate videos of seniors doing somersaults. This is an actual task that we supported. You start to get the idea of how specific some of the data collection work is that we do, and it goes back to that class imbalance issue that we spoke about, that it needs to be representative of even the extreme version of seniors doing somersaults. The last version is about long-tail languages.
COVID created a unique challenge where there was a lot of information that needed to be shared digitally, in almost real-time around the globe. This included some specific languages where there may be only not a lot of people who natively speak that language. The translation text didn't support all of the languages. We worked in a consortium with a lot of other large tech players to go and collect and annotate some very long-tail languages to make sure that the information about COVID was being disseminated, not just for the common language, but across the board. A really important project for us, that last one. We spoke again about this good AI model equals model instructions and high-quality training data.
There is this question around, okay, if I'm an engineer and I'm looking to improve performance in my models, should I focus on spending time around the model instructions or training data? This is a bit of a long-standing question in the AI community. There's a quite respected AI practitioner, Andrew Ng, who has a company, Landing AI, and he tried to answer this question. He had built a model to detect defects in steel sheeting. A computer vision model, it takes photos of steel and automatically identify is that a defect or is that a piece of dirt, et cetera. They built a model, and they got to a baseline performance of 76.2%. He split his team into two tasks. One was, he got a group of people say, "Hey, go out and improve the code.
Get the latest research possible from the largest tech players and do whatever you can to apply this new architecture and this new model code to the existing data and see how you can improve it. He got another set of his team to go and improve the data. Let's not change the code, let's just go out and collect more data, improve the labels, improve the quality of the data. You can see the difference here. This is one example. Improving the code had almost, well, no impact on the performance of the model, whereas improving the data had a really significant uplift, and the average human performance for these types of tasks was 90%. They actually got it to above human performance in identifying steel defects.
One example, but more of an illustrative view of how there's this performance improvement benefit from looking at the training data composition. Another example here is the performance of a competition called ImageNet. ImageNet is a bit of the gold standard competition in computer vision, where you've got a few million labeled examples, and the task is to create a fairly general computer vision model where you load in an image, and it will tell you what's contained within the image. Over the past seven or eight years, starting with that core dataset, the performance of the model is able to get to 86.5%. These are serious heavy hitters who are investing time in this competition. By providing extra training data, you can see, particularly in the outer years, that there's been a significant uplift in the performance of ImageNet.
Another example of the benefit of more training data, and how that yields to better accuracy and model performance. This really comes to this shift from model-centric to data-centric AI. In a model-centric world, AI engineers will use the available data and try to develop models that compensate for any noise or inaccuracies in the model. You can think about it as you hold the data fixed, and you try to improve the model. Data-centric AI flips that on its head. It's all about improving the volume and/or the quality of the data, the training data that's used to train the model. You try some different models, but the focus is on improving the data. It's holding the model fixed and iteratively improving the data.
What we see as a shift that's occurring in the model development world is that this shift from model-centric, where the constraint has been, "Here's the data I have," to a data-centric view, where there's a lot more focus being placed on: How do I improve the data? How do I expand my datasets and enrich the data? This is the shift that we spoke about from model-centric to data-centric AI. The fourth trend we see is the need for training data management. We've spoken about the AI life cycle, and there's a really important part here, particularly that we focus on, which is the data collection, labeling, and preparation piece. In the pink down the right-hand side, the kind of the tasks that we see.
Doing these tasks are really important, but there's an entire set of capabilities that are emerging around how to support the development and the management of training data. Things like version control get really important. If you build an AI model on one set of training data and the training data's changed, you'll never get that same performance again. Managing the version of the training data is really important for, one, experimentation, so you can figure out what composition of training data worked better than others, but also traceability. If there is issues in real-world production, it can be linked back to and quickly identify what the core training dataset is. Training data security is another important issue that's emerging. We spoke before about the difference in traditional software development being code and AI being data-centric.
If you are a hacker, you can go in and change the code in traditional software. In the new world of AI, data is the most important piece. Placing security around the training data becomes really super important, and there's going to be a lot of focus in that. There are a whole bunch of other issues, but they're just two examples around how there's an ecosystem being built around the management and the controls of training data. Finally, one that's really important for us and where we're placing a lot of focus is applying automation to the labeling process. There's three main buckets of automation that make sense for data labeling. The first is pre-labeling. This is where AI performs an initial pass on the annotation, and the workers, it's doing more of a check and a correction of the pre-labeling, if correction is required.
It's still human-annotated data. Humans have done that validation and the correction, but we're using AI to speed up that process by having the first pass. It significantly reduces the annotation time. We also see a fairly positive quality uplift through pre-labeling. The next is what we call speed labeling. This is where AI is used to assist the crowd worker in the labeling process. You can think about this similar to an auto-complete function, where it's humans plus AI working together to get to a fast outcome and a higher quality outcome. Finally, where we use automation in the labeling process is in what we call smart validators. This is when the crowd worker has completed the annotation, it will be a layer of checking that completed work prior to sending the file back and moving on to the next stage of the annotation.
The benefit of validators is, of course, it improves the quality of the model, but it also acts as a guide for the crowd workers around how they may want to do things differently in the future to get to a better performance. We've spoken a fair bit about training data. We'll move on now to some observations around training data. Sorry, some of the modeling techniques and how that is evolving. One of the things which I think is really important to understand is that AI-enabled applications typically involve a large number of models that rely on a large number of modeling techniques. The examples here are for a voice interface system. Think about your favorite at-home voice interactive product. There are three main blocks, technical blocks, and these are in the dark blue on the left-hand side, where you've got language processing.
This is the models that hear what you're saying while there's a wake word, then they listen to what you're saying. That's processed then to text. All right. This is the speech-to-text component. The next is intent handling. It's one thing to transcribe audio to text. The other is the natural language understanding component, which is highly complex and requires a lot of different types of models. One is understanding the intent, but it's also then matching to the knowledge of the system. Finally, it's the response generation, and the response typically involves different types of responses. One is the spoken audio that is returned to you. Let's say you wanted to start a timer on your phone, the voice interface system would be, "Okay, I've started the timer." It also needs to then go into the application and start the timer.
The actual activity that's involved. On the right-hand side, and I know this is very hard to see, but all of these smaller boxes are different types of techniques and algorithms that are used for each of those processes. You can see that there's a lot of different models that are required to be brought all together to deliver an AI-enabled application. In the real world, what we see is that there's not one modeling technique that's typically used end-to-end for a model. Across the top here are some techniques. This is not exhaustive. You've got transfer learning. Transfer learning is when you take a piece of a model that's been trained on something else and slot it in, which will get you some benefit that's out of the model.
Self-supervised learning, it's a modeling technique where there's no training, or there's no data annotation required for the training. The next three are examples of supervised learning, where the first is, it might be what we call off-the-shelf data, which is data that can be bought from a marketplace, or it's already data that's applicable but not specific, not necessarily totally specific for the application. Supervised learning, that's using AI-assisted human annotation. There may be a requirement for supervised learning where it's human annotated only. The composition of how the different modeling techniques are brought together varies, as you could imagine. In the example of a U.S. English chatbot for a bank. U.S. English is very common language. Retail banking is quite a common industry.
There might be a fair amount of models that can be used for transfer learning, specific techniques for self-supervised learning, and even a set of off-the-shelf data that might be specific for U.S. English and retail banking. That will get a long way in the model development stage. You start to get into the more specific. For a specific bank, it's going to have different product sets, that it's called differently. Terms and conditions, a whole set of company-specific taxonomies, and that's where data needs to be collected. In this case, supervised learning using AI-assisted annotations might get a long way, and there might be some requirement at the end for human-annotated data where AI-assisted modeling hasn't been developed yet. As you work down the specificity, the next example is a French chatbot.
You start to become more reliant on custom data collection and custom data annotations, just because the existing models and the existing off-the-shelf data don't exist. The third example is a Qatari Arabic chatbot for a marine insurance company. You start to get the idea that more specific AI models don't have the luxury of a lot of pre-existing work that's been used. It starts to get very custom in the type of data and the techniques that are being used. Having saying that, we've spoken about a limited number of techniques. There's a huge amount of research being put into new AI approaches. On the left-hand side, these are the number of papers which are being posted to arXiv, which is a quite common place for researchers and other academics to post their papers. These numbers on the chart are in thousands.
In 2019, there was almost 30,000 publications on AI posted to arXiv. On the right-hand side, there's a lot of research being done by new teams, and we're in a nascent industry, and it's emerging very quickly. There's a lot of forward progression in the types of modeling techniques that are being used. What we see, though, is the popular AI techniques that are used actually in mature AI practices still rely on human involvement. Here on the left-hand side, it's for a bunch of companies that were surveyed, well over 3,500. What are the different modeling techniques that they're using? You'll see the first is supervised learning. That's where examples are being provided and examples that have meaning assigned. Deep learning, a subset of supervised learning, but another way where humans are required for the preparation of the labeling process.
Human in the loop and active learning, and knowledge base and knowledge graphs. These are all different techniques where a level of human annotation is required. While there's a lot of research being put in advancing how AI evolves, we see in the real world that humans still playing a big role in the creation of high-quality training data. Mark mentioned, we've been on a journey, right? We're moving from this transformation into an AI-powered provider of AI data and solutions. We've gone from data types being language-focused to very AI-focused and supporting a wide variety of use cases. Our service-led delivery model to something that relies heavily on our products. This comes with a shift from project-based to more committed revenue. Our customers have been concentrated. Our products have allowed us to support a greater diversity of customers.
Mark spoke more yesterday around the org structure and reporting. This evolution has not occurred overnight. We've been on a journey. Through the acquisition of Butler Hill and Leapforce, Appen Connect became a really important part of our tooling and our infrastructure. Phase II was we acquired Figure Eight, and that gave us a very strong base set of capabilities. We've continued to invest and evolve our products, and this has led us to being very focused on a product-led expansion. Products are really important, but it is one part of the capabilities that we offer, and it's the combination of our crowd of well over a million strong, our deep internal expertise on how to deliver high-quality training data, and the product that's the real differentiator. We're going to focus today a lot on the product and Wilson will talk more about this.
I'm going to give a quick intro. There's five main components to our product suite. The first is Appen Connect. Appen Connect is our product that manages our global crowd workforce and does a lot of the matching from the crowd to the task. It's a really smart marketplace that matches the global crowd with projects. We're applying a lot of AI and building a lot of smarts to make that as seamless as possible. The next is the Appen Data Annotation Platform. This is the real engine of the company, where the crowd workers complete their task, and our customers can set up and monitor performance and create real bespoke annotation tasks for our crowd workers. We've got a set of new products that we're really excited about and are going to make a real step change in the performance of the business.
The first is Appen Intelligence. Appen Intelligence is the set of models that we use to improve automation throughout the business. This includes, like what we've spoken about in the annotation process. How do we improve the productivity of our crowd workforce and deliver better quality for our customers? It also includes processes to manage the crowd and those workforce tools. It's a really big part of what we're focusing on. The next is In-Platform Audit. Wilson will talk a lot more about this, but In-Platform Audit enables our customers to understand the composition of their training data better. We spoke a lot about class imbalance and quality errors. These can be very hard to diagnose and navigate when you've got data sets of hundreds of thousands of images, as an example.
The In-Platform Audit is a way for our customers to really easily navigate and narrow in on areas that need to be addressed and where performance needs to be improved or more data might need to be collected and brought into the system. Finally, and this is one which I think is super exciting, is Appen Mobile. A really great mobile interface that serves a couple of purposes. One, it's a way for crowd workers to engage with us. Log into the system, identify what jobs are available to them. Secondly, it serves as a different form factor for data collection and annotation. There are a bunch of features in the mobile-specific domain that aren't available for desktop. Things like location-specific, and other sensors which are inherent in mobiles but not in other areas.
Again, Wilson will talk through all of these, but this is just a quick intro. What's really valuable is though, that these products create a huge amount of value for our customers. First, we spoke a lot about AI-augmented data labeling and collection. That really improves the speed, quality, scale, and unit economics of the work that we do. AI-enabled crowd management. Increases our internal productivity and the experience of our crowd. We've got a lot of expertise in the company. We're trying as much as we can to productize that expertise and build that into our products. It automates a lot of the high-quality work that we're able to do. We've got a lot of in-built crowd management features.
This reduces risk for customers, particularly those that are looking at different crowd solutions and thinking about how they work with very large crowd. Finally, we spoke about the combination of the crowd with technology. That's a real competitive differentiator for us and enables us to do a lot of the work to solve some of those problems that we spoke about early on around data quality, diversity, and bias. We'll have a break now. Wilson's got a really exciting presentation and set of demos. After that, we'll go into some Q&A. We'll leave now and come back in around 25 minutes.
[Break]
Hello, and welcome back. We'll take you over to Wilson shortly, but before that, just a brief recap of some of the things that Ryan spoke about. He took us through the evolution of the company from a language service provider to an AI data provider, from a services-led company to a product-led company. He also took us through the importance of training data and the importance of quality in particular. He mentioned the number of different techniques that are used and the number of different training data types that are available. Overall, we're in a complex and evolving space, that requires a rich set of technologies. We'd like to take you through those now, so I'll hand you over to Wilson, who's in our Bay Area location, and he's pleased to join us via the technology. Take it away, Wilson.
Thank you, Mark. Welcome back, everyone. Ryan has shared that the AI industry is moving from model-centric to data-centric AI. To support data-centric AI, we have evolved our product suite significantly. We upgraded existing products to give our customers and crowd a better experience. We built new products to support new use cases and broadly a lot of machine learning capabilities to drive efficiency and unit economics. We now have an intelligence platform with a lot of automation capabilities, and the human only needs to be involved when necessary. Let's take a look. We have two existing products, Appen Connect and Appen Data Annotation Platform. Appen Connect is the platform where we match our global crowd to annotation tasks. Appen Data Annotation Platform is a platform where the crowd can deliver tasks. They can collect data. They can annotate data.
It's also the platform where our customers can manage their tasks in a self-service manner. Both Appen Connect and Appen Data Annotation Platform has evolved a lot with a lot of new features, better experience, a lot of AI capabilities. We also developed three new products. This is what excites me the most. Appen Intelligence, In-Platform Audit, and Appen Mobile. They make huge difference to our business, our customers, and our crowd of contributors already. Appen Intelligence, it includes the proprietary machine learning models to empower other products. It has models to automate the labeling tasks. It also has models to automate the project management tasks. To support data-centric AI, just collect data and annotate that data is not enough. In-Platform Audit helps the data scientists to really analyze the training data, so that they can understand the quality, distribution, and potential bias from the data.
It is essential, you probably already heard from Ryan. It is really essential to get the data right so that they can have a better AI performance. Last but not least, Appen Mobile. It's our new mobile app. It upgrades the crowd experience, help them to do different type of data collection tasks. It also helps Appen to increase our reach to even broader crowd group. The AI data industry, it values quality, speed, scalability, security, and unit economics. Our product suite can support all of them and really keep our business ahead of the competition. Now, let's look at some details of those different products. Let's first look at Appen Connect. Appen Connect is used by over 1 million crowd workers, as well as the Appen internal teams. Project managers set up projects and tasks. Crowd workers find the project and deliver tasks.
There are two major focus for Appen Connect. Number one is efficiency and scalability. We optimize the user experience so that both the crowd workers and our internal team members, they can be very efficient and all necessary effort can be saved. Number two is automation. We want to automate the project management effort as much as possible and let the platform to manage the crowd instead of human. Let's look at a very typical project life cycle. A project manager, they will set up the project, and then sourcing the workers or candidate to work on their project. They will have those workers to ramp up their skill and pass the qualification, then the worker can start to work on the project. After worker work on the project, the project manager need to really track the progress, track their productivity, their quality progress.
If the worker bump into any issue, the project manager need to support them to fix those issues. Meanwhile, the project manager also need to detect the fraudulent users constantly and kick those fraudulent users out of the projects. You can see it's pretty complicated, flow and life cycle, and some of those tasks are very time-consuming. Sourcing candidates, supporting workers when they bump into issues, and also do the fraud detection. Those tasks can take a lot of human effort. We are using Appen Intelligence to automate them and also make Appen Connect a intelligence marketplace. Let's look at the automation of the task to source crowd workers. Within Appen Intelligence, we have built a crowd data lake, which it contains a lot of crowd data, their behavior data, their project histories, their skills, their quality and productivity data.
Based on those data, we build a machine learning models to recommend workers of projects or recommend projects for workers. We also build a machine learning model to detect fraudulent users. With those AI capabilities, when a project manager, they finish set up the projects, Appen Intelligence can understand the sourcing requirements, then find those workers for the projects. They will send out personalized notification to the worker, if the worker is interested, they will apply for the project. Once the worker applied, Appen Intelligence will further screen them to check if they are eligible or is that a potential fraud. Based on those information, Appen Intelligence pass them or fail them. If the worker pass through the auto screening, they will be activated to the project automatically. If I see all those steps here in green color, this can be done by Appen Intelligence.
Steps in the gray color, those are manual steps from the contributor. With Appen Intelligence, with this automation, this will now save huge amount of effort for project managers. They only need to get involved when Appen Intelligence, when our machine learning model is not sure about the decision. Majority of those time, those tasks all got automated. Let's look at another example, fraud detection. Given Appen Connect is a marketplace, there can be fraudulent users. If you look at the example at the left side, those two accounts, they are from the same IP, and one user normally works from three AM to five AM. It's very suspicious, and it can be a fraudulent users. It is key to remove those users so then the product quality is not compromised, and we don't really pay unnecessary costs.
However, you can also understand that analyzing the activity from over 1 million workers is not possible by human. Fraud detection models from Appen Intelligence help us to do the job. It processes more than 1 million users every day and handling 200+ signals for every user. The fraud detection model, the accuracy is pretty good. It's around 95%. Those models are used in a lot of places. It checks user during the new user registration. It's used to screen the product application. It also runs in the back end all the time to detect any suspicious activity. Fraud detection with machine learning not only automates the huge amounts of human efforts, but also support a scale human just cannot handle, right? Talking about like 1 million workers, it's just so hard for human to check every day.
Appen Connect, as you can see, it creates huge value for our customers and our crowd. It connects customers with our global crowd. It automates a lot of product management work and reduce the overhead costs. It enables our business to scale and support the future growth. Future investment focus on two areas. First, we will continue to optimize and make a very good user experience for both the crowd and also the internal team members. Second, we will just continue to add more automation, so then the manual management efforts become less and less. Now, let's take a look at Appen Data Annotation Platform. Majority of the data annotation platform in this AI data industry only focus on certain areas. Some focus on computer vision, some focus on audio and language, while the Appen Data Annotation Platform has the breadth and depth to support all kinds of use cases.
It has tools to support different type of data collection, tools to do content relevance, tools to annotate audio and text data, tools to support image, video, and 3D point cloud data processing. Meanwhile, no matter how many tools you have, there will always be some special customer need you haven't heard before. We also have a powerful tool called Job Designer, and it will help the customers to design new tool very easily. The Job Designer also provides a programming language called CML, which is loved by the developers, they can program a pretty powerful tool pretty quickly. Those annotation tools, they are pretty powerful and work really well in a single task. While some AI data use case is very complicated, it need multiple steps and different operations to get the data right.
Those operation can be a human labeling job or a machine learning model or a script to process the data. Appen Workflow enables those use cases. It stitch all those different operations into a flexible workflow. Let's see how it works. Please have to play the first demo video.
AI application can be complicated, and customers often need the same data to pass through multiple jobs to satisfy the project requirements. Before Workflows, linking multiple steps for complex annotations used to be a manual process. Not anymore. Let's see a real-world example. A restaurant review platform must first categorize a continuous stream of user-generated content to identify key attributes. When a photo contains a menu, the customer would like the menu transcribed. When a photo contains food, the customer would like to know if it's a main course. When a photo is outdoors, the customer would like to know a bit more about the restaurant's outdoor amenities. Workflows make it simple for customers to break their complex projects down into smaller basic steps that are connected by flexible routing rules.
Routing rules can be set based on specific answers, result confidence, or to route random samples into QA jobs for further review. As always, settings such as targeting, quality controls, and pay can be customized on a per job basis. Today, most steps in workflows are jobs, but customers can leverage Appen's growing catalog of machine learning models and scripts to automate a variety of simple tasks. With workflows, it is easy to streamline even the most complex annotation tasks.
I hope that give you a better understanding on how those AI data use cases, how complicated it can be, and it's great to be able to support those complicated tasks. It's also very important to guarantee the quality. Quality is always one of the most important factors for training data. Appen Data Annotation Platform has a rich set of features to do quality control. The number one form of proactive quality control is test question, which customers can define ground truth data. Those data can be used to qualify the worker before they start or monitor their quality performance during the job. QA workflow uses a different methodology, where we ask high-qualified workers to review and correct annotation from other workers. Dynamic judgments collects judgments from multiple workers and aggregates the results to get a high confident answer.
Machine learning validation, this, the Smart Validator Ryan mentioned earlier, is to using machine learning predicted results to validate the annotation from the workers. It's very useful in certain use cases. Normally, when we handle projects, we use multiple of those features to achieve high-quality output. Security is another core consideration. You probably know, can easily understand how important security is for the AI data. Appen Data Annotation Platform provides very flexible deployment options. Customers can use the platform in our public cloud or deploy the platform in their private cloud, or in a completely air-gapped environment. For the customers, they use our public cloud. They can use a feature called Secure Data Access, they don't need to move their data into our platform. We only access their data when the worker is labeling them. Those access will expire after the data is labeled.
This just created another additional layer of data protection. Our platform also meets all those major security and privacy compliance standards like SOC 2, GDPR and HIPAA. With all those security features, our customers' data are well protected. Appen Data Annotation Platform, it creates huge value for our customers. The full suite of tools supports different customer use cases. Appen Workflow, it enables complex training data preparation, and those quality and security options help our customers to get high-quality training data and also help them to really protect their data. Future investment for Appen Data Annotation Platform focus on several areas. We will continue to evolve the tools to support newly surfaced use cases. For example, now our team is working on building a new tool to annotate all those satellite imagery tasks.
We're also working more to provide a better API to have a tighter integration with our customer systems. Quality and security are never-ending effort. We will just continue to invest more and more on those features and offerings to help our customers to get high-quality training data and also protect their data. Let's look at Appen Intelligence. We build a product to prepare training data for AI companies. Meanwhile, Appen itself is also AI company, and machine learning is used in all our products. Appen Intelligence provides those machine learning capabilities. We have seen earlier how Appen Connect is using Appen Intelligence to automate sourcing workers to do fraud detection. Let's see how it helps the automation of the annotation efforts. Appen Intelligence provides proprietary machine learning models across different data categories.
It has models to identify speakers, to detect languages, segment audio files, convert audio to text to do voice recognition. It also has models to analyze text data, to detect gibberish, to extract the entity, and to do text classification. Those are the models commonly used in the natural language processing field. It also has models to process image and video data, transcribe text from an image, detect objects, and generate face landmark or blur faces to protect privacy. It also has a lot of models to handle 3D data for object detection and tracking. Those models are used to pre-label data, so then a human only need to review the pre-labeled results instead of labeling those data from scratch.
Those models are also used to check the annotation from human, to check their quality, to validate their results, and helps data quality, labeling speed, and saves a lot of labor costs. To better understand how those models are used, let's see a few examples. Understanding documents with machine learning becomes very popular now. Finance companies, they want to process receipts. A law firm want to find some legal information from a document. To train those machine learning models, we need to transcribe text from images, from PDFs or other files. Now let's see how those training data are labeled for OCR transcription. Can you help me to play that?
Now customers can get OCR training data with machine learning assistance. First, let's see how it works manually. As a customer, I upload images that I want transcribed, configure the tool, and launch the job. When the job is live, contributors see the image and draw bounding boxes on every word. The platform transcribes the word automatically, but contributors need to review and modify the word if needed. Contributors handle words one by one, and the whole document will take a long time to finish. Now let's see how machine learning-assisted pre-labeling can make this process more efficient. As a customer, I will use the workflows feature to set up the job. The workflow contains two steps. First, we use a machine learning model to automatically draw bounding boxes and transcribe the text in each box.
The second job is to route annotated images to contributors to review and modify as needed. Now I'm ready to upload the data and launch the workflow. After the machine learning model pre-labels the data, contributors will see the image with bounding boxes and transcribed text already in place. Their job is to review the results and make edits if needed. Thanks to machine learning assistance, OCR transcription can be 5x faster when compared with a manual alternative.
Clearly, machine learning assistance has helped the OCR data labeling. Let's see another example. Voice recognition is another widely used AI technique. Let's see how to prepare training data for voice recognition machine learning model.
Let's look at a voice recognition example. We provide a comprehensive suite of tools and models to enable the information extraction from audio data, including Appen's ML-powered audio annotation tool to speed up the labeling process. With manual labeling, there is a two-step process, starting first with audio annotation, where speech segments get identified and classified. The segmented audio gets loaded into the audio transcription tool, where they get transcribed.
What was it today?
I think we're going to talk about weddings.
Okay. What was it today?
As you can see, the end-to-end manual annotation process is labor-intensive and time-consuming. Now let's take a look at Appen's ML-powered audio annotation tool. This uses a diarization model to automate the audio segmentation step, then uses an automated speech recognition model to generate transcription hypotheses. Finally, we add a human review job to correct any possible mistakes in the model hypothesis. By this final step, the transcriber will see all the transcription hypotheses generated, and all they need to do is to correct any occasional mistakes. We have estimated that this is 20%-87% faster than manual transcription.
I think we're going to talk about weddings.
Okay.
I'm far away.
Let's switch gears to computer vision. Autonomous driving is probably the most exciting AI use case in computer vision field. To do autonomous driving, the cars need to understand the environment surrounding them. They use sensors like cameras and lidar to collect data. Lidar is a special type of sensor which collects 3D point data of objects in the surrounding environment, and the machine learning model needs to understand those 3D data, those 3D points, and classify them as cars, pedestrians, bicycles, or other object types. To prepare those training data, crowd workers need to operate in a 3D environment, and it requires some special skills. Normally, it takes a long time to label the 3D data. Let's see how does 3D point data get labeled. Please help to play the first demo video.
In this video, we will demonstrate two scenarios, labeling lidar data manually and labeling lidar data with machine learning assistance. This example has 26 frames, and the contributor needs to label all the objects in all those frames. First, the contributor needs to familiarize themselves with the 3D environment. They can navigate around and refer to the image data on the upper left. They need to find the car and draw a 3D cuboid, specify some attributes of the car, and then adjust the size to make it a tight fit. They go onto the second frame, add a cuboid, and repeat the process again. They continue this process for all 26 frames. Once all the cars have cuboids, the contributor steps through each frame and makes adjustments frame by frame. After adjustments are made, they can play back the video one last time as a final review.
Let's look at how machine learning assistance can help. The contributor still needs to do the same process for the first frame, find the car, add the cuboid, specify some attributes, and make some adjustments. When they move to the next frame, they find the cuboids of that car are already there, and they just need to make some small adjustments. They can also switch to grid view, where they'll find that the cuboids have been generated automatically in all 26 frames. They can just navigate across multiple frames, make some adjustments, and the tool will interpolate backward to add or modify the annotations in between the manually adjusted frames. Thanks to machine learning assistance, lidar annotation can be 4x faster when compared with the manual alternative.
Self-driving is pretty challenging and needs to handle different situations. We are seeing how this cuboid, how this object is labeled. Besides detecting those objects surrounding the vehicle, the car also needs to detect the lane lines on the road. Let's see how lane lines can be labeled. Please help to play the first video.
In this video, we will demonstrate two approaches for lidar lane line segmentation, labeling lane lines manually and labeling lane lines with machine learning assistance. Let's start with manually annotating. First, the contributor needs to zoom in so they can see all the points of the lane line. They'll then draw a tight bounding box to include those points, find another lane line, and repeat the process. Navigate to another part of the road and continue with this process. It's not a very complicated task, but it is time-consuming, and our contributors have to label thousands or tens of thousands of these lane lines. Now let's look at the machine learning-assisted approach. Switching to intelligent mode, the contributor can draw one large polygon around the whole lane, and the machine learning model will detect the lane line to create annotations for each individual line.
Thanks to machine learning assistance, lidar lane segmentation can be 6x faster when compared with the manual alternative.
We have seen from those demos, machine learning assistance is very powerful. Here's a quick summary of all the productivity difference we have observed. Audio and speech. Machine learning-assisted annotation can be up to 1.6x faster. Even in OCR, the results are even better. The 2D image bounding box labeling with machine learning assistance can be 30% faster, while OCR with machine learning assistance can be 6x faster. It also works really well with 3D data. It can be 4x-6x faster, but labeling in the 3D environment is really complicated for humans. Machine learning assistance is very powerful. Meanwhile, it doesn't really work well for content relevance tasks. Content relevance tasks are very often subjective. It often requires people with certain cultural background, and super hard to automate.
Overall, a lot of those data annotation effort from the crowd workers are now being automated by Appen Intelligence. Those automation improve the data quality, labeling speed, and also saves huge crowd costs for us. Clearly, Appen Intelligence creates huge value for our customers and also adding all those machine learning capabilities to other Appen products. It automates the crowd effort and lowers the unit costs. It also helps to improve the delivery speed as well as the data quality. It also automates product management efforts so that our business can easily scale. You may recall earlier how Appen Intelligence is used in Appen Connect side. In the future, we will just continue to add more AI capabilities to automate more use cases for both the worker side and also for the internal teams.
Let's now move to Appen Quality Audit, which is a new product we released last month. It's still in early stage, but I'm super excited about Appen Quality Audit. It already brings a lot of value for our customers. As you have seen from the slides Ryan shared earlier, training a good AI model can be expensive. It needs a lot of training data. It needs computation power. It needs effort from a data scientist team. If there's problems with the model, it's better to find those earlier instead of later, so that you don't need to redo all this work. Redoing all this work, that really introduces a lot of costs. AI model performance is driven by the training data. Debug and detect problems from training data early on is key to the model success.
In-Platform Audit is designed to help data scientists to analyze the training data. They can analyze the raw data before labeling or analyze data after labeling. With the ground truth data, they can also use them to evaluate model performance. In-Platform Audits will help the data scientists to detect all the data problems, like class imbalance, accuracy or quality, or label imbalance. Those data problems might not be that straightforward to understand. Let's use the example to explain. Let's say I want to train a machine learning model to classify if a tweet is a positive tweet or a negative tweet. To train the machine learning model, I first need to collect the training data. I scraped 10 million tweets from the internet.
If 9 million of them are from male and only 1 million are from female, then the model might not work well for female tweets using that data set. This is a class imbalance problem. I detected the class imbalance problem and fixed it. Now I have 5 million tweets from the male and 5 million tweets from female. I'm getting people help me to label these tweets. When I review this label result, I found a lot of positive tweets got labeled as negative. I got data quality problem. The accuracy is not high. Now I detected the accuracy problem and fixed it. However, for those 5 million tweets from male, 4 million of them are positive, while 1 million are negative.
Although the data labels are accurate, I get a label imbalance problem, which will cause a lot of problem for my model later on. I also detect the label imbalance problem and fix it. The data set is now well-balanced and has high quality. You can imagine the model trained using this data set are likely to have a good performance. I hope this give you a good sense of how training data insight can help to detect and fix data problems. I think In-Platform Audit, it provides additional value to our customers. It essentially enables customers to understand their training data, find problems, and fix them, which in turn will help them to improve their AI performance. We just released the In-Platform Audits last month. There's a lot more to do.
Currently, In-Platform Audits focus on training data analytics. We will expand it to support model performance evaluation in the future. Mark also mentioned there's a trend where people need a lot of tools to manage all this training data. We're also adding more training data management features into In-Platform Audits. This is a super exciting product. It can evolve to be a powerful training data analytics tool loved by every data scientist. Now let's move to Appen Mobile. We released Appen Mobile early this year. This new mobile app provides an upgraded experience. The crowd workers, they can engage with Appen at any time from any places now. The new mobile app provides a very intuitive user experience. A crowd worker, they can register to become a user quickly, find products easily, and work on all kinds of data collection tasks.
The app has made data collection easier than ever. Location-based app also becomes very popular, especially during the pandemic. Those apps, they need location-based data to train their AI, and our new mobile app supports those needs. The new mobile app is great. It provides better experience and also supports more data collection use cases. That's not the only benefit it brings. The new app also increase our reach to the mobile-only crowd workers. The population actually is pretty big. You know that there's a lot of people, they're only using mobile. This is a pretty big population in Asia and also other developing countries. Enough talking about this app. Let's see how it works. Please have to play the sixth video, the sixth demo.
Appen Mobile provides a simple and straightforward registration flow for contributors. New contributors just need to input their email, name, country, and state or province as appropriate, and then submit the registration request. They receive a notification code to their email, and after verifying that code, the registration is complete. A contributor can easily log into the app and see projects recommended for them. They can simply click the project to learn more details, and they can apply for those projects if interested. Some projects may ask for additional information from the contributor as appropriate. After the contributor qualifies for the project, they can start to work on project tasks. This particular project will ask the contributor to complete multiple steps, including collecting audio data, transcribing that audio data, tracking their eye movement, and then collecting handwriting samples. The first step is for the contributor to collect audio data.
They will need to speak as directed by the prompt, record the audio, and then transcribe the audio. The app supports the collection of a conversation between multiple people. One contributor will send the invitation link to the other contributor, and with that link, those two people can start a conversation which will be recorded. Let's see how video data tracking eye movement is collected. In this example, we will record a selfie video where a contributor will watch a bouncing ball video on their screen while we capture their eyeball movements. The data helps our client to train models to detect user sight focus. Contributors can even do handwriting tasks within our app. In this case, we are collecting contributor handwriting strokes start and end time with key point coordinates while they write. The data can be quickly and easily uploaded via the app.
Appen Mobile creates huge value for both our customers and also the crowd workers. It gives crowd members a much more intuitive experience. They can engage with Appen at any time from any places. It enables a lot of different data collecting use cases. It also help us to reach to a much bigger crowd population. In the future, we are going to invest more on this mobile app. We're going to support new data collecting use cases and also supporting other data annotation tasks. Whatever task, if can fit into a mobile screen, we want to try that in the mobile too. With the mobile app, we are just going to actively expand the crowd to support the diversity and also impact the sourcing. This wraps up my presentation. I will now hand it back to Ryan.
Thanks, Wilson. I'll spend a little bit of time on a recap and a close before we head into some Q&A. Wilson took us through the product suite from Appen Connect, which is used for our crowd management, the Appen Data Annotation Platform, used for our crowd workers to do the labeling and also our customers to set up and customize jobs. Some of the great new features that we've rolled out more recently, Appen Intelligence, In-Platform Audit, and Appen Mobile. We spoke about how these capabilities unlock huge value for our customers, from AI augmentation in data collection and labeling, delivering speed, quality, scale, and unit economics, the crowd management and some of the AI that we're using in that, including fraud detection. That really increases our internal productivity, but also the crowd experience.
We've embedded a lot of expertise in these tools, and that's really helping us deliver high-quality annotation work for our customers. We've got inbuilt crowd management features, and doing the crowd management on behalf of the customers is really important for them. That native integration with our crowd, it creates the competitive differentiation where we've got a complete set of tools and capabilities, both from a technology standpoint and from a crowd. That's the real differentiator at Appen and how we unlock a lot of value for our customers. It's not about having the right tools, and it's not about having the large crowd or the expertise. It's bringing all three of those together, and that's what our customers really value from us, and that's what we'll continue to focus on in the future. Product is going to be a very large part of what we do.
I'll now hand it to Mark, who will moderate our Q&A session. I think we've got about 30 minutes or a little bit more allocated for some questions.
Thanks, Ryan, thanks, Wilson. I hope you all enjoyed the presentations from Ryan and Wilson and the demonstrations as well. We have some questions. I'll read through them, throw them first to Ryan, first of all, and he can loop in Wilson as required. The first question is, how does Appen's own data labeling platform and AI investments compare with the competitors such as Scale AI? How are they different? Are you investing enough in R&D to keep up with new entrants? Ryan?
Thanks, Mark, and it's a good question. We monitor all of our competitors, as you can imagine, from information that's externally available and often do feature comparisons to understand where we are from the market and also speak to our customers too. A lot of our customers obviously will look at different products in the market and see how we compare. To the best of our knowledge, we have a very comparable set of products and some areas where we have a lot of deep expertise that is built into our products and creates a lot of differentiation for us. I think from a technology standpoint and our product suite that Wilson just took us through, it's comparable, and in some areas leading the market. I think like what we were just talking about before, there's a huge value about the combination of the product suite with the crowd.
We've got the products, and that's comparable to the market. It's that combination with the crowd and our internal expertise that really makes a huge differentiation for us.
Thanks, Ryan. I'd also add that we do know that our breadth of functionality is superior to many of our competitors who tend to focus in on one area. Recall the evolution of our business from a language data provider through to a multimodal data provider. Earlier on in our evolution, we were focused on language and speech data. Similarly, the new entrants are focused on a particular area, mostly image data. The question also asks about the rate of investment. Clearly, there's visibility into our investment, through our publicly available accounts. We don't have that same visibility into our competitors. We see the private competitors. We see the money they raise, we don't know how much they're putting into R&D. Ultimately, what we try to do is work with our customers to make sure we've got the range of products they want.
Per Ryan's feedback, our view is we are comparable, if not superior across a broad range of use cases. The next question is: Are any of your technologies being implemented in the electric vehicle industry? If so, what and how are they being implemented? Who are your business partners in this market, and what about autonomous vehicles? A few parts to that question, Ryan.
A few parts there. I'll focus on the autonomous vehicle part because I think the electric vehicle is more about the drivetrain. Autonomous is more about the perception, where there's a lot more need for training data. I think through the demos that Wilson just showed, we've got an advanced set of capabilities in the annotation market for autonomous vehicles. Wilson showed lidar, but there's also computer vision is used a lot also in this space. We have a range of customers that we work with to support their autonomous driving models. We definitely have the capabilities and the depth in that market, and it's an important focus area for us.
Yeah. Thanks, Ryan. I'd add that it's a very big problem for autonomous vehicles. As you can see from the demos, just the challenge to annotate road lines and then think about road furniture and other elements that you have to deal with as a driver. It is a big challenge and requires a lot of data. Okay. The next question also touches on autonomous vehicles. Is recency of training data for application like autonomous vehicles important, or is it an issue? Sorry, Ryan.
Thanks, Mark. Recency is super important across pretty much every AI model there is. Autonomous vehicles are a good example where it's really important. I'll give an example around recency and why it's important. Last-mile commuting is becoming really important. Two years ago, there weren't too many electric scooters on the road. There's not that many today. If you go to San Francisco, it's a bit of a different story. If you think about the annotation of electric scooter, particularly for someone that's upright. If you're using data from two years ago, you might treat them as a pedestrian. Now all of a sudden, you've got these pedestrian-looking objects that are traveling at 20 km an hour down the road. Just a basic example of how real-world environment's changing, and in the context of autonomous vehicle, that has a massive change, right?
A massive set of implications. I think the other thing Mark alluded, autonomous vehicle is really difficult. The specific environment changes geography by geography and country by country. Different sets of road rules, different sets of buildings in the background. Right-hand side, left-hand side. A lot of the work that's being done to build the models today are quite U.S.-centric. There's going to be a huge long tail of market-specific training required to make autonomous driving a truly global approach.
Yeah. Thanks, Ryan, and I might hand it over to Wilson to chime in on this one as well. As many of you know, the majority of the work we do is around search, and recency is really important in search. Wilson, given your background, perhaps you could add something on the importance of recency in search data.
Yeah. In search, actually, recency is core to search. If I remember my old days, when we trained the search algorithm, we basically released a new model almost every week just to catch up all those recencies. Search touches on a lot of different areas. There is culture change, there's a society movement, there's all those new keywords, popular music. There's a lot of new stuff that keep coming out. The search has to be able to support all those new stuff. That's the reason recency is super important for search. That's also the reason we retrain our model almost every week.
Yeah. Thanks, Wilson. If everybody on the call thinks about their own experiences driving, just to go back to autonomous vehicles. Certainly, if I go back to the area in Sydney where I grew up, the road's changed and sometimes it changes very quickly. Even humans need recent data. Of course, we can join dots very well, whereas the AI needs the training data to learn. It's much harder for an autonomous vehicle to learn than it is for a human. Recency is super important. Great question. Thank you. Okay. The next question. We talked a lot about the impact of Appen Intelligence on speech, text, image, video, et cetera. Spoke very little about the impact of it on content relevance. Can we go into this a little deeper? Ryan?
A good question. Content relevance is highly subjective. It's highly specific to a demographic. We need that subjectivity of a human, and the context of that individual's awareness around a specific environment, which is typically driven by demographics, where they live, et cetera. A lot of the work that we do in the automation space is around improving the speed of the crowd worker. We spoke about pre-labeling and validation and support during the labeling process to improve the speed of the work. With content relevance, there's typically less of a need to do some of the time-consuming tasks, like draw a polygon around a shape, for example. A lot of the AI that's being used in speech and image-related training data support is to really speed up the process, which helps with throughput and quality.
One of the things we are focusing, though, on the content relevance part is more of the user experience changes. How do we, not necessarily using AI in the process, but improving the environment that the workers are operating with to try and get those incremental step changes in the time that it takes to complete the task. That's an important focus for customers we support on our platform.
Yeah. Thanks, Ryan. Again, I'll hand it over to Wilson because we've talked a lot about this, Wilson, trying to automate content relevance. Perhaps you can share some of your thoughts on that and maybe some examples that bring it to life.
Yeah, sure. I think this is a really great question. Trust me, there's no lack of trying. This is a big part of our business, and we try very hard to see how we can save more labor effort there. It is hard. Content relevance normally requires a human to have a certain culture background, certain knowledge to do those. Sometimes you can use machine learning to try some of those and say, now I have a search keyword, I want to see some results, some search result or some product results. Is that super relevant to the keyword or not? We can use machine learning to try those. Sometimes you can see some success. There's a big problem with that. If you do that, and potentially can introduce a lot of bias.
You don't really want to using a machine learning trained to have a lot of How can I put this? Let's say you have a kind of machine learning classifier results, then the worker, because for them, the task is easy, right? It's relevant or not relevant. They will just pick whatever you said there, and introduce some bias, which is not good. The other part, I can't see a good example, but the machine doesn't really have those deep understanding on the culture element or a lot of subjective component only human possess. It's really hard. We tried a lot, but we haven't seen a lot of success there. Meanwhile, go back to the point Ryan mentioned.
We did have a pretty good success when we tried to design a new workflow or maybe a different UI, so then for the workers, when they do content relevance, it is much easier for them to deliver the results. We see some success there.
Yeah. Thanks, Wilson. Perhaps you may recall, earlier in Ryan's presentation, he had the three pictures of breakfast, the black pudding, the chocolate sprinkles, and the Vegemite. Depending upon what country you come from, you may think that's breakfast or not. That's an example of a cultural type question. The search, the relevance task, could be very simple. Is this breakfast or would you eat this for breakfast? The majority of people may look at the black pudding and say no. Whereas depending upon culture and where you come from, you'd have a different answer. Automating that is super tricky. No lack of trying.
Keep in mind also, the companies who ask us to do this for them, the largest search and social media companies in the world. They've got some pretty smart data scientists, and I think if they could have automated it, they would have. There's still a need for that human element there. Okay, the next question. Sorry, this is quite dynamic. It's a life of its own. Here's the next question. This may be one for Wilson, but I'll throw it to Ryan first of all. Feature engineering is one of the major time-consuming tasks undertaken by data scientists. Does Appen plan to invest in this area? Any investment in medical/biological data annotation technologies? Fairly specific. Ryan, do you have a response there?
I'll start with the medical, and maybe I'll throw to Wilson for the feature engineering part of the question. Our tools today are capable of supporting medical imagery. A lot of the imagery work is a computer vision-related task, and the flexibility of our tools can support those types of applications. That is an area of support today. Feature engineering, I'll pass on to Wilson for that one.
Sure. I think this is a great question. Data scientists really spend a lot of time on feature engineering. When you say feature engineering, most of those times related to the training data. All the products we shared today, all the examples we gave today, in broad term, is also feature engineering. They're really preparing all those features, which is also the training data to really have to train the model. We are working on a lot on those. Besides all those traditional products that we have help people to collect data, to analyze data, which will become the feature later on to use to train a machine learning model. That's already a big part of it, and also the new product, like Appen In-Platform Audit. That's actually help you to understand the training data, that's help you to understand the features.
What's the distribution of feature? Is there any bias for your feature? Basically, all our work, all our product is on helping people to do better feature engineering.
Yeah. Thank you, Wilson, and thanks for that question as well. The next question is: Can Appen Mobile technology be used for data capture as well as being a crowd-focused tool, Ryan?
Yeah, absolutely. That's one of the real core components of Appen Mobile. Wilson spoke through the two main features, one being the interface for our crowd, where they can sign on, view their tasks, and manage the relationship with Appen. The second is as a really powerful data capture tool. I'll give you an example around some of the features which are enabled in a mobile device that aren't in a desktop. Things like GPS. If the task was to go out and take a photo of a real-world environment, the GPS data is automatically tagged within the metadata of the image. That becomes a really important part of the metadata of the training data. There are a whole raft of different native features of handheld mobile devices that open up a different set of data collection capabilities.
Thanks, Ryan, and thanks for that question. The next question. Slide 46 in the pack refers to different AI technologies that are used in mature practices and highlights, quote, "Techniques that typically require some level of human annotation and/or data preparation." Unquote. Where it says some level, is that level of human involvement the same, higher, or lower than two years ago? Are lower levels of human annotation positive, pardon me, negative, or neutral to Appen?
A good question. A lot of AI varies, and there are different use cases, and things evolve differently to others. I think that in those specific areas around supervised learning, there are some supervised learning techniques where data can be taken directly from CRM systems or other which are automatically annotated, which may not require as much human annotation to complete the feature engineering. Whereas there are other supervised learning techniques that are heavily reliant on human annotations to complete the labeling process that's used to train the systems. I think that's part one of my response. The second part in how that has changed. Our market is growing, and we see an increasing need for human-annotated data. There's also a lot of need for training off data that's already prepared because it comes from CRM or other structured data systems. AI's growing. I think it's growing everywhere.
To answer, is it more or less? I think it's definitely more.
Thanks, Ryan. The next question. With the rising concerns on privacy issues with big tech, big tech are working on reducing data accessibility. Do you think this would hurt Appen's AI business model? If yes, what would be a solution?
We're still seeing how this is playing out. A lot of these changes are quite new. You're right in saying that seems to be a general view that there's a restriction on the data sharing and how that's used outside the broader ecosystem. One of the views that we have is that with this restriction of sharing, there'll be less available information to train the models that are used for things like advertising targeting, search results, et cetera. Again, we're yet to see it play out. It's very live, but there is potential that this could be a net positive for us as more data is required to fill those gaps that have been created by the restriction driven by privacy.
If you also think about it, what's changing is, for want of a better word, the unsolicited harvesting of data. We'll move to an environment where more permission is needed, where more protection is provided around personal data, more rights are provided around personal data. What's constant is that AI is the center of many product developments in technology, and it's also constant that AI needs training data. I hope we've illustrated that today. What's changing is the way that firms acquire data. There's no reduction in the need for data, but it's how companies acquire data. As Ryan says, that could play to our advantage because we could see companies coming to us saying, "We can't harvest this data anymore.
How do we get this data in a manner that protects the owner of that data? Yet to play out, but it is an important part of the AI industry going forward. Okay. The next question, what proportions of models would you think are done by self-supervised learning?
It's a tricky one to answer. Self-supervised learning is an interesting technique, and it has applicability, and it has its benefits because there's a lot less of the feature engineering and the data labeling required. It does have its drawbacks, though. It really is restrained on the assigning of meaning to the data. For instance, it will group a whole bunch of different shapes together, and that can be inferred to have a certain meaning. That doesn't quite have the same benefit as what we see through human-annotated data used to support supervised learning. The other thing is that with a lot of the self-supervised learning techniques, particularly in the large-scale applications, there's an opportunity for bias to be introduced, and it's quite hard to control. Unsupervised requires huge amounts of data. It's very difficult to filter out the inputs that create bias.
I'll ask Wilson whether he has a view on that specific question around the proportion of models that rely on self-supervised learning.
Yeah, I do. Actually, this area I put a lot of effort, monitor the progress, and also see how this technology evolves, right? Self-supervised learning is not a new thing. It's been there for a long time. I use that term from 10 years ago together with other technique. One thing super important for self-supervised learning, it also called self-supervised representation learning. What that going to mean? It really means, let's say, how you learn those representative of a word, right? Let's say, for example, back to my Twitter example, like the tweet classification example. I want to classify if a tweet is positive or negative. I'm using a language for labeling data, like this tweet positive, this tweet negative. I'm using that to train the model. Meanwhile, I also use a lot of self-supervised learning to prepare the features before that. What that mean?
I can use a lot of technology to make sure converts, let's say, this particular word, what does this word mean, right? I can use self-supervised learning to really convert one word into a vector. That's kind of a feature engineering step. I will convert that word into vector, and then using that vector as an input for my supervised learning to train the end result. I know it's a little bit complicated, but it's just one step of the overall machine learning training progress, and also we use those techniques together. It's not I use self-supervised learning that replace supervised learning. That's not the case. I use them both to train one model.
Yeah. Thanks, Wilson. I think, overall, that as You recall the slide in the deck for the speech application. It was a very complicated slide, blue-shaded slide. There are many different models in one application or one product, and there are many techniques that go into those models. Training data is essential for AI, but it's expensive. If the only training data was available was human-annotated data, it could be potentially prohibitively expensive. The developers of AI are looking for any technique they can to accelerate and improve the cost of the development of their AI products. Typically, there's a mix of techniques that goes into building one product. And that's reflected in the way pardon me, that we're going about our business.
Rather than rely on the simple technique of human-annotated data, we're looking to use AI ourselves to accelerate and improve the unit economics of the production of that data. Okay, the next question. Why do customers use your platform and tools rather than build their own?
A good question. Customers rely on us for a variety of areas. Firstly, and I think we covered some of this today, data labeling is difficult. Data labeling and annotation at really high quality levels can be very difficult. That's point one. I think the next is that managing a crowd is very difficult also. It's one thing to assemble a 1+ million people. It's actually another to allocate work, manage the quality, do the payments, et cetera. Customers could, and some of our customers or people in the industry, I should say, do build out their own annotation platforms, and it's largely to support or it's initiated by supporting a very specific use case. They will build a data pipeline and a workflow within the business and some annotation tools that support a specific use case.
Comes the step of, okay, we want to do some different things. It's not just this narrow use case. We want to expand beyond that. That's when it starts to become very apparent to our customers that this is a big investment. It's difficult to bring the expertise across that wide variety of use cases. During our sales processes, we have many customers who have been down this journey where they start with a narrow use case, have built something internally, and quickly realized that it is very difficult to manage quality in particular, and it's difficult to support a breadth of AI use cases. Customers come to us when AI is getting serious, and they really want to move into production and support high-quality training data for high-performing applications.
Yeah. Thanks, Ryan. I think it's like any developing industry, and ours is still relatively early. There are many techniques that people try on their own. Could even be developing their own platform to do this work. It gets to a point where it's too complicated. The scale of the operation is too large. At the same time as people are learning how to do this, there are companies like Appen emerging to bring specialist expertise to the industries. I'm sure there was a time when every company made their own payroll system, for example, whereas now you would never do that. I think it's a bit of that evolution as well. I'd also add that every one of our customers benefits from all of our experience and knowledge that's embedded in the platform as opposed to just covering their particular use case.
Yeah, we see a departure from people building their own platforms to wanting to work with a specialist provider. Okay, the next question. How is the role of crowdsource workers changing for Appen as the model changes to a product-led, committed revenue model? What does crowdsource efficiency mean for Appen?
A good question. There's a large amount of work for our crowd. One evolution that we're seeing is that the demographics are getting more specific, so the ask from our customers. One of the things that we're doing is ensuring that we're able to serve the customer's needs by fitting the right demographics. That's very important for us. The other thing that we're working very hard on is what we spoke about with Appen Mobile. Making that crowd experience a lot more seamless. There's greater visibility into the tasks that are available and greater matching of a person's skills to the task. When they do come and work with us, it's tasks that they are able to deliver high-quality work and do more on those tasks and support in a really strong approach. Our crowdsourcing approach, we continue to build our crowd.
We continue to find ways to better match the people in the crowd with the right task, and that's a good experience for our crowd. It's a good experience for our customers and ultimately leads to stronger growth in the business.
Yeah. The crowd is a big expense. It's the cost of goods expense that goes through the business. If we can get more data per crowd worker to improve the unit economics, then that goes to our gross margin and ultimately to the bottom line. This is a multi-part question. Firstly, data ownership and licensing. If a client owns their data library, does that mean they no longer need that type of service again? Is ongoing data maintenance required? If so, is this typically provided by Appen or the client? Can data libraries from existing clients be sold to other clients wanting the same type of data? Is each dataset private and mostly customized to client needs? That was just part one.
Okay, let me handle part one. We spoke a lot about data recency. Again, the general view that all AI models degrade in terms of performance. It's not a matter of if, it's a matter of when. The view is that a store of data needs to be refreshed. We support a lot of our customers in updating and supporting additional collection, or if they have the data themselves, the labeling to refresh those data assets and those features so that the models can be retrained and the performance continues over time. The second part is around ownership of the data. It varies customer by customer. Some customers where we are doing the data collection on their behalf, and determining on the arrangement with the customer, we have the ability to access that data either for internal uses or to on-sell. Other customers, we don't.
It is very specific at a case-by-case basis.
Yeah. Thank you. Thank you, Ryan. The second part of this question is about semi-supervised learning, and I think we've covered that. In the interest of time, we'll move to the third part, which is: in relation to your expense item services purchased data collection, do you expect long-term cost improvements here? I think that relates to Appen Mobile, for example.
Yeah. Definitely. We continue to invest in, like Mark said, getting more data from per crowd worker. Appen Mobile is one of the big areas that we're focusing on. One, to improve the unit economics of data collection. I think more importantly is to create a more feature-rich set of data that we're collecting from the field. There's a lot of exciting projects that we're working on in this space.
I wonder, Wilson, do you have anything to add on ways that we lower the cost of data collection?
Yeah. There's a few areas that I'll look into. One is really just to make the data collection work much easier for the current worker. They just pick up their phone, open the app, and then task get done. Super easy. By improved experience, we can drive down some costs. We are also trying to using machine learning in some of those data collection cases. I give you an example. Some data collection tasks, they need a worker to record some voice data and then transcribe that voice data. One technique we use there is really when they record the voice data, we're using our machine learning capability in the back end, kind of transcribe those data automatically, but we don't show them to the worker because we just want to make sure the worker also provide their input.
What we do there, using all the pre-transcribed data in the back end, we provide an auto-complete feature for the worker. When they transcribe that data, we will show them, "Is this what you are going to say?" Yes. That save them time to really finish the data collection tasks. That's the second area. Basically, besides the better experience, easy to use, we are also applying machine learning to help data collection tasks.
Yeah. Thanks, Wilson. Data collection is becoming more important because recall Ryan's example about the chatbot. The U.S. English language chatbot, there could be a lot of off-the-shelf data. When you get down to a much more specific use case, a different language, a more specialized area, you've got to collect a lot of data for that. The easier it is, and the more cost efficient it is to collect data, the more value it is for the customer. The final part of this question is, what metrics in relation to Figure Eight, the Figure Eight acquisition, what metrics are you using to measure its success? The key one is one we took everybody through yesterday, which is the growth in that new market figure.
The new market figure is revenue that we derive from the enterprise sector, from the government sector, from China, and also revenue that flows through our platform from our major customers. You can see from yesterday's presentation that that's growing nicely. None of that revenue would be available without that acquisition. The next question further down the page is. The machine learning doing the OCR and audio transcription our proprietary software or off-the-shelf? I think I'll throw this one straight to Wilson. You built it. You should know.
It is proprietary models. We started with off-the-shelf models. It didn't work for a few reasons. One is a lot of use cases we are handling are very specialized, and we need to find the right training data to support any use case. Those off-the-shelf model doesn't really work well for those use cases. We have to train our own model. That's one reason. Second reason is also our model is little bit different from the end model. You can see, for example, audio transcription. Our model not only need to transcribe the audio to text, we also need to flag this is the background noise, this is a different gender, this is some pause. We need to label all those different activities. The off-the-shelf model doesn't need to handle those. We have to train our own proprietary model.
It will be more difficult than off-the-shelf model, but it just give us an advantage. Only us can do this type of job
Yeah. Thanks, Wilson. Thanks for that question. I hope that provides a clear answer. The next question, do the Appen products integrate with client-side data pipelines and applications? Again, I'll throw this one straight to Wilson.
Yeah, that's a great question. The answer is absolutely yes. We provide a very rich set of API so the client can, using our API, set up job, upload data, download data, and just make our system part of their overall pipeline. That is used a lot. That's also a big focus for our product engineering team.
Thanks, Wilson. The next question is, will this webinar be uploaded for replay? The answer is yes. A recording of today's event will be available on our Investor Center on the Events and Presentations page early next week. The next question, one for Ryan. As use cases become more niche, will you have to develop more tools? Can AI models be transferred between use cases?
Yeah, good question on the tooling. Like what we said throughout this presentation, we believe we have a complete set of tools. New AI use cases and with specific data techniques continue to emerge. We will continue to invest in the breadth of our tooling. Some examples there which are live at the moment is different light spectrum. Non-visible light spectrum is a good example where there's a lot of interest in AI applications. Our tools support it today, but there's more that we could be doing in that space. An example of an emerging area that we'll be focusing on. On the transferability of models, there is a very common technique in AI model development, which is called transfer learning, and transfer learning is used pretty much across the board for every model. It only gets you a small part of the way, though.
There is still a lot of fine-tuning required, and that's really where the supervised learning comes in and the requirement of high-quality training data.
Yeah. Thanks. You may also recall during the presentation, Wilson mentioned that we're doing some work on satellite image data, which is not just another data type, but of course, there's a sort of a tiling nature of that data that requires certain tooling, et cetera. Okay, the next question. Regarding moving from model-centric to data-centric, how far into this move to data-centric are customers and, in particular, the big global customers? How much of a difference can this shift make to Appen's financial performance? Ryan?
I think that we're well into this shift. It does vary industry and customer by customer. I think that in the more forward AI companies, including our largest customers, they're probably further along that shift, whereas there might be a set of customers who are more used to using internal data for their AI development, and they focus more on the models, partially because of the realization that if they're able to use human-annotated data or different data sources, that will have a big unlock in terms of value for their AI models. I might also get Wilson to chime in on this question.
Yeah. I think the whole industry is moving more to data-centric AI. That basically applies to almost every company who's working AI. It just become a sort of common understanding or common sense in the machine learning community. Data just plays super critical role to the AI model performance. No matter if you are a professor or you are from a big company or a small company, I think all those data scientists, they just know the importance of data.
Thanks, Wilson. Thanks, Ryan. The next question. Does everyone need the quality of the data that Appen provides? Ryan.
Well, it depends on the quality of the model that they're looking to produce. If a customer wants to build a low-quality model, and that's sufficient for the needs of the application, then they may not need high-quality training data. If you want to build a model that has high quality across not just a small subset, but a broad subset of inputs, then you will need high-quality training data. Maybe another way to put it, if you want to build a high-quality model and you've got low-quality training data, it's kind of not possible. You need high-quality training data to build a high-quality model.
Wilson, maybe you have some examples of when you might choose to use low-quality data.
I do. I do. Actually, when I work my chatbot project with my daughter during weekend, I don't need Appen to provide service to me. It's good enough to have a toy. If you are really using AI to do any serious business, high quality data is a must.
I think that sums it up, folks. Building a chatbot with my daughter. I can tell you, Wilson's daughter is quite young. That's the extent of the knowledge there. Okay. This is, I believe, the last question we have. Does Appen market off-the-shelf data libraries for chatbots? What is the extent of the service that Appen contributes to chatbot setups?
We absolutely do off-the-shelf data. We have a rich catalog of data that we've collected. One of the big differences for us is that we have very large volumes of data collected. It's used by a lot of customers to kickstart their development of chatbots. Yeah, it's an important part of our product offering.
That's all the questions we have. I'd like to take this opportunity to thank all of you for attending our webinar today. I hope it was useful. If I can leave you with three thoughts from today's presentation. The first is that, the future of AI is very robust, and it absolutely relies on large volumes of high-quality training data. I think the examples we provide make that very clear. I think also, that we've provided a lot of information on the need for technology to provide those large volumes of high-quality data. The complexity of the use cases, the volumes required, dealing with millions of crowd workers, it's not possible without a good, strong product foundation. The third thing is, I hope you see that we're investing into this. We've made a lot of progress in this area.
There's lots to do, over time, we are building much more of a product first business. Over time, building more competitive advantage and resilience into our business as well. Thank you once again. Thank you to my co-presenters, Ryan and Wilson, for all of their input to this. I'm looking forward to the next time that we all meet. Thank you, and good day.