FIR 123: Nutritional AI That YOU CAN EAT !!

Financial Investing Radio

In this episode, I have a guest who discusses nutritional AI that you can eat.

Grant Everybody, this is Grant Larsen. Welcome to another episode of ClickAI radio. Wow. Today I have in the house with me. Mr. Daniel DeMillard, he is the CTO of FoodSpace, and I am just honored for the opportunity to meet him and to get to hear his story and what he and his organization has been doing with AI. Alright, let's jump into this. Okay, so, Daniel, first of all, welcome.

Daniel Thanks for having me, Grant. Great to be here.

Grant It is, it is really great to have you here and to have another another patriots in AI, right things that you guys are doing, as I got to know you. And before we started this recording, I was fascinated with the kind of work that you guys are doing and where you focus your use and application of AI. But before we get too deep into that, would you step back in time and tell us? Who's Daniel, where do you come from? And what is it that's brought you to this point to be CTO applying AI in such a cool way?

Daniel Yep, so I actually studied economics and finance in school and came across an online class by Andrew Yang on Coursera. And at that point, I just absolutely fell in love with machine learning and artificial intelligence. And I was like, wow, this is absolutely what I'm wanting. Yeah, with my life. So, you know, started studying a lot. Got a job eventually at IBM Watson, and worked at a small company doing what type classification, then I was doing some consulting on the side, where I actually got connected with iosshare, Nike, the CEO of foods, but at the time, it was a lunchbox. And they were developing a consumer facing app that, you know, they were trying to pair people with recipes. And you could set up a diet profile for yourself, and instantly order things online through instacart, based on recipes that you find. And I initially got engaged with them, building a wine pairing recommendation application, where, given a certain recipe will automatically recommend a wine pairing that would go well, really well I

Grant ...need that certain kind of food, you're like, Hey, this is the right, right,

Daniel ...we're gonna wanna medium red wine with that right or a very sweet dessert wine.

Grant Until it this where lunchbox started, they were focusing on solving that problem.

Daniel They were focused on solving the one stop shop for keeping all of your recipes together, ordering food very easily. And then also being able to manage your diets, and allergens, and just make making sure all of that was really seamless. And they also had a great mission of trying to mitigate food waste, so that they could recommend given all of the stuff that you have in your fridge that would normally sit there and forget about they could recommend a recipe and for you to make of the things that you've already selected. And so

Grant all right, very good. All right. So so then you bumped into them, and you started to work with them. Tell us a little bit about the transition over to food space. How did the vision change?

Daniel Yeah, it was actually pretty serendipitous, and rather abrupt. So back in 2019, io was a part of grocery shop, which is a big conference in the CPG space and food industry. And he's trying to pitch this idea of lunchbox to brands and retailers to get them to sign up for it. And they all have basically kept telling them the same thing of like, hey, it's a great idea. It's super interesting. I would love to but what you're asking for with all of these like dietary profiles, and The information necessary to build these types of recommendation algorithms. It's we just don't have it, we don't have ingredients and digitized nutrition labels. And, you know, we have the very minimum. So given that that information is like, Okay, well, we've got to take a step back. So he calls me up. And he says, you know, hey, Daniel, would it be possible given a bunch of computer or a bunch of e commerce images, to extract the product information from there and actually read it from those images. And I had done a little bit of stuff with computer vision by that point, using, like pre trained, convolutional neural net models on image net, and then using transfer learning to identify key regions. And so I stopped and thought about it. And I was like, yeah, you know, I think the problem is tractable enough. And the technology's in such place, now that we can absolutely solve this problem. And so yeah, two years later, we now have a model that can classify things that matter of seconds. And we're gonna extract that product information and seconds with 99.7% accuracy 99.7. So if I bring I can assume to you, and you can see the label on it, you can figure out what this is, you can identify it. Exactly. So we'll extract things like brand name, product, name, net weight, the ingredients, the nutrition label, any certifications, such as whether or not it's recycled, kosher, non GMO, vegan, all of that awesome stuff, including marketing claims, like low sodium or, you know, contains less sugar, those those types of things, well, pull out all of the relevant information from the product label. And we'll read it in the same way that a person would read it. A lot of other products that do something similar or are entirely based on the universal Product Code code, the UPC barcode on the back, where they're basically just looking things up from a database, a database might have inaccurate information, it might be out of date, so might have been accurate at one point, it might have been transmis, transcribe with whoever transcribed it in the first place, we're gonna read that label and not image, the same way that a human would label it so that it's, we're going to the ground truth.

Grant So PC based, you're actually extracting the actual text, you're figuring out what this is, and then the semantics of it. What does that mean? Oh, it means this ingredient, this so much in the, you know, in terms of the amount of the ingredient, and so forth.

Daniel Exactly. And then we will also derive new information from that information that we've extracted, such as whether or not it's going to be had certain allergens. If it contains peanuts, we're going to let you know if it has a peanut allergy. We're also going to determine whether or not it conforms to different diets. So I'm a vegetarian. And I'm constantly reading labels for obscure things like whether or not my cheese contains rennet in it, which is a animal derived enzyme. So we will read all of that for you, and then derive whether or not it's going to correspond with with your diet.

Grant So can you talk about some of the use cases around this? So are you targeting b2b scenarios? Are you doing b2c? Is that something that I as an end consumer comes in interacts with it, let's say through through my cell phone? or How are you? How are people going to consume this this cool platform?

Daniel Yeah, absolutely. So right now, we are primarily focused on optimizing the e commerce experience. So if you're on Walmart, or Albertsons or target, and you are using your favorite grocery delivery app, or you're going in to do pickup, all of that purchasing decisions are happening upfront on the e commerce website. And at the very least, we want to make sure that that information is present and accurate so that you can make the decisions yourself whether or not at the very least, you can see that ingredients label and search to see if that rennet in gradient is there, or if you're trying to, you know bulk up make sure that it has enough protein or has low sugar, low fat, whatever your dietary needs are. We want to make sure that information is there. But we really want to enable a more optimized ecommerce experiences where you know in your little left side of the toolbar, you can select vegetarian or pescatarian or low sodium diet or a South Beach diet, or I'm allergic to shellfish, and automatically only be shown products that correspond to your dietary needs. So we really think that optimizing that e commerce experience and the search is where we can have the largest impact right away.

Grant So So some of the health profile of the person intersects with this, is it coming off? Like, I don't know, like, like the fitness app? Or is it coming off of other sort of apps? And then are capturing that health information? How do you integrate with that?

Daniel Yeah, absolutely. So right now, we are basically providing the data to the retailer so that they can make those optimizations. But certainly being able to integrate with, you know, My Fitness Pal, or Weight Watchers would help optimize these experiences. And we are in discussions with those types of companies as well to improve their databases. So that you aren't, you don't have to manually type in all of that information on your fitness app, you can basically just look it up in the database, and it's accurate and recent. One problem that we've seen is that 30% or so of data is of grocery products are updated every year. So anyone I think use one of these apps has the experience of typing in their information, finding out finding the correct product, but it's a little bit outdated, the calories are a little off the nutrient profiles a little bit off, we're gonna make sure that it's updated. And in the right place.

Grant That's interesting. So you talked about accuracy, the model accuracy for AI? And I think you should say 97%. Right. 99.7.

Daniel Yeah, we are absolutely religious about that is, wow, you know, that is the problem that we're trying to solve. Right now, if you look up any product, on, say, a large, very large e commerce website, like walmart.com, there is a somewhere between 40 and 70% chance that there is at least one mistake on that website, regarding just the ingredients and nutrition information. So if you're trying to base you know, your health profile on that, it's it's an inaccurate, so we are just absolutely religious about getting every single piece of information. Correct, at least as so far as it corresponds to the product images.

Grant So is this is this just for humans? Or is this also food for animals and pets? And how does this work?

Daniel Yep, so we've definitely just, we started with humans, we are expanding to pet food and being able to build attributes around that two things like wet versus dry pet food, whether it's for a large size dog, or a small size dog. And all of those attributes we're hoping will also assist in that product search and discoverability so that you're not being shown a dog food, that's, you know, too too big for your small small dogs. Right.

Grant Right. Right. Okay. All right, that makes sense. And then in terms of what we're talking about, who it's relevant to terms, your current market, so it's for humans, obviously, animals in the future. But as we think about the humans, this English base, is it other languages, Spanish or Mandarin or others? Where are you in terms of multilingual?

Daniel Yep. So, you know, food is, I think, sacred to everyone everywhere. And as we move from this, in store grocery experience, where you're, you have the product in front of you, you can pick it up, and you can read the label to an e commerce experience, where somebody might just be dropping that off to your doorstep, and you don't see the product until it's there. We really think it's important that we have as larger reach as possible. So we definitely are working on expanding our algorithms to apply to different regulatory regulatory environments. You know, Europe has, I think, 12 allergens, whereas the united states currently has nine, and they just added sesame, to their allergens. They also have different nutrition labels and different information that they require to be on those. And then in addition to that, the different languages that are actually present there, and all of that obviously presents different technological issues, custom models for each of those markets, but really what we've spent a lot of time Building and working on is creating models that can quickly adapt to these new domains and building a really robust training pipeline. So that basically all we have to do is collect more data, instill a little bit of domain expertise, where we have to learn a little bit about that new market or that language. But after that, we can adapt our models very quickly to that new.

Grant You know, I just have to ask, given that I love the AI piece of this, as well as just the benefit that you're bringing to human family. I mean, that's, that's huge. When I think about the AI portion of this, I think, how, how was building that model? I mean, how you have a lot of cans in your food storage now. I mean, how much? How many boxes of Cheerios did you buy? I mean, that's amazing. How did you get through all that? That's just that, right? There is a big challenge, right? To get through enough instances?

Daniel Yeah, um, I Oh, and Dan, my business partners, they spent a lot of time getting kicked out of grocery stores, because they kept picking out prod products and taking pictures with their phones. And so they were kicked out of a few grocery stores, I think they learned to, you know, explain what, what they were doing their first after a little bit, but certainly a lot of time, taking pictures of your entire pantry. Going around the grocery store, just pick it up as many random things as possible. That's creative.

Grant Yeah, that's, that's really great. If you have any particular challenges in terms of the kinds of food and other words, some things don't have labels, right. So certainly asparagus typically right or decent, things like that. So how do you deal with that?

Daniel Yeah, absolutely. So currently, we only support branded foods. So it does need to have that product label. But it's interesting that you should mention certain types of foods, we were doing a analysis an audit of our accuracy. And we were noticing that a certain product category, yogurt, in particular, was creating a lot of issues for us and was very low accuracy. And it turned out that the curvature of the yogurt container, and then the fact that it kind of tapers down, creating a lot of issues for OCR model, where the text is kind of getting bunched up at the edges of that, you know, yogurt container. So we actually had to like build a specific model just to handle those types of containers. So certainly, you know, a lot of our time and effort has been focused on the corner cases in those weird scenarios where that are particularly difficult. The like, very simple run of the mill cereal box, where it's a nice rectangular box and the nutrition labels very prominent. And it's a very usual format that's easier to solve that most of our time has been focused on these weird one offs, like these tobert, tapered yogurt containers.

Grant So so let me think about because I love the, again, this problem that you're solving and how it benefits people and their dietary needs. When I think about how people can consume this, what's the way that they will be able to interact with this standard? And what's the state of what space is doing today? Is it? Is it out there ready to be used? Or Where are you guys?

Daniel Yeah, we're currently working with brands to get their data to the retailers and some retailers are a little bit further along than others and optimizing the, you know, experience for you where you can set up those dietary profiles for yourself and only be shown the products that correspond to your values, or do you only want organic food or you have a gluten intolerance, only being charged on those foods that correspond with those values or dietary needs to just getting the product information out there to the retailers in the first place. We're also working with some initial engagements with smart appliance manufacturers, things like smart fridges and smart micro microwaves, where you can simply scan the product, either using the barcode or just the front of the product and instantly have your oven or your microwave set the time timer or the temperature for you to cook that product for you. Additionally, being able to do things like recipe planning based on the products that you have in your fridge, being able to order products from I'm a retailer directly using the feature on your fridge that is based on your dietary profiles and just you never needed to get on your computer. And you could just order, you know, your gluten free pizza directly from your smart fridge that is linked to a product database with information that we're providing, we really think that more and more people are going more and more of our purchasing Our food is going to happen in this virtual digitized space, whether that's through your computer, your smart fridge, and the more that information is available, the more that we can build a more customized experience, and really make shopping easier as well, so that you aren't ever being shown products that don't correspond to your dietary dietary needs or your values. You know, even being able to set timers and things for microwave, it might sound trivial, but it really should make the entire cooking, cooking experience that easier for you.

Grant You know, I certainly could benefit from walking up to the fridge and say, what are the possibilities of what I can create from what's in there, my wife will do that she's got that AI model already in her head, but I don't have that model, same set of food and go, there's nothing in there. And then she can craft you know, miracles out of it.

Daniel So yeah, I'm the exact same way. And, you know, you could you can set user profiles for everyone in your family and say, Hey, you know, I'm a pescetarian. And my daughter's gluten intolerant and my son really only it's organic food, and being able to mix and match all of those constraints, we can figure out what recipe and you know, what to eat for dinner, right?

Grant And so it sounds like, like, like, we've done that, that South Beach diet multiple times. Sounds like you know, you can literally walk up to your, to your fridge at some point and say, Hey, what is it that I can make that is in compliance with the South Beach diet?

Daniel Exactly. And then things like, you know, macros counting, like calorie counting, and counting how much protein that you're consuming, would be a lot easier using if all of this information is digitized, and you're interacting with it in a smart fridge type environment where it can track what you're picking up and making. So I think entering information into one of those calorie counting apps is often a pain and I think, a limitation for a lot of people. So anything that can mitigate some of that barrier to getting healthier and keeping track of what we're putting in our bodies, to me is very much welcome.

Grant So we've talked about the art of the possibilities around this right? What is it that this can bring the people that dramatically influences and impacts their health? What do you see in terms of the downsides? What hurdles or challenges? What could get in the way of either people adopting this or getting value from it? What what concerns or challenges do you see there?

Daniel Yeah, so some of the things that we've seen in the industry about the difficulty to use this type of data is, every retailer kind of has a different format for how they ask for data. Some retailers want the units and the nutrition and the value to be separate. So if you have seven grams for protein, sometimes they want us separate key for seven and a separate one for brands. They might call things different. Some people might think call things, UPC, other ones call it barcode. Other ones call it product ID. So that's some of the work in transit translating the data mapping or the data model to each of those retailers can be a major bottleneck for a brand say wants to get their data to Walmart to Albertsons to target. And they basically had to look at these like massive Excel spreadsheets, but like 70 columns or 150 columns, and manually copy that data over and it's a huge pain. And that that is one of the major reasons why only the largest of brands have the resources to get their data digitized in the first place. So what we do is, you know, we're going to first extract that information for you automatically from your images. You don't have to hire a team of people to do that extraction in the first place, where we've also built these mappings for the top 10 retailers where we can automatically syndicate and get the data in the format that they want to see. Whether that's directly through an API, and just automatically updating your information through an API, fortunately, some of the grocery industry isn't quite as forward thinking. So a lot of updates are just made through Excel spreadsheets. But we'll create that Excel spreadsheet for you. So that it's basically just a matter of sending that over an email. And I think that should mitigate a vast majority of the bottlenecks currently faced in the industry. Because some of the, I could just imagine being a brand manager and be like, Alright, well, here's my data mapping. But then there's these close lists for Walmart, where, you know, I'm supposed to put in a certain beef cut type for this product. And doing that, for every single one of my 150 500 products, that is going to be a huge ass.

Grant Yeah, it has said that. It's one of the things that drew me to this. And when you and I were first talking about this recently, which was, I feel like the work that you're doing is not only scales to the larger brands, but also it's pulling out all this information that makes it available, even the small to medium business space as well. And so feels very scalable, therefore approachable to benefit a lot of people, lots of different scenarios.

Daniel Yeah, absolutely. And we try to make things as easy as possible to get integrated with our system. So, you know, our simplest use case, if you already have data and a list of URLs for your product, you're going to send us over a CSV with your URL links and the product IDs associated with those. And we'll download those images for you and process them through the system. And now you can download it and whatever data format you want, you know, CSV or JSON, or an Excel or in target specific taxonomy format, or Walmart's or Albertsons. Or you can upload it through a, you know, drag and drop upload portal where you can just drop, drag a folder of your product images into that upload portal, interact with an API, or even give us access to your put them up on an FTP server and point us to it and we will download the images there. So it's really trying to make things as simple as possible. So that whatever your tech stack is, and whatever the size of your organization is, we can help you get up and running as quickly as possible.

Grant Lots of integration strategies for if that's powerful. That's awesome. Alright, so let me ask you, if the for the people that are listening to this, where are you going to direct them to what's what's, where are you going to invite and where do they go find out more about this?

Daniel Yep, so a FoodSpaceTech.com is the place to find all of the information.

Grant Okay, FoodSpaceTech.com; Awesome. That's great.

Daniel Okay, we actually just released a brand new website. So it looks great. And you can look at it now.

Grant It looks awesome. I've asked you a ton of questions. What questions Haven't I asked you? What would you like to share that I haven't prompted you?

Daniel Yeah. So I think that can be skepticism and the world of AI. And, you know, whether or not we can do what we say that we can do. And we are, again, just absolutely religious about product accuracy. And I think it's good for anyone who knows a lot about AI to know that AI can only take you so far and the machine learning is only going to get you so far. So we've spent a very large amount of our time building a very sophisticated human in the loop process, were really trying to figure out where the ML system is doing well and can be trusted, versus when a human needs to come in and take the reins and make a more educated more critical thinking decision about things with things like building known rules between the nutrition label. So calories is a very direct calculation from total carbohydrates and protein and total fat. So we can basically just cut check to see if that calculations done well. We can cross check our nutrition information against our ingredients where we've actually built models where we can predict certain nutrition elements based on the ingredients. You know, we know that a cookie were the first ingredient might be butter or sugar is going to have more fat content than something where the first ingredient is carrots. So if anything falls outside of those ranges, we can alert it and say, Hey, something's gone off the rails here, we should make sure if human takes a look at it. For certain container types, we know we are struggle a little bit more things like that yogurt container. So instead of relying on the ML models that work most of the time, but not all the time, we can just flag that certain product type for review by a human just to get another check on it. But we really think that the just to solve a problem, at least in the near term, using AI involves humans in that in the loop and being able to really distinguish that the easy cases, the happy path that I like to call from, hey, we've seen a new domain, you know, maybe it's a it, both English and Spanish is written on the back. So our models are getting a little bit confused. Let's flag that for review.

Grant Yep. Yeah, I really appreciate the qualification around AI. I tend to prefer to think of it as augmented intelligence than artificial intelligence, I feel Yeah, I feel like that's the state of where it really is. There's so many things out there, like, Oh, you know, ai robots and Terminator that give a real misperception. But, but today, this stuff around deep fake, right, is really starting to become, you know, a bit of a challenge, right, in terms of creating even less trust around this. So it's a real misuse, if you will, of that. So in this particular case, this is obviously real, honorable use of AI itself. But the whole if we can keep people's perceptions to this is to augment your thinking process, right? Your cognitive behavior. So even though it's coming to you and saying, This is what you could or should eat, or this is what makes sense, you know, from a nutritional value, you still own the responsibility yourself, right to end up saying, Yeah, this is something I'm gonna do, I'm not passing that off, you know, to the AI model and say, do all the thinking for me, right?

Daniel Oh, absolutely. And I could not agree more, you know, we are just providing the information to you. But it still requires that critical thinking and decision about your own values and your own goals to make the final decision about what you're going to put into your body. Right. We're just trying to make that easier. Make that whole decision process simpler. Yeah. Powerful.

Grant They're very cool. Okay. All right. Any last comments, before we wrap up here, Daniel? No, it was a great to be here at Brampton. Thank you very much for having me. Yeah. Thanks for taking the time and for sharing this cool platform that you've put together everyone, go take a look at food space tech.com. Thank you for joining and until next time, go get some nutrition.

Thank you for joining Grant on ClickAI Radio. Don't forget to subscribe and leave feedback. And remember to download your FREE eBook visit ClickAIRadio.com now.