Rogan Shannon

Hello, I’m Rogan and welcome back! I know, it’s been a while. Some new things in the background! Things are still changing in this room, I still have a lot to go. My Patrons already know this, but I’ve been busy with a new job. I’ve been figuring out what the process looks like, what my process looks like, and having work from that. It’s going good. That’s why I’ve been gone for a while. So that aside, let’s get into this video topic.

As you’ve seen from the title, today I’m talking about sign language to text recognition. I want to quickly talk about this, because I’ve been getting a few emails. I’m still really conflicted about this. Basically, I’ve been getting emails asking if the person can use my videos on this channel for training their AI to recognize what sign means this, translating into text. This is because all of my videos have captions included, so they can use that in parallel to figure out what a sign can mean in English. I have problems with that. For several reasons.

First, I’m sure this is often not their intention, but when they contact me asking that… Basically, they’re looking at it as sign, ASL, to English. Like that can be done just looking and translated. I’m like, hmm. That is pretty much interpreting, translating, from one language to another. You’re not trying to create something that will take sign and make it into sign writing, for example. This is not equivalent to vocally spoken English becoming English text. That’s because they already have equivalency, while sign doesn’t. And my captions don’t always exactly match what I say.

Second, my signing has a range. Sometimes it’s very English, sometimes it’s more ASL. It depends on the day, what my brain process is, and so on. So… Your trained AI will learn that changing range, it won’t be able to identify that difference. Also, it’s learning from my signing. It will learn how to translate *my* signing style, not other people’s. Some people are more strongly ASL than I am, some people are more English, or use SEE. All of that. Learning from me is a tiny step, and barely even scratches the surface of all that ASL is.

Third, for those who are watching, you can clearly see that I’m a white, masculine-presenting person. I’m also queer. So that means my signing style, for those who aren’t skilled in ASL, often shifts from masculine to feminine. I tend to stay around the middle. So because of that, it means your AI will learn… It won’t learn how to read someone who’s very feminine or very masculine. They’re very different in how they sign certain words.

Fourth, like I just said, I’m white. You will be training your AI to learn from a white person. I don’t sign the same as a Black man, or a Black woman. Or even queer Black people. I don’t sign the same. This runs into the same problem as facial recognition. Facial recognition is biased to white people, because white people designed it in the first place, developed it, trained it on white people’s faces. It often makes mistakes with Black people. I foresee this sign language recognition will be the same with Black and people of color.

Fifth, I know the reason why they contact me is because I have *many* videos, almost 400 now. So that’s a big resource for training an AI, and out there, there are limited resources for the training. I get that, but like I’ve said, I am one person. Training your AI on one person won’t lead to effective understanding.

Vocal speakers are diverse – high, deep, lively, monotone, accented, regional, fast, slow, and the list goes on. It’s the same in sign, the same. Signing fast, slow, feminine, masculine, with facial expressions. Expressions are important. All of that… That’s why even speech-to-text recognition struggles sometimes. If you have a strong accent, it doesn’t understand. Even your register, it can’t understand some registers. You will run into the same problems in sign, and maybe even more so, because sign, specifically ASL, is very varied in our language structure, our linguistic features, how we express, and all the different things. So training your AI on one person is not enough.

I just mentioned facial expressions. That’s one of the things that I feel like many of these people who want to do sign language to text recognition work forget about the expressions. They tend to focus on the hands, how the hands flex and move, focusing on that. Facial expressions are very important. For example, if I were to say: I’m happy! Or: I’m happy. It’s the same sign, happy, exactly the same, but the expression changes the whole meaning. Happy! I do mean I’m happy, but if I say happy. Obviously I’m not. That sign language recognition tends to be missing that and not include expressions.

All of what I’ve said so far, I haven’t even mentioned coding. Coding is the most tough part of this whole thing. I personally don’t understand code, I don’t understand how to do it. I may understand really basic things, but in-depth typing, no. But I do know people who do. And I know of one group with deaf people that are working on trying to develop a method of recognizing sign language. They’ve really struggled, it’s tough. They’ve been working on this for several years, and they got nowhere near. And they’re deaf, know the language, and so on. So when I have hearing people–hearing people!–contact me saying that they’re doing this as a project for my university, my last semester, or my thesis project, whatever. That caused conflict for me, because they’re hearing. They probably don’t really know sign. This is their temporary, their final project. They probably won’t continue it after graduating. So… I almost feel like, what’s the point? I’ve had several, three or four different people contact me asking for this. I already understand enough that I know it’s not possible, even with in-depth training and such. This is a long-term project, if you really want to invest in it. I did have one group from the UK reach out. They are aware that BSL and ASL are different. They’re working on both ASL and BSL, okay. And they are planning on having this be a long-term project, alright.

I’m not knowledgeable on coding, but I know enough to recognize that what these people are trying to do is very tough. Because they’re trying to teach an AI or program a full language that has varying ways of signing it, teach it that when the programmer themself may not be knowledgeable with this language. Then taking that language and translating to a full other language. I don’t know if our coding or tech capabilities are there yet. Or if they will ever be.

So this has happened often enough recently, that when people reach out saying that they want to do this… I’m inclined to ignore it or say thanks, but no thanks. Because–especially if it’s a university project. I’m like, you will not be able to do this in one semester, you will not.

So from now on, any time people contact me for a university project or thesis or whatever, I will be saying no. Because… You do it, then what’s next? Will you make that available to the deaf community? High chances are no. Because it didn’t succeed, or with what you did, but you graduated so there’s no need to continue. So where’s your exchange for my sharing? All of that was just me throwing out things, unscripted. So I’m probably missing a few things, I don’t know. Leave in the comments what you think about this and how you feel about people doing this, the concept of this. Whatever. Leave comments below.

If you want to support my content financially, I would really appreciate it if you joined my Patreon or made a one-time donation to my Ko-fi tip jar. Subscribe to my channel. Follow me on my socials – Facebook, Twitter, Instagram. Thanks for reading, see you next time.