I've been learning Japanese for just over a year now, and its a community that might have the highest concentration of amazing free learning apps with sentence mining capabilities, many of which do things very similarly to what you're building. Have you tried some of these?
asbplayer
Manatan
Yomine
Anki Miner
AnimeCards
Nagare
These are just for video, there are other similar solutions for games, manga and ebooks, and some of them are all in one
I've only heard of asbplayer and Anki miner before, the others I haven't. It's a great space tbh, people are building all the time. For me specifically it's about trying to serve all languages to the best of my ability, alot of the ones I've come across only focus on Japanese or a small selection of languages. Also I think there's scope to improve UX and reducing friction for non tech savvy people
Migaku is far from perfect, but it's fairly polished and supports a ton of languages. Honestly, I wish it just focused on a couple of languages and did them well instead of trying to support as many as possible, though.
In my experience, apps for learning japanese is a hugely saturated niche because the venn diagram of people learning japanese and people who like building software is a circle
I think it would be interesting to try this with another language though. Picking something like Russian or something obscure like Turkish could be a more undeserved segment
Thanks! There is a whole like subsection of language learning called immersion based learning. Fictional sources can be OK just depends on how specialised and unique the vocabulary is.
Yeah that is my suspicion too, I am considering adding an option to transcribe system audio too so it could hook into whatever video or audio is playing on the computer. That would be quite a big change so would like feedback first! hahah
Interesting. I've been working on the same problem (you need stuff at a kid's level, but not aimed at kids, if you are interested in the content then it will be easier to learn): https://tailoredreaders.com/samples
I think I like your solution better than mine, which is basically generating children's books from user prompts.
When you say you generate stories from the films, are you summarising / simplifying the narrative of scenes in the film (to down level the language), or creating something new, just with the vocabulary the film uses?
It's a similar methodology, my problem with things like this is that they do give you comprehensible input, but it's very boring. I don't engage at all. The idea of Katarineko is to create comprehensible input from content that interests you. Say you want to watch Spirited Away but it's a bit too advanced for you at the moment, we create stories at your level for Spirited Away to teach you the core words you need to achieve your target comprehension level.
This is my main issue with tools like these.
Automated subtitles are inherently inaccurate, because much of language depends on context which you cannot get just from a brief snippet of something. This wouldn't be much of an issue if the tool was targeted at people dedicated to creating subtitles, but if you're trying to learn a new language, how are you supposed to know when the output is wrong or not?
Yeah that is a very good point, it’s something I thought about a lot when positioning the app. Essentially there are 2 mitigations to this, one is that it’s not really ideal for beginners and mainly targeting intermediate learners who can recognise or at least question errors to a certain extent but don’t want to create the entire subtitle file themselves. There is an inline editor for them to fix the errors.
The other thing is the speech recognition models are very sensitive to the audio. If the audio quality is good then the model would do a good job, I’m not sure how much experience you have with Whisper Large but it is very capable on normal speech. The issues arise when there are many competing sounds overriding each other
I build the exact same system, but for (mandarin, simplified) Chinese which i'm learning.
Wanted to create such a system for a while, but it's a lot of hooking up the correct libs and making a UI etc. never found the energy.
Thanks to 'ai' I was able to do this finally, think more people needed this push :-)
It's web-based and can be self-hosted in a docker.
Nice! Yeah I think the functionality is never very complicated, I'm more so interested the UX and trying to add features that reduce friction and slowly build an app that can be stable and relied upon.
I've been somewhat studying Japanese for... quite a few years now. I have yet to find the "ideal" method, but I know it must be mobile (I'm continually migrating away from sitting in front of my laptop) and something I can randomly get in and out of at any time (anything taking up dedicated batches of time eventually gets dropped or barely touched).
Yeah I can relate haha, that is one of the things I questioned, if a desktop app was the best format for this app. I do feel the majority of people are drifting more towards mobile tbh so may have to make a companion app lol
> I'd find a video I wanted to learn from, hear a useful sentence, and then realise that turning that sentence into something I could study later was both time consuming and draining at times.
How does not study a sentence? You mean memorizing it later?
I love having auto-generated subtitles available, but I am still skeptical that we can call them "accurate" yet. For many languages the models are quite weak, and even for English and German on YouTube I see errors all the time.
Even for English (which surely has the best support) anything outside of urban American accents is still incredibly weak - you can pretty much guarantee very noticeable mistakes on even highly intelligible mainstream urban British/Irish/Aussie/Kiwi/SA accents. Stronger regional / rural accents & dialects usually render generated subtitles entirely useless.
Definitely a domain a million miles from being solved if even English is this bad.
Yeah I agree the Youtube ones are not very good, I am not sure what they have implemented behind the scenes for it. However this uses the Whisper Large model which is quite a bit better than the Youtube auto generated ones. Only catch is you need to have the video on disk and wait a bit longer lol
Youtube is very bad, but I have honestly not yet seen anyone implement this well on any platform, which indicates to me that it's not a solved problem. Very open to being proven wrong here though - do you have examples of Whisper Large in use in a large consumable sample?
It's not fully solved, there are still errors but its improved a lot the past few years especially compared to Youtube. The best approximation is the Whisper models atm as far as I have seen. There is a write up here on hugging face which talks more about the models and the benchmarks against it.
Yeah that is a fair point, it can be replicated but not many people want to replicate an app and would rather pay for it. The functionality is not hard to replicate but the UX side of things and general support and improvements over time is what people have paid for. There is a lifetime option also - yearly was about offering choices.
Same goes for Dropbox - you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CSV on the mounted filesystem.
Yet most don’t. Because they don’t wanna spend time on it and they want a tested and proven software. Heck why even use Mac/linux/windows when you can just vibecode your own OS.
During my English learning process, I often come across some slang expressions used in videos. Could this application handle them properly and mark them out?
Potentially, it’s about the audio quality more than anything. If there are not multiple voices talking over each other or music playing over it then it has a good chance
That’s fair enough! I am curious tho, are there any issues you face while using the app that could be handled better? Or any features that would make it easier to use?
I tried Language Reactor a couple of years ago and bounced off of it because of its absolutely atrocious support for traditional character Chinese. Its page literally says there is none on Netflix despite most their Chinese content being from Taiwan.
Migaku is flawed, but better. This tool seems like a much less polished version of Migaku or Lexirise with just a local media player and no ability to use it on YouTube, Netflix, Bilibili, etc.
This is really cool idea. But don't you think taking material and saving specific sentences may add unnecessary mental pressure to the process of learning which is said is ideal when the student is in a playful mood?
True, I would say though from my personal experience there is a serious study element inherent to learning languages to a high level too. So it depends on what the user wants, there are gamified apps like Duolingo but the efficacy of it has been questioned over the years
Yes! It supports many languages, you’ve made me realise a gap in the messaging in terms of communicating that value.
It can support up to 99 but the subtitle generation quality tends to decreases with more obscure languages as they wouldn’t have been in the training data
I've been learning Japanese for just over a year now, and its a community that might have the highest concentration of amazing free learning apps with sentence mining capabilities, many of which do things very similarly to what you're building. Have you tried some of these?
asbplayer Manatan Yomine Anki Miner AnimeCards Nagare
These are just for video, there are other similar solutions for games, manga and ebooks, and some of them are all in one
I've only heard of asbplayer and Anki miner before, the others I haven't. It's a great space tbh, people are building all the time. For me specifically it's about trying to serve all languages to the best of my ability, alot of the ones I've come across only focus on Japanese or a small selection of languages. Also I think there's scope to improve UX and reducing friction for non tech savvy people
Migaku is far from perfect, but it's fairly polished and supports a ton of languages. Honestly, I wish it just focused on a couple of languages and did them well instead of trying to support as many as possible, though.
even for "tech savvy" people, it's great to have an easier to use UX, or at least have the option to
In my experience, apps for learning japanese is a hugely saturated niche because the venn diagram of people learning japanese and people who like building software is a circle
Yeah, all in one apps are now getting built every other week
I think it would be interesting to try this with another language though. Picking something like Russian or something obscure like Turkish could be a more undeserved segment
Cool idea!
While I'm not so sure about learning a language with content from fictional sources, that should at least make it more relevant for the learners.
And you can use it on podcasts too, where people talk normally.
The biggest hurdle is probably that users need to get hold of the media so they can use it as input.
Thanks! There is a whole like subsection of language learning called immersion based learning. Fictional sources can be OK just depends on how specialised and unique the vocabulary is.
Yeah that is my suspicion too, I am considering adding an option to transcribe system audio too so it could hook into whatever video or audio is playing on the computer. That would be quite a big change so would like feedback first! hahah
I've been building something similar but targeted at bringing content that interests you to your level: https://katarineko.com.
I've been learning Japanese as well and the content at my level is boring while the content I'm interested in is a wall.
Interesting. I've been working on the same problem (you need stuff at a kid's level, but not aimed at kids, if you are interested in the content then it will be easier to learn): https://tailoredreaders.com/samples
I think I like your solution better than mine, which is basically generating children's books from user prompts.
When you say you generate stories from the films, are you summarising / simplifying the narrative of scenes in the film (to down level the language), or creating something new, just with the vocabulary the film uses?
Ahh nice, hope it goes well!
How does this product compare to the methodology of dreaming.com?
It's a similar methodology, my problem with things like this is that they do give you comprehensible input, but it's very boring. I don't engage at all. The idea of Katarineko is to create comprehensible input from content that interests you. Say you want to watch Spirited Away but it's a bit too advanced for you at the moment, we create stories at your level for Spirited Away to teach you the core words you need to achieve your target comprehension level.
>Drop any file. Get accurate subtitles.
This is my main issue with tools like these. Automated subtitles are inherently inaccurate, because much of language depends on context which you cannot get just from a brief snippet of something. This wouldn't be much of an issue if the tool was targeted at people dedicated to creating subtitles, but if you're trying to learn a new language, how are you supposed to know when the output is wrong or not?
Yeah that is a very good point, it’s something I thought about a lot when positioning the app. Essentially there are 2 mitigations to this, one is that it’s not really ideal for beginners and mainly targeting intermediate learners who can recognise or at least question errors to a certain extent but don’t want to create the entire subtitle file themselves. There is an inline editor for them to fix the errors.
The other thing is the speech recognition models are very sensitive to the audio. If the audio quality is good then the model would do a good job, I’m not sure how much experience you have with Whisper Large but it is very capable on normal speech. The issues arise when there are many competing sounds overriding each other
I build the exact same system, but for (mandarin, simplified) Chinese which i'm learning. Wanted to create such a system for a while, but it's a lot of hooking up the correct libs and making a UI etc. never found the energy. Thanks to 'ai' I was able to do this finally, think more people needed this push :-) It's web-based and can be self-hosted in a docker.
Nice! Yeah I think the functionality is never very complicated, I'm more so interested the UX and trying to add features that reduce friction and slowly build an app that can be stable and relied upon.
Care to share with a fellow learner? :)
i published it to github today, its still very alpha but it does the job https://github.com/enimatek-nl/shuoshuo
Link please?
I've been somewhat studying Japanese for... quite a few years now. I have yet to find the "ideal" method, but I know it must be mobile (I'm continually migrating away from sitting in front of my laptop) and something I can randomly get in and out of at any time (anything taking up dedicated batches of time eventually gets dropped or barely touched).
Yeah I can relate haha, that is one of the things I questioned, if a desktop app was the best format for this app. I do feel the majority of people are drifting more towards mobile tbh so may have to make a companion app lol
> I'd find a video I wanted to learn from, hear a useful sentence, and then realise that turning that sentence into something I could study later was both time consuming and draining at times.
How does not study a sentence? You mean memorizing it later?
By study I mean adding to Anki for example as a flashcard
I love having auto-generated subtitles available, but I am still skeptical that we can call them "accurate" yet. For many languages the models are quite weak, and even for English and German on YouTube I see errors all the time.
Even for English (which surely has the best support) anything outside of urban American accents is still incredibly weak - you can pretty much guarantee very noticeable mistakes on even highly intelligible mainstream urban British/Irish/Aussie/Kiwi/SA accents. Stronger regional / rural accents & dialects usually render generated subtitles entirely useless.
Definitely a domain a million miles from being solved if even English is this bad.
Yeah I agree the Youtube ones are not very good, I am not sure what they have implemented behind the scenes for it. However this uses the Whisper Large model which is quite a bit better than the Youtube auto generated ones. Only catch is you need to have the video on disk and wait a bit longer lol
Youtube is very bad, but I have honestly not yet seen anyone implement this well on any platform, which indicates to me that it's not a solved problem. Very open to being proven wrong here though - do you have examples of Whisper Large in use in a large consumable sample?
It's not fully solved, there are still errors but its improved a lot the past few years especially compared to Youtube. The best approximation is the Whisper models atm as far as I have seen. There is a write up here on hugging face which talks more about the models and the benchmarks against it.
https://huggingface.co/openai/whisper-large-v3
In the age of LLMs, AI assisted programing and vibe coding can't this all be replicated?
I'm confused why is this a yearly subscription or paid at all?
This tool might as well be open source to be brutally honest.
Yeah that is a fair point, it can be replicated but not many people want to replicate an app and would rather pay for it. The functionality is not hard to replicate but the UX side of things and general support and improvements over time is what people have paid for. There is a lifetime option also - yearly was about offering choices.
Same goes for Dropbox - you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CSV on the mounted filesystem.
I understand your jest, but you can really vibecode your own Dropbox in this day and age.
Yet most don’t. Because they don’t wanna spend time on it and they want a tested and proven software. Heck why even use Mac/linux/windows when you can just vibecode your own OS.
I'm pretty confident that Grok CLI 4.6 could one shot this.
During my English learning process, I often come across some slang expressions used in videos. Could this application handle them properly and mark them out?
Potentially, it’s about the audio quality more than anything. If there are not multiple voices talking over each other or music playing over it then it has a good chance
Been a pretty happy user of https://www.languagereactor.com
Not seeing a reason to switch atm tbh
That’s fair enough! I am curious tho, are there any issues you face while using the app that could be handled better? Or any features that would make it easier to use?
ASR in LR is down often from my experience.
I tried Language Reactor a couple of years ago and bounced off of it because of its absolutely atrocious support for traditional character Chinese. Its page literally says there is none on Netflix despite most their Chinese content being from Taiwan.
Migaku is flawed, but better. This tool seems like a much less polished version of Migaku or Lexirise with just a local media player and no ability to use it on YouTube, Netflix, Bilibili, etc.
I can't even login to languagereactor.
This is really cool idea. But don't you think taking material and saving specific sentences may add unnecessary mental pressure to the process of learning which is said is ideal when the student is in a playful mood?
True, I would say though from my personal experience there is a serious study element inherent to learning languages to a high level too. So it depends on what the user wants, there are gamified apps like Duolingo but the efficacy of it has been questioned over the years
does it support Spanish? what other languages does it support?
Yes! It supports many languages, you’ve made me realise a gap in the messaging in terms of communicating that value.
It can support up to 99 but the subtitle generation quality tends to decreases with more obscure languages as they wouldn’t have been in the training data
Super cool, great job!
Thanks, I appreciate it!
account required != free to try.
Technically its still free since there is no card requirement but we can agree to disagree here
yeah, this comment was such a nitpick, hn! You know!