Mahjong Detection with Yolo Models (Draft)
date
Mar 25, 2024
slug
mahjong-detection-yolo
status
Published
tags
Learning
Tech
Mahjong
type
Post
URL
summary
Mahjong Detection with Yolo Models
Introduction
Recently I started an interesting endeavor to detect and recognize mahjong hands from a given image and calculate the score of the hand. This idea pops to me because in
Riichi Mahjong, scoring is very difficult and requires a lot of familarity with the rules. Especially for beginners, it is very hard to calculate the score of a hand because one has to be really adequate with the concept of Fu and Han to calculate the base point. Then from the base point one can finally derive the actual hand score.- When a non-dealer (ko, 子: child) goes out by self-draw
tsumo, the dealer (oya, 親: parent) pays the winner 2 × basic points, and the other two non-dealers pay the winner 1 × basic points.
- When a non-dealer goes out by
ron, the losing player pays the winner 4 × basic points.
- When the dealer goes out by
tsumo, all the three non-dealers pay the winner 2 × basic points.
- When the dealer goes out by
ron, the losing non-dealer pays the winner 6 × basic points.
The Han and Fu is different from hand to hand. Fu is particularly difficult to remember.
For curious readers, here’s a break down of Fu calculations.
Fu Calculations
Type | Description | fu |
ryanmen-machi (両門待ち) | the wait for either side of two simple tiles (e.g. 1 or 4 for 23; similarly, 34/45/56/67/78) made into a shuntsu | 0 fu |
kanchan-machi (嵌張待ち) | the wait for the middle of two simple tiles (e.g. 2 for 13; similarly, 24/35/46/57/68/79) made into a shuntsu | 2 fu(waiting for one kind of tile) |
penchan-machi (辺張待ち) | the wait for the inner side of outermost two simple tiles (i.e. 3 for 12 or 7 for 89) made into a shuntsu | ㅤ |
tanki-machi (単騎待ち) | the wait for another single tile (e.g. 1 for single 1, East for single East) made into a toitsu | ㅤ |
shanpon-machi (双碰待ち) | the wait for either tile in two toitsu (e.g. 1 or 2 for 1122) made into a kōtsu | 0 fu(fu of wait is 0 fu itself, however, fu of melds (2, 4 or 8 fu) for the new kōtsu is still valid.) |
simple tiles | terminal or honor tiles |
minkō (明刻),or min-kōtsu (明刻子)(open same three tiles meld, an open triplet) | 2 fu |
ankō (暗刻),or an-kōtsu (暗刻子)(closed same three tiles meld, a closed triplet) | 4 fu |
minkan (明槓),or min-kantsu (明槓子)(open same four tiles meld, an open kan) | 8 fu |
ankan (暗槓),or an-kantsu (暗槓子)(closed same four tiles meld, a closed kan) | 16 fu |
shuntsu (順子)(sequential meld, a run) | 0 fu |
toitsu (対子)(two tiles meld, the pair) | 2 fu for seat wind tiles, prevailing wind tiles or dragon tiles. 4 fu (or 2 fu in some rules) when the seat wind and prevailing wind match. 0 fu for other tiles |
Therefore, I started to think what if I can just take a picture with my phone and the app can tell me the score of the hand. That would be very convenient especially for beginners.
My deployed model to try out
- Some sample models I've developed are deployed at Roboflow. Feel free to try this out! You can take a photo or simply use your phones camera (note this is buggy on phones that it causes memory leak if you use it for more than 30 seconds)
Other mahjong apps I deployed
First Step: Struggle to train locally
- Found a random repo on github that geneartes mahjong tiles on random background.
- The quality of the images are not very good, and I trained Yolo v5 and v8 on it, the recognition result is rather terrible.
- Todo: talk about yolo v5 and v8
- Todo: talk about training on Apple silicon.
Second Step: Roboflow is the Saver
- Found really great training data on Roboflow with mahjong tiles.
- Training on local takes forever but now performs much better.
Third Step: Training on Roboflow
- Thanks to Eli from Roboflow team who geneoursly provided me with 20 free training credits.
- Hitting mAP, Precision, Recall over 99% but performs terribly on my own mahjong
Fourth Step: Training on my own mahjong
- Take pictures
- Found human lablellers for cheap on taobao. $0.01 per detection box.
- Training on Roboflow again
- Does it much better now.
- Add screenshots below
Step 5: Stretch Goals
- Mahjong Scoring, talk about different scoring packages I found
- Add inference code demo
- How to calulate SHanten etc.