Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/eve-ning/ppshift_ml
vsrg machine learning
https://github.com/eve-ning/ppshift_ml
Last synced: 13 days ago
JSON representation
vsrg machine learning
- Host: GitHub
- URL: https://github.com/eve-ning/ppshift_ml
- Owner: Eve-ning
- License: mit
- Created: 2019-02-23T16:56:10.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2019-03-31T04:36:22.000Z (over 5 years ago)
- Last Synced: 2024-10-27T18:24:52.069Z (2 months ago)
- Language: Python
- Homepage:
- Size: 54.1 MB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### PPSHIFT
ppshift is meant to be an overhaul on how analysts look at map difficulty, where I take a detour towards machine learning instead of pure algorithms
## Is it good
No, it's not good enough for production, however, feel free to reference this as learning from mistakes. Read **Issues** if you want to know why it failed.
## Research Steps in a nutshell
- Get replays & beatmaps
- Match the replays to beatmaps
- Parse Beatmaps, deconstructing them into small parameters
- Teach the machine that the pattern at a certain offset results in player X performing well/badly
- Score the results, and fine tune until it's the best
- Evaluate
## Issues
The issues are described in the PDF link under this, but I'll write them here too.
### The model isn't reliable
While I think the model is good to a certain degree, I don't think it's stable enough compared to a non-black box model. In other words, I don't think this model is production ready for a few reasons:
- The model doesn't calculate less popular key counts well. **The main reason for that is because there's too little ranked maps for uncommon keys**
- The model leans to the Top 50 population: **osu!API only provides Top 50 scores, which are very biased against leaderboard dominators**
- The model is bad at calculating easy maps as there's little to no data with regards to deviation in those
- The model is bad at matching situations where 2 maps are actually the same difficulty, but due to low interaction, the medians are vastly different.
- The model doesn't have enough results for bad deviation situations.
- The model seems to be reliably calculating how hard is it to get a very high score (i.e. Varying LN ends are hard to perfect, compared to hard jacks), however, most player's perception of difficulty rating doesn't fall in line with that.
- The model has very strong bias towards maps that are popular, e.g. Triumph \& Regret, which seems to be heavily affected by this issue **Similar issue with top 50, popular leaderboards will have less variance but high consistency, which makes it an outlier compared to other maps**
- Even if the model works, it's hard to explain what happens in the model since the neural network acts like a black-box.
### How to fine-tune the model
I think the main reason of this issue is the **Assumption of the Top 50**. In other words, we assumed that the Top 50 results' median can be treated like a player. However, this assumption proved to be very disastrous in terms of reliability.
- One way to prevent this is to look at specific player's replays and do the same modelling again, however, we will pivot from the player's perspective.
- We could have also, instead of looking at Top 50, we would focus on the a specific range of scores (which is painfully annoying to grab from the API), e.g. 750000 - 850000, in which will give us more information on bad deviations instead of extremely high scores with less bad deviations.
- We could have dropped the idea of looking at key counts over 7 due to the small amount of data we would have gotten from them. This has created a lot of noise in the model.
- We did not take into account in-depth reading difficulty (i.e. how hard is it to read a broken stair compared to a smooth one). If we could've calculated that one reliably, it would've smoothed out the LN map difficulties
- If we were able to grab Double Time results, it'll double our data, and increased the pool of harder maps, this makes it significantly easier for the machine to rate them.
#### [Research Steps (For my own reference)](https://github.com/Eve-ning/ppshift_ml/blob/master/research_tex/main.pdf)
# Results and [Plots](https://github.com/Eve-ning/ppshift_ml/tree/master/out/e50_50_96_48r2_1_24r2_1_s10)
These are the results.
Lower the score, the better.
The score is the mean of deviation between the original and actual graph
p(X) is the percentile
| Score | Metadata | p15 | p25 | p50 | p75 | p85 |
|-------|---------------------------------------------------------------------------------------------------------------------|------|------|-------|-------|-------|
| 1.15 | typeMARS - Triumph & Regret (Regret) <[ A v a l o n ]> | 6.28 | 6.41 | 6.70 | 7.29 | 8.00 |
| 1.80 | ginkiha - Anemoi (Daybreak) <[ A v a l o n ]> | 6.28 | 6.42 | 6.75 | 7.35 | 7.64 |
| 1.43 | ginkiha - Borealis (GRAVITY) <[ A v a l o n ]> | 6.26 | 6.44 | 6.85 | 7.50 | 7.91 |
| 1.22 | xi - over the top (Extra) <Zan -> | 6.68 | 6.80 | 7.04 | 7.55 | 7.84 |
| 0.97 | Hermit - Dysnomia (Tidek's 4K Monochrome) <-Kamikaze-> | 6.53 | 6.69 | 7.04 | 7.76 | 8.08 |
| 1.00 | Imperial Circus Dead Decadence - Uta (Tragic Ends) <pkk> | 6.52 | 6.71 | 7.05 | 7.55 | 7.85 |
| 2.15 | Utsu-P feat.Kagamine Rin - Tokyo Teddy Bear (SHD) <puxtu> | 6.53 | 6.72 | 7.19 | 7.92 | 8.38 |
| 1.43 | Team Grimoire - G1ll35 d3 R415 (Shana's Extra) <Allezard> | 6.80 | 6.99 | 7.47 | 8.12 | 8.54 |
| 1.19 | MAZARE - Mazare Party (Ash's 4K Extra) <Insp1r3> | 6.95 | 7.14 | 7.52 | 8.11 | 8.42 |
| 1.54 | Team Grimoire - G1ll35 d3 R415 (L45T C4LL) <Allezard> | 6.89 | 7.13 | 7.55 | 8.24 | 8.85 |
| 0.87 | Colorful Sounds Port - ETERNAL DRAIN (Black Another) <Wh1teh> | 6.90 | 7.09 | 7.55 | 7.96 | 8.09 |
| 1.18 | TeamGrimoire+amaneko - croiX (GRAVITY) <Takane6> | 6.80 | 6.98 | 7.63 | 8.35 | 8.94 |
| 1.44 | Helblinde - Memoria (Original Mix) (Memories) <Manwon> | 6.54 | 6.82 | 7.64 | 8.25 | 8.55 |
| 1.20 | Hypernite Industries - Speedcore 300 (Extra) <nowsmart> | 6.93 | 7.16 | 7.64 | 8.49 | 8.94 |
| 1.03 | kamome sano Electric Orchestra - HE4VEN ~Tengoku e Youkoso~ (HEAVENLY) <Fresh Chicken> | 6.84 | 7.06 | 7.71 | 8.61 | 9.23 |
| 1.10 | ZOGRAPHOS (Yu_Asahina+Yamajet) - Verse IV (INFINITE) <Ichigaki> | 6.83 | 7.11 | 7.77 | 8.46 | 8.98 |
| 1.45 | Team Grimoire - C18H27NO3(extend) (4K Capsaicin) <[Shana Lesus]> | 6.66 | 7.01 | 7.78 | 8.37 | 8.72 |
| 0.75 | Kobaryo - Dotabata Animation [feat. t+pazolite] (Ultra) <YaHao> | 6.83 | 7.24 | 7.86 | 8.56 | 8.92 |
| 0.99 | Camellia as "Bang Riot" - Blastix Riotz (Jinjin's INFINITE) <Fresh Chicken> | 7.15 | 7.46 | 7.88 | 8.65 | 9.14 |
| 1.29 | Kurokotei - Galaxy Collapse (Cataclysmic Hypernova) <Mat> | 6.91 | 7.36 | 7.98 | 8.77 | 9.35 |
| 1.28 | Team Grimoire - Sheriruth (Arzenvald's EXtinction) <DoNotMess> | 6.97 | 7.32 | 8.02 | 8.68 | 9.19 |
| 0.64 | Colorful Sounds Port - ETERNAL DRAIN (Eternal) <Wh1teh> | 6.97 | 7.37 | 8.02 | 9.21 | 9.82 |
| 0.67 | UNDEAD CORPORATION - The Empress scream off ver (Zenx's SHD) <TheZiemniax> | 7.16 | 7.35 | 8.07 | 8.73 | 9.00 |
| 1.02 | Chroma - I (Chicken AI's Maximum) <Lude> | 7.21 | 7.56 | 8.08 | 8.89 | 9.58 |
| 1.60 | Kaneko Chiharu - Zettai Reido (GRAVITY) <SpectorDG> | 7.21 | 7.52 | 8.10 | 8.87 | 9.38 |
| 2.57 | S-C-U feat. Qrispy Joybox - anemone (Kawa & Julie's 7K Extra) <Julie> | 7.52 | 7.74 | 8.10 | 8.46 | 8.77 |
| 0.98 | RoughSketch feat.Aikapin - Grimm (Kobaryo's FTN-Remix) (Grimmpossible) <CHARGE> | 6.83 | 7.37 | 8.12 | 8.93 | 9.58 |
| 1.10 | AAAA - Hoshi o Kakeru Adventure ~ we are forever friends! ~ [Long ver.] (Epochal Adventure) <Lude> | 7.24 | 7.57 | 8.23 | 9.01 | 9.43 |
| 1.12 | Camellia as "Bang Riot" - Blastix Riotz (Shirou's EXTRA) <Fresh Chicken> | 7.09 | 7.49 | 8.26 | 9.18 | 9.99 |
| 2.51 | Seiryu - Time to Air (Bruce's 7K Another) <Spy> | 7.66 | 7.96 | 8.45 | 9.01 | 9.42 |
| 1.10 | P*Light feat. mow*2 - Homeneko*Sensation (8K Lv.12) <victorica_db> | 7.57 | 7.87 | 8.51 | 9.43 | 10.34 |
| 1.99 | S-C-U feat. Qrispy Joybox - anemone (Kawa & Julie's 8K Extra) <Julie> | 7.43 | 7.86 | 8.54 | 9.25 | 9.69 |
| 1.05 | Kaneko Chiharu - iLLness LiLin (HEAVENLY) <Fresh Chicken> | 7.34 | 7.91 | 8.64 | 9.38 | 9.65 |
| 1.59 | Amane - Space Time (Amane Hardcore Remix) (7K Lv.40) <Lieselotte> | 8.07 | 8.28 | 8.73 | 9.43 | 10.00 |
| 0.97 | Ryu* - Yukizukiyo (victorica's 8K Lv.12) <Spy> | 8.04 | 8.25 | 8.73 | 10.50 | 11.20 |
| 2.42 | GReeeeN - Shinobi (7K Burst!) <ExUsagi> | 8.23 | 8.40 | 8.74 | 9.16 | 9.49 |
| 0.77 | xi - Double Helix (Nucleic) <Level 51> | 8.30 | 8.46 | 8.75 | 9.08 | 9.33 |
| 1.63 | Yooh - Ice Angel (Euphoria) <pporse> | 8.24 | 8.42 | 8.76 | 9.28 | 9.71 |
| 1.34 | Ogura Yui - Baby Sweet Berry Love (3R2 Remix) (Sweetness Overload!) <KawaEE> | 7.91 | 8.21 | 8.83 | 9.53 | 9.89 |
| 1.64 | Ryu* Vs. Sota - Go Beyond!! (7K Black Another) <DE-CADE> | 8.16 | 8.35 | 8.87 | 9.53 | 10.20 |
| 1.63 | void - Valedict (Black Another) <pwhk> | 8.18 | 8.42 | 8.91 | 9.44 | 9.93 |
| 1.55 | Cres - End Time (Epilogue) <PyaKura> | 8.23 | 8.48 | 8.92 | 9.33 | 9.68 |
| 1.10 | High Speed Music Team Sharpnel - M.A.M.A. (ExTra) <arcwinolivirus> | 7.80 | 8.33 | 8.92 | 9.41 | 9.67 |
| 0.72 | XeoN - Xeus (LV.12 Leggendaria) <My Angel Azusa> | 8.35 | 8.54 | 8.97 | 9.39 | 9.66 |
| 1.20 | D(ABE3) - MANIERA (Collab Another) <iJinjin> | 8.39 | 8.55 | 8.97 | 9.55 | 9.90 |
| 1.20 | orangentle / Yu_Asahina - HAELEQUIN (Extended ver.) (Jinjin's Dissociative 7K Identity Disorder) <Fullerene-> | 8.32 | 8.53 | 8.98 | 9.54 | 9.93 |
| 1.25 | penoreri - Everlasting Message (GRAVITY) <Julie> | 8.23 | 8.49 | 9.01 | 9.53 | 9.78 |
| 1.68 | Gekikara Mania - Deublithick (Blocko's Extra) <_underjoy> | 8.20 | 8.48 | 9.02 | 9.88 | 10.37 |
| 1.30 | Cres - End Time (Afterword) <PyaKura> | 8.38 | 8.64 | 9.02 | 9.62 | 10.15 |
| 1.03 | Sisterz - Inverse World (Universe) <Kyousuke-> | 8.50 | 8.63 | 9.02 | 9.69 | 10.06 |
| 1.34 | w_tre respect for AT&HU - Schur's Theorem (Black Another) <Nivrad00> | 8.56 | 8.74 | 9.05 | 9.76 | 10.70 |
| 1.63 | Hermit - Dysnomia (8K Melodie de tristesse) <-Kamikaze-> | 8.39 | 8.61 | 9.07 | 9.80 | 10.67 |
| 1.37 | Sharlo & yealina - Kakushigoto (Rumi's 7K MX) <Sharlo> | 8.18 | 8.46 | 9.07 | 9.76 | 10.19 |
| 1.66 | Shoujo - Reminiscing (Black Another) <- R e b a -> | 8.40 | 8.66 | 9.08 | 9.68 | 10.11 |
| 3.66 | xi - Happy End of the World (9K Collab Insane) <Blocko> | 7.88 | 8.26 | 9.08 | 10.31 | 11.10 |
| 1.25 | BlackY - Harpuia (INFINITE) <Ichigaki> | 8.42 | 8.58 | 9.09 | 10.04 | 10.53 |
| 1.59 | penoreri - Preserved Valkyria (GRAVITY) <Takane6> | 8.10 | 8.43 | 9.09 | 9.79 | 10.21 |
| 1.10 | Yooh - LiFE Garden (Extended Mix) (Eutopia) <Wonki> | 8.28 | 8.48 | 9.10 | 10.53 | 11.25 |
| 1.33 | kamome sano - archive::zip (GRAVITY) <Fresh Chicken> | 8.29 | 8.52 | 9.10 | 9.86 | 10.34 |
| 1.12 | Soleily - Violet Soul (D's Extra) <pocket-Gao> | 8.41 | 8.64 | 9.12 | 9.77 | 10.46 |
| 1.67 | LeaF - LeaF Style Super*Shredder (SC) <nowsmart> | 8.22 | 8.61 | 9.12 | 9.64 | 10.05 |
| 1.61 | xi - Quietus Ray (Heaven) <Kuo Kyoka> | 8.36 | 8.61 | 9.14 | 10.04 | 10.73 |
| 1.51 | DJ TOTTO VS TOTTO - Vajra (Emiria's 8K Leggendaria) <biemote> | 8.37 | 8.61 | 9.15 | 10.10 | 10.84 |
| 1.95 | Hermit - Dysnomia (D's 8K Another) <-Kamikaze-> | 8.39 | 8.62 | 9.16 | 10.24 | 11.04 |
| 1.58 | MiddleIsland - Achromat (7K Black Another) <iJinjin> | 8.52 | 8.72 | 9.18 | 9.95 | 10.41 |
| 1.19 | gmtn. (witch's slave) - furioso melodia (7K furioso maratona) <Harbyter> | 8.19 | 8.52 | 9.20 | 10.18 | 10.88 |
| 1.58 | DJ SHARPNEL - Touch the angel (K-ON!!) <pporse> | 8.36 | 8.64 | 9.22 | 9.94 | 10.61 |
| 0.81 | MAZARE - Mazare Party (Ash's 6K Extra) <Insp1r3> | 8.28 | 8.59 | 9.23 | 9.95 | 10.30 |
| 1.87 | DJ Mashiro - Prismatic Lollipops (Lv.20) <Shinzo-> | 8.13 | 8.41 | 9.28 | 10.24 | 10.76 |
| 0.95 | Soleily - Renatus (Another) <Multiple Creators> | 8.30 | 8.57 | 9.28 | 10.09 | 10.70 |
| 1.21 | BlackY - FLOWER -SPRING Long VER.- (As Flowers Bloom And Fall) <Kawawa> | 8.31 | 8.57 | 9.29 | 10.39 | 11.31 |
| 1.81 | Nekomata Master - Sennen no Kotowari (GRAVITY) <Critical_Star> | 8.53 | 8.81 | 9.30 | 10.36 | 11.73 |
| 0.74 | UNDEAD CORPORATION - The Empress scream off ver (Jepetski's Empress) <TheZiemniax> | 8.12 | 8.53 | 9.31 | 9.78 | 10.02 |
| 1.37 | Team Grimoire - C18H27NO3(extend) (7K Axities) <[Shana Lesus]> | 8.46 | 8.71 | 9.36 | 10.15 | 10.59 |
| 1.69 | Camellia as "Bang Riot" - Blastix Riotz (GRAVITY) <Fresh Chicken> | 8.15 | 8.64 | 9.38 | 10.21 | 10.78 |
| 1.65 | Kucchi- - Remilia ~Kyuuketsuki no Tame no Kyousoukyoku (Rebanada's Black Another) <Garalulu> | 8.52 | 8.75 | 9.39 | 10.26 | 10.84 |
| 1.80 | Kucchi- - Remilia ~Kyuuketsuki no Tame no Kyousoukyoku (Concerto Cruento) <Garalulu> | 8.64 | 8.89 | 9.51 | 10.29 | 10.81 |
| 1.26 | Halozy - Kanshou no Matenrou (Eternity) <Rumia-> | 8.37 | 8.73 | 9.52 | 10.38 | 10.82 |
| 1.63 | xi - Garyou Tensei (Million's 7K MX) <LNP-> | 8.36 | 8.70 | 9.54 | 10.46 | 10.88 |
| 1.29 | Mitsuyoshi Takenobu no Ani - Amphisbaena (Fallen Heaven) <Kawawa> | 8.63 | 8.96 | 9.54 | 10.16 | 10.53 |
| 1.59 | Chroma - Hoshi ga Furanai Machi (Meteor Shower // pporse's 7K) <Wonki> | 8.31 | 8.58 | 9.56 | 11.46 | 12.82 |
| 1.72 | LeaF - Alice in Misanthrope -Ensei Alice- (Alice in Wonderland) <Kawawa> | 8.40 | 8.70 | 9.58 | 10.52 | 10.97 |
| 1.33 | Umeboshi Chazuke - Panic! Pop'n! Picnic! (Picnic!) <jakads> | 8.69 | 8.93 | 9.58 | 10.38 | 10.77 |
| 1.10 | xi - Ascension to Heaven (Elysium) <Jinjin> | 8.82 | 9.05 | 9.65 | 10.36 | 10.73 |
| 1.57 | Ayane - Endless Tears... (CrossOver) <richardfeder> | 8.78 | 9.14 | 9.65 | 10.40 | 10.98 |
| 1.07 | CLIMAX of MAXX 360 - PARANOiA Revolution (Expert) <Tornspirit> | 8.98 | 9.23 | 9.65 | 10.14 | 10.46 |
| 1.22 | Jun Kuroda + AAAA - Ultimate Fate (The Apocalypse) <Kawawa> | 8.59 | 8.83 | 9.67 | 10.70 | 11.22 |
| 1.35 | Warak - REANIMATE (Reanimated obj. Kamikaze) <Rayz141> | 8.58 | 9.00 | 9.69 | 10.70 | 11.46 |
| 1.29 | Ice - citanLu (pporse's Lunatic) <Lass> | 8.72 | 9.01 | 9.69 | 10.62 | 11.20 |
| 1.28 | Yooh - FIRE FIRE -DARK BLAZE REMIX- (GRAVITY) <Shinzo-> | 8.83 | 9.07 | 9.72 | 10.53 | 10.94 |
| 1.55 | xi - Happy End of the World (ajee's 5K Armageddon) <Blocko> | 8.74 | 9.10 | 9.75 | 10.45 | 10.92 |
| 1.74 | m108 - * Crow Solace * (KK's 7K Extra) <Critical_Star> | 8.60 | 8.93 | 9.79 | 10.95 | 11.49 |
| 1.22 | Cardboard Box - The Limit Does Not Exist (Extra) <iJinjin> | 9.05 | 9.25 | 9.79 | 10.33 | 10.61 |
| 1.63 | DragonForce - Symphony of the Night (Rhapsody of the Warriors // 6K) <ajeemaniz> | 8.65 | 9.00 | 9.86 | 11.18 | 12.12 |
| 1.97 | xi - F (X) <pwhk> | 8.69 | 8.92 | 9.88 | 11.65 | 12.98 |
| 1.89 | DJ Genki VS Camellia feat. moimoi - YELL! (Solitude) <Turrim> | 8.57 | 8.93 | 9.91 | 11.48 | 12.39 |
| 0.94 | Gekikara Mania - Deublithick (Ultra) <_underjoy> | 8.43 | 8.77 | 9.93 | 11.32 | 12.35 |
| 1.14 | KRUX - Illusion of Inflict (7K Kruxified) <Reba> | 8.58 | 8.93 | 9.98 | 11.80 | 12.26 |
| 1.22 | Toromaru - Enigma (GRAVITY) <Kawawa> | 8.86 | 9.20 | 10.02 | 10.96 | 11.57 |
| 1.46 | sakuzyo - AXION (Ex-ray) <ljqandylee> | 8.81 | 9.13 | 10.02 | 10.88 | 11.41 |
| 1.53 | xi - Garyou Tensei (Million's 7K SC) <LNP-> | 9.07 | 9.45 | 10.06 | 11.03 | 11.70 |
| 1.92 | DJ Genki vs. Camellia feat. moimoi - Sunshine (7K Luminescence) <Turrim> | 8.79 | 9.16 | 10.06 | 12.07 | 14.42 |