Marani
Marani is an open research project building Georgian speech datasets, evaluation tools and models for recognition and synthesis. We document the whole process, from raw audio to evaluated models, so it is reproducible and helps others build speech technology for low-resource languages.
github.com/wtfnukee/maraniupdated 19:27 UTC · commit 1081a1f · 4 jobs running
Audio collected10,606 h38,831 recordings · 60 sources
Cleaned3,060 h+190 h/dayclips from 3,263 h raw · 5,847 of 38,831 recordings
Usable speech2,064 h+921 h this week2,064 h with transcripts · 1,294 h TTS-grade
Spent$270$0.13 per usable hourModal $254 · Vast $16 · $9 Vast credit left
01
Data
selection Nov 03 17:37Raw audio processed
3,263 of 10,606 h raw · 5,847 / 38,831 recordings
02
Growth
Cleaned and usable
cleanedusable speech
Where cleaned audio comes from
hours of clips by source
03
Pipeline
CollectDownload recordings from public sourcesin progress
25 of 27 source recipes acquired · 38,831 recordings
CleanCut each recording into clean speech clipsin progress
5,847 of 38,831 recordings
LabelTranscribe clips and score their qualityin progress
5,400 of 38,831 recordings
Build datasetsDrop duplicates and test-set overlap, split, selectrunningupdated Nov 03 17:37
Train & evaluateTrain models, score them on held-out testsin progress29 trained models
50 jobs today, 15 failed
04
Models
Best so far: NVIDIA model fine-tuned on Marani, strictest labels gets 16.9% of words wrong on held-out Parliament sittings, against 21.5% for NVIDIA's pretrained model (both with the language model).
| model | Parliament test | FLEURS test | W&B | ||
|---|---|---|---|---|---|
| no LM | with LM | no LM | with LM | ||
| NVIDIA FastConformer, pretrained pretrained | 29.1% | 21.5% | 18.3% | 13.5% | |
| NVIDIA model fine-tuned on Marani, more labels asr-omni-010 | 24.6% | — | 20.2% | — | run |
| 27M model from scratch, plus human-written text asr-scratch-010-human | 24.6% | — | 37.6% | — | run |
| NVIDIA model fine-tuned on Marani, strictest labels asr-omni-005 | 24.9% | 16.9% | 19.3% | 13.2% | run |
- Numbers are word error rates: the share of words recognised wrongly. Lower is better; the best in each column is bold.
- Parliament test: 15 sittings, 2,622 clips, 5.5 hours held out from training, scored against their official transcripts.
- FLEURS test: 979 clips from a public benchmark of read sentences. NVIDIA trained its model on FLEURS training data.
- With LM: decoding helped by a Georgian text model built from Wikipedia, web text and transcripts of other Parliament sittings.
- Shown: the best 3 of 11 models with test scores (29 trained in all).
Human review
Transcripts checked
not started · 180 clips ready
Clips listened to
36 of 200 clips
05
Sources
All audio is used privately for research. Public releases include only sources that allow redistribution.
Can go into a public release
yes 4.5 h · no 7,586 h · not cleared 3,016 h
| source | recordings | raw h | public release |
|---|---|---|---|
| Georgian Public Broadcaster archives and Teleskola | 15,054 | 6,286 | no |
| Georgian YouTube channels (TV, radio, education, podcasts) | 1,947 | 919 | no |
| Parliament of Georgia plenary sittings | 307 | 711 | not cleared |
| Radio Palitra programme archive | 2,033 | 553 | not cleared |
| მამა თეოდორე გიგნაძის ქადაგებები | 1,083 | 525 | not cleared |
55 more sources · 1,612 h
| Adjara TV news videos | 16,500 | 477 | not cleared |
| Fight Hop | 301 | 268 | not cleared |
| მეცნიერება ყველასათვის | 123 | 164 | no |
| შემდეგი პოდკასტი | 101 | 123 | no |
| საუბრები ნაპოლეონზე | 88 | 70.1 | no |
| პირადი საქმე | 64 | 58.8 | not cleared |
| Duty Free Talks • გულახდილი პოდკასტი | 83 | 58.5 | not cleared |
| წითელი კედელი | 34 | 49.0 | not cleared |
| პრინციპში ყველაფერზე | 47 | 44.0 | not cleared |
| დირექცია/Film Office | 45 | 25.5 | not cleared |
| Internet Archive audio items with Georgian language metadata | 331 | 23.9 | not cleared |
| Vegan Georgia - ვეგანური საქართველო | 28 | 23.3 | not cleared |
| ბალანსი | 20 | 18.7 | not cleared |
| Marketer.ge | 22 | 17.6 | no |
| სახელოსნო • sakhelosno | 13 | 15.1 | not cleared |
| სოციალური სამართლიანობის ცენტრის პოდკასტები | 25 | 14.4 | not cleared |
| GZAZE • გზაზე | 22 | 13.7 | not cleared |
| პოპკასტი • Popcast | 13 | 13.4 | not cleared |
| პოლიგონი | 11 | 13.0 | not cleared |
| მამიკოს ზღაპრები | ზღაპრები ქართულად | @mamikoszgaprebi | 94 | 11.2 | not cleared |
| AWR ქართული - ქრისტეს იგავები [Georgian poc] | 37 | 11.0 | not cleared |
| პოდკასტი რელიგიაზე | 22 | 9.7 | not cleared |
| სიმშვიდის ლაბორატორია | 9 | 9.5 | not cleared |
| ახრაშუნე სლათუნა a Queer Talk Show | 11 | 9.2 | not cleared |
| ოქტოპოდი | 13 | 9.1 | not cleared |
| შენი აუდიო სივრცე | 43 | 6.1 | not cleared |
| რომან აფაქიძის პოდკასტი | 7 | 5.0 | not cleared |
| ღია პოდკასტი — ცვლილებები თანაბარი უფლებებისათვის | 8 | 4.3 | not cleared |
| დეპეშა პოდკასტი | 5 | 4.1 | not cleared |
| ირინა ტორონჯაძის პირისპირ | 3 | 3.8 | not cleared |
| დელფის ორშაბათის პოდკასტი | 10 | 3.7 | not cleared |
| იაფფასიანი პოდკასტი | 9 | 3.4 | no |
| ორჯერ ორი x Orjer Ori | 52 | 3.2 | not cleared |
| ცარიელი აზრები / Empty Thoughts | 34 | 3.1 | not cleared |
| Archive.org Georgian commentary collection | 17 | 2.9 | not cleared |
| თეას პოდკასტი | 6 | 2.7 | not cleared |
| Peace Corps Georgian course audio | 33 | 2.7 | yes |
| პროსპექტის პოდკასტი - როგორ გავზარდოთ გაყიდვები | 3 | 2.3 | no |
| ეს არ არის პოდკასტი | 6 | 2.1 | not cleared |
| რავა ხარ, შჩმ? | 7 | 2.1 | not cleared |
| mikheil online | 4 | 2.0 | not cleared |
| Peace Corps Georgian course video | 21 | 1.8 | yes |
| Megrelian Podcast - მარგალური პოდკასტი | 6 | 1.7 | not cleared |
| მარკეტინგის სკოლა | 5 | 1.5 | not cleared |
| Tabula - გრძლად საკითხავი | 2 | 0.8 | not cleared |
| იაბ იუმ/Yab Yum | 2 | 0.5 | not cleared |
| Constitutional Court of Georgia public hearings | 1 | 0.5 | no |
| ucla-phonetics-georgian | 8 | 0.5 | not cleared |
| psychologist.ge | 3 | 0.2 | no |
| მარტივი პოდკასტი | 2 | 0.1 | not cleared |
| ფანტას მაგორიას პოდკასტი | 3 | 0.1 | not cleared |
| Georgian shortwave radio airchecks (Dave J. Valko, Internet Archive) | 2 | 0.1 | not cleared |
| ბურთი და მოედანი | 1 | 0.1 | not cleared |
| ქებაი და დიდებაი | 1 | 0.0 | not cleared |
| Wikimedia Commons Georgian pronunciations (Lingua Libre) | 46 | 0.0 | not cleared |
06
Spend
since 2026-10-01| work | where | share | cost |
|---|---|---|---|
| Training models | Modal | $86.18 | |
| Cleaning and labelling | Modal | $79.41 | |
| No run record (before 11 October) | Modal | $27.58 | |
| Collecting audio | Modal | $26.24 | |
| Evaluation and research | Modal | $23.57 | |
| Rented GPUs | Vast | $15.73 | |
| Building datasets | Modal | $11.00 |
Total $269.73 · Modal $254.01 · Vast $15.73 ($9.27 credit left) · $0.13 per usable hour, as billed by Modal and Vast.
07
Latest changes
Nov 2 – Nov 8
- acquireRerun recipes in campaigns and add Court and deeper YouTube recipes
- evalFreeze an 80-sitting Parliament stenogram test set
- deployServe Dagster on gamarjoba.dev behind Access
- launchFetch Parliament stenograms on the Mac fallback
- deployServe the dashboard and Label Studio on gamarjoba.dev
- evalCap held-out clips per recording; restart the tunnel when its routes change
- dashboardTrace spend to work, compare models by decoder, plain labels
- studiesScore with the 6 GiB parliament LM by default
- + 6 fixes and refactors
Oct 26 – Nov 1
- studiesBuild a never-built decoder LM inside the study
- lmBinarize on a Vast host inside the language_model step
- + 19 fixes and refactors
Operator detail
| asset | status | partitions | progress | clip h | gates ok / failed | worker time | last |
|---|---|---|---|---|---|---|---|
| source_candidates acquisition | saved outputs | 27 / 27 | — | 0 / 0 | 2.175 h | Nov 03 19:23 | |
| acquired_recordings acquisition | saved outputs | 25 / 27 | — | 2 / 0 | 11.908 h | Nov 03 19:14 | |
| clips preparation | saved outputs | 5,847 / 38,831 | 3,059.794 h | 1 / 0 | 7.973 h | Nov 03 05:03 | |
| transcripts enrichment | saved outputs | 5,400 / 38,831 | — | 2 / 0 | 33.862 h | Nov 03 05:03 | |
| quality enrichment | saved outputs | 5,411 / 38,831 | — | 2 / 0 | 24.885 h | Nov 03 05:03 | |
| music enrichment | saved outputs | 5,441 / 38,831 | — | 2 / 0 | 31.444 h | Nov 03 05:03 | |
| speakers enrichment | saved outputs | 5,347 / 38,831 | — | 2 / 0 | 34.532 h | Nov 03 05:03 | |
| fingerprints preparation | saved outputs | 5,847 / 38,831 | — | 1 / 0 | 0 | Nov 03 05:03 | |
| eval_references evaluation | in progress | 4 / 5 | — | 1 / 0 | 1.657 h | Nov 03 00:02 | |
| contamination corpus | complete | 1 / 1 | — | 0 / 0 | 10.635 h | Nov 03 18:58 | |
| duplicates corpus | complete | 1 / 1 | — | 0 / 0 | 3.388 h | Nov 03 16:37 | |
| splits corpus | complete | 1 / 1 | — | 1 / 0 | 1.201 h | Nov 03 19:22 | |
| selection corpus | complete | 1 / 1 | — | 3 / 0 | 9.980 h | Nov 03 17:37 | |
| package corpus | not run | 0 / 1 | — | 0 / 0 | — | — | |
| checkpoint training | in progress | 29 / 37 | — | 0 / 0 | 0 | Oct 31 19:44 | |
| checkpoint_eval evaluation | complete | 1 / 1 | — | 0 / 0 | 0 | Oct 30 10:40 | |
| contamination_references corpus | complete | 1 / 1 | — | 0 / 0 | 8.6 s | Oct 16 14:41 | |
| curation operations | complete | 1 / 1 | — | 0 / 0 | 135.7 s | Oct 27 18:26 | |
| data_audit operations | complete | 1 / 1 | — | 0 / 0 | 1,060.2 s | Oct 27 18:09 | |
| dataset corpus | complete | 10 / 10 | — | 0 / 0 | 6.780 h | Nov 03 04:21 | |
| eval_api_opinions evaluation | in progress | 3 / 5 | — | 0 / 0 | 1,136.3 s | Oct 29 14:51 | |
| eval_hypotheses evaluation | in progress | 1 / 5 | — | 0 / 0 | 126.5 s | Oct 19 14:12 | |
| eval_opinions evaluation | in progress | 2 / 5 | — | 0 / 0 | 2,963.1 s | Oct 19 14:17 | |
| evaluation_score evaluation | not run | 0 / 1 | — | 0 / 0 | — | — | |
| experiment_comparison evaluation | complete | 1 / 1 | — | 0 / 0 | 173.1 s | Oct 23 20:55 | |
| experiment_plan training | in progress | 27 / 37 | — | 0 / 0 | 4.775 h | Oct 31 17:42 | |
| found_alignment acquisition | complete | 1 / 1 | — | 0 / 0 | 4.666 h | Nov 02 21:24 | |
| found_text acquisition | complete | 1 / 1 | — | 1 / 0 | 1.091 h | Nov 02 17:55 | |
| label_contamination corpus | complete | 1 / 1 | — | 0 / 0 | 2.240 h | Nov 03 19:11 | |
| labels labels | in progress | 3,094 / 38,831 | — | 0 / 0 | 23.286 h | Nov 01 11:42 | |
| language_model evaluation | complete | 3 / 3 | — | 0 / 0 | 3.196 h | Nov 01 22:19 | |
| models acquisition | not run | 0 / 1 | — | 0 / 0 | — | — | |
| recording_catalogue operations | complete | 1 / 1 | — | 0 / 0 | 65.4 s | Nov 03 04:03 | |
| retention operations | complete | 1 / 1 | — | 0 / 0 | 74.8 s | Nov 02 04:32 | |
| review review | complete | 1 / 1 | — | 0 / 0 | 1,631.4 s | Nov 03 19:01 | |
| review_sample review | complete | 1 / 1 | — | 1 / 0 | 672.9 s | Oct 15 15:23 | |
| second_opinion research | complete | 4 / 4 | — | 0 / 0 | 2,130.7 s | Oct 28 12:27 | |
| source_corrections operations | complete | 1 / 1 | — | 0 / 0 | 367.2 s | Oct 28 08:50 | |
| study_report research | in progress | 17 / 21 | — | 0 / 0 | 3.109 h | Nov 01 22:59 |
Throughput and run overhead
| step / worker | provider | runs | audio h | seconds / audio h | GPU step seconds / audio h | USD / audio h | missing hours / cost runs |
|---|---|---|---|---|---|---|---|
| acquired_recordings | modal | 17 | 0.000 | unavailable | unavailable | unavailable | 17 / 0 |
| acquired_recordings | queue | 18 | 0.000 | unavailable | unavailable | unavailable | 18 / 0 |
| contamination | modal | 75 | 77,486.832 | 0.494 | unavailable | 0.000 | 0 / 0 |
| contamination_references | modal | 1 | 0.000 | unavailable | unavailable | unavailable | 1 / 0 |
| curation | local | 3 | 0.000 | unavailable | unavailable | unavailable | 3 / 0 |
| data_audit | local | 8 | 0.000 | unavailable | unavailable | unavailable | 8 / 0 |
| dataset | modal | 78 | 0.000 | unavailable | unavailable | unavailable | 78 / 0 |
| duplicates | modal | 63 | 57,641.995 | 0.212 | unavailable | 0.000 | 0 / 0 |
| enrichment | modal | 275 | 264.570 | 144.509 | 144.509 | 0.056 | 12 / 185 |
| enrichment | queue | 144 | 586.310 | 557.962 | 115.701 | unavailable | 0 / 144 |
| enrichment | vast | 2 | 22.054 | 78.664 | 78.664 | 0.008 | 0 / 0 |
| enrichment | yandex | 65 | 50.899 | 1,150.740 | unavailable | unavailable | 0 / 65 |
| enrichment/music | modal | 275 | 264.570 | 57.930 | unavailable | unavailable | 12 / 263 |
| enrichment/music | queue | 144 | 586.310 | 137.587 | unavailable | unavailable | 0 / 144 |
| enrichment/music | vast | 2 | 22.054 | 55.449 | unavailable | unavailable | 0 / 2 |
| enrichment/music | yandex | 65 | 50.899 | 313.600 | unavailable | unavailable | 0 / 65 |
| enrichment/quality | modal | 275 | 264.570 | 34.607 | unavailable | unavailable | 12 / 263 |
| enrichment/quality | queue | 144 | 586.310 | 113.381 | unavailable | unavailable | 0 / 144 |
| enrichment/quality | vast | 2 | 22.054 | 28.584 | unavailable | unavailable | 0 / 2 |
| enrichment/quality | yandex | 65 | 50.899 | 261.315 | unavailable | unavailable | 0 / 65 |
| enrichment/speakers | modal | 275 | 264.570 | 44.893 | unavailable | unavailable | 12 / 263 |
| enrichment/speakers | queue | 144 | 586.310 | 162.356 | unavailable | unavailable | 0 / 144 |
| enrichment/speakers | vast | 2 | 22.054 | 16.278 | unavailable | unavailable | 0 / 2 |
| enrichment/speakers | yandex | 65 | 50.899 | 331.584 | unavailable | unavailable | 0 / 65 |
| enrichment/transcripts | modal | 275 | 264.570 | 54.802 | unavailable | unavailable | 12 / 263 |
| enrichment/transcripts | queue | 144 | 586.310 | 158.158 | unavailable | unavailable | 0 / 144 |
| enrichment/transcripts | vast | 2 | 22.054 | 31.663 | unavailable | unavailable | 0 / 2 |
| enrichment/transcripts | yandex | 65 | 50.899 | 274.318 | unavailable | unavailable | 0 / 65 |
| eval_api_opinions | modal | 3 | 0.000 | unavailable | unavailable | unavailable | 3 / 0 |
| eval_hypotheses | modal | 1 | 0.000 | unavailable | unavailable | unavailable | 1 / 0 |
| eval_opinions | modal | 3 | 0.000 | unavailable | unavailable | unavailable | 3 / 0 |
| eval_references | modal | 5 | 0.000 | unavailable | unavailable | unavailable | 5 / 0 |
| experiment_comparison | modal | 4 | 0.000 | unavailable | unavailable | unavailable | 4 / 0 |
| experiment_plan | modal | 33 | 0.000 | unavailable | unavailable | unavailable | 33 / 0 |
| found_alignment | modal | 9 | 0.000 | unavailable | unavailable | unavailable | 9 / 0 |
| found_text | modal | 4 | 0.000 | unavailable | unavailable | unavailable | 4 / 0 |
| found_text | queue | 1 | 0.000 | unavailable | unavailable | unavailable | 1 / 0 |
| label_contamination | modal | 36 | 0.000 | unavailable | unavailable | unavailable | 36 / 0 |
| labels | modal | 36 | 37.562 | 218.463 | 218.463 | 0.120 | 0 / 4 |
| labels | queue | 919 | 56.173 | 60.933 | 60.297 | unavailable | 886 / 33 |
| language_model | modal | 5 | 0.000 | unavailable | unavailable | unavailable | 5 / 0 |
| prepare | modal | 286 | 289.855 | 25.370 | unavailable | 0.006 | 0 / 184 |
| prepare | queue | 168 | 900.171 | 19.622 | 11.494 | unavailable | 0 / 168 |
| prepare | yandex | 123 | 86.462 | 42.625 | unavailable | unavailable | 0 / 123 |
| recording_catalogue | modal | 6 | 0.000 | unavailable | unavailable | unavailable | 6 / 0 |
| retention | modal | 4 | 0.000 | unavailable | unavailable | unavailable | 4 / 0 |
| review | local | 470 | 0.000 | unavailable | unavailable | unavailable | 470 / 0 |
| review_sample | modal | 1 | 0.000 | unavailable | unavailable | unavailable | 1 / 0 |
| second_opinion | modal | 4 | 0.000 | unavailable | unavailable | unavailable | 4 / 0 |
| selection | modal | 77 | 75,549.600 | 0.476 | unavailable | 0.000 | 0 / 0 |
| source_candidates | modal | 38 | 0.000 | unavailable | unavailable | unavailable | 38 / 0 |
| source_candidates | queue | 26 | 0.000 | unavailable | unavailable | unavailable | 26 / 0 |
| source_corrections | local | 5 | 0.000 | unavailable | unavailable | unavailable | 5 / 0 |
| splits | modal | 69 | 63,569.560 | 0.068 | unavailable | 0.000 | 0 / 0 |
| study_report | modal | 14 | 0.000 | unavailable | unavailable | unavailable | 14 / 0 |
| study_report | queue | 1 | 0.000 | unavailable | unavailable | unavailable | 1 / 0 |
Audio hours are decoded raw hours; historical corpus steps use prepared hours at materialization. Reruns count again.
Per-step dollars use provider-reported charges for isolated Modal apps or Vast endpoints serving one step type. Only runs with both billing and audio hours enter the cost denominator. Billing can lag. Account totals are not allocated by runtime. Shared Modal cohort apps retain application-level billing; their full charge is never attributed to each individual run.
| run | status | wall s | reported steps s | outside reported steps s |
|---|---|---|---|---|
| eb80f712 | success | 86.202 | 1.100 | 85.102 |
| 13e69f02 | success | 2,175.694 | 2,119.600 | 56.094 |
| 5f2eb376 | success | 230.092 | 1.100 | 228.992 |
| 03951d5f | failure | 2,645.728 | 299.700 | 2,346.028 |
| 83395d4c | success | 118.312 | 0.900 | 117.412 |
| 6c73cbfb | success | 4,221.635 | 3,785.300 | 436.335 |
| b10d56cd | success | 5,458.657 | 5,397.800 | 60.857 |
| d44b775a | success | 167.845 | 1.400 | 166.445 |
| 30fabc5c | success | 178.994 | 0.900 | 178.094 |
| 5bc57386 | failure | 4,726.270 | 324.200 | 4,402.070 |
| f6506028 | failure | 1,787.321 | 255.100 | 1,532.221 |
| 1c7b2692 | success | 100.712 | 0.900 | 99.812 |
Outside reported steps includes startup, checks, Pipes, unreported work and retries. Concurrent step times can overlap.