CrocoSign

The Polish Association of the Deaf estimates that about 40-50 thousand people in Poland use PJM. It has its own grammar, and meaning is carried by, among other things, the direction and force of a movement and facial expression.

There are few interpreters. 236 sign language interpreters took part in the 2023 census by STPJM and the University of Warsaw. Regional registers hold 508 entries, but they are not kept up to date, so there are probably far fewer active interpreters. An hour of interpreting costs about 132 PLN on average. In an emergency, such as a visit to the doctor, it is often impossible to book one.

We translate straight from video to text. Many systems first write signs down in an intermediate notation called glosses. No manual gloss annotation exists for PJM, so those methods could not be used directly. Pseudo-glosses could be generated automatically, but we chose a fully gloss-free approach: no extra stage and no noise from automatic annotation errors.

We found no existing PJM-to-text translator and no set of recordings ready for training models. We built the dataset ourselves.

The recordings come from the Open Repository of the PJM Corpus (Otwarte Repozytorium Korpusu PJM), run by the Sign Language Linguistics Lab at the Faculty of Polish Studies, University of Warsaw. We downloaded and fixed the transcriptions, cut the recordings into one-sentence clips and worked out who signs in each clip. The corpus is licensed under CC BY-SA 4.0, and so is our dataset. The model weights are under CC BY-NC-SA 4.0, because the models we started from are for non-commercial use only. Our code is under the MIT licence.

Open Repository of the PJM Corpus (University of Warsaw)

video-sentence pairs

29,92829,928

of video, approximately

25 h25 h

Deaf signers

151151

This dataset exists thanks to the Deaf people who recorded themselves for the corpus and the University of Warsaw team who collected and shared it.

The reference sentences we compare model output with are in English. We translated the Polish transcriptions with DeepL, so they are machine translations, one per clip.

Dataset on Hugging Face: CrocoSign/pjm-data

Dataset preparation code: crocosign/pjm-data (GitHub)

In hours we are mid-pack, but we have 151 signers, second only to OpenASL. With that many people, a model can be tested on signers it never saw in training.

For PJM the table shows the cleaned part we trained on: 29,928 pairs, about 25 h (28 h before cleaning). Figures for the other datasets come from their publications.

We compared three models: SpaMo, our modification of SpaMo (“SpaMo (mod.)” in the tables) and Uni-Sign. We trained each one either from scratch or starting from a model already trained on another sign language. We tested them in two ways: in the MS split the test signers also appear in training, in the SI split the model sees them for the first time.

The best single models came from Uni-Sign fine-tuned on American Sign Language: OpenASL reaches 42.4 BLEURT on MS and 38.2 on SI, How2Sign 42.0 and 37.8. On BLEU-4 the two are almost identical (8.53 and 8.61 on MS); a second training run of each moved BLEU-4 by 0.14 and swapped their order, so we do not name a single winner. Our modification of SpaMo scored close to the original.

Transfer matters most: on MS, without it every model ends up around 2 BLEU-4, and with it the score rises several times over, to 5-8.6 points. For scale, two Deaf signers fluent in ASL and English (not professional interpreters) reach about 19.8 BLEU on How2Sign, and for a third of the clips they could not make out key details without wider context.

The result depends more on the starting point than on the architecture: the best models went through training on American Sign Language before PJM. Part of that gain may come from that stage also teaching the model to write English. On signers the model did not see in training (the SI split), all models do worse: the best, Uni-Sign, gets 6.09 BLEU-4, and both SpaMo variants about 2. This is the weakest point of these models today.

Models on Hugging Face: CrocoSign/pjm-models

Run the translation yourself

The pjm-models package turns a PJM video file into an English sentence. It downloads the model weights from Hugging Face the first time it runs. The example below uses Uni-Sign; in our test it downloaded about 3.5 GB including dependencies. In our measurements on a 6.6-second recording, Uni-Sign took about 11 s on a CPU and 1.6 s on a graphics card. SpaMo is much heavier: about 110 s on a CPU and 13 s on a card, which needs 24 GB of memory.

git clone https://github.com/crocosign/pjm-models
cd pjm-models
uv sync
uv run python -c "import signsdk; print(signsdk.from_pretrained('unisign-pjm-openasl-ms').predict('clip.mp4').text)"

Put your own PJM recording in place of clip.mp4. You need Python 3.10 or newer and uv.

• pjm-models (GitHub)
• Model weights (Hugging Face)

The full app, with the interface and the API, lives in two repositories:

• croco-sign-backend (API)
• croco-sign-frontend (interface)