AI Models Tested on Rabbinical Ordination Exams in HalachaBench Project
A new project named HalachaBench tests advanced AI models using real rabbinical ordination exams from Israel's Chief Rabbinate. Anthropic's Claude Fable 5.1 topped the evaluation with 79.7 out of 100, highlighting both the potential and current limitations of artificial intelligence in religious jurisprudence.
Artificial intelligence is increasingly intersecting with complex religious rulings through a new project named HalachaBench, which puts leading AI models to the test using actual examination questions from the Chief Rabbinate of Israel. The experiment evaluated whether advanced language models could navigate intricate rabbinical rulings under closed-book conditions without internet access.
The benchmark assessed the models' ability to analyze halachic issues, reference classical sources from Chazal, Rishonim, and contemporary authorities, and issue legal determinations. The test utilized 60 questions drawn from five recent rabbinical examinations across key subjects including family purity, marriage laws, agricultural commandments, ritual baths, and daily prayer.
Commercial Models Outperform Open-Source Alternatives
Results revealed significant performance disparities between advanced commercial models and open-source equivalents. Anthropic's Claude Fable 5.1 secured first place with a score of 79.7 out of 100, followed by GPT-6 Astra with 73.3, GPT-6.1 Sol with 72.8, Claude Opus 5.5 with 69.1, and Gemini 3.8 Flash with 64.4. Conversely, downloadable open-source models scored below 11 points. Despite leading the ranking, the top-performing model still missed approximately one-fifth of the requirements for a complete halachic response.
Due to the absence of official answer keys for rabbinical exams, project creators developed a rigorous evaluation methodology. Detailed rubrics incorporating historical commentaries, required rulings, and common pitfalls were drafted using Anthropic's Claude Fable 5.1, peer-reviewed by additional models, and cross-referenced with approximately 2,000 sources via the Sefaria digital library. Scoring was subsequently administered by GPT-6 Astra acting as an impartial judge.
Rabbinical Reception and Future Prospects
Commenting on the findings, Rabbi Avraham Stav described the initiative as fascinating while raising critical questions regarding testing parameters. He noted that unlike human candidates whose examinations test memorization and internalization, AI systems derive immense utility from accessing extensive external databases, raising questions about open-book performance.
Project organizers emphasize that HalachaBench remains an exploratory intellectual exercise as rabbinical authorities have yet to manually review the evaluation rubrics. Future phases aim to incorporate human rabbinical oversight to determine whether artificial intelligence can ultimately serve as a reliable auxiliary tool in religious jurisprudence.
