{"slug":"41-open-source-llms-evaluated-on-19-benchmarks","name":"41 Open-Source LLMs Evaluated on 19 Benchmarks","summary":"A repository of local benchmark evaluations comparing 41 open-source large language models across 19 tasks, with rankings, scripts, notebooks, results, and utilization logs.","publisher":"jayminbhan","homepage":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","kind":"eval","hasApi":false,"hasMcp":false,"costRange":"unknown","costNotes":"Pricing is not stated. Evaluations were performed locally on a personal computer, and compute costs are not provided.","functions":["rank","search","evaluate","generate","reason"],"slices":[{"functionId":"evaluate","label":"LLM benchmark evaluation","note":"Evaluates open-source language models across reasoning, math, commonsense, natural-language inference, knowledge, and reading tasks using lm-evaluation-harness.","cost":"Unknown"},{"functionId":"rank","label":"Model leaderboard ranking","note":"Provides overall and category rankings calculated from task scores, including mean scores and runtime fields.","cost":"Unknown"},{"functionId":"search","label":"search","note":"Inferred from executor","cost":"Unknown"},{"functionId":"generate","label":"generate","note":"Inferred from executor","cost":"Unknown"},{"functionId":"reason","label":"reason","note":"Inferred from executor","cost":"Unknown"}],"interfaces":[],"composabilityRationale":"The repository provides scripts, notebooks, CSV and Excel tables, JSON results, stdout logs, and GPU utilization logs that can serve as static evaluation artifacts. The page does not state an API or MCP interface.","signals":[],"functionProbabilities":[{"functionId":"rank","value":0.8,"source":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"lexical","observedAt":"2026-11-04T06:16:15.272Z"},{"functionId":"search","value":0.8,"source":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"lexical","observedAt":"2026-11-04T06:16:15.272Z"},{"functionId":"evaluate","value":0.72,"source":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"lexical","observedAt":"2026-11-04T06:16:15.272Z"},{"functionId":"generate","value":0.72,"source":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"lexical","observedAt":"2026-11-04T06:16:15.272Z"},{"functionId":"reason","value":0.72,"source":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"lexical","observedAt":"2026-11-04T06:16:15.272Z"}],"claimVerification":[{"key":"mcp","status":"unverified","asserted":"false","evidenceSource":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"fallback"},{"key":"api","status":"unverified","asserted":"false","evidenceSource":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"fallback"},{"key":"cost","status":"unverified","asserted":"unknown","evidenceSource":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"fallback"},{"key":"license","status":"unverified","asserted":"","evidenceSource":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","method":"fallback"}],"tweets":[],"notes":[],"origin":"intake","createdAt":"2026-11-04T06:16:18.195Z","updatedAt":"2026-11-04T06:16:18.195Z","sources":[{"kind":"web","label":"executor","url":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","excerpt":"BranchesTags\n\n## Latest commit\n\n## History\n\n20 Commits\n\n## Folders and files\n\n| Name | | Name | Last commit message | Last commit date |\n| --- | --- | --- | --- | --- |\n\n## Repository files navigation\n\n# 41 Open-Source LLMs Evaluated on 19 ","at":"2026-11-04T06:16:18.195Z"}],"recordId":"cmp_41-open-source-llms-evaluated-on-19-benchmarks","claims":{"name":"41 Open-Source LLMs Evaluated on 19 Benchmarks","summary":"A repository of local benchmark evaluations comparing 41 open-source large language models across 19 tasks, with rankings, scripts, notebooks, results, and utilization logs.","kind":"eval","publisher":"jayminbhan","homepage":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","costRange":"unknown","costNotes":"Pricing is not stated. Evaluations were performed locally on a personal computer, and compute costs are not provided.","functions":["rank","search","evaluate","generate","reason"],"slices":[{"functionId":"evaluate","label":"LLM benchmark evaluation","note":"Evaluates open-source language models across reasoning, math, commonsense, natural-language inference, knowledge, and reading tasks using lm-evaluation-harness.","cost":"Unknown"},{"functionId":"rank","label":"Model leaderboard ranking","note":"Provides overall and category rankings calculated from task scores, including mean scores and runtime fields.","cost":"Unknown"},{"functionId":"search","label":"search","note":"Inferred from executor","cost":"Unknown"},{"functionId":"generate","label":"generate","note":"Inferred from executor","cost":"Unknown"},{"functionId":"reason","label":"reason","note":"Inferred from executor","cost":"Unknown"}],"interfaces":[],"composabilityRationale":"The repository provides scripts, notebooks, CSV and Excel tables, JSON results, stdout logs, and GPU utilization logs that can serve as static evaluation artifacts. The page does not state an API or MCP interface."},"relations":[{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"minimax","kind":"fills-same-slot-as","note":"Shared functions generate, reason. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"grok-xai","kind":"fills-same-slot-as","note":"Shared functions generate, reason, search. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"alibaba-model-studio","kind":"fills-same-slot-as","note":"Shared functions generate, reason. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"byteplus-modelark","kind":"fills-same-slot-as","note":"Shared functions generate, reason. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"hunter","kind":"fills-same-slot-as","note":"Shared functions evaluate, search. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"typesafe-jev","kind":"fills-same-slot-as","note":"Shared functions evaluate, reason. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"toloka","kind":"fills-same-slot-as","note":"Shared functions evaluate, generate. Can fill the same function slot."},{"from":"41-open-source-llms-evaluated-on-19-benchmarks","to":"kogiqa-mcp-web-browser","kind":"fills-same-slot-as","note":"Shared functions generate, search. Can fill the same function slot."}],"explain":{"score":3,"formula":"100 × sum(weight × value) over observed factors only. Unobserved factors are omitted, not zeroed.","factors":[{"id":"github_health","label":"GitHub health","weight":0.18,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"agent_use","label":"Agent use","weight":0.16,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"human_notes","label":"Human notes","weight":0.14,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"composability","label":"Composability","weight":0.16,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"checklist","label":"Checklist","weight":0.1,"value":0,"contribution":0,"why":"No MCP surface is declared. No API is declared. Cost is unknown. 0 of 3 checklist fields are complete.","sources":[],"evidence":"observed"},{"id":"evidence_quality","label":"Evidence quality","weight":0.14,"value":0.18107144632819594,"contribution":0.025350002485947435,"why":"1 sources cover 1 evidence types. The latest dated evidence is 2026-11-04. Freshness is omitted unless observed as a signal.","sources":[{"kind":"web","label":"executor","url":"https://github.com/jayminbhan/41-llms-evaluated-on-19-benchmarks","excerpt":"BranchesTags\n\n## Latest commit\n\n## History\n\n20 Commits\n\n## Folders and files\n\n| Name | | Name | Last commit message | Last commit date |\n| --- | --- | --- | --- | --- |\n\n## Repository files navigation\n\n# 41 Open-Source LLMs Evaluated on 19 ","at":"2026-11-04T06:16:18.195Z"}],"evidence":"observed"},{"id":"benchmark","label":"Benchmarks","weight":0.12,"why":"insufficient evidence","sources":[],"evidence":"insufficient"}],"missing":["GitHub health: insufficient evidence","Agent use: insufficient evidence","Human notes: insufficient evidence","Composability: insufficient evidence","Benchmarks: insufficient evidence"]},"score":3}