The Prym representations of the mapping class group are an important family of representations that come from abelian covers of a surface. They are defined on the level-$\ell$ mapping class group, which is a fundamental finite-index subgroup of the mapping class group. One consequence of our work is that the Prym representations are infinitesimally rigid, i.e. they can not be deformed. We prove this infinitesimal rigidity by calculating the twisted cohomology of the level-$\ell$ mapping class group with coefficients in the Prym representation, and more generally in the $r$-tensor powers of the Prym representation. Our results also show that when $r\ge 2$, this twisted cohomology does not satisfy cohomological stability, i.e. it depends on the genus $g$.