b
(b) Given, $A = \left[ {\begin{array}{*{20}{c}}1&{ - 1}\\2&{ - 1}\end{array}} \right],\,\,B = \left[ {\begin{array}{*{20}{c}}a&1\\b&{ - 1}\end{array}} \right]$
==> $A + B = \left[ {\begin{array}{*{20}{c}}{1 + a}&0\\{2 + b}&{ - 2}\end{array}} \right]$
${A^2} = \left[ {\begin{array}{*{20}{c}}1&{ - 1}\\2&{ - 1}\end{array}} \right]\,\,\left[ {\begin{array}{*{20}{c}}1&{ - 1}\\2&{ - 1}\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{ - 1}&0\\0&{ - 1}\end{array}} \right]$
${B^2} = \left[ {\begin{array}{*{20}{c}}a&1\\b&{ - 1}\end{array}} \right]\,\,\,\left[ {\begin{array}{*{20}{c}}a&1\\b&{ - 1}\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{{a^2} + b}&{a - 1}\\{ab - b}&{b + 1}\end{array}} \right]$
Also, ${A^2} + {B^2} = \left[ {\begin{array}{*{20}{c}}{{a^2} + b - 1}&{a - 1}\\{ab - b}&b\end{array}} \right]$
==> ${(A + B)^2} = \left[ {\begin{array}{*{20}{c}}{1 + a}&0\\{2 + b}&{ - 2}\end{array}} \right]\,\left[ {\begin{array}{*{20}{c}}{1 + a}&0\\{2 + b}&{ - 2}\end{array}} \right]$
Also, ${(A + B)^2} = \left[ {\begin{array}{*{20}{c}}{{{(1 + a)}^2}}&{\,\,\,\,0}\\{(2 + b)(1 + a) - 2(2 + b)}&{\,\,\,\,4}\end{array}} \right]$
Also, ${(A + B)^2} = {A^2} + {B^2}$
==> $\left[ {\begin{array}{*{20}{c}}{{{(1 + a)}^2}}&0\\{\,(2 + b)(a - 1)}&4\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{{a^2} + b - 1}&{a - 1}\\{ab - b}&b\end{array}} \right]$
By equating, $a - 1 = 0 \Rightarrow a = 1$ and $b = 4$.