maybe you can add another criteria using if inside the match something like

=OFFSET(Pivot!$B$1, MATCH($A5, IF(Pivot!$C$2:$C$17="N",Pivot!$A:$A, 0))-1, , $B5, )