The troublesome code in Jacks is at AF:
AF LBI 0,9 ;set B to 9
B0 INL     ;M=L<7:4>, A=L<3:0>
B2 SKE     ;skip if A=M
B3 JMP 10A
b5 LB 1,13

When this code is hit, EN2 is set, so L outputs are enabled. Normally, you would disable outputs while getting input. I thought I read something in the data sheet about this, but I can't find it now. I think any output bits set to 1 will read as 1s unless they are driven low. L<5:4> are the muxed button inputs and L<1:0> are the LED muxes. If I'm correct about the output bits getting read, then that SKE will compare those.