I feel stupid.
I think I have solved the M/S + intensity joint stereo problem.
I had thought the issue had to do with an ambiguous standard since my first attempt produced output quite different from two other references (mpg123 and Xaudio). After tweaking the algorithm (OPT_ISKLUGE) I ended up with something that sounded like Xaudio. However, it was inferior to the sound from mpg123.
While I still think ISO/IEC 11172-3 is ambiguous if not outright wrong -- it was clarified somewhat in ISO/IEC 13818-3 -- I discovered the cause of the inferior sound as a bug in my code.
I feel stupid because the fix is trivial:
--- layer3.old.c Mon Nov 6 01:56:35 2000 +++ layer3.c Mon Nov 6 01:57:57 2000 @@ -1112,7 +1112,7 @@ lsf_scale = lsf_is_table[right_ch->scalefac_compress & 0x1];
for (i = bound; i < 576; ++i) { - unsigned int is_pos = 7; + unsigned int is_pos;
if (f-- == 0) { if (w-- == 0) {
This is actually what I had written originally, but I had changed it due to a compiler warning. :-/
With this fix, the sound is now more like mpg123, much improved, and hopefully, correct in conjunction with OPT_ISKLUGE -- which may not really be a kluge any more.
Ironically, Xaudio must have a similar bug, even though I can't look at their code! :-)
Cheers, -rob