Thanks for running this test!

I suspect there is a bug with TRANSMITTANCE. I don’t know exactly what it is supposed to do, but it appears from the first part of the code that a given fraction of particles are transmitted without change in direction. Maybe, for many particles, it averages out. That part seems to make sense.

However, later, the probability of Fresnel reflection and refraction are calculated. If TRANSMITTANCE is 0, this is based on electrodynamics. If transmittance is non-zero, however, the ratio of Fresnel refraction to Fresnel reflection is simply TRANSMITTANCE. This makes no sense to me.

You could try changing the code here:
https://geant4.kek.jp/lxr/source/processes/optical/src/G4OpBoundaryProcess.cc#L1199

from

1199       if(fTransmittance > 0.)
1200         transCoeff = fTransmittance;
1201       else if(cost1 != 0.0)
1202         transCoeff = s2 / s1;
1203       else
1204         transCoeff = 0.0;

to

if(cost1 != 0.0)
      transCoeff = s2 / s1;
else
      transCoeff = 0.0;

I hesitate to change this in the repository, because it’s been like that for 10 years and no one has complained. You could file a bug report.

Also, as you note, my explanation of how absorption/transmission/Fresnel is chosen was incorrect. The part of the Book for Application Developers you quoted is correct, as far as I know, and aside from the above. Post back if it’s not clear.