I would not set DYLD_LIBRARY_PATH here, it should never be needed to point to Homebrews’s own install directory. It could well be this that’s interfering as it might be overriding a system path, hence the permissions error.
It’s a known issue that Xquartz/Homebrew X11 can interfere with each other, or rather, CMake can pick up a mixture of the two. This can be checked by looking for variables containing “X11” in the CMakeCache.txt file in the build area. For best behaviour, these should all point at the Xquartz install location(s).
Whilst X11 on macOS is still currently supported, my personal suggestion would be that unless you 100% require X11 for your use case, the pure Qt drivers (QtOpenGL, Qt3D, ToolsSG) give an easier/better experience on macOS.