I'm baffled as of why this happens, and I believe this might be gtk related, however not all gtk3 programs I've tested do this so I'm reporting this directly against firefox: On x11, file->open (or ctrl+o), select any file, attempt to double-click to open: nothing happens. File is is only selected. This happens regadless if the file in the dialog has been selected previously or not. Attempt to press enter: also nothing. I have to explicitly click "Open". Try the same with gimp which links to the same gtk library: both work as expected.
I can confirm this bug, and it also happens in the Save As dialog that opens when you press Ctrl+S. I tried downgrading Firefox to 151.0.4-1 and this makes the issue go away. I tried analyzing this a bit with ChatGPT and found that the Save button in the Save As dialog and the Open button in the Open File dialog can still be activated by Alt+S and Alt+O respectively, so it's not a focus issue. According to ChatGPT, this suggests that the dialog response associated with GTK_RESPONSE_ACCEPT is not activated by Enter anymore.
So apparently this is actually an upstream "feature": https://bugzilla.mozilla.org/show_bug.cgi?id=2047865 logging this here for others.