- The AI that are mostly in use in medicine are not LLMs

- Yes. All doctors advice should be taken cautiously, and every doctor recommends you get a second opinion for that exact reason.