Are you using the new AI / Gemini search? It seems to work better when I turn that OFF and use the old search. It's buried in the settings.
Mine brings up hundreds of photos of cats, both mine and random ones that I didn't even know were in the photos.
It's even semi decent at telling apart individual pets (you can give them names just like people), though it sometimes struggles between our different black and white cats.
I don't tag mine at all and don't use any geolocation stuff and it works really well finding cats or finding cars or whatever. I wonder if I have different options set or something. I don't think I've ever customized it.