Some people will get annoyed if you ask. Others will want money. If they get annoyed, that's their problem. At least you're asking.
One thing that comes to mind is: what, exactly IS a reference photo.
If you look at 50 photos, and choose elements from each from memory, could you say that you are using general knowledge found on Google, and not copying any one work?
If you choose a wing here, a head there, general pose over there, a color palate from over there would any copyright be enforceable?
That said, I can recognize copies of most of my photos with very high reliability, in spite of hundreds of other hummingbird photographers out there, many with similar poses. Some of my poses are fairly unique.
But I have a shot at antelope canyon that looks like it was shot from exactly where a more famous photographer shot the exact same keyhole at the exact same angle. I should sue! Not really. We both went to an iconic location and found "the" iconic feature. I've stood very close to Ansel Adams, too. Many thousands of photos have been shot at tunnel view in Yosemite, for example.