I've thought about fishing in very similar ways to you before, particularly sport fishing where the fish is thrown back. I suppose it's nice that it does get put back but, in the meantime, it's rough on the fish who did not actively choose to participate.
Meat does need to come from somewhere, though. Personally, I like to stick my head into the sand and not think where it comes from. I know the people who own a nearby farm (they deliver a dozen eggs to me weekly - farm fresh eggs are pretty tasty) and they also grow cows and chicken. I've met some of the cows and chickens, and realised during their last meat sale that I could not buy any, having once actually patted one of those cows. Is it different buying from a store? Well, at least I didn't look those cows in the eyes.
Many shift their thoughts around the fact that something died and someone had to kill it to put meat on their table...it's those that then go on to criticise hunters that comfound me.
With fishing...putting fish back after catching them...I wonder how many people get into the water and find that fish to ensure it actually survived that ordeal? (None obviously). Also, I wonder how many would stick a hook and line into their cheek and tolerate being dragged around by it to understand what it feels like for the fish...I imagine it not to be any fun at all.