I cleaned as much as I could with paper towel, break cleaner didn't help. Hope, diaphragm will not fall apart from this sealer.
I've always veered away from using Hylomar Blue on thin elastomer seals as the instructions do not venture into that subject, some people have used it in that area with great success. So using it is more than likely okay for thin elastomer seals, it's definitely okay on metals.
Looking at the manufacturers MSDS for the recommended removal product, it has acetone in it, so means raiding someone's beauty kit. But I'm not sure whether elastomer seals can stand acetone as it's a very strong product. I generally clean seals and metals/plastics (and so on) with IPA (pure alcohol) as it leaves no residue, it's a bit weak on removing strong products like Hylomar but is my first choice and I usually try that first.
Having said the above, here is what I would do, and it is purely experience nothing more. Wipe the area of the seals under question with cellulose thinners to remove unwanted products, then have ready some IPA to wipe down immediately after. I've never damaged a seal using this method.
Hope it goes okay.