If you download the plugin files it is located in:
woocommerce-quantity-increment > assests > js
Both those .js files need to be changed. The <input type="button"> is located near the top, and there is one for plus, and one for minus.
However I will note, once I took out the button text, and tried to click on the '+' or '-', it prompted me to enter text on a mobile device. So I'm not sure this is the best fix.
I am still trying to find a solution as my client needs this fixed ASAP.