Call restore_remote_modifiers after key up, to make sure the modifier (such as NumLock is correct for the key up event as well.