In NTSA TIMS portal, mother’s maiden name is your mother’s surname (family name) before she was married. It is likely to be the name of your grandfather’s name (from the mother’s side).
If that does not work, try interchanging names, using any of your mother’s three names.