Mark Attendee as Optional on Google Calendar Invite
Hover your cursor over the attendee’s name or email address after you add it. To the right, you’ll see a gray contact (person) icon. Click that icon to mark your guest as optional. Once you do this, the icon will appear white indicating the optional status.