I like that invitations sent for meetings will update the attendee list to 'apologies' if the invitee declines the invitation. It would be nice if the list of attendees started out as "Pending" or "Invited" and would then be designated as "Yes" or "Will Attend" and "Apologies". This way, invitees who had not responded can easily be identified.
Additionally, with split in person and Zoom meetings, noting how someone is attending is also required... something along the lines of "Attending via Zoom" or "Attending In Person"
I agree and think this is an important feature.
The current status where an invitee is assumed attending unless otherwise is counter intuitive - it gives no confirmation of acceptance. For multiple reasons the invite may have got lost or missed - which does not constitute acceptance of meeting and having it on their calendar.
The suggestion for all stages is ideal.
In the short term a "not attending unless confirmed" logic would make more sense.