Cannot replace table in Database Query Builder

So, I have a Server Action that queries the database for some user information. This Action is one of my older ones and now needs some updating. I have a “users” table that extracts 6 columns of data once the query executes. I am no longer using the “users” table and this has been replaced with a table called “people”. I thought I could goto the Query Builder, add the new “people” table and adjust all of the conditions and sorting rules with the newly added table fields and then just delete the “users” table. But when I try this, the delete “X” is not clickable. So it appears that the first table added to the Database Query Builder is not removable. Is this correct or am I missing something? I really don’t want to have to recreate a bunch of Server Actions.

Thanks for the help.