In short the idea is

if a field always submits a value,

• then if its value is missing (===null) then maintain value

if a field does not submit any value (e.g. when being empty like the checkboxes)

• then for it add field presence information into the form (an HTML input tag)

Root issue is that fields plugin is trying to save custom fields values

• without knowing if that record with custom fields that is being saved is

a. via a JForm submit (controller save task), which may or may have not skipped the field from adding it to JForm (actually this was the only case, we could make this differently)
b. via other code that is to saving the record without setting the custom field values inside the $data['com_fields'] array

Note that after this PR

• for case (b) to clear a field value like checkboxes your should add a value array() because missing value means maintain current value, but this is exactly as it should have been in the first place, just making a note of it

There HAS to be a way to do this without making form field objects presence aware. The form field objects are NOT data validators!

@mbabker

yes there are other ways

We could do this

1. Create a random id RANDID for every form load
2. Add to form as a single field to contain RANDID (similarly like we add 1 field for check session token)
3. Create and add into session the presence array using index
   e.g. com_users.user.record_id.RANDID
4. Then we field presence data from session during save

In such a case this PR would work the same

• will just change the parts of this PR that use the form inputs to use session

Honestly i would prefer to use the session solution if other people like it too,
it seems to me that it will be a more clean / proper solution

:sigh:

Sometimes I think I'm talking to myself.

You do NOT need to be aware of presence. The form API does not need to be aware of the pre and post submit states. Your request handler should be doing something very close to what I commented in #19742 (comment) as it relates to fields which can be nullable from the request (i.e. checkboxes) to ensure all fields have values and toggling a checkbox from true to false is handled correctly as a deliberate action.

If you have to put the pre-submit data into the session, you've screwed up in so many ways that I don't even want to try counting.

To be blunt, unless someone is going to fix the code that is reading the data from the request and ensuring that it gets correctly validated/structured BEFORE $form->bind($data); gets called, I really don't want to see another PR or a modification on an existing one. Because at this point I feel like everybody continues to ignore the broken code and pushes for complete ? fixes that give further weight to comments like #19749 (comment)

i thought you did not like the field presence stuff -- added inside the form --

anyway
if we would want to play with words , we can call it 'state' "information" about fields, and forget the "presence" stuff

lets not forget what we want to succeed

• we want to have code save records without providing custom fields data (values lost) or even by providing ... some of the custom field values and maintaining the other values
• we want to have code save records but some custom fields do not submit values
• we want to have code save records that are added to JForm ... but then the code decides to skip them in frontend OR backend form based on configuration !! and YES this is yet another case that values are lost !!

anyway i gained some good knowledge about the joomla custom fields inner stuff,
would like to use them for cases like users (when the issue gets fixed)

i ll close this PR and someone else can have a different approach