Duplicate records can be caused due to lack of checks in the creation process for existing values that handle whether or not a record is updated or a new record is created
- A lot of times you won't have the identifier of the record until after it is created which causes complications
Not understanding the workflows in the API
- There might be flags in the backend that says whether or not an order should be updated and just creates a new one every time.
CEF utilizes a CustomKey field on every entity
- This value is usually reserved for the remote system's identifier
- Some CEF entities set the CustomKey internally or an admin is able to change the CustomKey manually. However these are exceptions to the rule and should be checked for.
When syncing a record (Customer)
- List all Customers from remote system
- iterate through the customers
- Call a GetByKey endpoint to see if record already exists
- If the response is null then create a new instance otherwise just map against the existing record
- CEF integration service utilizes the "Upsert" endpoint.
- Note that the only check for Create vs Update is whether the incoming record contains a valid ID value.
- What NOT to do.
- Sending a record from remote system to CEF without validating if the record already exists causes a few problems
- The upsert command will not find the ID value and therefore cause duplicate records by always creating a new record
- Some entities (Accounts and others) utilize a separate check for CustomKey/Email/Name to prevent duplicates, however skipping the validation process still could cause errors
- If properties on the entity are set within CEF then just mapping from the remote system and syncing them in without validation could overwrite and nullify those properties causing them to be lost
- Good practice is to incorporate Digest Model (See documentation on Digest Model implementation in Connect) TODO: link to documentation once created
- This list can be used to easily validate whether or not an incoming remote system record exists
- If the record doesn't exist then you can easily just create a new instance and map
- If the record DOES exist
- Hash the incoming version of the record and compare hashes vs the found digest model
- If hashes match you can skip the record
- If mismatched get the existing record from CEF → map → upsert to CEF