0:02
The times of the days of yore when orders can be that simple, great, it can do that.0:13
However, we ran into a lot of clients where it was not so simple.0:19
We needed ways to split orders based upon a series of facets that were gonna be different but all needed to be possibly working at the same time.0:34
So we had one client who needed it, split by the vendor who was assigned to the product, one who needed it split by a manufacturer, one who needed it split by a store, one who needed it, split by the pairing of store and vendor, et cetera.0:51
Not to mention the storefront ones like he just mentioned, with the multiple target addresses where you want to split your shipment to go to two different locations yourself, so to for proper tracking purposes and everything.1:05
When we built the targets order checkout provider and it is not the first version of itself, there was another one that came before it, but it doesn't matter.1:12
It just know that we had to change it a few times over time to get it as simplified as we have it now.1:19
And if you've ever looked at that code, you know that it is anything but symbol.1:25
The the point being that for every facet that can cause the split, it builds a string internally that has all of these different pieces together as a key in a dictionary, and then in that dictionary, each key value is a list of sales order items.1:45
That would be That would describe what's supposed to go to that location from that originating location, which could be a different warehouse, a different store, a different vendor, a different manufacturer, whatever.1:58
So with all that information together, it internally builds this Dictionary of items and then analyzes it all together for the splits that it's required to make display based upon the fact that there's back end data the user can't change.2:12
So like it's assigned to a different store, it's assigned to a different manufacturer, whatever.2:15
And then physically puts all of those to the UI and says hey user, these are the separate orders that are gonna be created because you ordered something from store A and something else from store B, et cetera.2:29
Then the user goes in and picks those items and says I want these items to go to this address.2:34
I want these items to go to the same address.2:36
That's fine, they could do that, but it would be physically selecting it twice in the UI.2:41
They could also then split one of those orders, one of those sections that were already split.2:47
So like all the items that were going to store A, maybe they want to split them in half and have three go here and 2GO here to these two different addresses that the customer has.2:56
So that in that case, we have a total of three separate orders being created.3:02
Two of them came from store A, one of them came from store B.3:04
The two items from from or, sorry, the same item or different items from store A got split into two separate orders because they're going to two different addresses.3:14
Everybody got that straight.3:15
Great, awesome.3:15
Not even gonna wait for a reply.3:18
So all of that information together is really hard to figure out mathematically, programmatically, and storing it in the database and recalling it to the UI safely and all of that stuff.3:31
We had to propagate that same info not just to the order section, but into the carts, into quotes, into invoices, into potentially sample requests.3:43
Potentially returns.3:46
Although returns, it just ends up working physically differently and potentially purchase orders.3:51
So we had to make sure that this wall worked generically enough that it all could come together.3:56
Then you throw in the fact that a client might actually have invoices and quotes turned on inside their website and returns.4:05
For a customer to jump into the website and find an order and see all of the related data for all of that stuff would have been a nightmare worth of data calls to the server asking, hey, this is the order I'm starting from.4:19
I need every piece of information that was ever relevant to this order from all directions.4:25
What quotes it came from, what returns have been made, what invoices have been paid against it because it might have been set to building later during checkout.4:32
All of that stuff to prevent having to physically jump through the in the entire web of relationships of all of these different tables we birthed the sales group.4:46
The sales group is meant to be the central hub of all these different spokes of data, so that all of that information can be gathered quickly and easily.4:56
So the most important thing in checkout these days is not the order itself, but the sales group, because from there you can access everything else that was relevant.5:06
If you're dealing with a customer who had quotes, there's possibly a, you know, one or more quotes that the customer went through and either approved or denied in order to make that information.5:17
And then the quote may have been said during the approval.5:22
They've been converted into an order with a payment attached, or maybe not a payment attached and therefore an invoice was generated.5:29
All those things get told about the sales group ID and get a put into appropriate lists.5:39
Because of the way that Entity Framework works for pulling data out of the database.5:44
There's the N + 1 problem.5:47
You're you may be aware of it.5:49
It it's basically says that every join that you make in data and everything that you you pull out of the database, especially as you go multiple levels deep worth of joins makes the data slower and slower and slower to pull it out To the point where it instead of taking nanoseconds or milliseconds, it could take you know 10s of seconds or even minutes to physically run that data and pull it out.6:11
So trying to map or all of that data forcibly and get all of it at once in one call is going to make it an extremely slow process if you can ask for the sales group because it's going to have at least the IDs of all the referral event stuff.6:27
And probably some other basic info too like custom keys, dates.6:31
The basic stuff that you have on the on the list mapping of most of these objects are gonna come with the sales group when you ask for the a full the the sales group for by ID.6:43
So therefore with all of that information there, you could then use the IDs on those things to go out and ask the server for an order by ID, an invoice by ID, et cetera in order to provide the syncs for those items through.6:58
With some clients, we've had it where the only thing that needs to go into the third party system are the suborders or it or only the master orders.7:07
It doesn't really matter which one.7:09
You just need to write the appropriate logic to pull the appropriate data for what you're trying to do in your integration.7:15
So in this integration, what order data are you, or what orders records are you expecting to need to push into in for each individual split sales order?7:27
Or just a master that has all the info in it at once, but but no information about this actual separate addresses or anything that all That's one.7:39
That's why I was thinking maybe it'd be easier just to do it as a straight Sequel query rather than trying to pull from the slow EF.7:47
Just use Dapper and pull make a SQL query, get the information that way, create a custom model and try to push the information that way.7:56
That should be faster because then you're not making multiple calls.8:03
If you wanna go if.8:05
If that's an appropriate solution for connect these days, yes.8:08
OK.8:10
As you were mentioning that EF is slow if it has to make multiple calls, and you said we're gonna have to make multiple calls with this one, it might be easier.8:18
Yeah, just to wait.8:20
That's not quite what I said, correct?8:21
If you're trying to ask for a ton of data with a single call, meaning multiple levels of data and getting all the information from all the orders and all the stuff on the singular call, that's gonna be slow.8:33
If you ask for the sales group, which will give you back the ID and it'll probably take a couple few milliseconds or like 100 milliseconds to get that sales group.8:41
You can then use all those IDs and ask for all those orders using like a 4H async and ask the server for all the orders on it at the same time.8:48
You'll get all those data that points back in probably less than a second okay individually, but you'll have all the data but and.8:57
But if you were to try to combine all that into one call on Steph, that call will probably take 30 seconds to a minute.9:05
So even though you're having to make separate calls, if you're asking for smaller pieces of data that is more exact to what you need by IDs or by custom keys, you're gonna get that data a lot faster.9:17
Okay, at least so far, right?9:25
It's sending like we may need two separate get common functions then because if we're doing it on the child orders we have to inevitably get the master order to get stuff like the billing info, but then on other ones we have to.9:40
If we were seeing you should just ask for the master order to get the billing info because the billing contact should be replicated on the sales group itself.9:49
Just when I ran into an instance where it wasn't, but maybe that was project specific, yeah it it could be project specific, but the out of core what's supposed to happen is the billing address should be linked on the sales group itself so that you don't have to go ask for a master order to get that info.10:07
Well, if we're going by child order only, we wouldn't be grabbing the sales group and the difference of whether we're going through this because if we're going through the sales group that doesn't work if there's multiple orders.10:20
So that would only work for directly doing the master order through sales group, yes, because we have to have one ID that identifies each order we're syncing.10:29
So either if we're doing the child one, it'd be the child order or if it's, you know, the master order, we could do either the master or the sales group.10:40
Yes, if whatever, everything you just said I understood correctly.10:48
So then what does that look like from our perspective from connect with the common model?10:55
Sorry, I joined this one late, I was in a ACI meeting.10:58
Are we able to create different settings to handle this sales group stuff given what the the Seth client wants with what's in the Erps or is it gonna be project specific?11:11
In ERP specific, I think you're gonna have to have like he said, I think you're gonna have to have two of them and it's gonna depend on which one you use for which client because the client may need the master orders or they may need the suborders the it's I it almost guaranteed you'll never need both.11:31
So when you have one that's set up to read all the suborders, the the suborder one is going to have to ask out for the for the either the sales group or the master order in order to figure out what the billing address is.11:47
Because that billing address info is not on the suborder.11:52
We could make a specific endpoint in Seff that said for suborder X, get me the billing address that's off the sales group or the sales or or the the master order.12:04
That way you don't have to make a call for an entire order just to get the billing address off of it.12:10
Got it.12:12
So if there are things like this where you know you'd be asking for something just to throw away most of the data because you only were interested in one property of it, we can make an endpoint in SEF that would do the thing that you needed to do more directly so that you could, yeah, call it.12:29
I think that kind of coordination would be awesome to have like a just a connect override that follows core around that has all the endpoints that we use on a daily basis.12:41
Any objections to that from the rest of the team?12:43
Go ahead, Austin, I I, I do have a question on the suborders.12:46
So in a case where we were doing an integration of suborders only, does the suborder contain that one suborder specific?12:56
Like total calculations, tax and such?12:59
Yeah, like if okay, they're supposed to.13:03
Taxes is entirely provider specific.13:09
So if the client didn't want any kind of taxes, there will be $0.00 for taxes.13:14
If they did basic taxes, it will have a rough calculation using just the simple math of like Texas is it's 8 1/4% sales tax, so it would literally just be the amount of those items times 8.25% on it.13:30
The if it's Avelera where we have full stuff, Avelera gives us a really detailed data about the taxes.13:38
So we actually have very exact values.13:40
So we can stick it into all the line items on the orders about their taxes on their separate orders and that's it and that becomes safe to you.13:48
So you'll be you should be really heavily populated in that case.13:54
Same thing for discounts.13:55
Discounts because of the complexity and how the discounts get applied, there should be discount values on there.14:04
But the discount value might be the like a generalized state like amount because of what happens with how discounts get applied.14:13
Because like if you had your total order was $900 and you got split into like 5 different suborders but your discount was a $40 off coupon, you know what's the logic behind how the discount is applied to a suborder?14:31
It's not, it's not $40 to one order.14:34
It could right?14:34
It's theoretically going to be, you know, $40 / 5 suborders to be applied, so that's $8 on each suborder to be applied.14:44
It gets weird when you have to think about that because if it's a dollar amount off followed by a percentage off followed by individual line items having their own discounts that are just specific to that line item, and that line item got actually split into two separate orders because it wants to go to two different addresses it gets, it gets really complex on how that stuff is applied.15:03
We have a math place in in.15:06
We have some math in place that should apply values reasonably, and they've worked so far as I know for the clients that have tried to use them.15:18
It gets especially complicated then when you have to go then do a return because then you have to apply any amounts that were being done via discounts against the items across the orders because the discount may no may may be fussed with at that point and the taxes may be fussed with at that point.15:38
It gets, yeah it it's exceedingly complicated that for those things.15:43
But generally speaking, yes, you should have taxes on your line items on the suborders.15:48
Yes, you should have discounts on your line items of your suborders, Okay.15:56
So as it stands right now, we don't do anything with sales groups, correct, whether it's from or to common model.16:10
So far as I know that's correct.16:12
OK.16:15
In Austin were you you were doing it on Rydell, correct.16:19
Correct.16:20
OK.16:23
Granted that was not common model though that was a custom integration, right.16:30
OK.16:32
So I wonder how often do we see where it's not used sales groups and this is maybe a question for the Connect team in general.16:39
And if James doesn't know offhand, I mean this is the first time I've ever the the single order checkout provider in Seth has been deprecated for more than a year.16:51
It's been OK deprecated since like 2020.16:53
Like we we physically stopped using it and no one's allowed to use it.16:57
Gotcha.16:57
So this column model is just not going to work for some.17:00
Yeah, the everyone has to use targets order checkout.17:04
Now whether you are going to use the master orders or the suborders on your end it with is dependent on your integration for with the client.17:13
Because it may be that the UI was set up to where you know and the and the data was set up where there nothing would ever cause a split, in which case there would only be 1 suborder and so that's effectively the same as asking for the master order.17:26
It's just which data you're getting back because you're you're not gonna have the billing address on the suborder itself.17:33
Got it, Okay.17:37
So then, so it's next theme.17:39
Yeah, go ahead.17:40
So I guess the effective question there was how many projects have we seen that have it or that have actually needed a split, right?17:50
Because I've actually needed to split.17:51
I'd say it's probably one in every four clients that needs some kind of split and then it breaks down further to like which split they were actually needing to use, whether it was a vendor or a store or something custom.18:05
And I do have it in there for that customer because because like I said with that dictionary key earlier, that key is just a string value.18:12
And I have an override spot there that lets you override and write your own logic to build into that string.18:17
And so as long as that string becomes unique or in cases stays the same, those items get put into that list for that target to attach to on it.18:30
So if the thing says that you know all the all 10 issues should come up in the same order, then regardless of other splits, 10 issues will get grouped together in one spot And it because it says 10 issue inside of the string, you know whatever custom thing you did for that client, I say you if it's probably the Seth side of the team, you know doing that, but whatever.18:52
Yep, okay.18:56
So then I suppose the question becomes what do we support with this as a common model solution?19:07
What do we support as far as a common customization and what just becomes a straight up customization and maybe you guys, the Connect team can think about that based on what we learned from James here.19:18
I would recommend we just consider suborders and or you know like the single orders.19:25
Obviously not single single order but you get the idea.19:28
Single order that came from target order to be the common model and then we just have some extra thing in there that it can pull the master to get the extra details that it needs from the master if it needs to and it just always thinks them as separate orders.19:45
I'm just thinking about other systems here.19:46
I'm not sure really any of them are going to have an actual common way of handling this.19:52
Other stuff like, I mean, for example, if if they have different shipping addresses between the different suborders, that's gonna be information loss when we go to the ERP without more customizations.20:05
Obviously Shopify could handle that, but I mean, we're not gonna sync stuff to Shopify, right?20:11
Out of curiosity, why is sales order so much different than like invoices and stuff?20:17
It's not.20:19
I think invoices can work the same way.20:22
Yeah, technically.20:26
Well, technically they each work their own way, like quotes do.20:28
One have a whole set of customizations and weirdness that that like invoices don't do, and invoices have sets of things that they can do that orders don't do kind of thing.20:38
So for instance, at the time of checkout, if you say bill me later, instead of actually making a payment during checkout, or you make a partial payment so you're only paying for like part of what the total amount would be.20:51
And then you you're supposed to get a a bill later automatically in out of box automatically an invoice would get generated and that invoice would be for the remaining balance of whatever that order was and that would get attached to the sales group.21:07
There are clients who might be on a net 30, in which case that invoice would then eventually get rolled into another invoice.21:17
That would basically void all older invoices and give them one new invoice that says this is tied to all the older invoices and all these other sales groups that would be attached to.21:27
And that you if you're paying this invoice in full, you're paying off all this other stuff.21:33
And that would be like a monthly recurring job that would go in and consolidate all older invoices into one new invoice every month kind of thing.21:44
And what could have happened is maybe accounting at the client and came in and said, ohh, I'll go ahead and pay these other two invoices now so I don't so that they end up not rolling into the net 30 later.21:55
That kind of thing can physically happen.21:57
I've seen it.21:59
So invoices have that kind of stuff and invoices have also had other weird stuff where like they did like a partial payment on the invoice.22:07
So what ended up happening was it closed out the invoice that was there and automatically generated a new invoice for the new remaining balance and said this balance amount is due by X date or whatever the the client's logic was.22:21
So there's there's stuff like that.22:22
And then quotes is a whole other ballpark because you have quotes that are, you know sent by separate like a request for a quote from the from the customer saying I need pencils that meet this certain standard.22:36
Like they they've got to be #2 or #3 pencils.22:39
And then four or five different stores respond to those quotes saying I can sell you #3 pencils at this price and #2 pencils at this price on their quote.22:50
And if you're buying in bulk like this many, I'll I'll knock the shipping off, so it'll be a shipping discount or free and that that'll be that part of that quote.23:00
And then another store comes in and replies saying that they can do them at these two prices.23:04
And then and then the customer comes into the sales group UI and sees the the quotes that are on there and sees those replies from stores and goes OK of these different stores I'm gonna pick store A because I think they have the best price for me even though they didn't mark shipping free or whatever on it.23:24
So all the other quotes then get closed as rejected and the one that they accepted was accepted and then that one got converted into an order.23:33
So like there's that's a whole suite of of craziness that ends up happening inside quotes in a multi store setup.23:40
And that could happen from either stores or from vendors or from manufacturers because we had to make it generic enough that it had to work from any of those different points of view depending on how any client could come into us and ask us for that info later on down the line.23:55
Quotes also have it where not only can you acceptable quote, you can also like push back on a quote and say no, no no I don't like that price, You know if you'll make it down, you know another $0.30 off per pencil, then then I'll accept it and then the manufacturer or vendor or store whoever it is that had had that quote has to respond in the affirmative or the error or reject it.24:19
And then basically they can go back and forth as many times as they want until they arrive at A, at A at a negotiated value that they want, and then that quote becomes their order.24:29
Then there's also subscriptions and all this other stuff too.24:31
I have to consider all of these different things all at the same time because they all have to be able to be functional within the system if the if the right settings were turned on all at the same time with stuff.24:44
So yes, they do have things that that work differently, but they all have specialized stuff because we've had a client at some point who needed that particular feature.24:54
Yeah, functional, because I don't think we've even tried to come model against the multistore setup, have we?25:00
Well, I guess we have in Rydel, are they multistore Rydel?25:08
Yeah, they do.25:09
They've got multiple brands at least I think.25:11
Yeah, multibrand brand is a is a split facet.25:16
But generally speaking on a brand, you don't have cross data in there because you're only able to or like you go to the that particular brand's website and then you could only order items from that brand on that site.25:32
In that case, because I believe you, they had it separated where like the skates were on one site and the ice skates were on the other one.25:37
Yeah, kind of, yeah thing.25:39
So basically you're not buying the same stuff, you're buying different things on those different websites.25:42
As far as the customer's concerned, they don't know that there's anything together about them really other than maybe the fact that they that they see the word bridl on both sites kind of thing.25:54
But like I had a the original client where we did all this split stuff from was was Tal Tyndale Advisors LLC.26:03
They were a conglomerate chain that basically they bought en masse from the from the manufacturers and the vendors for tools and stuff for hardware stores and then they took ownership of multiple multi chain mom and pop hardware stores.26:24
So there might be like three or four in a particular city that don't exist anywhere else but and they and they're going up against you know Home Depot on prices.26:33
Well the only way to get that cheap enough is to buy and big enough bulk to to compete with Home Depot and Lowe's.26:40
So all of these little mom and pop chains come together and they're all ordering from Tau.26:45
And Tau has these warehouses and then the warehouses in stock and everything from all those mom and pop stores are all get integrated together.26:52
And so then you had all these brand domains that were like this is mom and Pop A's branded shop.26:59
Like it's a totally separate website.27:01
Nobody has any idea that they're affiliated with somebody else or with Tyndale.27:04
But now their prices can meet the same level that you can get at you know for your DeWalt, you know Buzz saw you know from from Home Depot or from Lowe's.27:16
So they were the ones who originally got us to to build this stuff up.27:20
And then they they wanted it so that customers could order like you know there would be contractors who were trying to buy all the supplies for like multiple houses within a within a neighborhood that they were that was going up.27:35
So they would buy, you know, lumber.27:37
They would buy tools, They would buy, you know, whatever supplies they needed.27:41
And then they would split them into the separate orders.27:44
And then they also had them like scattered out to where like we need this order to arrive by this date.27:48
And then 30 days later, we're gonna need this order to arrive here at this date because that's when we'll be done with the with the previous orders for the stuff like building the house.27:58
And then we'll move on to the next house.28:00
At that point, that's when we'll need the supplies.28:02
But they're making the purchase all at once for it, which helps with the discount because of because of volume discounts for buying all those same lumber.28:13
Got it.28:15
Okay.28:17
Well, all right, I'm trying to think.28:21
I do wanna add some of those advanced invoice features.28:25
I'm not sure how well those will work with common model, at least for the time being.28:29
So if those are being used, I'd call it a potential risk factor slash customization.28:35
In most of those advanced cases, what usually is happening is that it's not happening on our end, it's happening in the ERP and the ERP was required to send us some kind of notification to say that invoice XYZ was dead or voided because it had been rolled into another one kind of thing.28:53
So most of those cases were having over there.28:55
We had I I believe it was only one case that it we had to actually had to do it on our end to consolidate the the invoices even then yeah to sync that data back that would be a customization because the COM model would not handle that well at least currently right.29:13
I'm just trying to think of what what cases the column model covers at all right now then for orders and invoices obviously invoices going fine right we've been doing that for pay hub but orders, quotes, invoices, returns theoretically although no ones are actually managed to pay for it.29:33
Purchase orders and sample requests or the, Yeah, two.29:37
We focus on orders and invoices only at this point for column model anyway.29:48
So what changes do we need to make here to make this work?29:50
A for PBC and B for the majority of staff projects going forward or don't we know yet?29:54
So I guess moving back for a moment, how does PBC work again?30:01
Do they actually have a split going on like multiple different suborders plus a mass order on the side they do.30:10
On the enforce side, they do not.30:15
OK, that's interesting.30:15
So how are they being split on the subside right now?30:19
Is it just by shipping address, by the shipping ID I guess that's a shipping address.30:29
OK.30:31
So that's I mean that the that's something we have to ask the client no matter what like that that has to be customization because in four can't have multiple shipment addresses in the single order.30:48
So I don't think there's any way PVC in particular could become a model which I would stand by my original recommendation that we just do like 1 sub to one order in the ERP as common and then anything else is just simple customization because I would support that.31:11
Yeah, most, most systems don't really have a direct way of representing this stuff.31:16
I got out of the Shopify, but we're never going to stick to Shopify for this.31:23
Yeah.31:24
And there are infinite variations that we can go through here.31:30
OK, so that's what we have right now, covers that case, right?31:34
Nope, not quite.31:35
So what we do need to still add into this one is we would need to pull back the master order just to grab billing information and then I guess whatever else is purely on the master order or I guess master order and or sales group itself.31:55
So we just need to know what's on this model right here, these billing contacts and stuff that we're getting here.32:00
Well that model is just the child order.32:02
And I guess the other thing is it would probably be useful just thinking about this here.32:13
Ohh no, I guess James, the normal search model lets you search for only child orders, right?32:21
Or ohh no, I guess what?32:24
Probably there there is an argument that on the search model that will let you say that it must be a an order that has a sales group as either the master or the suborder.32:34
There's it's two different properties and they're boolean okay or they're they're they're Tri states so they're nullable boolean that we'll say that so that you can just we we use them to display the the correct way that the the client wants to display them in the UI.32:51
Because sometimes the the client wants to to display the master orders inside the order list of these user dashboard versus the suborders.32:58
So there are two boolean values and each one will set, you know, give me back only masters or give me back only Subs.33:05
OK.33:06
So I that since we don't have any sort of common whole thing just for the listing function itself, we'll probably just have to make that part of the documentation that like if you're on a SEF project and you're syncing orders across, remember that you have to do this for suborders.33:24
Yeah, and that's only one line.33:25
An order will never have both of them assigned at the same time.33:29
So what I would say is if you're looking at a sales order model and you did model dot sales group as master ID, if that has a value that's the master order and so it'll give you the group ID and then you can use the group ID to to query for any any sales orders that have that as the sub order sales group as sub order ID value.33:54
And Jeremy, the the part that we also need to add to the documentation is that when listing them we need to you need to manually look for only those that are Subs in the sales groups.34:10
Okay it won't be no, it'll be a separate problem.34:15
So this yeah this.34:17
This is where where if you were to ask the server for the sales group, the sales group full will come back with the at least a list mapping of all the suborders.34:31
So you have all their ID's, all the custom keys right there.34:34
You won't have their stills, items or anything onto each individual suborder.34:37
If we're syncing you can at least use those to go then call for the individual suborder from from Seth.34:43
Well, what I think is probably we would go with just like this typical scanner and child sync pattern that we already have going where just one sync spawns off a ton of different child jobs.34:55
Each child job handles one order.34:57
That way you know error reporting is per per order and all that stuff.35:01
So each of those child jobs would just be handling 1 suborder.35:05
So on the master job we would just be doing the list call, I guess normal sales or endpoints for those that are suborders.35:14
So instead of sales or search model here he would need to look for a sales group.35:18
Nope.35:18
Still be sales or search model.35:20
But then it sounds like there's just a property down there that we can say only give me those orders.35:24
Yeah.35:24
That are sub orders.35:25
Can you.35:26
Yeah.35:27
Where where's your sales order search panel looks like it's on 501.35:31
Where's what?35:32
Sorry, on line 501 there you have a sales order search panel.35:36
We can use that to just get the property name off of it real quick for sure by typing after that custom key null.35:42
Yep.35:43
Yeah.35:44
So then just start typing like sub.35:48
Yeah, just sort of note that.35:49
So what the?35:51
Yeah, scroll back up what's coming in what I got is it said has has sales group as sub.35:57
Yeah yeah that's the boolean value that will say has sales group as sub.36:01
That will mean that it is a sales order.36:03
A sub order if with that value is true, the the same thing should be there for master as well.36:10
So where it says has Sales group as master, true would be in the only giving back masters and then the both the the no order could ever be both at the same time.36:27
And if target group the target orders just checkout is turned on correctly, which it should be for the last several years.36:34
No order should be in the system without being at least one of those two things.36:49
Okay.36:52
So then he's gonna call and you said it's 11 invoice for info right?37:06
So he needs the has sales group as.37:09
So in BBC we need to talk to the client because as it stands it's impossible because they want one many or rather they want many sales orders on the sub side that have different shipping addresses to go over as a single sales order on the info side that it has or that can only ever have one shipping address.37:32
So if we do it as is we're going to lose data.37:35
We need to talk to them about how that's going to work First, if the splits are controlled not by via the UI where the user could select multiple addresses and they could only ship to 1 address and then the splits are instead being handled by data like separate store IDs or separate vendor IDs etc.37:59
That should be doable because the only because there will only be one shipping address on there.38:04
It's just they're all coming from different origins, which I'm guessing they would be handling internally through some other mechanism on that system.38:12
So that would be OK.38:14
Origins you'll need to verify again, you'll need to verify that just like you said with Actually, I believe the origin would be probably the warehouse code on the info side, and there's also only one of those, or actually I take it back that might be handleable per line item, but I don't know it or I don't know how many other Erp's allow that as well.38:39
Not very many, yeah.38:41
So again, I stand by my original thing, just leave the simple case of pure suborders as common and then anything else is just custom.38:52
And like once we've done enough of these to have an idea of what all the different systems actually support, then we can maybe take a look at trying to make something a common customization.39:03
And and if you guys can collect information together amongst yourselves about times where you're calling Seth to get a large piece of data in order to access a particular property off that data.39:16
I can then go bake in calls that will more directly provide that information that you're looking for so that you don't have to lose the performance of making the bigger call just to get the little piece of data.39:32
That would be awesome.39:33
Thank you.39:34
Yeah, yeah, that'd be great.39:34
Unfortunately, just with the way the EF adapt and you know even Dapper's gonna still gonna have like problems with accessing data a certain way.39:42
But when you're trying to get data that is nested like multiple layers deep and there's a whole lot of different branches of whatever you're trying to do in your object graph that's that's the slower your response time's gonna be.39:54
Whether you're going database direct or you're doing it through the mapping system that's that has Seth, so it the more targeted you can be.40:03
About what you're asking for, and the more specific around how you're asking for it, the better.40:13
I'm actually running into this really hard with one of the the Service request form in the Wbs's website right now.40:20
They had written a call originally to ask for all the accounts that are affiliated with the main account that the user was logged in as so that they could order on behalf of, and then all of the addresses for all of those accounts all to be embedded and pulled back On that single call.40:40
It took more than 3 minutes to load the page because of that.40:46
There were because there were 164 references.40:48
Imagine if they had thousands, right?40:50
I mean it's it's it gets ridiculous.40:52
But instead if I ask for, I.40:55
If I change the service call to say, give me on this call, give me the list of accounts I have access to on this call, give me the last the list of each of the contacts for this account that I now selected.41:07
Inside the UI of which one I'm going to do.41:09
I now have much more targeted stuff that gets done in less than a second each, and I no longer have a three minute page to wait time to load.41:25
Makes sense.41:32
So I'm trying to think in this particular case, I just got a get up with PVC and say hey let's go ahead and clarify what we're looking for with the sales orders and then we can work from there because I'm probably, we're not probably not gonna find an exact solution right now, I imagine, Right, right.41:52
And that's fine details of how they want the order to be put into import, yeah.42:00
And honestly, I this is so much farther than I would have gotten myself anyway.42:03
So I really do appreciate you coming, James, and talking us about this.42:08
It really helps explain this puzzle to me a lot because looking at the data itself, it was not making a whole lot of sense.42:16
Thank you for helping make sense of that.42:18
Yeah.42:21
Why is it there?42:22
Because somebody asked for it at some point in the last 10 years.42:26
Yeah.42:30
Okay, cool.42:31
So then I guess no changes to the sales order, stuff today, correct?42:35
JL Scarlett, what do you guys think?42:37
We do need a change overall to have the like I'm saying they get common function has to know to pull the sales group and or master order in order to properly get all the information about the child order.42:53
But other than that it should just be business as usual.42:57
So are these contacts not accurate then?43:00
The ones are on the sales order model.43:03
The shipping one is, it's the billing one that we have to grab from the sales group, which that, yeah, we don't assign it to the suborder because it's we only need to assign it once.43:12
So we assign it on the sales group and on the on the master order itself, which I think this can also be optimized quite a bit already.43:20
We don't need to do those reads if the billing contact property is already not null, which it wouldn't be because in this case like 99% of the time this is gonna be coming from a a recall to the sales order.43:35
So we'll we should already have the contact being mapped back.43:38
Yeah yeah.43:39
So PBC is on an older version of the Seth connector, so it has inefficiencies like this still in it.43:45
Yeah it is.43:46
It is pre ACTDB.43:47
Ohh sure.43:50
Well, let's take a look at what we have currently armed a video.44:17
Yeah, this only reads if it's null.44:21
You're saying not zero today, right?44:34
Allow a negative number.44:36
So you would actually want greater than 0.44:46
Well, and if you haven't seen it yet, in the develop branch, the contracts that we use in CEF have been all pushed to their own project and it's that project builds in.net48and.net seven.45:01
So if you want to reference that project like copy it into connect, it's got all these contracts in it that says like contract dot check valid ID, contract that requires valid ID, those kinds of things in there that will simplify a bunch of that stuff.45:13
And Brendan Lyon has reworked them with all the latest math versions of that.45:20
So if they are number based, it actually uses the core like hyper efficient pattern checking on it now, which is even more efficient than what we what you just typed there.45:32
OK, but we're actually still missing something there because we want to take the oh oh nevermind.45:42
I see there's no closing assignment now.45:44
That's right.45:47
Yeah.45:52
So in this one, not necessarily the PBC code but this one we'd add this to do, Yep.46:01
So there we would need to say like I guess it'll be a full set of if branches probably.46:10
Yeah.46:10
Where if the orders was it like sales group ID as sub is non null, then we need to use that to go grab like the actual sales group's billing address or rather the I guess the master order of the sales group's billing address.46:31
Unless the customization has occurred within the client that model that's shipping that's currently on line 49 will never actually be populated.46:40
Ohh, really?46:42
Yeah.46:42
That so like that would only ever be populated by a customization.46:45
It's a dead column from a really old setup.46:49
Ohh, do we just drop it entirely then?46:52
I I would from the common model part because it would only ever be instead of a custom thing that a client did.46:58
Sure.46:59
I think the old thing tried to fill in common fulfillment before we decided to move that on to a later version.47:06
Yep, makes sense.47:16
OK, And then this billing contact to become that masters, right?47:28
Instead of what's on here normally, right.47:32
If this order is a sub, then we have to pull its master and grab billing contact off that.47:38
OK, got it.47:43
Then in terms of actual mapping itself, does any change here?47:47
Yeah, back in that function, because you're making several separate calls for different pieces of data and different things.47:53
That's a candidate for like doing a like threading all those so they can all run at the same time.47:59
Yeah, and that'll improve the performance because now it'll it'll ask the server for all these separate pieces at once.48:05
And your total time isn't sequential, it's it's concurrent.48:09
Yeah, I would not use test dot went all directly.48:13
I would probably use like the 4H async after putting them all all the tasks into a collection.48:17
But yeah, especially as you can admit the 4H that's on 52 where you're you're asking for every payment.48:27
Generally, there's only gonna be one, but there could be multiple if it was a partial payment each time For that, yeah, client okay, we're we are already hitting Seth pretty hard though, because we will have multiple child jobs running at once.48:44
I don't care.48:44
Hit it as hard as you want.48:48
If there's a problem with Seth coming back because it's being hit too hard, then get with me and I will figure out how to make Seth run better.48:57
I want you to hit Seth hard.49:02
Well, you came to the right place.49:04
There's a quote.49:05
Hit Seth hard.49:06
Seth's daddy.49:07
Yeah.49:08
I want you.49:08
Yeah.49:09
I want you to hit Seth hard.49:10
I I don't want you to hit Seth stupid.49:14
OK, Perfect.49:22
Prepare it to be inundated because we we do things wide open usually.49:27
I know.49:28
I I've made genetics that run screaming millions of records in 15 minutes.49:32
Yep.49:34
I've.49:34
I've done it.49:35
So I know you guys can too.49:39
Yep.49:40
OK, stuff crash from a really fast product that gives a rite of passage over here.49:45
Exactly.49:49
OK, perfect.49:50
Anything else we should change about sales orders and maybe we get into some of the other problems you were seeing Scarlett, while we've got James here, the product stuff for for that, it's the because the product endpoint resolves by name by default.50:11
And so for whatever reason since Nat decided to duplicate their products thousands ohh of times that kind of causes an issue.50:19
Yeah yeah.50:21
They're really setting in SEF to make it so that it cannot resolve a product by name and must use the key.50:28
Yeah, that's not the because Nat went from having 8000 products to having 200 and thousand products in Sage 100.50:39
So and all of them were created on a specific date.50:42
So it's very likely that just something weird happened with Sage 100.50:50
So if there's not 200,000 valid ones, but there's only 8 thousandish valid ones, then I would be running up the flagpole over there and say, hey ******* fix your ****.51:00
Yeah, we are plugged until you fix that and we agree we're not doing categories anymore, right?51:08
Yeah, that it's just not very commentable in general at least.51:16
Yeah, especially with PVC.51:17
That's that is complete custom.51:20
Okay okay, what about the the contact model that you saw with users missing in contact?51:36
Scarlett.51:39
That one.51:40
Let me look at the error again.51:46
Failed to read contact model by D831.51:50
Yeah, so for that one, the only thing I can think of is either because it was hitting a QA site, so either it died mid transit or with that one.52:01
When a contact is deleted, does it not delete or reset the user thing because that was on where was it?52:12
Yeah, that was trying to read the contact on a user and for a reason that contact didn't exist was that error.52:23
That's not something we should ever see, is it?52:24
James User should always have a contact.52:28
A user should always have a contact if it was currented through standard registration process.52:34
If it was pushing, maybe not specific if it was created in the database.52:40
If it was manually created in the database and no contact was put shut on it, that's a bad state for that user.52:45
I would go create a contact and throw it on the user even if it had no data in it.52:50
Okay, OK Dimension.53:08
When you created directly in the database, you can't create the password hashing or any of that other stuff correctly, right?53:17
OK, so maybe just a project specific thing.53:21
What else can you guys think of that?53:25
We'd like to pick James' brain about override bool 28 minutes before I blow up.53:32
What was that, Dale?53:33
Override bool equals?53:38
Oh yeah, yeah yeah that's in step side right?53:43
Is that a question?53:46
Yes.53:46
So we've run into issues where.53:50
So obviously we don't always have to push with the custom key, especially nowadays since we just resolved things on our side.53:59
The standard implementation for the base models equals is ID is equal to the other ID, which if they're both null, that's always obviously always true.54:10
And custom key is equal to the other custom key, which again if they're both null or empty, that's always true.54:16
Active is equal to the other active, so largely going to be true.54:20
And serials will attributes equals others serials by attributes by reference.54:24
So if we don't actually have to send any serials any serializable attributes, then that's always going to be true as well.54:31
So basically any two things in Seth will compare as true.54:36
Yeah, and a lot of the cases Connect handles them.54:50
But there was 03 models and mappings, 03 Clarity ecommerce dot models actually, if you want just look for base model dot cs.55:01
If you control P, give Azure five hours.55:10
Admit it, you're confident of this.55:13
There, there we go.55:16
And it's line 233.55:22
Yeah, this one.55:27
So that causes issues with Newton soft JSON serialization because it will keep.55:34
I believe it was a hash set of all of the OR channel markers hash set exactly, but it keeps some sort of internal list of all the items that it it has seen in the current serialization session property, the dollar ID property.55:58
If you were looking at the raw JSON string, you would probably see dollar ID on every object inside there at every at every level of depth.56:06
But when, like when we're just calling Newtonsoft the JSON convert dot serialize object on a Seth object, so let's say that we're calling that an Account.56:16
None of its account contacts have custom keys, sets, none of it's serialized by attributes or anything other than null.56:24
They're all active.56:25
They don't have an ID yet because we're pushing in for the first time.56:28
Then we actually peeked in the Newtonsoft source code and it will keep a list of all the objects that has seen so far in order to look for reference loops.56:38
And because it relies on the like normal equals implementation.56:44
Then, for example the the account itself can compare as true to the can compare as equal to its account contact, its account contact is equal to its contact, its contact is equal to its address, etcetera.56:59
So we end up getting immediate errors about reference loops, even though there's not actually a reference loop because they're all coming from base model, right?57:13
Yep.57:13
And in particular they don't have custom keys since like, we don't really need to assign those typically.57:19
Yep, yeah, the idea would be 0 custom keys.57:21
Null.57:22
Active is true, serialize attributes is null.57:25
So those two things are inherently equal.57:28
Yeah yeah so you're OK.57:37
I'm sorry.57:37
I'm trying to make sure I understand what what you're saying.57:40
So you're comparing an account with an account account rather than Newton.57:45
Self dot JSON is doing the comparison.57:48
Why select Where's Newton?57:49
Stop doing the doing the comparison inside JSON convert dot serialize object.57:55
Why is it comparing them?57:58
That's a very good question for doing self.58:01
Like where where is this comparison happening?58:04
Yeah, it's doing it for reference loop checking or yeah, reference loop checking.58:09
Inside there we've had a fun little debugging session going through the ND compilation.58:16
Let's figure that one out.58:18
OK, do you have the raw JSON string on each side or do you have the?58:23
We have the C# object and we are trying to turn it into JSON like in order to send in the staff right in in here bring up serializable attributes dictionary extensions dot cs.58:38
You see those JSON serializer settings I've changed?58:51
I have changed the reference loop handling in a version of CEF that was within the last two years.58:56
I don't remember which one so that instead of ignoring them, it now serializes them as a dollar ID.59:05
That might be part of where you're running into is that if you're not using those same settings on your end when you call utsoft dot JSON serialize or deserialize, it might be getting confused.59:16
Thought we had tried the different reference loop handling options.59:20
You can you can try them again, but I I wouldn't just do like the one property.59:25
I would take that entire JSON settings value set so that you're doing it identically because of the way the contracts resolve and all that stuff is in there.59:36
And obviously this doesn't work the same way in STJ.59:38
So if we're talking about STJ, that's a totally different we we've already been that, not been on that road.59:43
We know we only use Newtonsoft dot JSON on stuff.59:47
OK, where can I set the default settings for HTP client?59:53
Nope, it's not.59:54
This wouldn't be the HTTP client.59:56
If you scroll up, go to.1:00:01
I thought we had just JSON settings search for ohh shoot.1:00:05
Why does new Insoft call it again?1:00:07
Does it call up the JSON serializer setting settings or JSON serializer options or JSON options?1:00:14
If you go back a little stuff for search text for JSON options I think that would be it.1:00:21
Nope.1:00:22
It looks like JSON options is system text JSON since I I saw it popping up in Ekdb and we definitely don't use new Insoft there.1:00:31
Then just look for JSON convert dot with the control F yeah serialize object right there.1:00:45
So that's a JSON convert serialize object.1:00:46
It has the search model that you're passing into it, but you're not passing the secondary argument which would have this other piece.1:00:52
Since you're in SEF service, you should have access to the serialized blocks of dictionary extensions right there.1:00:57
So you should be able to add a second argument to the serialize object.1:01:05
What was it called?1:01:06
Jason, Serializable Serializable dictionary extensions.1:01:11
And let's actually probably make a property up on self-service here that just stores the JSON options so we we don't have to type out like the full qualified name every single time.1:01:19
Well that's that's why I also have an extension method called JSON serialize width settings no which internally calls JSON convert dot whatever and then passes that into it.1:01:33
So if you do like search model dot JSON serialized with settings, okay.1:01:42
So you're not seeing it there.1:01:42
You might need to pull in the name space from so it might be in another dll perhaps?1:01:49
Yeah go go go tab back over to the browser window where you've got the thing up.1:01:54
It is the namespace of Clarity ecommerce interfaces, models.1:01:57
Try adding that namespaces using in your file real quick because you're you should be referencing this DA little bit added.1:02:06
It was there from when I brought in just directly.1:02:09
And writer definitely should be.1:02:11
So then if you go back down to search model dot.1:02:15
Yep yeah so search model dot JSON serialize just start typing JSON.1:02:31
So to fill it to narrow it down.1:02:33
OK, I don't know why it's try try accessing try accessing them not to the other way.1:02:38
Try what Jill, try accessing them not to the other way just to double check.1:02:42
So like do the serializable attribute dictionary extensions dot.1:02:48
Nope not not JSON serializable serializable attribute dictionary extensions.1:02:53
Yep yes dot okay.1:02:55
So the version you're referencing is older than where I've had the fix.1:03:00
Yeah so we we probably just need to like I was saying before, just make a property up there that just references the JSON settings from sevs dll and then just swap that because we this is only gonna work for newer projects then right?1:03:13
Looks like JSON settings are gonna be in the old ones too.1:03:15
The JSON settings have been inside SEF since 12,016.1:03:21
Okay they changed around 20/20/2021 and then the latest versions of Seth like 2023 should have the extension method so that you don't have to call JSON convert dot serializable object with two with two arguments.1:03:37
You could just call the one extension method and it just does it okay.1:03:40
So right now if you cut that prop that value on 167 and add it as the second argument and serialize object on the next line, it would work.1:03:50
Not there, not undo what you just did as a second argument to the serialize object function call.1:04:03
Oops, clean up your white space.1:04:11
Which for this just so we can retain compatibility with older CEF projects, how about we just add a serialize CEF object method?1:04:24
But this this shouldn't break anything right?1:04:25
Because the JSON settings are in the Dlls that's gonna be referencing directly so we're not having to repeat that everywhere.1:04:31
Ohh sure, because then we're also repeating that in the post and other stuff and if it changes then we also have to change it across the many locations.1:04:39
Yeah, the way that you have formatted this line is going to confuse the person who looks at this the next time because serialized object has two, sure, two arguments to it and you put them in the same.1:04:49
I would put search model on its own line as well.1:04:51
Yeah, there, now that's the part that makes sense, yeah.1:04:55
And then if you were to convert that content into its own extension method that just internally called the internalized box for dictionary extensions, then you wouldn't have to worry about whether you were on a newer or older version of CEF, because right, the JSON settings would still be there.1:05:29
Well, not just Seth's methods, but Seth's JSON settings.1:05:33
Yeah, because we may or may not have the methods.1:05:36
Yep, Yep, perfect.1:05:47
And funny enough that that issue that we saw what made us find that equals thing was on PBC for Jim the older version of stuff.1:05:58
So now you have it serializing it and then you're gonna be you're using this to send it over the wire on the post call next and then Seth is on the that side and it's trying to do the equals compare and then you're running into a problem there.1:06:12
Or is it that equals comparison was happening on connect side so connect was calling JSON convert dot serialize object and then just like stepping into that.1:06:22
So then where is that code?1:06:24
Because that's not what we just changed.1:06:27
We were just calling zeros objects second ago right.1:06:30
And when we did it on a call that was to a post A to to a post async, where's the call where you were actually doing an equals comparison?1:06:37
You know we do not directly compile call equals.1:06:41
So basically here.1:06:44
Yeah.1:06:45
Can you try just F-12 ING into zeros object?1:06:47
Yep.1:06:49
Or whatever the heck writer uses.1:06:51
All right, now I'll go into internal.1:06:55
Well, let's see if you can find this.1:06:56
Now go into serialize, serials internal.1:07:03
We can have plenty of those.1:07:06
All right, so where are you?1:07:10
I don't remember if it's this one.1:07:13
I forgot to keep going further in.1:07:17
There's something that will keep a list of or list or hash that or something like that of objects that is seen.1:07:30
That's work I think.1:07:39
And hover value for a second value is a JSON type right?1:07:44
Where?1:07:44
Line 1146 Object.1:07:48
OK Nope.1:07:48
So go into that dot serialize perhaps?1:08:01
Nope, That's talking about like JSON references.1:08:04
Yep, there's an actual reference loop like Detector which that that's talking about C# references, and then obviously would like to have to go into using JSON references if it finds them.1:08:16
If it's allowed to, where are you?1:08:22
Wait, Scroll back up a little bit.1:08:26
Search a little bit more.1:08:28
Is this in deserialize right there?1:08:30
That's serialize value.1:08:30
OK, go into.1:08:33
Wait, stop.1:08:37
The bigger law going to serialize object right there.1:08:40
181 here we are.1:08:44
Serialize stack.1:08:45
I think that's our guy there.1:08:47
Yeah.1:08:48
So just a list of objects.1:08:49
And now if you look for references to serialize stack, everything's about the references today.1:08:58
Holy crap, yeah, so we need to find the place that's you have the check for circular reference one right there.1:09:07
Yeah, it's probably it.1:09:09
Yeah, contains and then yeah.1:09:12
So it's because of the quality comparer, but I think the default one there is going to use the normal object dot equals which will just end up going straight back to the CEF virtual Google One where like two things can be equal despite being different types and all of that.1:09:33
So that's where we run into it.1:09:35
So we were not directly calling equals ourselves.1:09:38
We were just calling Newton S at JSON and telling it to serialize something.1:09:44
And that was failing because according to the like default CEF equals method, two things can be equal.1:09:54
Well, I guess just when they're not actually equal.1:09:57
And that's why we had to have the ignore, right?1:10:00
Nope.1:10:01
I think the only way we were, we were able to fix it last time around was we had to like explicitly new up 0 attributes in every place that we knew to up a model.1:10:10
Yeah, Yep, I remember doing that.1:10:12
And that's that's carried over to the connector.1:10:15
Now when you got that, that's in the current connector.1:10:17
So that like that's already working mostly.1:10:21
Technically.1:10:22
How long has this been going on?1:10:25
I've seen, I saw this like based a year or two ago and then obviously again with is that that on in Seth's side on the base model, the equals override has been there since 2015.1:10:39
Yeah.1:10:39
And I've never heard of this being a problem, probably because we always set keys on stuff, right?1:10:44
Yeah, yeah.1:10:45
But there's no real common way to always set a key, so we can't like we don't always do that anymore.1:10:51
Previously we would have the reference loop handling by default on our end set to dot ignore right?1:10:57
Because out of box their default is error.1:10:59
We set it to ignore.1:11:01
And then recently I changed it so that it does a serialize which is where the if you look at the JSON output string, you'll see a dollar ID property.1:11:10
That's the reference to each like to each reference of another object within the JSON.1:11:18
So yes, like this.1:11:20
So like right now the CCA Scantran submitted attribute value has a dollar ID of two.1:11:26
Something else inside here could reference dollar ID 2 as a property value and it would know to read that CCA one from the top there.1:11:36
So and that's how that that referencing works.1:11:39
It also means that if you ever duplicate the dollar IDs in the JSON, because you've manually modified the JSON yourself, you've broken the JSON and it will never work again until you undo whatever you've done.1:11:50
The only safe way to really like modify the JSON manually at that point is then to, especially if you're adding new objects, is to then strip all of the dollar IDs out so that it then becomes like the regular JSON that we used to have, and then it basically just default handles them as null for stuff.1:12:07
Will it repopulate them when it sees on the stuff side If if you reserialize it back to a string, yes it will repopulate them as a new thing, but okay, I've never tried it like where like you took it off of three or four objects and like left the rest.1:12:20
Yes as far as I've done.1:12:21
Do it wholesale throughout the entire JSON.1:12:23
Yeah, just strip them all out.1:12:25
Now the specific action where you're trying to actually having to to to to make it react differently was that if how?1:12:34
Was there a situation where like an account was being compared with an account contact?1:12:39
Because if it is serializing an accounts then it will descend into the list of account contacts and that serialized stack will still contain the original account so then it will be checking so here so here OK so so internally it's 1 hash set or whatever and yeah every individual object is served in there and that it's in the the compare so serialize it should evaluate to true.1:13:02
So it's not storing by the.1:13:03
OK, let's say that we just to send it into the account contacts list serialized that contains the account and probably only the account and then value here would be the account contact so that it would be doing that contains and then account contacts like dot equals the accounts.1:13:21
So I'll go over to the base model virtual bull equals and then it would be seeing that well the custom keys are both null, ID's are both null, active is true and Cos of attributes are both null so they must be the same thing.1:13:35
Ohh no we have a reference loop.1:13:38
Yep OK which I don't know.1:13:42
So why?1:13:43
Part of the part of the problem that I'm seeing here is I think this protected virtual boot equals was meant to be handled for this level of of the model.1:13:54
And then like if it was an Amo base it had an override of the equals again that would call this base to take care of those properties and then they would add on name a description on the name.1:14:03
Ohh sure so so and so on up the line to the chain.1:14:07
What I think we're missing is like an equals that is actually at like the account contact level that would say that it would didn't compare all the individual properties on those things on it.1:14:16
Because of that, it's weird that I have never seen this actually, because a problem before now.1:14:23
Again, a lot of time we probably would have been sending custom keys, yeah, but so it's because if we've never been able to match because it would whenever there.1:14:31
But now you've got a a newer and and arguably better way of doing things and that has unfortunately caused side effect of uncovering this issue.1:14:39
Yep, okay.1:14:40
And in general we just can't.1:14:41
I think I understand what's happened now and I think I understand what I would do to fix it.1:14:47
The fix would probably involve writing something into the T4 for the models and then having that spit out and then basically it wouldn't be fixed until the next version of Seth.1:15:01
I can get it into the develop you know as soon as I can get the approval for to do the work.1:15:06
But you know the sucks in this app and the and the whole goal of this equals was basically so that we could ignore certain problem certain properties in the compare because like compare the created date and updated date didn't matter for comparison.1:15:19
It wasn't part of the data set that we were actually trying to check and that's why there was also like if you look at some of these the the get hash code, the the we were overwriting the hash code to ignore those kinds of properties and only grab the ones that we cared about to actually compare against.1:15:35
So this hashing No2 because yeah we didn't want to compare two different created dates when all the other data was the same so the hash code could align.1:15:45
That was an earlier form of doing the digest CRC either 32 or 64 calls when we were doing hashing for connect earlier on for comparison and it was so that Also I I know this because it it it gets to a variety of other things that we were trying to compare but we wanted to be able to compare like the entity record with the model within equals so that it would compare you know the properties we cared about active custom key attributes, name, description, but not created date updated date or any other like time stampy type things that didn't matter.1:16:24
We we were trying to solve that problem as we were doing it and I think that maybe where some of these problems that like had been have been fine because we haven't actually used them in anywhere for different stuff.1:16:34
But now it's actually affecting you then that you guys have this newer process in place.1:16:39
Was that the simpler fix of just including the type and equals?1:16:43
Obviously that's not gonna work between two account contacts since that was still compares equals but it would at least solve like the basic setup of obviously you know Account isn't is never gonna be equal to a Account contact.1:16:58
If we can sign someone up to physically go through and make a PR to each release yeah hopefully to the 2019 releases of Seth with that type at check put on there.1:17:12
I'm fine with that.1:17:14
I'll try to get that approved to get somebody to actually physically do that and then everyone who is on each version of Seth has to pull that update from that release into their client project and then everyone.1:17:27
So it's going to be as well propagates all all through everything.1:17:31
Unfortunately I spent like going forwards at least for develop and onwards might just be easier to.1:17:37
I will just definitely get smarter stuff in develop because I can put the I'll put the type check on at at a minimum I can do that like tomorrow.1:17:46
And then I wanna add these smarter equals overrides to the T4 so it's spitting out those equals correctly for everything because I didn't realize this was causing a problem for you guys on this end.1:17:56
Now with it awesome.1:18:03
And so with the smarter equals we'll get an override and then the equals will then check all these things.1:18:07
What I'll probably really do is I'm gonna look into the get hash codes that have been because there's a modern version of that stuff now.1:18:16
If you scroll back up you see the the difference between the netfiber greater and the other one.1:18:22
Yeah netfiber greater one is much smarter for that.1:18:26
I wanna look at best practices for for overwriting hash code and equals for those things and I'll see about like what the what the the modern version of that is for both.net frameworkand.net core.1:18:39
Because this is gonna affect us when we switch over to to Phoenix as well for this stuff because the models are all gonna be comparable somehow and I wanna make sure that's comparable for connect as well.1:18:52
Adding the on there type to the hash code should probably be enough since anything that's gonna use a hash code is still gonna do a dot equals comparison afterwards.1:19:01
So keeping the hash code super fast to run is probably better.1:19:07
Yeah.1:19:08
So I wanna I wanna make that hash code doing the correct properties and then the equals basically is just gonna use the hash code to do it.1:19:15
Unless that's not the best modern practice.1:19:17
So I'm I wanna do research that before I actually implement a change.1:19:21
At minimum I'm gonna add the type check to the equals right now so that we have that to get around the immediate issue.1:19:27
And then following that, I'm going to try to extend our equals and get hash codes calls to be smarter overall at all levels of the model, not just what's been inherited, so that they're running as quick as possible and with the best practices.1:19:48
We don't generally do any hashing on our projects nowadays, do we?1:19:52
Well, that's for I checking like you changed.1:19:56
Yeah, so.1:20:06
What are you doing instead of what the hashing was providing in connect then?1:20:10
Because the goal of the hashing was realizing that you already had the same data in SEF, so you didn't need to try.1:20:16
Try to push it up cert caller or whatever to SEF, which could theoretically cause problems on Sef's end with bugs when you could have just avoided sending the data altogether.1:20:28
So what What are you doing instead?1:20:32
Usually go by modify dates.1:20:35
It's just modified.1:20:36
Yeah, modified date, unfortunately could stand, just just say that it was modified, but nothing actually changed.1:20:43
In which case, right, you're sending something that's that's not changing anything, right?1:20:47
Which could happen, depending on the client, 10s of thousands or millions of times on a single sync run.1:20:54
Yeah.1:20:54
The issue is there's no way we can really commonize detecting of the changes, at least currently.1:21:00
Well, it's not.1:21:01
The way that we were doing that was we were taking the relevant properties, adding them into a digest dot CRC hash call which created a value that was a U long was what was stored as a long.1:21:12
And then you could, you could recheck with the digest that hash code just because if it had the same hash code that was generated by that, then you knew that you didn't need to send anything further over the wire to Seth.1:21:23
And that's great for Seth, but what about other systems you would figure out something to to store it something somewhere in in theory, I mean obviously doesn't work for every system I'm.1:21:39
But I mean if if most of the clients that we're working with are talking to Seth, then that that they're at least not inundating Seth with extra calls.1:21:47
I would imagine if it was like Dynamics GP or something.1:21:50
There are like comment fields on a lot of stuff where you could store that kind of value in there.1:21:56
If it's just about storing, you determined on every system to to figure out where it needs to store that info.1:22:01
I mean, if it's just about storing then we can always just stop that the EKB key table, right?1:22:06
Yep, yeah, that's the only issue there is still handling customizations and stuff like we would need more than just the two hour base to do sync stuff at that point.1:22:18
Yeah, yeah.1:22:18
I mean, from the next side, we don't have the luxury of time.1:22:26
Yeah.1:22:30
So my question there is as far as commonality is concerned, you're gonna go sync a product you're pulling certain properties out of, let's say in four or whatever.1:22:46
All the properties you're pulling happen in one spot.1:22:50
Like I see a map to common product here on the screen or an invoice or whatever they're they're usually in one maybe 2 spots depending on how many sub objects you have to like pull in with it.1:23:02
You can pull each of those properties and then like make a make a function right next to this one that says well I pulled the invoice dot created date, invoice dot last modified date, invoice dot shipping cost, invoice dot disc account amount, invoice dot billing address and I'll I'll go in and look at my map to common address.1:23:24
Every one of those properties you just add it to your string that's going into your hash and then that that becomes part of the digest CRC call and we were using.1:23:35
You could use a different algorithm as well as long as you're consistent.1:23:37
But we had built in those algorithms into our thing as digest CRC 32 and 64 because they were quick and they were relevant to what we were doing and they and they were easily storable as a as a long inside the database for the stuff.1:23:56
So every one of those properties that you're reading, you put into the hash, and it should be consistently loading it out of that thing.1:24:05
And it doesn't matter what the other end stores because this is where you pulled from that, and that's what you're actually trying to check against is what did you pull and what was the data?1:24:15
Look, what value did it arrive at with the data that you pulled from on it?1:24:22
And then you store the final value, which is just a long in some field somewhere on the other end.1:24:29
And Seth, we have an actual hash column, so we have it on every table because it's part of the base model.1:24:34
So we we could easily store that all over the place.1:24:36
And if we needed to store extra ones because we had multiple sources of truth for different pieces of data, we would just then we would usually then add them to the serializable attributes.1:24:46
In one case, I actually added an extra column to a table because of the performance problem of doing that, but that was a that was a special handling case for that one.1:24:57
So if you need to store that information in, let's say that your source of truth was Shopify and your target was Dynamics GP, you could store that that hash value in Gp's comment field of the product that we have.1:25:17
We have, we can, we can just yeah, add a new new column to the yeah and yeah, you guys have that EKB, the DB, whatever thingy.1:25:24
Now that you could store all those values inside there and no one else has to care about it.1:25:30
That way, even if you get a new modified date, if the data that arrives is the same, it doesn't matter that the date was different.1:25:36
Especially for systems where there is no modified date because they're giving you a text file of CSV or custom data that you have to that has literally 9,000,000 records in it because they're stupid.1:25:48
Yes, it's actually happened, or the OR the data with no dates on it.1:25:53
Then you're using the hash value and it doesn't matter what the date says or or date could have said because the hash value's arriving at the same data.1:26:00
I don't have to sync 9,000,000 records, I don't have to mount it move like 2000 because two only 2000 actually changed of where the stuff.1:26:07
And I've saved myself all that processing time of talking to the output side of the of the equation for for any of where where my targets are for my data.1:26:17
Because my source didn't change, you know my in number of targets don't have to take data just to realize that nothing changed.1:26:33
So that's how we turned a 7 hour sync on Tel to 3 minutes, right?1:26:49
Anything else in the last?1:26:50
I know we're a little over while I'm here.1:26:54
Anything else you want to ask if you want to stick around?1:26:56
I think that's a lot of it.1:26:59
I think maybe what we ought to do then is set up like once every six weeks or something, we'll have you come in and we'll kind of discuss the problems we ran into and see how we can optimize it.1:27:07
I think the the next thing we want to do is get you involved with creating some 4 Connect endpoints.1:27:14
That'll help make things, especially on the pay hub side, make things come together smoother.1:27:20
Okay.1:27:22
Awesome.1:27:23
This is great.1:27:24
Thanks, James.1:27:25
Yeah.1:27:26
Yeah.1:27:26
Thank you.1:27:26
All right.1:27:29
Awesome.1:27:29
Have a good weekend, guys.1:27:30
Ohh.1:27:33
And welcome back, Austin.1:27:35
Glad to have you here, buddy.1:27:36
Yeah, buddy.1:27:38
Thanks.1:27:39
Glad to be back.1:27:41
Yeah.1:27:41
It's gonna be a good time.1:27:46
So down that you're back.1:27:47
We have these five stage X3 projects coming in.1:27:50
No, no, no.1:27:51
They're Eclipse projects.1:27:52
Epicore, Eclipse man.1:27:54
They're not X3.1:27:57
Ohh, that's that's the easier system, right?1:28:02
Easier.1:28:05
Especially since you'll get to make the Eclipse connector.1:28:08
Aren't you joyous for that?1:28:12
I.