00:00:02.260 --> 00:00:05.443 Anybody has any questions, feel free to just unmute, interrupt
00:00:05.443 --> 00:00:08.778 me or raise your hand, whatever. I don't care. I'll be keeping an
00:00:08.778 --> 00:00:11.709 eye on it so that I can answer questions as they come up.
00:00:12.440 --> 00:00:13.780 Um, so.
00:00:14.690 --> 00:00:19.599 Um, so the way that uh. The way that uh, Elasticsearch works and
00:00:19.599 --> 00:00:24.433 the way that we're using it in self is broken out into two main
00:00:24.433 --> 00:00:28.210 steps. Number one is indexing and #2 is querying.
00:00:29.470 --> 00:00:33.703 So you've all heard by now index the products or reindex your
00:00:33.703 --> 00:00:37.050 products or is that data in the index, whatever.
00:00:37.720 --> 00:00:42.030 Uh, that's all relevant to step one. Um, so uh. Specifically,
00:00:42.030 --> 00:00:46.271 under the searching provider, the files that handle indexing
00:00:46.271 --> 00:00:49.400 are the hilariously named dump reader files.
00:00:50.100 --> 00:00:53.718 Um, and I think I heard James say last week. That's just what
00:00:53.718 --> 00:00:57.220 they called him in like the elastic search documentation or
00:00:57.220 --> 00:01:00.956 like sample code, which makes it even funnier. In in the sample
00:01:00.956 --> 00:01:04.516 code, which works off of running an index against the nugget
00:01:04.516 --> 00:01:08.310 package repo, that's the name of the file that they had for this
00:01:08.310 --> 00:01:11.928 was dump reader. So they were reading a dump of nugget. Yeah,
00:01:11.928 --> 00:01:15.664 that makes sense. It's a dump of data, but it's doesn't make it
00:01:15.664 --> 00:01:16.889 any less funny to me.
00:01:18.160 --> 00:01:21.487 And it will always be funny. Yeah. Because children. Yeah.
00:01:21.487 --> 00:01:24.814 Yeah. Well, you know, if you can't laugh at that, what can
00:01:24.814 --> 00:01:28.311 you laugh? Well, the 5th grader and all of them. Yeah, that's
00:01:28.311 --> 00:01:28.650 right.
00:01:29.670 --> 00:01:33.507 Um, so anyway, uh, the first half of the operation for of
00:01:33.507 --> 00:01:37.676 using elastic is to index the data. And to do that, what we're
00:01:37.676 --> 00:01:41.844 doing is, and these dump reader files is there's gonna be this
00:01:41.844 --> 00:01:45.616 giant link query, and it's obscenely long, but what it's
00:01:45.616 --> 00:01:49.851 doing is super important. So the first thing that it's doing is
00:01:49.851 --> 00:01:53.887 it's applying some filters, and we're going to come back and
00:01:53.887 --> 00:01:57.923 talk about these filters in a minute because this is kind of
00:01:57.923 --> 00:01:59.710 your first line of defense.
00:01:59.790 --> 00:02:02.865 Debugging products not showing up in catalog and whatnot, but
00:02:02.865 --> 00:02:05.842 we'll come back to that. So we're gonna filter out products
00:02:05.842 --> 00:02:08.768 that we don't want to show up in the catalog, stuff that's
00:02:08.768 --> 00:02:11.844 inactive, set to not be visible or it's discontinued or we're
00:02:11.844 --> 00:02:14.870 filtering out that products type key, which again we'll talk
00:02:14.870 --> 00:02:18.045 about that more later. Once we filter down to the ones we want,
00:02:18.045 --> 00:02:21.170 we're going to select just the properties we care about off of
00:02:21.170 --> 00:02:24.395 that product and it's related or associated objects. So we index
00:02:24.395 --> 00:02:27.470 information about the brands that the product is assigned to,
00:02:27.470 --> 00:02:30.100 the categories. That's in franchises, manufacturers.
00:02:30.250 --> 00:02:34.043 Doors, vendors, ratings and some information about the variations
00:02:34.043 --> 00:02:37.664 of the product, the variance, which again, we'll talk a little
00:02:37.664 --> 00:02:40.998 more about that later. Once we've pulled all that off, we
00:02:40.998 --> 00:02:43.469 map it onto the concrete types and memory.
00:02:44.120 --> 00:02:47.504 And I'm I'm kind of blessed through this part because this
00:02:47.504 --> 00:02:50.889 section here I'll I'll talk a little bit more about things
00:02:50.889 --> 00:02:54.504 here later. But suffice to say this giant LINQ query what it's
00:02:54.504 --> 00:02:58.003 actually doing is it's defining the products that we care to
00:02:58.003 --> 00:03:01.560 read out of the database for the index and what properties we
00:03:01.560 --> 00:03:05.175 want. So this is gonna be your first place to go if you have a
00:03:05.175 --> 00:03:08.330 client customization or a feature for core whatever if
00:03:08.330 --> 00:03:11.715 you're changing anything about elastic and you need to and
00:03:11.715 --> 00:03:14.240 somebody's like we need to make it so that.
00:03:14.310 --> 00:03:17.529 Plastic can filter against um like for example what we did
00:03:17.529 --> 00:03:20.967 last week, the starting price on the lot. This is where you're
00:03:20.967 --> 00:03:24.133 gonna go first because that starting price is not indexed
00:03:24.133 --> 00:03:27.625 and you can find out if it is by going in this file and looking
00:03:27.625 --> 00:03:30.845 at what's in this select right here, this first select. If
00:03:30.845 --> 00:03:34.174 starting price isn't in here, then you know that you need to
00:03:34.174 --> 00:03:36.030 pull that value out and index it.
00:03:36.690 --> 00:03:39.478 Um, and so I don't have this is on an older version of stuff. I
00:03:39.478 --> 00:03:42.310 should probably open my develop local so that it's a little more
00:03:42.310 --> 00:03:42.790 up to date.
00:03:43.480 --> 00:03:44.080 Um.
00:03:46.150 --> 00:03:46.970 But uh.
00:03:47.610 --> 00:03:51.528 So this all is all the stuff that you're going to be indexing
00:03:51.528 --> 00:03:52.540 on that product?
00:03:53.430 --> 00:03:56.420 Um, so you're gonna add it to here and then uh, you'll have to
00:03:56.420 --> 00:03:59.316 make sure that you come down in the second select and map it
00:03:59.316 --> 00:04:01.500 onto the concrete types, whichever one it is.
00:04:03.010 --> 00:04:06.492 And so that defines what we're actually reading out of the
00:04:06.492 --> 00:04:10.033 database. Now that's all cool and good, um, but the body of
00:04:10.033 --> 00:04:13.870 this for each down here is where we actually build up what we're
00:04:13.870 --> 00:04:17.470 indexing and storing in and any kind of post processing or I
00:04:17.470 --> 00:04:20.775 guess I should say pre processing we want to do on that
00:04:20.775 --> 00:04:24.139 data before it goes into the index. So for example this.
00:04:25.500 --> 00:04:28.954 Here is a a nice little tool we have where if the if the brand
00:04:28.954 --> 00:04:32.409 name coming out of the database isn't valid so it's null or an
00:04:32.409 --> 00:04:35.755 empty string or all white space or whatever, or replacing it
00:04:35.755 --> 00:04:39.045 with the string unbranded. And that makes it so that on the
00:04:39.045 --> 00:04:42.555 catalog we can bundle all of the unbranded ones into a friendly
00:04:42.555 --> 00:04:44.310 string that just says unbranded.
00:04:46.460 --> 00:04:49.367 And then we build up the and so there's different kind of
00:04:49.367 --> 00:04:52.526 similar preprocessing or post processing we can do on the data
00:04:52.526 --> 00:04:55.584 here, um, that sometimes comes up with all like handling the
00:04:55.584 --> 00:04:58.793 filterable attributes, varying attributes, categories, suggests
00:04:58.793 --> 00:05:01.500 for like the type of head results and all that stuff.
00:05:01.500 --> 00:05:04.358 That's all handled in here. So if you have values in the
00:05:04.358 --> 00:05:07.667 database that are stored in kind of a weird way, but the way that
00:05:07.667 --> 00:05:10.926 you would want to query against them is different, by all means,
00:05:10.926 --> 00:05:12.380 simplify as much as you want.
00:05:14.170 --> 00:05:18.031 What we're storing into elastic only has to be queryable
00:05:18.031 --> 00:05:22.163 against. We're not displaying this data anywhere. So you can
00:05:22.163 --> 00:05:26.160 you can filter down to and and modify this into the least.
00:05:26.900 --> 00:05:30.919 Amount of information that needs to be here to get back the
00:05:30.919 --> 00:05:35.073 results you need, and that's encouraged. Um, because the more
00:05:35.073 --> 00:05:39.160 there's no point in storing things in elastic that we're not
00:05:39.160 --> 00:05:39.830 gonna use.
00:05:41.500 --> 00:05:44.562 Uh, so we don't need to, you know, you don't need to store
00:05:44.562 --> 00:05:47.468 like an entire record, you know, like if I was gonna do
00:05:47.468 --> 00:05:50.790 categories, I wouldn't do like a list of category models, which
00:05:50.790 --> 00:05:53.956 we don't. You can actually go and look and look to find them
00:05:53.956 --> 00:05:57.330 in here, handle categories. And this is kind of a messy one, but
00:05:57.330 --> 00:06:00.392 ultimately what it returns is indexable category filters I
00:06:00.392 --> 00:06:01.950 think, or something like that.
00:06:05.780 --> 00:06:09.672 Uh, indexable categories, um, so. And that indexable category
00:06:09.672 --> 00:06:13.438 is like kind of pared down description of a category, which
00:06:13.438 --> 00:06:16.954 is just the stuff that's relevant to actually filtering
00:06:16.954 --> 00:06:19.340 or displaying filters in the catalog.
00:06:20.740 --> 00:06:21.120 So.
00:06:22.110 --> 00:06:25.694 Um, so that pretty much covers that step one. Um, step one is
00:06:25.694 --> 00:06:29.220 indexing, storing data into elastic. At this stage, the only
00:06:29.220 --> 00:06:32.746 thing we're doing we're not querying against anything. We're
00:06:32.746 --> 00:06:36.446 not filtering any results. All we're doing is storing data from
00:06:36.446 --> 00:06:39.914 the database into the Elastic search index so that later in
00:06:39.914 --> 00:06:41.880 step two we can query against it.
00:06:42.520 --> 00:06:45.852 Does anybody have any questions about any of that or any
00:06:45.852 --> 00:06:49.535 thoughts or or uh anything to add but I'm gonna real quick I'm
00:06:49.535 --> 00:06:53.277 gonna popcorn pick somebody to explain the kind of the just the
00:06:53.277 --> 00:06:56.844 next level down steps for that. So that was step one is just
00:06:56.844 --> 00:07:00.527 indexing a product and I know Nick you've been working on this
00:07:00.527 --> 00:07:04.210 so I'll call on you for you've been working with Brian on this
00:07:04.210 --> 00:07:07.894 so I'll call and you first. So can you can you summarize the I
00:07:07.894 --> 00:07:11.402 have like 3 steps in that in that step one of how you would
00:07:11.402 --> 00:07:12.629 add a new product to.
00:07:12.770 --> 00:07:16.410 For a new, uh, new value. Uh to the dump reader.
00:07:17.390 --> 00:07:18.550 And the indexable products.
00:07:19.860 --> 00:07:24.138 So sorry, say again what you want me to do. Yeah, so, so
00:07:24.138 --> 00:07:28.416 Brennan said the first major step is to index, you know,
00:07:28.416 --> 00:07:32.844 index the data, right? So how would you go about modifying
00:07:32.844 --> 00:07:37.348 this? What are the steps to modify this method to add a new
00:07:37.348 --> 00:07:39.749 property to the indexable model?
00:07:43.710 --> 00:07:45.020 OK, so.
00:07:46.320 --> 00:07:47.450 The first step.
00:07:48.510 --> 00:07:50.720 To add a property to this we.
00:07:53.960 --> 00:07:57.780 Not entirely sure I still understand, but I think.
00:08:01.600 --> 00:08:04.740 Yeah, I'm still not entirely sure where. Yeah, yeah. So it's
00:08:04.740 --> 00:08:07.983 kind of what we what we covered earlier today. So we we talked
00:08:07.983 --> 00:08:11.072 through all of this stuff and like the giant query and then
00:08:11.072 --> 00:08:14.315 how we map it on to the product indexable model and stuff like
00:08:14.315 --> 00:08:17.352 that. So on that pull request that we had that we did last
00:08:17.352 --> 00:08:20.338 week and that we fixed up, we're kind of finished up this
00:08:20.338 --> 00:08:20.750 morning.
00:08:22.100 --> 00:08:25.884 The added starting price on to the lot or sorry not onto the
00:08:25.884 --> 00:08:29.731 the index and so basically we're we're just going looking for
00:08:29.731 --> 00:08:33.702 what were the steps we had to do to get starting price onto the
00:08:33.702 --> 00:08:35.130 index when we did that.
00:08:36.240 --> 00:08:39.079 And I know last week we went through, we kind of blasted
00:08:39.079 --> 00:08:42.218 through it really fast. Um, so um, but if you had to summarize
00:08:42.218 --> 00:08:45.157 real quick, I'll summarize kind of the answer here. Unless
00:08:45.157 --> 00:08:48.246 actually I'm gonna pick one more person and then I'll try and
00:08:48.246 --> 00:08:50.040 summarize and no worries, go ahead.
00:08:51.380 --> 00:08:55.462 I was going to say we added the starting price into the lots
00:08:55.462 --> 00:08:57.270 into that select statement.
00:08:58.840 --> 00:09:02.618 And then a little bit lower if you Scroll down, I think it's
00:09:02.618 --> 00:09:02.990 under.
00:09:04.530 --> 00:09:08.191 Yeah. May not be here because yeah, there's another one. I'm
00:09:08.191 --> 00:09:11.972 on an old version of stuff that doesn't have the lots in here.
00:09:11.972 --> 00:09:15.873 So, um, but, but yeah, you're on the right course where there be
00:09:15.873 --> 00:09:19.414 a second one down here. Yeah, there was two different lots
00:09:19.414 --> 00:09:23.376 that we had to add that starting price equals like X dot lots dot
00:09:23.376 --> 00:09:26.977 starting price or something along those lines. And then all
00:09:26.977 --> 00:09:30.818 the way down towards the bottom in the product indexable model,
00:09:30.818 --> 00:09:34.179 there was the starting price equals version dot lot dot
00:09:34.179 --> 00:09:35.080 starting price.
00:09:35.420 --> 00:09:39.026 Yep. So that's and I can show you guys in the actual, the pull
00:09:39.026 --> 00:09:42.518 request that's here, uh, what that ended up looking like was
00:09:42.518 --> 00:09:46.238 I'll open the actual file so we can see the whole thing and real
00:09:46.238 --> 00:09:49.501 quick just a high level summarize that. So ultimately we
00:09:49.501 --> 00:09:52.993 need to change the select to actually get the value from the
00:09:52.993 --> 00:09:56.657 related, whether it's a property directly on the product or any
00:09:56.657 --> 00:10:00.091 related entity that the product has access to. And then the
00:10:00.091 --> 00:10:03.698 reason that there's two steps there, the select on top and the
00:10:03.698 --> 00:10:05.530 select on bottom is that's just.
00:10:05.600 --> 00:10:09.227 Their normal. That's our normal standards and practices for UM.
00:10:09.227 --> 00:10:12.799 For the pattern of selecting it, we selected the first time to
00:10:12.799 --> 00:10:16.200 optimize the SQL and then the second time to actually build
00:10:16.200 --> 00:10:19.544 the outputted object. Typed. Yeah, the typed object below.
00:10:19.544 --> 00:10:23.059 Yeah, because if it's self in the SQL query will reject it if
00:10:23.059 --> 00:10:26.630 it thinks that you're trying to create a custom type or a type
00:10:26.630 --> 00:10:29.861 that it doesn't want to allow. So we have to do this SQL
00:10:29.861 --> 00:10:33.432 version of it, bringing the SQL data back into memory and then
00:10:33.432 --> 00:10:35.870 assign it to actual concretes that we can.
00:10:35.950 --> 00:10:39.022 It's a sign out. And the difference is that yeah the the
00:10:39.022 --> 00:10:42.094 the the performance difference is significant because it
00:10:42.094 --> 00:10:45.490 doesn't have to load all data in order to assign to the object
00:10:45.490 --> 00:10:48.939 first. It just has to load the data we wanted in the 1st place.
00:10:48.939 --> 00:10:52.012 By doing it this way just modifies the select. So rather
00:10:52.012 --> 00:10:54.653 than do and just for anybody that has base level
00:10:54.653 --> 00:10:57.833 understanding SQL, rather than do select star from it says
00:10:57.833 --> 00:11:00.959 select just the fields that we're telling you. We need to
00:11:00.959 --> 00:11:03.977 optimize the data. Come and create a date. Yeah it just
00:11:03.977 --> 00:11:06.079 selects the basic information here so.
00:11:06.200 --> 00:11:10.150 OK, so that's that's the first part is to change the select to
00:11:10.150 --> 00:11:13.913 get just the fields that we're looking for. Then the second
00:11:13.913 --> 00:11:17.676 part is kind of potentially one or the other, right? You're
00:11:17.676 --> 00:11:21.752 either just say indexable model dot value equals whatever or you
00:11:21.752 --> 00:11:25.515 can write a custom handling. Basically you set the property
00:11:25.515 --> 00:11:29.466 to the indexable model through either a custom handling method
00:11:29.466 --> 00:11:33.480 or just if it's a single value directly to the indexable model.
00:11:35.160 --> 00:11:37.590 Yeah, that's that's pretty much correct, um.
00:11:38.840 --> 00:11:42.482 And again, I'm kind of showing in the pull request that, uh,
00:11:42.482 --> 00:11:46.245 that that Nick made earlier what we actually did last week. So
00:11:46.245 --> 00:11:49.947 you can see that we added it in the top section of the query,
00:11:49.947 --> 00:11:53.590 then we again have to pass it through in the bottom section,
00:11:53.590 --> 00:11:57.233 and then lastly store it onto the INDEXABLE model inside the
00:11:57.233 --> 00:12:00.816 foreach loop. Fantastic. So before I move on to querying, I
00:12:00.816 --> 00:12:04.698 wanna stick around in this file for just a minute and talk about
00:12:04.698 --> 00:12:08.341 some really important stuff that absolutely comes up all the
00:12:08.341 --> 00:12:08.640 time.
00:12:08.750 --> 00:12:11.922 Uh, and I want to make sure that everybody is paying attention or
00:12:11.922 --> 00:12:14.950 listening. So if you're paying attention, last listening, last
00:12:14.950 --> 00:12:18.074 listening, can you please raise your hand on teams so I can yell
00:12:18.074 --> 00:12:19.180 at anybody who doesn't?
00:12:22.630 --> 00:12:24.670 OK, cool. Alright, looks like everybody.
00:12:25.410 --> 00:12:27.000 Fantastic. Brandon's not.
00:12:27.750 --> 00:12:29.960 Yeah, he's, he's humbled that.
00:12:31.620 --> 00:12:34.874 Uh, OK, so thank you for everybody for for paying
00:12:34.874 --> 00:12:38.779 attention listening. Um, because this is gonna be debugging
00:12:38.779 --> 00:12:42.750 Elasticsearch, not showing your products in the catalog 101.
00:12:44.110 --> 00:12:47.972 And this is something that happens all the time. I just had
00:12:47.972 --> 00:12:51.964 to do this yesterday. Um. And so this is extremely important,
00:12:51.964 --> 00:12:56.149 super useful. So to get started, the 1st place that you're gonna
00:12:56.149 --> 00:13:00.076 check is, are we even indexing my product, right. So if it's
00:13:00.076 --> 00:13:03.939 not in the index, it's not going to come up in the catalog.
00:13:03.939 --> 00:13:07.802 That's, I mean unless somebody's made customizations to the
00:13:07.802 --> 00:13:11.343 catalog to do some wild stuff like just showing random
00:13:11.343 --> 00:13:14.369 products without going through elastic, which.
00:13:14.490 --> 00:13:18.591 Ideally, we would never do. Your product does not show in the
00:13:18.591 --> 00:13:22.098 catalog unless it's in the index. So this section of
00:13:22.098 --> 00:13:25.736 filters that I talked about earlier dictates, at least
00:13:25.736 --> 00:13:28.780 partially, what does and doesn't get indexed.
00:13:29.590 --> 00:13:33.002 So what that means is you need to go check the products you're
00:13:33.002 --> 00:13:36.469 not seeing. Are they active? Are they visible? Did you actually
00:13:36.469 --> 00:13:38.040 set his discontinued to true?
00:13:39.390 --> 00:13:42.755 Those are kind of the first three easy ones to check, and
00:13:42.755 --> 00:13:46.585 what I'll often do is I'll build a SQL query myself directly like
00:13:46.585 --> 00:13:49.950 in SSMS, and I'll run this against my database and see if
00:13:49.950 --> 00:13:53.025 the product I'm looking for comes back in the in the
00:13:53.025 --> 00:13:53.489 results.
00:13:54.970 --> 00:13:58.564 So I'm gonna open that up and kind of show that as we're
00:13:58.564 --> 00:14:02.284 going, uh, well, SSMS takes its sweet time to load, and of
00:14:02.284 --> 00:14:05.690 course as soon as I start talking smack, it opens up.
00:14:07.230 --> 00:14:10.942 But uh, the other one that's here is filter by type keys. Now
00:14:10.942 --> 00:14:14.534 this one is something that you're not going to see quite as
00:14:14.534 --> 00:14:18.306 often, but this absolutely does get used on a lot of projects.
00:14:18.306 --> 00:14:21.719 What this lets us do is filter out certain type keys, or
00:14:21.719 --> 00:14:25.251 specifically include only
00:14:25.251 --> 00:14:29.083 catalog. And this is useful for projects like actually show you
00:14:29.083 --> 00:14:30.640 perfect example GL veneer.
00:14:31.530 --> 00:14:35.753 Um, this? This is one of the sites we went into production
00:14:35.753 --> 00:14:37.400 with their catalog. Um.
00:14:37.950 --> 00:14:41.606 They have all these tons and tons of species of wood, and
00:14:41.606 --> 00:14:45.452 every one of these species of wood, if we just pick a random
00:14:45.452 --> 00:14:46.020 one, has.
00:14:46.760 --> 00:14:50.477 Different sizes, different papers, different, etcetera. Uh,
00:14:50.477 --> 00:14:54.317 different options. What they wanted to avoid was having their
00:14:54.317 --> 00:14:57.973 catalog be afromosia plane sliced, you know, 4 by 8 paper,
00:14:57.973 --> 00:15:01.566 and then the next one being everybody playing sliced 8 by
00:15:01.566 --> 00:15:05.654 1010 millimeter paper, whatever. They just wanted one instance of
00:15:05.654 --> 00:15:09.495 each species to be in a catalog so that you could just browse
00:15:09.495 --> 00:15:13.150 for the species you want and then click on it and find the
00:15:13.150 --> 00:15:14.390 cut you want, right.
00:15:15.050 --> 00:15:18.574 So we filter, we actually were able to do that completely out
00:15:18.574 --> 00:15:21.757 of box by just filtering the catalog to only be variant
00:15:21.757 --> 00:15:24.940 masters. These are the variant master products that tie
00:15:24.940 --> 00:15:28.237 together all these different variations. So by hiding the
00:15:28.237 --> 00:15:31.819 variations, the products where their type is just variant from
00:15:31.819 --> 00:15:35.343 the catalog, we get a nice clean catalog where all you see is
00:15:35.343 --> 00:15:38.924 your top level products and we don't have kind of the bloat of
00:15:38.924 --> 00:15:42.619 showing every possible variation of that product in the catalog.
00:15:42.619 --> 00:15:44.950 So that's what that filter by type keys.
00:15:45.020 --> 00:15:48.366 Option does in here. By default it's. There's nothing here so
00:15:48.366 --> 00:15:51.712 it's gonna show every product type key. But if you're on like
00:15:51.712 --> 00:15:54.896 a client project, this is very worth checking while you're
00:15:54.896 --> 00:15:58.026 going through and debugging to see if your product is not
00:15:58.026 --> 00:15:58.619 showing up.
00:15:59.950 --> 00:16:03.457 You can find the the app settings key by just clicking on
00:16:03.457 --> 00:16:07.266 this property and there it is and you can go check this in the
00:16:07.266 --> 00:16:11.015 app settings JSON files or you can check them in the admin on
00:16:11.015 --> 00:16:14.644 newer versions of stuff. On other versions 2021.4 and older
00:16:14.644 --> 00:16:17.969 you have to go check the app settings dot config file.
00:16:19.240 --> 00:16:22.293 But uh, you can check this file or check this uh setting and see
00:16:22.293 --> 00:16:25.206 if it's set to anything. If it is, then you need to make sure
00:16:25.206 --> 00:16:27.320 that your product is one of those type keys.
00:16:28.740 --> 00:16:31.245 Otherwise, this doesn't matter and you can, you can kind of
00:16:31.245 --> 00:16:33.876 skip over it, but that's just another, you know, good thing to
00:16:33.876 --> 00:16:36.424 know. So these four things are going to be the initial piece
00:16:36.424 --> 00:16:39.013 that filters out products that come back, but they're not the
00:16:39.013 --> 00:16:41.435 only thing that dictates if a product does or doesn't get
00:16:41.435 --> 00:16:41.770 indexed.
00:16:42.800 --> 00:16:45.638 The next thing that we need to look at after all of this. Uh,
00:16:45.638 --> 00:16:46.600 yeah, go ahead, Nick.
00:16:47.890 --> 00:16:51.890 I was just gonna ask, is there ever situations where like the
00:16:51.890 --> 00:16:55.954 website you just showed where they're filtering on the variant
00:16:55.954 --> 00:16:59.825 master, where they might also have individual products that
00:16:59.825 --> 00:17:04.019 don't have different variations to display at the same time? And
00:17:04.019 --> 00:17:08.083 is there, how do we handle that kind of situation? Yeah, great
00:17:08.083 --> 00:17:12.019 question. And in fact, GLV does that too. Yep. Yeah. So this
00:17:12.019 --> 00:17:16.019 type keys is actually a list of type keys that you can filter
00:17:16.019 --> 00:17:17.889 to. It's not like a singular.
00:17:18.610 --> 00:17:22.346 And So what we can do is we can have it show variant masters and
00:17:22.346 --> 00:17:25.911 general products. So general is the type key that we have for
00:17:25.911 --> 00:17:29.533 just standard singular add to cart. It doesn't have variations
00:17:29.533 --> 00:17:32.350 or anything type of products, just simple stuff.
00:17:33.950 --> 00:17:36.110 So by doing that we can show.
00:17:36.890 --> 00:17:40.685 Yeah. And kids as well. Um, so we have by doing that we can
00:17:40.685 --> 00:17:44.734 have standard products and just variant masters showing without
00:17:44.734 --> 00:17:48.657 showing all the variations in the catalog either. So that's a
00:17:48.657 --> 00:17:52.516 fantastic question. That's how you would do that is that the
00:17:52.516 --> 00:17:56.439 value of this in your settings would be 2 type keys, variant,
00:17:56.439 --> 00:18:00.361 master and general. And then if the project you're working on
00:18:00.361 --> 00:18:01.880 has kits, then also kit.
00:18:02.880 --> 00:18:06.505 OK, cool. Yep. So yeah, that's a great question. OK, so go back
00:18:06.505 --> 00:18:09.961 to what I was saying. So these filters are the first sort of
00:18:09.961 --> 00:18:13.474 piece that will block product from showing up in the catalog.
00:18:13.474 --> 00:18:17.043 The next place that you want to check is at the very beginning
00:18:17.043 --> 00:18:20.499 of the for each loop down here this early out. So by looking
00:18:20.499 --> 00:18:23.785 you can kind of read through these and see these are all,
00:18:23.785 --> 00:18:27.524 each one of these is checking if a setting is enabled and if that
00:18:27.524 --> 00:18:31.150 setting is enabled making sure that the value coming out of the
00:18:31.150 --> 00:18:32.849 database passes that settings.
00:18:32.920 --> 00:18:37.091 Requirement. So for example this one is does the product index
00:18:37.091 --> 00:18:41.196 require the product to have a brand and if it does, does this
00:18:41.196 --> 00:18:45.500 version not have a brand. If any one of these conditions passes,
00:18:45.500 --> 00:18:49.472 then we skip the product and that product won't be indexed.
00:18:49.472 --> 00:18:53.511 By default all of these are false except for category. So by
00:18:53.511 --> 00:18:57.550 default a product will not show up in the catalog unless you
00:18:57.550 --> 00:19:01.854 assign it to a category and that category specifically. If we go
00:19:01.854 --> 00:19:02.980 look at this has.
00:19:03.070 --> 00:19:05.090 Categories, um.
00:19:06.610 --> 00:19:10.051 Which comes from a little higher up I think. I probably passed
00:19:10.051 --> 00:19:13.110 it. I can't see here it is. Those categories have to be
00:19:13.110 --> 00:19:16.387 active. The product category record needs to be active. The
00:19:16.387 --> 00:19:19.665 category itself needs to be active, visible, and include in
00:19:19.665 --> 00:19:22.724 menu. All of these conditions have to be true about the
00:19:22.724 --> 00:19:26.329 category the product is assigned to for the product to show up in
00:19:26.329 --> 00:19:29.879 the catalog. That is critically important because it's something
00:19:29.879 --> 00:19:33.266 that I often overlook. I just assign it to a random category.
00:19:33.266 --> 00:19:36.380 So yeah, whatever, I'm trying to get some data in there.
00:19:36.470 --> 00:19:39.784 And it doesn't show up and I'm like confused, you know,
00:19:39.784 --> 00:19:43.631 whatever. I don't know why it's not showing up. It's because you
00:19:43.631 --> 00:19:47.300 know that that category is not included menu or it's somebody
00:19:47.300 --> 00:19:50.555 said it's active faults or something. So that's really
00:19:50.555 --> 00:19:54.106 important to check is if you're not seeing a product in the
00:19:54.106 --> 00:19:54.580 catalog.
00:19:56.080 --> 00:19:59.080 Number one, you can always just throw a break point in here, but
00:19:59.080 --> 00:20:01.895 that's not terribly ideal for projects where there's tons of
00:20:01.895 --> 00:20:04.757 products and you know you're gonna hit this breakpoint a lot.
00:20:04.757 --> 00:20:07.711 I guess this break point would actually be a great place to put
00:20:07.711 --> 00:20:10.711 it because you just see the ones that are getting skipped by the
00:20:10.711 --> 00:20:13.572 early out. So this is actually a fantastic. That's right. Put
00:20:13.572 --> 00:20:16.434 that right there. I believe it would still go, you would want
00:20:16.434 --> 00:20:19.434 one line lower so that it to get it only if it's only inside the
00:20:19.434 --> 00:20:22.019 if this does hit, if it's only on the inside the, it'll
00:20:22.019 --> 00:20:24.973 highlight this line and then if this condition doesn't pass, it
00:20:24.973 --> 00:20:26.450 just cuts straight down to this.
00:20:26.540 --> 00:20:28.972 Fine, but if you put it on the opening brace inside, it'll
00:20:28.972 --> 00:20:31.363 actually hit this. But yeah, either way, here or here, it
00:20:31.363 --> 00:20:32.270 doesn't really matter.
00:20:33.730 --> 00:20:36.748 Uh, this might be a little more obvious that you're trying to
00:20:36.748 --> 00:20:39.815 break on the continue. Um, but um, but anyway. Yeah, you could
00:20:39.815 --> 00:20:42.833 put a breakpoint here and then this would only get hit on any
00:20:42.833 --> 00:20:45.656 products that passed your initial filters. The type keys,
00:20:45.656 --> 00:20:48.675 active, visible, discontinued, but that didn't pass the early
00:20:48.675 --> 00:20:51.595 out. And then when you hit this breakpoint, you know, since
00:20:51.595 --> 00:20:54.711 it'll be broken. You can kind of mouse over these and see which
00:20:54.711 --> 00:20:57.778 one of these was the reason it failed, and then you can go and
00:20:57.778 --> 00:21:00.893 dig into the data a little bit more and figure out what's going
00:21:00.893 --> 00:21:01.040 on.
00:21:02.450 --> 00:21:05.575 Generally, if you're trying to find a specific product to do
00:21:05.575 --> 00:21:08.495 this with, you can modify the breakpoint to say that the
00:21:08.495 --> 00:21:11.570 version dot custom key equals the custom key of that you're
00:21:11.570 --> 00:21:11.980 looking.
00:21:12.800 --> 00:21:15.668 Yeah. So if it's, if it's ID, you can also Bob 123. Yeah,
00:21:15.668 --> 00:21:18.587 yeah, you could do that. And then the breakpoint will only
00:21:18.587 --> 00:21:21.604 fire off when it goes to that one. And then if it never hits
00:21:21.604 --> 00:21:24.720 that breakpoint, you know, it either didn't go to that spot or
00:21:24.720 --> 00:21:27.886 and went on and actually indexed it, but then you couldn't find
00:21:27.886 --> 00:21:29.370 any results or something else.
00:21:30.400 --> 00:21:33.637 Yep. So um, for those that haven't used and I, that should
00:21:33.637 --> 00:21:35.010 be a double equals there.
00:21:35.790 --> 00:21:38.737 Um, for those that, uh, haven't used the conditional
00:21:38.737 --> 00:21:42.128 breakpoints. They are very cool and very useful. Um, they do
00:21:42.128 --> 00:21:44.520 slow down the execution of the code a lot.
00:21:45.620 --> 00:21:49.498 Um. So sometimes it can be a little bit like you wanna be
00:21:49.498 --> 00:21:52.240 judicious about where you use these. Um.
00:21:52.950 --> 00:21:54.280 Um, but uh.
00:21:55.000 --> 00:21:58.610 These can be really useful where it will only hit this breakpoint
00:21:58.610 --> 00:22:02.002 if the condition that you've typed in is true. So pretty cool
00:22:02.002 --> 00:22:05.011 stuff. And to do that all you do is you can create the
00:22:05.011 --> 00:22:08.512 breakpoint, right click on it, and then click conditions and it
00:22:08.512 --> 00:22:12.068 pops this little menu up and you can type in whatever it is that
00:22:12.068 --> 00:22:13.710 you want to type in there. So.
00:22:17.750 --> 00:22:21.330 OK. So so that kind of talks through how to check if a
00:22:21.330 --> 00:22:24.260 product is being indexed in the first place.
00:22:25.460 --> 00:22:25.850 So.
00:22:26.870 --> 00:22:30.712 This is, I would say 99.9% of the time when a product isn't
00:22:30.712 --> 00:22:34.426 showing up in the catalog for one reason or another, it's
00:22:34.426 --> 00:22:38.012 gonna be either one of these early outs or one of those
00:22:38.012 --> 00:22:41.790 filters up there. Most of the time it's not related to the
00:22:41.790 --> 00:22:45.376 actual, like querying or searching the index, it's just
00:22:45.376 --> 00:22:49.346 related to one of those one of these early out skips. So does
00:22:49.346 --> 00:22:52.099 anybody have any questions on any of this?
00:22:53.280 --> 00:22:55.680 And if not, I will move on to.
00:22:56.960 --> 00:22:57.930 Part 2.
00:22:58.740 --> 00:23:00.690 Gonna show a cabana? Check.
00:23:02.740 --> 00:23:07.042 Uh, I don't have. I think I have cabana, but I I was just gonna
00:23:07.042 --> 00:23:11.076 go through the code and stuff. Um, I never, I never checked
00:23:11.076 --> 00:23:12.690 cabana because I'm lazy.
00:23:13.970 --> 00:23:17.142 I usually just I usually just either break point or I go check
00:23:17.142 --> 00:23:20.163 the database immediately to figure out which ones are being
00:23:20.163 --> 00:23:23.486 sent and then figure it out from there. But Kibana is a fantastic
00:23:23.486 --> 00:23:25.400 tool that I probably should use more.
00:23:27.730 --> 00:23:31.398 I can cover that next week if you want that. Yeah that would
00:23:31.398 --> 00:23:34.945 be really cool. I'm gonna I'm gonna popcorn one more time.
00:23:34.945 --> 00:23:38.734 Jeremy Novak So what he kind of can you summarize kind of what
00:23:38.734 --> 00:23:42.522 Brandon just talked about with the filtering. So how would you
00:23:42.522 --> 00:23:46.070 check that your what's actually being filtered in and out.
00:23:49.520 --> 00:23:53.840 Well, I think what I understand is that that product should up
00:23:53.840 --> 00:23:57.749 above it has to match those conditions like it has to be
00:23:57.749 --> 00:23:58.230 active.
00:23:59.080 --> 00:24:02.760 Um, and has to be in the filters etcetera to actually make it
00:24:02.760 --> 00:24:06.440 that far. And then there's that if condition. If any of those
00:24:06.440 --> 00:24:09.230 comes up true, it's just going to get skipped.
00:24:10.740 --> 00:24:13.885 But has to make it past all that to get to where we're at on
00:24:13.885 --> 00:24:15.020 screen right now? Yep.
00:24:16.650 --> 00:24:17.070 Beautiful.
00:24:18.160 --> 00:24:21.881 Awesome. So, so far we're, I'm just gonna summarize those two
00:24:21.881 --> 00:24:25.662 points. Um. So first you need to set up your product correctly
00:24:25.662 --> 00:24:28.963 and I guess I'll do this in reverse order kind of what
00:24:28.963 --> 00:24:32.804 Jeremy's talking about. Ensure that the product you're creating
00:24:32.804 --> 00:24:36.165 that you wanna see in the catalog has the correct type.
00:24:36.165 --> 00:24:39.767 And it's not to say it's a categories, if brands are turned
00:24:39.767 --> 00:24:43.488 on, it's associated to a brand just being total here and then
00:24:43.488 --> 00:24:47.269 once you have the product set up correctly so that the filters
00:24:47.269 --> 00:24:48.770 will actually pick it up.
00:24:49.410 --> 00:24:52.650 Then if you're adding a, if you're if you're changing what
00:24:52.650 --> 00:24:56.165 is indexed, then you would need to modify the select so that it
00:24:56.165 --> 00:24:59.460 can actually output so that you can set it to the indexable
00:24:59.460 --> 00:24:59.790 model.
00:25:03.760 --> 00:25:07.285 And I just want to confirm that you said the categories is the
00:25:07.285 --> 00:25:10.754 only piece of that conditional right there that's required by
00:25:10.754 --> 00:25:13.888 default. By default on by default, yes. Yeah OK and you
00:25:13.888 --> 00:25:17.525 can. Each one of these is just a normal app settings configured.
00:25:17.525 --> 00:25:20.994 You can have 12 into them and go look at the default value on
00:25:20.994 --> 00:25:24.519 them and the appsettings key to go check in a specific site if
00:25:24.519 --> 00:25:27.988 it's enabled. But still like the brand one is by default it's
00:25:27.988 --> 00:25:31.458 false. Category default true franchise is gonna be false. And
00:25:31.458 --> 00:25:34.200 anyway all the rest of these are gonna be false.
00:25:34.280 --> 00:25:37.206 Default. And I will say that I just just confirmed that because
00:25:37.206 --> 00:25:39.995 I didn't actually know how to box how turning off categories
00:25:39.995 --> 00:25:42.876 would handle. Basically it just hides the product drop down in
00:25:42.876 --> 00:25:45.757 the mega menu and we don't have any way that specifically goes
00:25:45.757 --> 00:25:48.637 to the catalog if that's turned off. So you just have to go to
00:25:48.637 --> 00:25:51.426 Slash catalog. But then all products are basically indexable
00:25:51.426 --> 00:25:54.353 at that point. If you have all of the other filters turned off,
00:25:54.353 --> 00:25:57.325 if the category is disabled, the products thing instead of being
00:25:57.325 --> 00:26:00.114 a menu, it should just be a button that will take you to the
00:26:00.114 --> 00:26:02.903 catalog. Yeah, it's not right now, but that's a bug that I'm
00:26:02.903 --> 00:26:04.229 going to put a bug ticket in.
00:26:04.830 --> 00:26:05.540 To fix.
00:26:06.410 --> 00:26:07.910 Good call, um.
00:26:08.330 --> 00:26:08.760 OK.
00:26:09.970 --> 00:26:13.629 Um, so fantastic. If nobody has any other questions, I'm gonna
00:26:13.629 --> 00:26:17.289 move on to the querying part, which I'm not probably gonna dig
00:26:17.289 --> 00:26:21.007 as much into. I'm going to talk through some of the kind of the
00:26:21.007 --> 00:26:24.493 basics of it. Um. And then I wanted to cover a couple other
00:26:24.493 --> 00:26:27.979 things that came up when with Nick this morning, there were
00:26:27.979 --> 00:26:31.755 some kind of interesting things that I hope people will get some
00:26:31.755 --> 00:26:33.440 mileage out of. So all right.
00:26:35.040 --> 00:26:38.655 So like I said earlier, elastic is 2 steps index query. We've
00:26:38.655 --> 00:26:42.213 just covered all the indexing. So now onto querying the main
00:26:42.213 --> 00:26:45.945 class that's going to handle the querying. And there's a couple
00:26:45.945 --> 00:26:49.270 of classes that are relevant here, but the main one that
00:26:49.270 --> 00:26:53.061 you're going to care about here is search module dot CS. This is
00:26:53.061 --> 00:26:56.502 the class that actually builds the query that we send into
00:26:56.502 --> 00:26:58.309 elastic. It builds it with the.
00:26:59.080 --> 00:27:03.667 Fluent API and I gotta be honest, I absolutely hate this
00:27:03.667 --> 00:27:07.530 API like this. Like syntax that that nest uses.
00:27:10.050 --> 00:27:14.636 It's, uh, I I just, I I hate it. But uh, you know, it is what it
00:27:14.636 --> 00:27:18.870 is. Um, so it makes almost no sense when you when you got a
00:27:18.870 --> 00:27:23.033 really strong link background because nothing that they're
00:27:23.033 --> 00:27:27.266 doing actually manage actually does what it says on the tin
00:27:27.266 --> 00:27:31.641 like this and equals plus Qi hate this so much it's so bad it
00:27:31.641 --> 00:27:32.700 makes me angry.
00:27:34.550 --> 00:27:39.777 Ohh it angers. Uh, anyway um so uh, but anyway. So there's
00:27:39.777 --> 00:27:45.448 documentation out there on how this whole nest fluent API thing
00:27:45.448 --> 00:27:45.980 works.
00:27:47.400 --> 00:27:50.207 Not gonna dig too much into that right now. Um, because the
00:27:50.207 --> 00:27:52.874 majority of the time anything that you're gonna be doing
00:27:52.874 --> 00:27:55.634 adding to this or customizing this, you're either gonna be
00:27:55.634 --> 00:27:58.348 modifying existing values, and often you're just gonna be
00:27:58.348 --> 00:28:01.202 modifying app settings that are already wired into all this,
00:28:01.202 --> 00:28:04.149 like the boosts and stuff. Or you're gonna be building, you're
00:28:04.149 --> 00:28:07.144 adding a new thing to query that there's already something else
00:28:07.144 --> 00:28:10.044 that queries pretty much exactly the same way and you'll just
00:28:10.044 --> 00:28:10.700 copy paste it.
00:28:12.340 --> 00:28:16.408 And so, uh, the the uh, the work that we did last weekend was uh
00:28:16.408 --> 00:28:20.352 or last week was a a perfect, a perfect example because it was
00:28:20.352 --> 00:28:24.233 one of those ones that actually didn't already have something
00:28:24.233 --> 00:28:25.109 similar to it.
00:28:26.900 --> 00:28:30.569 Uh, that we were building against. So we had to go and
00:28:30.569 --> 00:28:31.170 find. Uh.
00:28:32.800 --> 00:28:35.520 We had to go and and build kind of that one. Uh.
00:28:36.880 --> 00:28:40.075 From from the ground up I I wasn't here for that one what
00:28:40.075 --> 00:28:43.435 the cause and I fully remember what I put in the requirement
00:28:43.435 --> 00:28:46.741 for it when I was talking to James about it originally. But
00:28:46.741 --> 00:28:49.991 was it it was a starting price and so it was that you were
00:28:49.991 --> 00:28:53.407 picking a a range of starting prices or the way that we ended
00:28:53.407 --> 00:28:56.768 up the way that we ended up getting it to work this morning.
00:28:56.768 --> 00:28:59.577 So we had to make some adjustments was. So all the
00:28:59.577 --> 00:29:02.828 stuff that we did for storing it is still the same. So the
00:29:02.828 --> 00:29:06.023 pricing factory receives the credit limit which is stored
00:29:06.023 --> 00:29:06.960 into it and then.
00:29:07.060 --> 00:29:11.577 Service base we down here we store the credit limit onto here
00:29:11.577 --> 00:29:15.657 and then in the searching service we passed that credit
00:29:15.657 --> 00:29:20.321 limit into the filters and then ultimately in the search module
00:29:20.321 --> 00:29:24.766 we go down here and we built the starting price credit limit
00:29:24.766 --> 00:29:29.429 query modification, we pass all the goodies in and then we jump
00:29:29.429 --> 00:29:33.874 down into that search module based mods and this is the file
00:29:33.874 --> 00:29:36.059 that actually handles that so.
00:29:37.080 --> 00:29:40.641 It's basically that we just did a range on the field of this. It
00:29:40.641 --> 00:29:44.148 has to be less than that value. Um, so this was the way that we
00:29:44.148 --> 00:29:46.450 managed to get this working this morning.
00:29:48.300 --> 00:29:51.854 And so, um, all it's basically doing is saying that this query
00:29:51.854 --> 00:29:55.520 also has a range filter where on the field starting price and it
00:29:55.520 --> 00:29:58.962 has to be less than the credit limit we're passing it on the
00:29:58.962 --> 00:30:02.290 form. And there's a couple of things that come before that
00:30:02.290 --> 00:30:06.013 where we if we don't have filter by credit limit enabled, then we
00:30:06.013 --> 00:30:09.624 don't impose this filter. If the credit limit that we passed in
00:30:09.624 --> 00:30:13.291 is null or has an invalid value, we don't impose this filter and
00:30:13.291 --> 00:30:15.829 then otherwise we throw on the range filter.
00:30:17.490 --> 00:30:20.626 So that's how we ended up doing that for that, and it worked
00:30:20.626 --> 00:30:23.660 perfectly, um, this morning, which was pretty cool to see.
00:30:25.110 --> 00:30:28.937 So you just pick about, you just pick a value and it gives you
00:30:28.937 --> 00:30:32.887 all starting limits under that. Yep, yeah. So if my credit limit
00:30:32.887 --> 00:30:36.775 is 5000, then this filter will give me back all of the lots who
00:30:36.775 --> 00:30:40.360 starting prices less than 5000. This could be a less. This
00:30:40.360 --> 00:30:44.310 should probably be less than or equal now that I think about it.
00:30:45.750 --> 00:30:48.860 But yeah, so it basically just anything less than my credit
00:30:48.860 --> 00:30:49.690 limit I can see.
00:30:50.470 --> 00:30:53.853 So would you how would you modify this? Let's say we wanted
00:30:53.853 --> 00:30:57.293 this in core and it wasn't based off of credit limit. I just
00:30:57.293 --> 00:31:00.620 wanted to be able to filter on starting price between some
00:31:00.620 --> 00:31:04.060 input value, right? Like I have a you have a low range and a
00:31:04.060 --> 00:31:07.613 high range. How would you modify this to do that? Yeah, so you
00:31:07.613 --> 00:31:11.166 could pass in like. So instead of form dot credit limit, you'd
00:31:11.166 --> 00:31:14.718 probably pass in like form dot min starting starting price and
00:31:14.718 --> 00:31:18.215 form dot Max starting price and you can do dot field starting
00:31:18.215 --> 00:31:20.640 price dot less than form dot min starting.
00:31:20.730 --> 00:31:24.155 Rice dot greater than or sorry min form, dot Max credit or
00:31:24.155 --> 00:31:27.290 starting price and then dot greater than form dot min
00:31:27.290 --> 00:31:31.005 starting price. And those could be text boxes, number fields on
00:31:31.005 --> 00:31:34.430 the front end that somebody could actually physically type
00:31:34.430 --> 00:31:37.914 in a minimum maximum they care to see, and we can filter to
00:31:37.914 --> 00:31:41.687 exactly that range here. So the range. The range probably needed
00:31:41.687 --> 00:31:45.461 a switch to go between having a min and not having a Max, having
00:31:45.461 --> 00:31:49.234 a Max, not having a min, having both, having either, so that all
00:31:49.234 --> 00:31:50.570 four cases are handled.
00:31:50.750 --> 00:31:51.770 Yeah, yeah. Well, that.
00:31:52.590 --> 00:31:55.754 So the range, so the range extension here, just so I
00:31:55.754 --> 00:31:59.517 understand, the range extension for Elastic just allows you to
00:31:59.517 --> 00:32:03.338 have these extra, um, less than, greater than. Does it give you
00:32:03.338 --> 00:32:06.802 anything else less than is exactly and I think there's an
00:32:06.802 --> 00:32:07.340 equal to.
00:32:08.590 --> 00:32:11.937 And there's some other stuff you can do with it. But yeah, it's
00:32:11.937 --> 00:32:15.285 basically just numerical ranges. And the the only two, the only
00:32:15.285 --> 00:32:18.581 thing you have to do for this range, you have to tell it which
00:32:18.581 --> 00:32:21.929 field on the indexed object that you're operating on. So in our
00:32:21.929 --> 00:32:25.225 case, it's the starting price that we're indexing and then you
00:32:25.225 --> 00:32:28.416 have to tell it some kind of filter, less than, greater than
00:32:28.416 --> 00:32:31.764 whatever, but you only need one. So this is anything less than.
00:32:31.764 --> 00:32:34.902 So starting price, obviously this wouldn't be valid, but it
00:32:34.902 --> 00:32:37.780 could be negative a billion and this would still work.
00:32:39.090 --> 00:32:42.398 Uh, as long as anything less than the credit limit that
00:32:42.398 --> 00:32:45.885 you're passing in. Um, and we could always also put like a
00:32:45.885 --> 00:32:49.725 greater than zero, but obviously we should just validate against
00:32:49.725 --> 00:32:53.566 that and not index things where the starting price is a negative
00:32:53.566 --> 00:32:53.980 number.
00:32:55.660 --> 00:32:58.972 That's fair, that's fair. OK, cool. Yeah. I was just curious
00:32:58.972 --> 00:33:02.230 on what the if you know because I I've looked at a lot, you
00:33:02.230 --> 00:33:05.542 know, because I've done some elastic search work in the past
00:33:05.542 --> 00:33:06.899 and the documentation is.
00:33:07.950 --> 00:33:08.900 Subparts.
00:33:10.400 --> 00:33:13.787 Yeah, if it exists, yeah, I've, I have. I have found that
00:33:13.787 --> 00:33:17.292 especially a lot of it is that we're on an older version of
00:33:17.292 --> 00:33:21.147 elastic and it's kind of hard to find good documentation for this
00:33:21.147 --> 00:33:24.885 version. But often the most help I get with this is finding old
00:33:24.885 --> 00:33:28.624 stack overflow questions people have, like how do I do XYZ? And
00:33:28.624 --> 00:33:32.245 then somebody provides the code snippet. And most of the time
00:33:32.245 --> 00:33:35.867 that code snippet isn't one for one gonna work for me, but it
00:33:35.867 --> 00:33:39.664 gives me it gives me kind of the clues or like the pieces that I
00:33:39.664 --> 00:33:41.650 can use to piece it all together.
00:33:42.020 --> 00:33:46.026 Um and uh, when? When it comes to elastic search customizations
00:33:46.026 --> 00:33:49.781 like this, by all means, ask questions. Anybody I still ask
00:33:49.781 --> 00:33:53.537 questions on anything I'm doing with this, I have to Google
00:33:53.537 --> 00:33:57.042 stuff. I have to ask other developers that have done it
00:33:57.042 --> 00:33:58.670 before at the company. So.
00:33:59.840 --> 00:34:02.712 By all means just send a message or yeah, something in
00:34:02.712 --> 00:34:03.600 development team.
00:34:04.550 --> 00:34:08.730 Don't feel like you're, you're um bothering us with a question
00:34:08.730 --> 00:34:12.313 about this because it because this stuff is extremely
00:34:12.313 --> 00:34:16.427 complicated and it's very unique to how it has to work versus
00:34:16.427 --> 00:34:20.673 like just doing a regular LINQ query. It took me over a year to
00:34:20.673 --> 00:34:25.053 learn elastic and get a function catalog out in 2016. I mean that
00:34:25.053 --> 00:34:29.167 was really difficult to learn this. Yeah. So yeah so it's and
00:34:29.167 --> 00:34:33.148 to say the unsaid part of what James is saying, it's really
00:34:33.148 --> 00:34:34.939 easy to waste way too much.
00:34:35.020 --> 00:34:36.360 I'm trying to make sense of it.
00:34:37.470 --> 00:34:41.058 So if if you feel like you can give it a, give it a short try.
00:34:41.058 --> 00:34:44.647 But don't waste too much time on this. If you could get stuck,
00:34:44.647 --> 00:34:48.293 just ask for help. Uh, it's it's crazy complicated and it's not
00:34:48.293 --> 00:34:51.881 worth it's not worth spending 20 hours on one filter trying to
00:34:51.881 --> 00:34:55.527 make sense of how it works. Just just get some help with it. If
00:34:55.527 --> 00:34:59.172 you're stuck and you know we'll we'll do what we can to explain
00:34:59.172 --> 00:35:02.647 it so that people can as many people as possible can work on
00:35:02.647 --> 00:35:06.008 this. But at the end of the day, this stuff is just stupid
00:35:06.008 --> 00:35:06.920 complicated. So.
00:35:07.440 --> 00:35:10.010 Nick, you got your hand up. Is that from before or is that new?
00:35:11.830 --> 00:35:16.098 No, I actually have a question somewhat related but before. If
00:35:16.098 --> 00:35:20.299 this is, this functionality is for like an auction based site
00:35:20.299 --> 00:35:23.958 and we want to set a credit limit of like $500.00 for
00:35:23.958 --> 00:35:28.024 example. Why are we basing it off of the starting price and
00:35:28.024 --> 00:35:32.021 not the current bid that's on the product? Because if this
00:35:32.021 --> 00:35:36.290 starting price is like 450 and it is currently bidding at 600,
00:35:36.290 --> 00:35:40.626 is that product, do we want that product? Absolutely, I got you
00:35:40.626 --> 00:35:41.439 on this one.
00:35:41.610 --> 00:35:45.188 So specifically for PCs, the way that the reason that they're
00:35:45.188 --> 00:35:48.651 doing this is that you just don't have access to see it. So
00:35:48.651 --> 00:35:52.056 for example, if your credit limit is let's say now granted
00:35:52.056 --> 00:35:55.692 credit limit is a total number. So for example, let's say your
00:35:55.692 --> 00:35:59.386 credit loan is $50,000. If the starting price is is under that,
00:35:59.386 --> 00:36:02.849 then that's fine, they could still see it just so that they
00:36:02.849 --> 00:36:06.543 have a reference. If it becomes let's say 60,000 lot, then they
00:36:06.543 --> 00:36:09.659 can still see it because then that will give them the
00:36:09.659 --> 00:36:11.680 reference for future, future lots.
00:36:11.760 --> 00:36:15.550 Yeah. OK. Well, maybe I don't have access to that, um, but as
00:36:15.550 --> 00:36:19.218 far as bidding functionality after the fact, they are doing
00:36:19.218 --> 00:36:23.008 some things where, well, I guess you didn't ask that question
00:36:23.008 --> 00:36:26.370 specifically, but they do have an available credit and
00:36:26.370 --> 00:36:30.405 available credit is their credit limit minus any outstanding lots
00:36:30.405 --> 00:36:34.011 that they've won that they haven't paid for. And then also
00:36:34.011 --> 00:36:37.801 any Max bids that high bids that they have on any lot. So for
00:36:37.801 --> 00:36:41.469 example, if my, if my limit is $50,000, I have one lot that
00:36:41.469 --> 00:36:42.020 I've won.
00:36:42.100 --> 00:36:45.964 $10,000 that I haven't paid yet and I have a $20,000 high bid on
00:36:45.964 --> 00:36:49.531 another lot. Then my available credit is 20,000 and so on a
00:36:49.531 --> 00:36:52.920 $50,000 credit limit, I could still see another phone or
00:36:52.920 --> 00:36:56.071 another lot that is $40,000 starting price and I can
00:36:56.071 --> 00:36:59.698 actually all be able to see that. But I won't be able to bid
00:36:59.698 --> 00:37:03.443 on it because I have outstanding bids that are high and I also
00:37:03.443 --> 00:37:06.832 have outstanding lots that I haven't purchased, but they
00:37:06.832 --> 00:37:09.270 still want them to be able to see these.
00:37:10.630 --> 00:37:15.161 Kids. And then also at a bid level, it would be very tedious
00:37:15.161 --> 00:37:18.430 to reindex every single time a bid came in.
00:37:19.300 --> 00:37:22.354 Yeah, that that's the they're mainly. The key change is that
00:37:22.354 --> 00:37:25.508 data that fluctuates too often because this only updates every
00:37:25.508 --> 00:37:28.762 15 minutes by default. Even if I made an update every minute, it
00:37:28.762 --> 00:37:31.916 would still be out of date. If there's a bidding war active on
00:37:31.916 --> 00:37:34.720 a particular lot, it would there's no way to physically
00:37:34.720 --> 00:37:37.874 keep that up to date and keep the caching data from the server
00:37:37.874 --> 00:37:40.878 and all that stuff properly, excuse me, refreshed. It's the
00:37:40.878 --> 00:37:43.932 same reason. And this is also going to be something that you
00:37:43.932 --> 00:37:47.037 guys will all hear all the time. A lot of projects want this.
00:37:47.037 --> 00:37:49.190 There's various ways we've done it before.
00:37:49.290 --> 00:37:53.636 But that's why out of box we do not filter on inventory either
00:37:53.636 --> 00:37:57.706 is because again, the index is basically a snapshot of the
00:37:57.706 --> 00:37:59.499 database every 15 minutes.
00:38:00.260 --> 00:38:00.880 And that's.
00:38:01.780 --> 00:38:05.780 So you know if if a product has one thing left in stock.
00:38:07.030 --> 00:38:10.001 We take a snapshot and then somebody immediately purchases
00:38:10.001 --> 00:38:13.124 it for the next 15 minutes, a product that is in fact not for
00:38:13.124 --> 00:38:16.197 sale is going to be showing up in the catalog even if you're
00:38:16.197 --> 00:38:19.421 filtering by in stock. So that's why by default we don't or out
00:38:19.421 --> 00:38:22.342 of box, we don't have the ability to filter by inventory.
00:38:22.342 --> 00:38:25.415 There have been clients that we've told them, hey, we can do
00:38:25.415 --> 00:38:28.588 it. But here's the caveat. And you might have situations where
00:38:28.588 --> 00:38:31.611 you're seeing out of stock products in the catalog. Are you
00:38:31.611 --> 00:38:34.431 OK with that? And they said sure, why not. And so it is
00:38:34.431 --> 00:38:36.900 something that is physically possible for us to.
00:38:36.980 --> 00:38:41.266 It's just not accurate enough for us to have it as an out of
00:38:41.266 --> 00:38:42.110 box feature.
00:38:43.460 --> 00:38:46.329 So there's also other ways we've done it on other projects, but
00:38:46.329 --> 00:38:49.153 they're very, very specialized to the way those projects work.
00:38:49.153 --> 00:38:51.530 So it's certainly not worth bringing back into core.
00:38:54.410 --> 00:38:57.340 We should, we should the next time a client and then this is
00:38:57.340 --> 00:39:00.127 just a conversation really for just the core team, but we
00:39:00.127 --> 00:39:02.914 should definitely just bring that back inventory indexing
00:39:02.914 --> 00:39:05.748 back into core and then just wrap it in an app setting and
00:39:05.748 --> 00:39:08.727 keep it off and just tell people like yeah we can turn it on,
00:39:08.727 --> 00:39:08.919 but.
00:39:09.670 --> 00:39:13.123 Yeah. It may not be accurate. Yeah. Yeah. And just basically
00:39:13.123 --> 00:39:16.293 have a caveat on that one. That's feature flag of like,
00:39:16.293 --> 00:39:19.690 yeah, technically it's there. Yeah, that's a good idea. OK.
00:39:19.690 --> 00:39:19.860 So.
00:39:20.650 --> 00:39:24.072 Uh yeah. So looking back at the code, um, this, this query
00:39:24.072 --> 00:39:27.726 method and some of these other ones, again, super complex. I'm
00:39:27.726 --> 00:39:31.380 not gonna dive into absolutely everything, but I'm gonna go to
00:39:31.380 --> 00:39:34.918 the parts that you guys will probably run into. So something
00:39:34.918 --> 00:39:38.399 I mentioned earlier was like fuzzy searching and stuff like
00:39:38.399 --> 00:39:42.111 that. The amount of fuzziness is handled on these fuzziness. So
00:39:42.111 --> 00:39:45.649 some projects will want you to adjust how fuzzy the matching
00:39:45.649 --> 00:39:49.071 is, especially for stuff like custom key. We have a lot of
00:39:49.071 --> 00:39:50.580 projects that clients are.
00:39:50.660 --> 00:39:53.957 Uh, people are often like just like B to B stuff. People will
00:39:53.957 --> 00:39:56.988 come to the site and knowing that they're looking for an
00:39:56.988 --> 00:39:57.520 exact SKU.
00:39:58.210 --> 00:40:02.259 And so a lot of clients will say like for for custom key, can you
00:40:02.259 --> 00:40:06.124 like eliminate any fuzziness so that if I type in ABC D1234 as
00:40:06.124 --> 00:40:10.051 SKU and there's a product that matches that, I'm just gonna get
00:40:10.051 --> 00:40:13.916 that product and I'm not gonna get ABC 1235 even though that's
00:40:13.916 --> 00:40:15.880 really close to what I typed in.
00:40:17.470 --> 00:40:20.567 Obviously on like B to C sites like consumer facing stuff, we
00:40:20.567 --> 00:40:23.465 want that sort of fuzziness because most consumers aren't
00:40:23.465 --> 00:40:26.462 going to a site knowing the exact skew they're looking for.
00:40:26.462 --> 00:40:29.410 They just kind of know that the general topic. But for B2B
00:40:29.410 --> 00:40:32.457 sites, it makes a lot of sense. And so you may have projects
00:40:32.457 --> 00:40:35.605 that wanna adjust the fuzziness or just the boost of different
00:40:35.605 --> 00:40:38.652 things. This is what you're gonna handle the fuzziness these
00:40:38.652 --> 00:40:41.900 are you're gonna handle boosting specifically what boosting does
00:40:41.900 --> 00:40:42.200 is it?
00:40:43.730 --> 00:40:47.943 Uh, once elastic has determined all of the relevant results, it
00:40:47.943 --> 00:40:49.590 scores them based on how?
00:40:50.300 --> 00:40:54.453 Close, they match your original query. Boosting determines how
00:40:54.453 --> 00:40:58.541 much a match or a near match matters for a specific field. So
00:40:58.541 --> 00:41:02.497 for example, matches by custom Key and name have really big
00:41:02.497 --> 00:41:06.453 boosts as it says here with a massive boost. And the reason
00:41:06.453 --> 00:41:10.079 for that is obviously most of the time that's the most
00:41:10.079 --> 00:41:13.903 relevant thing. Somebody searching for a skew or the name
00:41:13.903 --> 00:41:18.189 of the product, that's the thing that they care to see the most,
00:41:18.189 --> 00:41:21.090 or they want to match the results the most.
00:41:21.190 --> 00:41:24.721 And so if you type in uh shoes and there's a product with the
00:41:24.721 --> 00:41:27.854 word shoes in the name and there's a different product
00:41:27.854 --> 00:41:30.988 that's sort of kind issue related but it has it in the
00:41:30.988 --> 00:41:34.462 description. Obviously the one we care the most about is one
00:41:34.462 --> 00:41:37.767 that has shoe in the name. That's what I searched. I want
00:41:37.767 --> 00:41:41.355 to see the one with shoe in the name and the one that has shoe
00:41:41.355 --> 00:41:44.830 in the description. It can come up but it needs to be scored
00:41:44.830 --> 00:41:48.191 lower. So that's what these boosts handle is they boost we
00:41:48.191 --> 00:41:51.040 can determine the relative relevance if you will.
00:41:51.170 --> 00:41:54.950 Of different fields and how much they match the query. Um, so
00:41:54.950 --> 00:41:58.609 again you may have projects where people are like, oh don't
00:41:58.609 --> 00:42:02.511 we don't worry as much about the name because people are coming
00:42:02.511 --> 00:42:06.353 in and typing in SKUs. Then you can go into these settings and
00:42:06.353 --> 00:42:10.195 you can drop down the boost by name a little bit and then bump
00:42:10.195 --> 00:42:13.853 up the boost by custom key to something a little bit higher
00:42:13.853 --> 00:42:17.390 and kind of adjusting the relative values between them to
00:42:17.390 --> 00:42:21.110 fine tune the results most of the time for things like that.
00:42:21.180 --> 00:42:24.725 Then, depending on the project, you often don't need to actually
00:42:24.725 --> 00:42:28.217 physically modify too much. You may need to eliminate fuzziness
00:42:28.217 --> 00:42:31.599 on some things, which it does and does involve coming in here
00:42:31.599 --> 00:42:35.090 and doing like I think you do. Like fuzziness dot edit distance
00:42:35.090 --> 00:42:37.600 0 and that does like 0 fuzziness on this one.
00:42:40.020 --> 00:42:40.620 Um.
00:42:42.390 --> 00:42:45.179 You may have to do that on some projects, but generally speaking
00:42:45.179 --> 00:42:47.925 you can get the results you're looking for by just tweaking the
00:42:47.925 --> 00:42:50.500 boosts um and occasionally touching. You know. If if the if
00:42:50.500 --> 00:42:52.988 touching the boost values doesn't get what you're looking
00:42:52.988 --> 00:42:55.649 for, then you can come in and play with the fuzziness and try
00:42:55.649 --> 00:42:57.280 it. You'll see if that if that helps.
00:42:58.000 --> 00:43:01.150 Um, what? Just kind of. Go ahead, Jamie.
00:43:03.750 --> 00:43:07.561 Just to help me understand the fuzziness, the auto, what is
00:43:07.561 --> 00:43:11.437 that equivalent? You just typed zero and I wanna say auto by
00:43:11.437 --> 00:43:15.375 default is like an edit distance of two and specifically it's
00:43:15.375 --> 00:43:18.869 it's levinstein distance I think or something similar.
00:43:20.170 --> 00:43:23.807 And it's it's. Uh, it's also by the length of the word. So the
00:43:23.807 --> 00:43:27.156 word is 12 characters long. It allows for a greater error
00:43:27.156 --> 00:43:30.794 distance, yeah, but if it's a 5 letter word, it's it will only
00:43:30.794 --> 00:43:32.179 do like two. OK, so and.
00:43:33.600 --> 00:43:36.667 A lot of it, like if you explicitly put in like a dot
00:43:36.667 --> 00:43:39.961
actually doing is like an edit
00:43:39.961 --> 00:43:42.290 distance of two means that in order for.
00:43:43.070 --> 00:43:46.440 For something to match based on that fuzziness.
00:43:47.330 --> 00:43:51.435 You would only have to modify two things about the destination
00:43:51.435 --> 00:43:55.541 to make it match the target. So for example like, um, icon and
00:43:55.541 --> 00:43:59.647 icon. Yeah yeah that's a good yeah. So if you misspelled this,
00:43:59.647 --> 00:44:03.427 this is 1 edit distance away because this one has an end.
00:44:03.427 --> 00:44:07.012 This one doesn't him. And another interesting thing is
00:44:07.012 --> 00:44:10.010 that this is also one edit according to most.
00:44:11.420 --> 00:44:14.163
Because it's just an, it's a
00:44:14.163 --> 00:44:17.156 transposition of those two letters, so it just swaps those.
00:44:17.156 --> 00:44:19.850 That counts as one. But then, you know if I had like.
00:44:20.730 --> 00:44:24.173 If I, you know, misspelled big time, this is 2. This is an edit
00:44:24.173 --> 00:44:27.240 distance of two, so this would still show up for an edit
00:44:27.240 --> 00:44:30.253 distance of two. If I was looking for Nikon and I typed
00:44:30.253 --> 00:44:33.427 Nikon with AC and an MI would still get results for Nikon,
00:44:33.427 --> 00:44:36.709 they would just be scored lower than something that directly
00:44:36.709 --> 00:44:40.045 matches Nikon for one reason or another through an extra M on
00:44:40.045 --> 00:44:43.435 the end of that would be 3 now because it has to change to and
00:44:43.435 --> 00:44:46.878 remove one. Yes, that would be 3. So at this point you wouldn't
00:44:46.878 --> 00:44:49.999 be matching Nikon results anymore if you only had an edit
00:44:49.999 --> 00:44:50.860 distance of two.
00:44:51.080 --> 00:44:54.258 Now edit this fuzziness auto might match Nikon to Nikon. It
00:44:54.258 --> 00:44:57.277 just depends on how elastic implements fuzziness auto in
00:44:57.277 --> 00:45:00.562 their system. And that's, you know, sort of just, you know, I
00:45:00.562 --> 00:45:04.005 don't know it might work but at some point you start to get to a
00:45:04.005 --> 00:45:07.342 point where you're like well, should Nikon with two M's and AC
00:45:07.342 --> 00:45:10.521 match Nikon with a K and 1 N like you're starting to get to
00:45:10.521 --> 00:45:13.699 point like it makes sense that it wouldn't if it didn't for
00:45:13.699 --> 00:45:16.931 some reason. So anyway so just real quick and this is for my
00:45:16.931 --> 00:45:20.056 application, everyone else as well, what is our out of box
00:45:20.056 --> 00:45:21.009 obviously we have.
00:45:21.080 --> 00:45:24.789 Auto on custom key and name do we boost. So we it is a massive
00:45:24.789 --> 00:45:28.558 boost right now, but that's all. It's on an app setting, right?
00:45:28.558 --> 00:45:32.150 So even though it says with a massive, massive boost, that's
00:45:32.150 --> 00:45:35.860 only for the out of box boost. Yeah right correct. Yeah so you
00:45:35.860 --> 00:45:39.747 could go in these and you could, I mean if you wanted to for some
00:45:39.747 --> 00:45:43.456 reason, you could go edit this setting in the settings file or
00:45:43.456 --> 00:45:47.107 in the admin and make this zero and just not boost by name at
00:45:47.107 --> 00:45:50.700 all. I don't know why you ever would, but that is completely
00:45:50.700 --> 00:45:51.230 possible.
00:45:52.310 --> 00:45:55.705 OK, so theoretically if I had a client that was like I my
00:45:55.705 --> 00:45:59.392 customers know their custom keys exactly and I could go set my
00:45:59.392 --> 00:46:02.963 fuzziness to zero on my name and really I could just set the
00:46:02.963 --> 00:46:06.241 boost to zero and it just wouldn't really wouldn't come
00:46:06.241 --> 00:46:08.700 back. But then I could set my custom key.
00:46:10.230 --> 00:46:14.083 Fuzzy fuzziness to edit distance zero and then a massive boost on
00:46:14.083 --> 00:46:17.469 the custom key. So 10,000 or 100,000 or whatever. I don't
00:46:17.469 --> 00:46:21.264 know what the Max is there, but super high so that if I'm typing
00:46:21.264 --> 00:46:24.650 that in, the only thing that comes back with at least the
00:46:24.650 --> 00:46:28.212 first thing that comes back is going to be the skew that I'm
00:46:28.212 --> 00:46:29.380 looking for exactly.
00:46:30.460 --> 00:46:30.830 Right.
00:46:31.650 --> 00:46:31.910 OK.
00:46:33.160 --> 00:46:36.597 So I'm gonna get a ticket in for changing the fuzziness by app
00:46:36.597 --> 00:46:37.470 setting as well.
00:46:38.720 --> 00:46:41.651 OK. Yeah, the the reason it's not there at the moment is
00:46:41.651 --> 00:46:44.736 because of the way that that function is called. It has. We
00:46:44.736 --> 00:46:47.873 would have to write separate things to make it actually that
00:46:47.873 --> 00:46:50.907 you could probably do something like and I'm just going to
00:46:50.907 --> 00:46:53.170 sketch this really fast, just like private.
00:46:54.090 --> 00:46:56.309 There's the type. Are there other options other than
00:46:56.309 --> 00:46:58.570 fuzziness auto and fuzziness fuzziness edit distance?
00:47:01.260 --> 00:47:01.830 Yes.
00:47:02.530 --> 00:47:03.050 Oh, there are.
00:47:05.900 --> 00:47:09.098 Yeah, and it it handles it differently on like it once an
00:47:09.098 --> 00:47:12.463 enum value and another is a class instance where you specify
00:47:12.463 --> 00:47:15.000 values and you're newing up a class instance.
00:47:15.670 --> 00:47:18.051 And stuff. So like the the function you're calling is
00:47:18.051 --> 00:47:20.829 different and the implying that inside that DSL tree is really
00:47:20.829 --> 00:47:21.050 hard.
00:47:23.150 --> 00:47:27.042 We'd have to make separate fuzziness things to have those
00:47:27.042 --> 00:47:30.531 reply correctly as they fuzziness object and and do
00:47:30.531 --> 00:47:31.270 weird ****.
00:47:32.250 --> 00:47:35.467 Yeah, I was thinking just literally just a theory. You
00:47:35.467 --> 00:47:39.153 could do something like this and then these would just be like
00:47:39.153 --> 00:47:42.956 get fuzziness and you're setting you'd be like plastic searching
00:47:42.956 --> 00:47:44.770 for other config dot searching.
00:47:45.600 --> 00:47:50.220 Name Buziness enabled and then.
00:47:51.690 --> 00:47:55.407 Uh, searching name fuzziness distance or something like that.
00:47:55.407 --> 00:47:59.244 And then this would go in here and return either fuzziness auto
00:47:59.244 --> 00:48:03.081 or the edit distance that you've enabled. And obviously, again,
00:48:03.081 --> 00:48:04.760 caveat on these settings of.
00:48:06.100 --> 00:48:09.352 You're gonna have a hard time tuning this but yeah. But you
00:48:09.352 --> 00:48:12.713 know by default it's gonna be one. I wouldn't do that. I just
00:48:12.713 --> 00:48:16.020 throw a -, 1 on there. And if you set it to anything else if
00:48:16.020 --> 00:48:19.490 you set it to, if you leave it, then it's just auto. And if you
00:48:19.490 --> 00:48:22.796 set it a value greater than zero, then that's just your edit
00:48:22.796 --> 00:48:26.374 distance and that's what we call a lot of box. Yeah. So basically
00:48:26.374 --> 00:48:29.410 it's like this, this values default -, 1. And then this
00:48:29.410 --> 00:48:32.608 would change to this and it would basically be if not this
00:48:32.608 --> 00:48:35.970 then start has value or distance equals whoopsie, Daisy -, 1.
00:48:36.060 --> 00:48:39.182 Then we're gonna go by auto, otherwise we return by the edit
00:48:39.182 --> 00:48:41.640 distance and then yeah, yeah, that makes sense.
00:48:42.890 --> 00:48:46.707 Yeah, that way all these are -, 1. All of these fuzziness
00:48:46.707 --> 00:48:49.670 distances would be -, 1 by default. Um, and.
00:48:50.640 --> 00:48:53.800 Otherwise you could uh, yeah, cool. I like that.
00:48:56.300 --> 00:48:59.220 I just pasted a screenshot of the docs about what auto does.
00:48:59.940 --> 00:49:01.840 And it's based upon the length of the characters.
00:49:02.580 --> 00:49:02.940 OK.
00:49:04.910 --> 00:49:07.750 So zero to two characters. It must match exactly three to
00:49:07.750 --> 00:49:10.541 five. It allows one edit. Anything greater than five, it
00:49:10.541 --> 00:49:11.570 allows 2 edits. Nice.
00:49:12.260 --> 00:49:16.236 OK, cool. And then you can specify an auto that has a low
00:49:16.236 --> 00:49:17.470 and high manually.
00:49:18.610 --> 00:49:19.290 As well.
00:49:21.690 --> 00:49:24.813 Alright, well, that's cool. That's pretty cool. So now we
00:49:24.813 --> 00:49:27.829 know that. Um, well, cool. I know this part's getting a
00:49:27.829 --> 00:49:31.222 little bit like. Yeah. I have another question real quick, and
00:49:31.222 --> 00:49:34.615 I apologize because I did get more more very specific for core
00:49:34.615 --> 00:49:37.792 stuff there. Can you tell me what's the difference between
00:49:37.792 --> 00:49:41.239 match, prefix and term? And you may have already said that, but
00:49:41.239 --> 00:49:44.040 I missed it. No, I can't, but James probably can't.
00:49:45.580 --> 00:49:52.230 Don't know the difference. Batch is done by looking at.
00:49:54.300 --> 00:49:57.172 Well, see the the suffix keyword part? Part kind of messes with
00:49:57.172 --> 00:49:59.820 this a little bit, but matches. You're matching the entire
00:49:59.820 --> 00:50:00.089 thing.
00:50:00.900 --> 00:50:04.496 Prefix is it starts with this value like it starts with blue
00:50:04.496 --> 00:50:07.090 when it when the value could be blue-green.
00:50:07.770 --> 00:50:10.925 Um, to make some kind of turquoise statement? Whatever
00:50:10.925 --> 00:50:14.368 um. And then term is, we're breaking it up into a series of
00:50:14.368 --> 00:50:17.811 known term values, ignoring words like ANS, toos. OK, I see
00:50:17.811 --> 00:50:21.196 stuff like that. But the key, the suffix keyword part also
00:50:21.196 --> 00:50:24.696 internally says don't just use the value, use the value, but
00:50:24.696 --> 00:50:28.196 also the value broken up into a set of keywords. So it makes
00:50:28.196 --> 00:50:31.984 them closer to what they are for each other. But I keep all three
00:50:31.984 --> 00:50:35.713 of them in there because any one of them could provide some kind
00:50:35.713 --> 00:50:38.639 of valid result that we're allowing to go through.
00:50:39.580 --> 00:50:44.372 I see, I see. OK, so I'm gonna. I'm gonna state that back to you
00:50:44.372 --> 00:50:49.018 then. So match is just any you provide the you provide the the
00:50:49.018 --> 00:50:53.368 value and you say anywhere in the whole string. So I would
00:50:53.368 --> 00:50:57.202 look down specifically. Specifically looking at the
00:50:57.202 --> 00:51:01.552 strings as a whole. Yeah, it wants to match entire string.
00:51:01.552 --> 00:51:03.470 Query query was cool hats.
00:51:04.150 --> 00:51:07.913 And there was a product that was just called cool car. Cool hats
00:51:07.913 --> 00:51:11.444 and cool car aren't gonna match on the match, but they might
00:51:11.444 --> 00:51:14.976 match on term because they both have cool and term breaks it
00:51:14.976 --> 00:51:16.249 into cool and exactly.
00:51:17.320 --> 00:51:18.710 Yes, separate terms.
00:51:19.360 --> 00:51:22.696 And whereas prefix if you searched hat with cool hats,
00:51:22.696 --> 00:51:26.640 that would not come back in that one because it's looking for at
00:51:26.640 --> 00:51:30.340 the start of the of the text. OK, so why don't? And this may
00:51:30.340 --> 00:51:33.920 just be our implementation of it. Why don't prefix in term
00:51:33.920 --> 00:51:34.830 have fuzziness?
00:51:36.250 --> 00:51:39.536 I don't think they. They might not be able. Yeah, they don't.
00:51:39.536 --> 00:51:42.822 They just don't have that as a feature from elastic. It's not
00:51:42.822 --> 00:51:45.737 part of their that feature. Yeah, gotcha. So if you're
00:51:45.737 --> 00:51:46.850 looking for a prefix.
00:51:47.520 --> 00:51:49.030 You better type it correctly.
00:51:50.440 --> 00:51:54.854 Yeah, and generally speaking, again like James was saying, um,
00:51:54.854 --> 00:51:59.338 the dot suffix keyword thing is breaking the name property into
00:51:59.338 --> 00:52:03.542 keywords, which are basically for gross oversimplification,
00:52:03.542 --> 00:52:07.747 it's splitting on spaces if you want to think about it like
00:52:07.747 --> 00:52:12.091 that. And then on that split, anything that wouldn't match to
00:52:12.091 --> 00:52:16.155 our regular word, like you just got a #2 or you got a% or
00:52:16.155 --> 00:52:20.149 something like that, it will ignore those as part of the
00:52:20.149 --> 00:52:20.780 keywords.
00:52:20.850 --> 00:52:24.321 It up and we've only the things that are actual words. So if I
00:52:24.321 --> 00:52:27.793 was looking for like a string of characters together, yeah. So
00:52:27.793 --> 00:52:30.879 does it cut because you mentioned like ANDS and ORS and
00:52:30.879 --> 00:52:34.241 so if I had like Barnes and Noble. Yeah. And I just searched
00:52:34.241 --> 00:52:37.382 Barnes noble it would find it because yeah it removes it
00:52:37.382 --> 00:52:40.689 filters out, yeah filters out and and and and the and A and
00:52:40.689 --> 00:52:43.720 basically any articles in general, they call them stop
00:52:43.720 --> 00:52:46.200 words. If I remember right. Yes, stop words.
00:52:47.700 --> 00:52:51.519 Cool. Yeah. So using that example, going back to like cool
00:52:51.519 --> 00:52:55.728 car or cool hat, if you typed in cooler as like a search option,
00:52:55.728 --> 00:52:59.548 would that hit the prefix? Would that match on the prefix?
00:52:59.548 --> 00:53:03.562 Because the beginning of the word is the same. If the product
00:53:03.562 --> 00:53:07.511 was cool hats and we searched cooler, it would not. Well, it
00:53:07.511 --> 00:53:11.719 would come back because it might come back on match because it's
00:53:11.719 --> 00:53:15.992 dropping 2 characters, unable to match that. But if it was, if it
00:53:15.992 --> 00:53:18.000 was cooler, you said cool list.
00:53:18.460 --> 00:53:20.881 That's three. That's three characters away. That would
00:53:20.881 --> 00:53:22.070 definitely not match, yeah.
00:53:22.860 --> 00:53:26.143 OK, gotcha with fuzzing this audio on match, right? If you if
00:53:26.143 --> 00:53:29.373 you cranked up the edit distance to like three or four, then
00:53:29.373 --> 00:53:32.445 yeah, but then you're gonna start getting some real weird
00:53:32.445 --> 00:53:35.834 results. Yeah, you start getting really bad results when you do
00:53:35.834 --> 00:53:37.370 too high of an edit distance.
00:53:38.560 --> 00:53:41.670 OK. Yeah. And obviously partially expects people to be
00:53:41.670 --> 00:53:45.290 messed up. Yeah. And obviously some of that bad results can be.
00:53:46.160 --> 00:53:49.457 Mediated with like, you know, sorting by the relevant score.
00:53:49.457 --> 00:53:52.700 But at some point people are just gonna start wondering why
00:53:52.700 --> 00:53:55.620 they're seeing hats when they typed in. I don't know.
00:53:56.610 --> 00:54:01.653 Uh, car I I can't. I don't know. Cars and hats the only thing on
00:54:01.653 --> 00:54:05.300 my mind right now for some reason. Um, but um.
00:54:06.670 --> 00:54:09.829 What that's about, um, so alright, so does anybody have
00:54:09.829 --> 00:54:13.271 any other questions here? Again, I know this stuff's kind of
00:54:13.271 --> 00:54:16.938 arcane and weird, but if anybody has anything else I can clarify
00:54:16.938 --> 00:54:20.211 while we're in here, by all means go down to the function
00:54:20.211 --> 00:54:20.550 score.
00:54:21.670 --> 00:54:25.124 Ohh yes this one's uh this is the doozy that that actually
00:54:25.124 --> 00:54:28.578 scores things before we get jumped down the function score
00:54:28.578 --> 00:54:32.209 on the on the one we were just looking at. If I have a client
00:54:32.209 --> 00:54:35.839 that's saying I don't want any fuzziness because I I've tried
00:54:35.839 --> 00:54:39.352 you know changing the edit distance to zero and things like
00:54:39.352 --> 00:54:41.870 that. Can I just remove can I remove like.
00:54:42.730 --> 00:54:47.744 Prefix and and and term and just say match on a fuzziness of edit
00:54:47.744 --> 00:54:48.580 distance 0.
00:54:49.890 --> 00:54:53.827 And that'll give me an exact match on exactly what they're
00:54:53.827 --> 00:54:58.032 saying, something like this. So yeah, here's, here's what I'll
00:54:58.032 --> 00:54:58.500 say is.
00:54:59.290 --> 00:55:02.748 And I assume you talking about Max, uh, because you and I tried
00:55:02.748 --> 00:55:05.990 and tried and tried and uh, I was actually gonna talk about
00:55:05.990 --> 00:55:09.341 what we ran into on that one here in a minute because I think
00:55:09.341 --> 00:55:12.421 it's super interesting and important for people to know.
00:55:12.421 --> 00:55:15.501 But what I'll say is, by all means, just screw with it a
00:55:15.501 --> 00:55:18.852 little bit, especially if it's something you just wanna learn
00:55:18.852 --> 00:55:20.959 and spare time. Just come in here and.
00:55:21.730 --> 00:55:24.693 They obviously don't PR this or anything but just comment stuff
00:55:24.693 --> 00:55:27.517 out and build it and then see what the difference is and the
00:55:27.517 --> 00:55:30.572 results you get back and you can start to understand what each of
00:55:30.572 --> 00:55:33.535 these individual pieces does for the query. So if you wanted to
00:55:33.535 --> 00:55:36.221 try that on your local, just come in here and comment out
00:55:36.221 --> 00:55:38.859 like prefix and term and just see what that does I mean.
00:55:39.520 --> 00:55:43.016 If if you've got the time to kind of experiment with it a
00:55:43.016 --> 00:55:46.874 little bit, by all means see if that, see if that gets with the
00:55:46.874 --> 00:55:50.672 clients looking for um, uh, a lot of the problem with what Max
00:55:50.672 --> 00:55:54.650 was looking for that we ended up discovering was they were one of
00:55:54.650 --> 00:55:57.906 the clients that said, yeah, it's OK if the inventory
00:55:57.906 --> 00:56:01.884 information is a little wrong as long as we can filter by it or a
00:56:01.884 --> 00:56:05.682 sort by it. And so we said OK and did that for them and that's
00:56:05.682 --> 00:56:09.420 fine, but the problem is that their default catalog sort and.
00:56:09.490 --> 00:56:12.754 In fact, the only catalog sort you're really able to get at
00:56:12.754 --> 00:56:15.040 without manually changing it is by stock.
00:56:15.920 --> 00:56:20.133 Even when you type in a query. So what that meant was we were
00:56:20.133 --> 00:56:24.007 typing in, let's say one of their skews was MT21, that I
00:56:24.007 --> 00:56:28.289 remember you type M21 with fuzzy matching, you're going to get
00:56:28.289 --> 00:56:32.367 matches for MT20 or MT28 or whatever. And obviously M21 has
00:56:32.367 --> 00:56:36.512 the highest relevance score of all of those. But when you're
00:56:36.512 --> 00:56:40.318 sorting by stock, we're not sorting by relevance, we're
00:56:40.318 --> 00:56:44.396 sorting by stock. So it didn't matter that M21 was what you
00:56:44.396 --> 00:56:45.620 typed in and that.
00:56:45.700 --> 00:56:49.604
there was less stock for MT21
00:56:49.604 --> 00:56:53.445 then something else it wasn't gonna show 1st. And So what we
00:56:53.445 --> 00:56:57.349 ended up trying to do at that time was basically making it so
00:56:57.349 --> 00:57:01.316 that if you searched by stock and had a query we did a sort of
00:57:01.316 --> 00:57:05.220 combination where we were going into this function score down
00:57:05.220 --> 00:57:09.187 here and trying to basically add a boost to the score based on
00:57:09.187 --> 00:57:12.209 the amount of inventory or product had so that.
00:57:13.310 --> 00:57:16.476 More relevant results would still come up closer to the top.
00:57:16.476 --> 00:57:19.746 And then of those more relevant results, they would be sort of
00:57:19.746 --> 00:57:22.860 internally or like grouped by the relevance and then sorted
00:57:22.860 --> 00:57:26.182 internally by stock. And I don't remember or think that we ever
00:57:26.182 --> 00:57:29.245 actually got that to work the way we wanted, but we didn't
00:57:29.245 --> 00:57:32.203 really play with the boosts too much. But that's kind of
00:57:32.203 --> 00:57:35.214 something you might run into as clients that have sort of
00:57:35.214 --> 00:57:38.432 conflicting asks with catalog results is they just want it to
00:57:38.432 --> 00:57:41.650 kind of magically know what you want and it doesn't work like
00:57:41.650 --> 00:57:41.909 that.
00:57:43.090 --> 00:57:46.459 Have it sort by like by relevance and by stock without
00:57:46.459 --> 00:57:50.319 having to figure out some very precise numbers that may or may
00:57:50.319 --> 00:57:54.241 not work for everything. And you know, it's one of those things
00:57:54.241 --> 00:57:57.978 that sounds simple, but it's really not that simple. And for
00:57:57.978 --> 00:58:01.654 the client, we were initially created the elastic build for
00:58:01.654 --> 00:58:01.900 Tau.
00:58:03.520 --> 00:58:06.251 They specifically wanted to be able to go into a particular
00:58:06.251 --> 00:58:09.027 product and say this product scores the highest whenever you
00:58:09.027 --> 00:58:11.940 search the word hammer. And it was a claw hammer, but they also
00:58:11.940 --> 00:58:12.759 had hammer drills.
00:58:13.870 --> 00:58:16.978 So hammer drills kept showing up first because a hammers earlier
00:58:16.978 --> 00:58:19.895 in the word, so we got a higher score and B the the customer
00:58:19.895 --> 00:58:22.907 just searching the word hammer, it's going to go anything that
00:58:22.907 --> 00:58:25.968 has anything to do with the word hammer. If they want to find a
00:58:25.968 --> 00:58:28.885 claw hammer first, they need to say claw hammer when they're
00:58:28.885 --> 00:58:31.802 searching and that's what we kept having to iterate to them,
00:58:31.802 --> 00:58:34.480 but they just would not understand it as we went along.
00:58:35.940 --> 00:58:39.092 Yeah. So, yeah, that's something to just kind of be aware of is
00:58:39.092 --> 00:58:42.245 that you will run into scenarios where clients have conflicting
00:58:42.245 --> 00:58:45.201 desires or, you know, maybe not conflicting, but they don't
00:58:45.201 --> 00:58:47.910 realize that what they're asking is substantially more
00:58:47.910 --> 00:58:51.112 complicated than it might seem. And that on the surface it seems
00:58:51.112 --> 00:58:54.167 like it should be something that isn't that difficult. But in
00:58:54.167 --> 00:58:57.320 reality, you're kind of, you're gonna run into a scenario where
00:58:57.320 --> 00:59:00.275 like, OK, well, I can't really do both of these at the same
00:59:00.275 --> 00:59:03.330 time or something like that, like the Max catalog issues that
00:59:03.330 --> 00:59:05.350 we thought I was just talking about, so.
00:59:06.470 --> 00:59:10.862 So yeah, it's fuzzy searching, and fuzzy matching is not an
00:59:10.862 --> 00:59:15.328 exact science. Uh, literally, that's the point of it is that
00:59:15.328 --> 00:59:19.647 it's not exact. Um, so trying to fine tune fuzziness to be
00:59:19.647 --> 00:59:24.040 exactly what you want is exactly the opposite of fuzziness.
00:59:25.380 --> 00:59:27.780 Like the opposite of the point of fuzziness. So just be aware
00:59:27.780 --> 00:59:28.710 that if somebody's like.
00:59:29.480 --> 00:59:33.477 You know trying to overly fine tune things or get weird sort
00:59:33.477 --> 00:59:37.541 order and whatnot that you're probably gonna need to you know
00:59:37.541 --> 00:59:41.866 just that's that's a warning for some bloat and complication that
00:59:41.866 --> 00:59:45.405 is hiding behind what sounds like a relatively simple
00:59:45.405 --> 00:59:49.272 features and build a Google level fuzziness matcher. Yeah.
00:59:49.272 --> 00:59:53.270 How hard could that be? But Amazon? Sure. OK do you have $10
00:59:53.270 --> 00:59:56.940 billion of Amazon money to give us Google? I just typed
00:59:56.940 --> 00:59:59.300 something and it says did you mean?
00:59:59.410 --> 01:00:02.816 This. Yeah, because they spent billions on that. Yeah. Can you
01:00:02.816 --> 01:00:05.736 guys, can you guys have it like auto correct, like my
01:00:05.736 --> 01:00:07.249 misspellings and stuff like.
01:00:08.330 --> 01:00:12.380 I mean, probably. How much money do you have? Yeah, exactly.
01:00:13.790 --> 01:00:16.220 Exactly. So anyway. Um.
01:00:16.920 --> 01:00:20.510 I know like I said we're we're getting in the weeds in it and
01:00:20.510 --> 01:00:24.217 I've asked about a million times but does anybody have any more
01:00:24.217 --> 01:00:27.750 questions? And then James I think you're gonna see something
01:00:27.750 --> 01:00:31.457 on the uh function score section here I have a dumb and obvious
01:00:31.457 --> 01:00:35.222 question. They're looking in the query multi match fields fields
01:00:35.222 --> 01:00:38.581 P goes to P dot custom key. Let's just look at that first
01:00:38.581 --> 01:00:42.056 line searching boost customer keyword function score factor
01:00:42.056 --> 01:00:43.910 that's coming from app settings.
01:00:44.730 --> 01:00:45.880 Yep, correct. OK.
01:00:46.640 --> 01:00:48.740 Anytime you see config or like.
01:00:49.500 --> 01:00:52.646 Provider config or just like Seth Config or whatever it is
01:00:52.646 --> 01:00:56.113 that's making. That's all gonna come from an app setting. Global
01:00:56.113 --> 01:00:59.633 app settings, Yep. And if you're looking at that in the code, you
01:00:59.633 --> 01:01:02.833 can also always just F-12 on that. And then you can go look
01:01:02.833 --> 01:01:06.193 at how that settings defined and you can see what data type is
01:01:06.193 --> 01:01:09.606 it, what's the app Settings key. So I can go find it in my file
01:01:09.606 --> 01:01:12.540 if it's defined. And if a setting is not defined, it's
01:01:12.540 --> 01:01:16.006 gonna be using its default value here. So like this one probably
01:01:16.006 --> 01:01:19.313 is not gonna be in your settings file, so you can assume that
01:01:19.313 --> 01:01:19.580 it's.
01:01:19.670 --> 01:01:23.898 He was going to be 500 because that's just it's default and
01:01:23.898 --> 01:01:27.985 real quick just so everyone knows cuz I know we have some
01:01:27.985 --> 01:01:32.495 new people in here as well. We do have settings on in the admin
01:01:32.495 --> 01:01:36.794 and I find at least it's much easier there. Now granted I do
01:01:36.794 --> 01:01:40.669 most most of the sites I'm searching for are not on my
01:01:40.669 --> 01:01:45.179 local so it's not easier but we do have admin site site setting
01:01:45.179 --> 01:01:49.689 updates here that now it's not fully built out so you just you.
01:01:49.760 --> 01:01:53.630 You can only search on the key itself, and it's a direct match,
01:01:53.630 --> 01:01:57.137 so you search you'd have to search for like you'd have to
01:01:57.137 --> 01:02:00.705 kind of know the key you're looking for in general, but it
01:02:00.705 --> 01:02:04.454 does edit it live, and it does make the change to the correct
01:02:04.454 --> 01:02:08.203 place, so you can do so if you know what you're changing, you
01:02:08.203 --> 01:02:12.012 can come in here to settings and so it'll it'll it'll actually
01:02:12.012 --> 01:02:15.882 write to your file. So let's say for example you're updating an
01:02:15.882 --> 01:02:19.390 app setting that is just using a default because it's not
01:02:19.390 --> 01:02:20.660 explicitly specified.
01:02:20.740 --> 01:02:24.000 In your app setting file, it will go right it to the app
01:02:24.000 --> 01:02:27.432 setting file, um so that it's an override and then and then
01:02:27.432 --> 01:02:31.035 update the site as necessary to be able to use it properly and
01:02:31.035 --> 01:02:34.582 real quick. I'll show you guys from the admin home page here.
01:02:34.582 --> 01:02:38.242 To get to that settings editor you can go home site maintenance
01:02:38.242 --> 01:02:41.732 settings and here it is. This will also show you the default
01:02:41.732 --> 01:02:45.164 value of a setting and the current value that's being used.
01:02:45.164 --> 01:02:48.310 There may be scenarios where you'll notice and this is
01:02:48.310 --> 01:02:49.740 really, really important.
01:02:50.980 --> 01:02:54.464 I'm gonna see if I can find a good example here. Um type type
01:02:54.464 --> 01:02:54.970 in panes.
01:02:57.700 --> 01:03:01.324 Because I found this one earlier. So like uh if you have
01:03:01.324 --> 01:03:05.266 like the billing and you see a billing enabled is false, Yep,
01:03:05.266 --> 01:03:09.018 billing position by default is 1, but the value being used
01:03:09.018 --> 01:03:13.151 right now is 0. Why? Because if you click on billing position it
01:03:13.151 --> 01:03:16.648 will tell you it depends on billing being enabled this
01:03:16.648 --> 01:03:17.030 value.
01:03:17.920 --> 01:03:21.482 Just is the default value of an int because the setting that it
01:03:21.482 --> 01:03:22.930 depends on is not enabled.
01:03:23.820 --> 01:03:27.100 And this one depends on clarity billing enabled, which is
01:03:27.100 --> 01:03:30.438 enabled and I believe I would, I would assume it's enabled
01:03:30.438 --> 01:03:33.832 probably by default, but so that's really important is if I
01:03:33.832 --> 01:03:37.508 was like yeah, I wanna go change purchase panes billing position
01:03:37.508 --> 01:03:41.072 from 1 to 20 for some reason I go in here and I do 20 and save
01:03:41.072 --> 01:03:44.636 it. It's gonna refresh this page and it's still gonna be 0 and
01:03:44.636 --> 01:03:46.220 I'm like well what the hell.
01:03:47.320 --> 01:03:49.820 It's because it depends on something. This UI is not
01:03:49.820 --> 01:03:52.745 perfect. As you can see this was kind of the first attempt at
01:03:52.745 --> 01:03:55.859 actually getting us like runtime
01:03:55.859 --> 01:03:58.737 will be improved as time goes on. But that's just a call out
01:03:58.737 --> 01:04:01.756 for people. If you're ever, if you ever change one of these and
01:04:01.756 --> 01:04:04.634 you're like I don't understand why this isn't working, check
01:04:04.634 --> 01:04:07.229 the depends on or the mute. There's also some that are
01:04:07.229 --> 01:04:10.201 mutually exclusive with other ones. So let's see. Sales quotes
01:04:10.201 --> 01:04:11.570 I think is one that has that.
01:04:13.410 --> 01:04:14.440 Split enabled.
01:04:19.060 --> 01:04:21.413 Damn, I can't remember where that is. I can't think of any
01:04:21.413 --> 01:04:22.970 mutually exclusive settings right now.
01:04:24.240 --> 01:04:27.606 Well, basically if you change one to turn it on, it will turn
01:04:27.606 --> 01:04:30.919 the other ones off. Or do you have to turn the other one off
01:04:30.919 --> 01:04:34.394 first before you turn this one on? Uh, I think you have to turn
01:04:34.394 --> 01:04:37.706 the other one off first before you can turn this one on. OK.
01:04:37.706 --> 01:04:41.073 And then real quick, Tristan, you have your hand up. Was that
01:04:41.073 --> 01:04:44.494 regards to this or go back as. Yeah, OK, cool. Yeah, one cool.
01:04:44.494 --> 01:04:47.969 Or I guess call out and again, I know this is work in progress.
01:04:49.210 --> 01:04:52.073 From what I saw, when you like blank out a field in there, it
01:04:52.073 --> 01:04:55.029 says it's like an empty string. It doesn't actually like set to
01:04:55.029 --> 01:04:55.260 null.
01:04:56.220 --> 01:04:58.741 Yeah, future would be cool if you could do that. Yeah. An
01:04:58.741 --> 01:05:01.220 explicit way to set something new. Yeah, that was, yeah.
01:05:03.280 --> 01:05:03.900 Umm.
01:05:04.620 --> 01:05:06.800 The little a little extra like.
01:05:07.510 --> 01:05:10.710 Uh, mini. But on the side, that's exactly what that would
01:05:10.710 --> 01:05:14.020 be like. Yeah, that's actually know it. Yeah. Yep. And then
01:05:14.020 --> 01:05:15.400 Nick, you had a question.
01:05:17.400 --> 01:05:20.769 Yeah, I just had tickets earlier in this Sprint that were
01:05:20.769 --> 01:05:24.313 specifically for updating apps settings and I just wanted to
01:05:24.313 --> 01:05:27.391 see if this was a option for making those changes or
01:05:27.391 --> 01:05:31.109 absolutely OK, there's optimal developer if you yeah so like if
01:05:31.109 --> 01:05:34.479 you had I would say that you should prefer this if you're
01:05:34.479 --> 01:05:38.197 like updating settings like a QA site or something because this
01:05:38.197 --> 01:05:41.856 is way faster than going Rep in the machine, opening the file,
01:05:41.856 --> 01:05:45.052 changing it, saving it, restarting the site, whatever.
01:05:45.052 --> 01:05:47.550 This is just you can just go log into the.
01:05:47.620 --> 01:05:51.550 Even for the. So let's say that I have a task that's like on.
01:05:51.550 --> 01:05:55.735 Damn, I can't think of any good. The one I had was validating the
01:05:55.735 --> 01:05:56.749 payment methods.
01:05:57.480 --> 01:06:00.459 The set of box, yeah, it's yeah. So that one you would just go
01:06:00.459 --> 01:06:03.155 through, um, check out on the site and make sure that it
01:06:03.155 --> 01:06:06.135 functions as you would expect. And you may not be aware of how
01:06:06.135 --> 01:06:09.256 it's supposed to function, since that would probably be the first
01:06:09.256 --> 01:06:10.770 time you're actually doing that.
01:06:12.230 --> 01:06:15.538 But uh so TMK quite to your question on, you're talking
01:06:15.538 --> 01:06:18.846 about and and this is another reason I call this out is
01:06:18.846 --> 01:06:22.390 because, Nick, this is for your question too, for if you're
01:06:22.390 --> 01:06:26.289 going to do it for a QA site for example, those are on a pipeline
01:06:26.289 --> 01:06:30.070 at least the new sites are. And so the way the deployment works
01:06:30.070 --> 01:06:33.792 used to work is that it would just take the backup configs and
01:06:33.792 --> 01:06:37.514 then it would write them into the new site. And so the way you
01:06:37.514 --> 01:06:41.117 had to do it in the past was you'd have to go into the file,
01:06:41.117 --> 01:06:42.240 make the change to.
01:06:42.550 --> 01:06:45.963 Um the the live app settings, but then also make the change to
01:06:45.963 --> 01:06:48.888 the backup config. Because if you didn't do that, the
01:06:48.888 --> 01:06:52.139 deployment would just overwrite your setting change and the
01:06:52.139 --> 01:06:55.390 backup configs. If you didn't change those would be good to
01:06:55.390 --> 01:06:58.857 go. What the deployment does now is it goes before it wipes and
01:06:58.857 --> 01:07:02.379 brings in the new site. It goes and backs up your configs to the
01:07:02.379 --> 01:07:05.846 backup config. So it takes your live configs, moves them to the
01:07:05.846 --> 01:07:09.043 backup configs, does deployment, and then moves the backup
01:07:09.043 --> 01:07:12.023 configs back to the live setting. Yep. So any settings
01:07:12.023 --> 01:07:12.890 that you change?
01:07:12.960 --> 01:07:16.445 They're directly in the solution items folder in the deployed
01:07:16.445 --> 01:07:19.817 site or through this admin
01:07:19.817 --> 01:07:22.010 deployments on on sites that are CCSD.
01:07:23.210 --> 01:07:26.462 OK. And and so for for future stuff, Nick any, any like you'll
01:07:26.462 --> 01:07:29.249 you'll probably get these tickets all the time in the
01:07:29.249 --> 01:07:32.604 future that it'll be like enable XYZ feature. And in the past it
01:07:32.604 --> 01:07:35.702 was like you had to do it on your local and you had already
01:07:35.702 --> 01:07:38.902 put into the QA box and you had already in the stage box, you
01:07:38.902 --> 01:07:42.103 had to tell every developer on the project like hey everybody
01:07:42.103 --> 01:07:45.251 turned this feature flag on or whatever. We still have to do
01:07:45.251 --> 01:07:48.503 the telling everybody to do it. There's a way that we can work
01:07:48.503 --> 01:07:51.807 around that. It's just that we have to make sure developers are
01:07:51.807 --> 01:07:54.750 diligent about not committing credentials into the file.
01:07:55.020 --> 01:07:58.441 Um, so we haven't done that yet because we've seen no evidence
01:07:58.441 --> 01:08:01.917 that people will be diligent at the moment. Uh, but now you can
01:08:01.917 --> 01:08:05.067 just go in here and change it on. So I can go to. I can't
01:08:05.067 --> 01:08:08.271 think of. Can somebody just throw out a QA site that isn't
01:08:08.271 --> 01:08:11.150 Ridel right now? That's the only one I can think of.
01:08:13.870 --> 01:08:16.270 So I can go into the admin for Nova here.
01:08:18.460 --> 01:08:22.990 And is it? Noah? Is one of the products on Nova? Uh, Johnny 5.
01:08:26.330 --> 01:08:30.906 I don't get a really old movie short circuit joke. Come on. I
01:08:30.906 --> 01:08:35.040 don't get it. I'm sorry. People are uncultured. I know.
01:08:36.220 --> 01:08:36.920 Um.
01:08:38.270 --> 01:08:41.400 And I guess it's probably spin it up because probably nobody's
01:08:41.400 --> 01:08:44.580 really gone to this one yet. But anyway, just to service, I can
01:08:44.580 --> 01:08:47.512 go to Nova QA, hit the same admin editor on this site, and
01:08:47.512 --> 01:08:50.593 go set the settings I want on QA and it's ready to test. It's
01:08:50.593 --> 01:08:53.823 ready to roll for QA to come in and validate that those settings
01:08:53.823 --> 01:08:57.003 are working as they need to and blah blah blah. I don't have to
01:08:57.003 --> 01:09:00.084 go and change it directly on the box and manually restart the
01:09:00.084 --> 01:09:03.115 site. I thought that we modified it so that environment, the
01:09:03.115 --> 01:09:06.146 environment, app settings does get client committed, client,
01:09:06.146 --> 01:09:07.239 client gets committed.
01:09:07.670 --> 01:09:11.666 Well, that's the thing that I was gonna that I was gonna say
01:09:11.666 --> 01:09:12.060 is um.
01:09:13.110 --> 01:09:16.036 If you're working on newer projects, there are two apps
01:09:16.036 --> 01:09:16.820 settings files.
01:09:17.730 --> 01:09:21.002 And I think the reason that we didn't go down this road first
01:09:21.002 --> 01:09:24.381 is we wanted to have this kind of live on projects for a little
01:09:24.381 --> 01:09:27.495 bit and get people used to the concept of two separate app
01:09:27.495 --> 01:09:30.926 settings files. And then there's also some stuff I wanted to get
01:09:30.926 --> 01:09:34.146 more buttoned down with the new setting stuff before we push
01:09:34.146 --> 01:09:37.261 this out. But there are two files, client and environment.
01:09:37.261 --> 01:09:40.323 Ignore the connection strings one for now, just these two
01:09:40.323 --> 01:09:43.490 apps, settings dot client and environment are the ones that
01:09:43.490 --> 01:09:44.810 matter. And because I am.
01:09:45.470 --> 01:09:49.306 I am who I am, and I won't apologize for it. I'm going to
01:09:49.306 --> 01:09:50.630 do this in here. Um.
01:09:51.670 --> 01:09:52.940 Client and environment.
01:09:53.760 --> 01:09:57.650 Your your client file over here on the right. This has all of
01:09:57.650 --> 01:10:01.604 your configuration values that are specific to the client, and
01:10:01.604 --> 01:10:05.056 that's it. Feature files, feature settings. Absolutely
01:10:05.056 --> 01:10:08.884 nothing in the client folder should be relevant to or in the
01:10:08.884 --> 01:10:12.775 client. JSON file. Should be relevant to folder paths on your
01:10:12.775 --> 01:10:15.913 local URLs, passwords, or usernames, or any other
01:10:15.913 --> 01:10:19.992 credentials. None of that should be stored in your app. Settings
01:10:19.992 --> 01:10:23.695 dot client dot JSON. This file should only be features and
01:10:23.695 --> 01:10:24.260 specific.
01:10:24.330 --> 01:10:26.280 Flags related to your client.
01:10:28.870 --> 01:10:31.772 And then on I'm not. Yeah, sorry, I sound like you're gonna
01:10:31.772 --> 01:10:32.450 say something.
01:10:35.760 --> 01:10:37.870 Might have been talking to a little one in the room, not us.
01:10:38.720 --> 01:10:40.910 Very possible. Um.
01:10:41.570 --> 01:10:44.640 So anyway, and then on the flip side, the appsettings
01:10:44.640 --> 01:10:48.165 environment file. This should be the one that contains all of
01:10:48.165 --> 01:10:51.406 your folder paths, URLs and credentials. Things that are
01:10:51.406 --> 01:10:54.932 either gonna be different for everybody because obviously not
01:10:54.932 --> 01:10:58.457 everyone has the same folder paths. Obviously right? Local is
01:10:58.457 --> 01:11:02.039 only relevant to my local and not to the QA site. So this file
01:11:02.039 --> 01:11:05.565 and then same thing with Prema with passwords. So it's either
01:11:05.565 --> 01:11:09.147 folder names and URLs where it may be different on other sites
01:11:09.147 --> 01:11:11.819 or passwords we do not want in source control.
01:11:11.910 --> 01:11:15.356 Because that's just a no no. It's you're asking for that
01:11:15.356 --> 01:11:17.110 stuff to be breached. Um, so.
01:11:19.050 --> 01:11:22.569 In the future what we want to do is make it so that this feature
01:11:22.569 --> 01:11:26.088 file, the client file can be in source control. But what we need
01:11:26.088 --> 01:11:29.337 to do first is make sure that every developer is being very
01:11:29.337 --> 01:11:32.639 diligent about not committing passwords and folder names and
01:11:32.639 --> 01:11:35.942 all that stuff into this. And part of the problem is that at
01:11:35.942 --> 01:11:39.407 the moment if you go and edit a setting and Eric this is to get
01:11:39.407 --> 01:11:42.872 your original question. If you
01:11:42.872 --> 01:11:46.229
like authorize that and go set
01:11:46.229 --> 01:11:48.719 my authorize net login transaction in secret.
01:11:48.990 --> 01:11:52.967 The only stores these into the client file right now. So what I
01:11:52.967 --> 01:11:56.820 need to do is I need to go into the settings editing code and
01:11:56.820 --> 01:11:58.250 have it so that if the.
01:11:59.790 --> 01:12:02.787 Setting is part of a credential, so if if it has like the
01:12:02.787 --> 01:12:05.991 credential property auditor or attribute on it that it should
01:12:05.991 --> 01:12:06.870 store those into.
01:12:08.340 --> 01:12:09.020 Door.
01:12:10.320 --> 01:12:13.387 Environment dot JSON file locally so that it doesn't
01:12:13.387 --> 01:12:17.092 persist them. And then of course you can see that it warns you,
01:12:17.092 --> 01:12:20.796 hey, this setting is using its defaults for this credential, so
01:12:20.796 --> 01:12:22.070 that's probably wrong.
01:12:23.730 --> 01:12:25.550 So anyway. Um.
01:12:26.210 --> 01:12:28.833 That's why we don't have that pushed out into, you know, live
01:12:28.833 --> 01:12:31.499 yet. I think it would be very nice if we if we could get that.
01:12:31.499 --> 01:12:34.080 But there's some things I have to do for the settings editor
01:12:34.080 --> 01:12:36.746 where I would need to go through and make sure that everything
01:12:36.746 --> 01:12:39.369 that is a credential is tagged with the credential attribute,
01:12:39.369 --> 01:12:41.824 which I think I already did that I may need to do like an
01:12:41.824 --> 01:12:44.574 additional pass for any changes in core that have happened since
01:12:44.574 --> 01:12:45.039 I did that.
01:12:45.770 --> 01:12:49.398 Um, but I'll need to go back and and make sure that all the
01:12:49.398 --> 01:12:53.268 credential stuff is tagged and then also anything that's folder
01:12:53.268 --> 01:12:56.776 name like folder path or URL needs to either be tagged as
01:12:56.776 --> 01:12:59.800 readonly if editing it would break your site, or.
01:13:01.210 --> 01:13:05.688 As uh, some of that Mark said, it's environment specific so
01:13:05.688 --> 01:13:10.466 that basically the editor knows which file to store the changes
01:13:10.466 --> 01:13:10.690 to.
01:13:12.120 --> 01:13:14.971 Based on the changes there are and then it would go through at
01:13:14.971 --> 01:13:17.505 the same time and just, you know, edit whichever update
01:13:17.505 --> 01:13:19.090 whichever file it needs to update.
01:13:20.690 --> 01:13:24.556 So, uh, very doable. Just haven't had the time to. And it
01:13:24.556 --> 01:13:28.889 kind of slipped my mind that we were not doing that yet. So, and
01:13:28.889 --> 01:13:32.821 I'm creating tickets for us in the back in the background.
01:13:32.821 --> 01:13:36.954 We're gonna get stuff working. Well, yeah. So yeah, hopefully
01:13:36.954 --> 01:13:41.287 that answers that question. Once we get to that, the whole point
01:13:41.287 --> 01:13:45.487 then, Nick, would be that you could just make the modification
01:13:45.487 --> 01:13:49.553 on your local for turning those features on and off and then
01:13:49.553 --> 01:13:50.019 commit.
01:13:50.090 --> 01:13:53.480 That client file, PR that file, that client file. And when QA
01:13:53.480 --> 01:13:56.981 gets it or anybody pulls Q, you know, pulls QA for their latest
01:13:56.981 --> 01:14:00.043 local, they'll have those feature modifications. And we
01:14:00.043 --> 01:14:03.434 don't have to say, hey everybody turn this feature on or turn
01:14:03.434 --> 01:14:06.879 this feature off or copy the app settings from blah blah blah.
01:14:07.600 --> 01:14:11.400 Yeah, yeah, that makes sense. And one of the tips that I think
01:14:11.400 --> 01:14:15.320 Brennan gave me when setting up my local environment was to grab
01:14:15.320 --> 01:14:19.301 the app settings file from QA to get a lot of the settings that I
01:14:19.301 --> 01:14:22.981 was originally missing. One concern that I have is because I
01:14:22.981 --> 01:14:26.419 had to make these changes earlier where I think I had to
01:14:26.419 --> 01:14:29.616 turn off one of the payment options that was set for
01:14:29.616 --> 01:14:33.295 anaerobe, and I did the old way of remoting into the server,
01:14:33.295 --> 01:14:36.854 manually updating the file, saving it, but I don't think I
01:14:36.854 --> 01:14:37.879 notified anybody.
01:14:38.630 --> 01:14:42.100 That I made that change. So I don't know. If not the first
01:14:42.100 --> 01:14:45.746 person, you're not the first person to ever do that about it.
01:14:45.746 --> 01:14:48.570 It happens, OK? Just that's not even why. Yeah.
01:14:48.640 --> 01:14:51.762 Yeah, probably not even the thousandth, but you know, that's
01:14:51.762 --> 01:14:55.089 why. That's why we wanted to get it. Where that feature set file
01:14:55.089 --> 01:14:58.211 is in app settings is because it's frequently something that
01:14:58.211 --> 01:15:01.179 everyone I forget to do all the time is tell people, hey,
01:15:01.179 --> 01:15:04.404 there's new settings that you're gonna need to enable for this
01:15:04.404 --> 01:15:07.679 feature to work, for the site to work the way it's supposed to.
01:15:09.020 --> 01:15:12.325 And so if we could just make it so people don't even have to it,
01:15:12.325 --> 01:15:15.377 that's that's not even we have to do. It's just part of our
01:15:15.377 --> 01:15:18.428 workflow is making sure that everybody's settings are up to
01:15:18.428 --> 01:15:20.869 date and that just makes everything easier, so.
01:15:21.920 --> 01:15:24.807 But yes, fantastic, uh, fantastic questions and call
01:15:24.807 --> 01:15:28.077 outs. I think we're just about out of time. But there's one
01:15:28.077 --> 01:15:31.237 last thing I wanted to cover that's not really related to
01:15:31.237 --> 01:15:34.452 elastic. It's just sort of a language thing that that Nick
01:15:34.452 --> 01:15:37.613 mentioned that I thought was it's something that I really
01:15:37.613 --> 01:15:41.046 only sort of recently made sense of and now I love it and it's
01:15:41.046 --> 01:15:41.700 really cool.
01:15:43.420 --> 01:15:45.290 Uh, so.
01:15:46.080 --> 01:15:49.208 Uh, I'm gonna cover that real quick for the next like 5
01:15:49.208 --> 01:15:52.671 minutes. And I just sketch it out in linqpad. So we this came
01:15:52.671 --> 01:15:56.191 up when we were looking at the Elasticsearch code earlier this
01:15:56.191 --> 01:15:59.431 morning. Nick saw the yield return and he's like, I don't
01:15:59.431 --> 01:16:03.062 really know what that does. And I think probably most people see
01:16:03.062 --> 01:16:06.469 this and it's like whatever magic and they just move on. And
01:16:06.469 --> 01:16:09.709 so I wanted to take a really quick moment to explain what
01:16:09.709 --> 01:16:12.950 yield return does and when to use it and why it's useful.
01:16:14.410 --> 01:16:19.050 Well, the example that I gave is and I'm going to clear some of
01:16:19.050 --> 01:16:19.340 the.
01:16:20.190 --> 01:16:22.601 Actually I'm going to cut this and paste it because it was a
01:16:22.601 --> 01:16:24.420 lot of typing. I don't want to type it again.
01:16:28.390 --> 01:16:32.517 OK so my first implementation of get letters here and I'm going
01:16:32.517 --> 01:16:36.709 to scooch this away so you can't see it. My first implementation
01:16:36.709 --> 01:16:40.578 of get letters right here it just creates a new results and
01:16:40.578 --> 01:16:44.641 then for each char or character in the input it adds it to the
01:16:44.641 --> 01:16:48.446 array and then returns the result. And then I can for each
01:16:48.446 --> 01:16:52.573 over that list that this returns and draw each letter. And if I
01:16:52.573 --> 01:16:56.701 run this it's just going to say hello just like that and that's
01:16:56.701 --> 01:16:58.120 all cool and fine but.
01:16:58.540 --> 01:17:02.533 There's some performance implication here, specifically,
01:17:02.533 --> 01:17:06.666 and it'll get a little bit in the hair, but if I made this
01:17:06.666 --> 01:17:10.799 some really, really long string, wow, this is really long.
01:17:10.799 --> 01:17:11.500 Seriously.
01:17:13.030 --> 01:17:14.800 How much longer does this thing go?
01:17:16.220 --> 01:17:21.435 Um, and I run this, you know, obviously again, this is of
01:17:21.435 --> 01:17:26.830 course successful 00.0 seconds. That's cool and funny, but.
01:17:28.060 --> 01:17:31.836 What what you don't really see happening here is these these
01:17:31.836 --> 01:17:35.799 like new lists right? Here #1. There's performance implications
01:17:35.799 --> 01:17:39.390 for allocating space on the heap for a new list and then.
01:17:40.230 --> 01:17:43.110 With things like list and other, uh, dynamically sized
01:17:43.110 --> 01:17:46.200 collections, as you're adding elements to that list in the
01:17:46.200 --> 01:17:49.029 background, the runtime is making sure that that list
01:17:49.029 --> 01:17:52.171 actually has enough allocated space for all the things that
01:17:52.171 --> 01:17:55.419 are in it. And if it runs out of room, it has to reallocate a
01:17:55.419 --> 01:17:58.666 larger block of memory and then copy what it already has into
01:17:58.666 --> 01:18:01.285 that new block of memory. And it's doing all this
01:18:01.285 --> 01:18:04.690 transparently, so you don't have to worry about it. As we're not
01:18:04.690 --> 01:18:07.832 writing C or C over here, we don't care about all that, but
01:18:07.832 --> 01:18:10.556 there's still performance implications because it's
01:18:10.556 --> 01:18:11.080 happening.
01:18:11.260 --> 01:18:15.584 So, uh, that can that can kind of. You know, it's it's unlikely
01:18:15.584 --> 01:18:19.571 that you're gonna notice it for this example, but what I'm
01:18:19.571 --> 01:18:23.625 getting at is basically that there is a performance penalty
01:18:23.625 --> 01:18:27.612 to dynamic memory like this, and so doing yield return can
01:18:27.612 --> 01:18:31.464 sidestep that. That's only one of the benefits of you'll
01:18:31.464 --> 01:18:35.586 return, but this is the same functionality implemented using
01:18:35.586 --> 01:18:39.167 yield return. So instead of a list, I'm returning an
01:18:39.167 --> 01:18:41.600 ienumerable. That's the same thing.
01:18:41.670 --> 01:18:45.563 Enable string, list, string um and for each result I yield
01:18:45.563 --> 01:18:49.918 return the letter and then all I do is change this to get letters
01:18:49.918 --> 01:18:52.690 yielded and we get the exact same result.
01:18:54.380 --> 01:18:58.840 So this code #1 it's more concise, which is you know.
01:18:59.910 --> 01:19:03.187 Sometimes a good thing. Uh, usually a good thing. Sometimes.
01:19:03.187 --> 01:19:06.302 Sometimes it's better for code to be verbose if it's more
01:19:06.302 --> 01:19:09.580 readable. Keep that in mind. Concise does not mean readable.
01:19:11.220 --> 01:19:15.417 I say that to some of the. I say that to some of the, uh, some of
01:19:15.417 --> 01:19:19.296 the really cool and yet very confusing pattern matching I've
01:19:19.296 --> 01:19:23.176 seen where it's like, yeah, you just turned 20 lines of code
01:19:23.176 --> 01:19:27.182 into one, but I can't tell what that one line of code is doing
01:19:27.182 --> 01:19:31.062 to save my life. So. Exactly. Yep. So keep that in mind that
01:19:31.062 --> 01:19:35.005 other people have to read your code. Yeah, yeah. I've said on
01:19:35.005 --> 01:19:39.075 more than one occasion there are two priorities to writing code
01:19:39.075 --> 01:19:40.920 for a business. It is. It is.
01:19:41.130 --> 01:19:45.405 It was 51% more. It's 51% that the code solves the problem at
01:19:45.405 --> 01:19:49.474 hand, and 49% that we can use that code again later. It is
01:19:49.474 --> 01:19:53.612 only barely more important that the code you write solves a
01:19:53.612 --> 01:19:57.130 problem than it is that we can use it again later.
01:19:58.620 --> 01:20:02.083 And you know that's yeah, sorry, go ahead further, further
01:20:02.083 --> 01:20:05.430 improving the performance and throughput on this, if you
01:20:05.430 --> 01:20:08.952 wanted to test the throughput, you could wrap the outer for
01:20:08.952 --> 01:20:12.592 each in the main with a for loop and have it like run it 1000
01:20:12.592 --> 01:20:16.115 times or 10,000 times to get yourself account that's higher
01:20:16.115 --> 01:20:19.813 and then see the comparison of the physical runs to instead of
01:20:19.813 --> 01:20:23.160 returning a list of strings, return a list of characters
01:20:23.160 --> 01:20:26.800 since they are characters. And three don't interpolate letter
01:20:26.800 --> 01:20:29.090 into a string which the interpolation.
01:20:29.160 --> 01:20:32.674 Alone is one thing and then the cost a lot. Yeah that's that's a
01:20:32.674 --> 01:20:35.972 lot of cost going into those things that could be that could
01:20:35.972 --> 01:20:39.378 be stripped out. Yeah those are just a super trivial there's a
01:20:39.378 --> 01:20:42.784 lot to show how it works and I was I was being lazy but anyway
01:20:42.784 --> 01:20:45.866 let's let's be I wanna be I wanna ask a question then so
01:20:45.866 --> 01:20:49.272 could you. So theoretically if you had let's say in Kinect and
01:20:49.272 --> 01:20:52.786 I know none of us are in Kinect development development land but
01:20:52.786 --> 01:20:55.759 theoretically there is some value there for instead of
01:20:55.759 --> 01:20:58.949 returning a list of account models you return a list of or
01:20:58.949 --> 01:20:59.219 an I.
01:20:59.320 --> 01:21:02.886 Innumerable of account models and yield return and new account
01:21:02.886 --> 01:21:06.339 model because there is a there is a pretty heavy performance
01:21:06.339 --> 01:21:09.396 loss in copying 1000 account models. Yes, exactly the
01:21:09.396 --> 01:21:12.453 difference into different memories and slots. Another
01:21:12.453 --> 01:21:15.907 advantage of yield return is it's returning as it processes,
01:21:15.907 --> 01:21:19.134 so the outer for each loop can begin to start processing
01:21:19.134 --> 01:21:22.530 console dot writeline on the interval on a character that's
01:21:22.530 --> 01:21:26.210 come back while other characters are still in processing to come
01:21:26.210 --> 01:21:29.437 back out of you get letters yielded, especially when you
01:21:29.437 --> 01:21:30.230 have an async.
01:21:30.310 --> 01:21:31.370 Uh, a weight yield.
01:21:34.800 --> 01:21:37.050 Ohh come on, does linqpad not have this?
01:21:38.480 --> 01:21:41.917 So I guess so OK so anyway, so rather than I could do here is I
01:21:41.917 --> 01:21:43.260 could do I could do like.
01:21:44.110 --> 01:21:49.295 Um, like this and with get letters every letter has to be
01:21:49.295 --> 01:21:54.927 processed first before any of this will run, but with with the
01:21:54.927 --> 01:21:56.090 yield return.
01:21:56.810 --> 01:21:59.896 It will process one single letter and then sleep for five
01:21:59.896 --> 01:22:03.248 seconds, and then process one single letter and then sleep for
01:22:03.248 --> 01:22:06.174 five seconds. And that's fantastic for scenarios where
01:22:06.174 --> 01:22:09.314 inside your loop you're doing some work, you're returning,
01:22:09.314 --> 01:22:12.559 you're doing some processing on the data that's taking time.
01:22:13.240 --> 01:22:16.785 You can instead of front loading all the processing of all of the
01:22:16.785 --> 01:22:19.901 loading of the data like you know, let's say this is like
01:22:19.901 --> 01:22:22.962 load data from something, something, whatever. This will
01:22:22.962 --> 01:22:26.293 only load and process one single value and then all your post
01:22:26.293 --> 01:22:29.623 processing can happen on that one value and it doesn't go and
01:22:29.623 --> 01:22:33.008 load another one until it needs to until the next time this is
01:22:33.008 --> 01:22:36.446 done and asks OK, what's next? What's my next letter? What's my
01:22:36.446 --> 01:22:38.380 next account model, whatever it is.
01:22:39.440 --> 01:22:42.775 And another one is like instead of using the the the standard
01:22:42.775 --> 01:22:46.165 for each there having our like a for each async function where
01:22:46.165 --> 01:22:49.340 you await and it turns it into partitions and stuff. It'll
01:22:49.340 --> 01:22:52.837 create partitions on the fly as it gets more data back from your
01:22:52.837 --> 01:22:56.335 get letters yielded through this thing. And to have all of those
01:22:56.335 --> 01:22:59.294 running in our first and first out so you could have 8
01:22:59.294 --> 01:23:02.791 characters processing in a time. Now that will throw off like if
01:23:02.791 --> 01:23:06.289 you actually want your output to always be in the order that the
01:23:06.289 --> 01:23:09.410 characters came out versus. I only care about that I got.
01:23:09.480 --> 01:23:12.424 The letters not ordered. They came in. Um, that's where that
01:23:12.424 --> 01:23:13.100 would come in.
01:23:14.160 --> 01:23:17.060 And in that you can then cut your processing time down
01:23:17.060 --> 01:23:20.489 significantly at the expense of CPU and possibly a little bit of
01:23:20.489 --> 01:23:23.601 memory. Because if you're doing it all at once, instead of
01:23:23.601 --> 01:23:26.660 allocating and deallocating, allocating and deallocating,
01:23:26.660 --> 01:23:29.719 you're probably having to allocate all of this at once or
01:23:29.719 --> 01:23:32.673 at once. It's in the same moment, I should say. Kind of
01:23:32.673 --> 01:23:32.990 thing.
01:23:33.690 --> 01:23:35.869 So there's there's a lot to play with here when you get to
01:23:35.869 --> 01:23:36.460 different stuff.
01:23:37.370 --> 01:23:40.738 Um, real quick. Uh, I know we're at time here, Tristan, you had
01:23:40.738 --> 01:23:43.896 your hand up and then I saw Jeremy Unmuted very quickly. So
01:23:43.896 --> 01:23:45.370 the Tristan, Jeremy call it.
01:23:47.630 --> 01:23:51.693 I think James touched on both of them because I was like, hey, I
01:23:51.693 --> 01:23:55.756 I should bring up the James said it. So that's OK. But yeah, no,
01:23:55.756 --> 01:23:59.320 this is cool. Cool to see us looking at stuff like this.
01:24:00.030 --> 01:24:03.017 Um, we should do this more often. This is all I'm going to
01:24:03.017 --> 01:24:03.220 say.
01:24:04.080 --> 01:24:06.904 Yeah, very super interesting. This is fantastic stuff.
01:24:06.904 --> 01:24:10.138 Unfortunately in this example it's. I don't think I have quite
01:24:10.138 --> 01:24:13.424 enough going on. Like chars are so small in memory that the the
01:24:13.424 --> 01:24:16.505 default size of list is probably large enough that it's not
01:24:16.505 --> 01:24:19.483 reallocating. So they're basically with an order that was
01:24:19.483 --> 01:24:22.307 another pieces. Instead of declaring a list, you could
01:24:22.307 --> 01:24:25.491 declare an array and give it the size of the input. Yeah, and
01:24:25.491 --> 01:24:28.520 that would be the size of the input in the list too. So it
01:24:28.520 --> 01:24:31.704 just allocates to that size and initially and then real quick
01:24:31.704 --> 01:24:34.219 you don't have to do this just in for each loop.
01:24:34.290 --> 01:24:37.413 Right, you could just set, you know, some value needs to be
01:24:37.413 --> 01:24:40.381 basically like I need to go get a full list of all these
01:24:40.381 --> 01:24:43.505 accounts and I'm gonna go get them from different places or
01:24:43.505 --> 01:24:45.380 build them based off of some input.
01:24:46.730 --> 01:24:50.551 You could just send us there in any for loop. So it's gonna be
01:24:50.551 --> 01:24:54.554 for each or four. Yeah, OK yeah. So you can. Yeah, you can, yeah.
01:24:54.554 --> 01:24:58.314 Because it's basically you're returning a sequence of results
01:24:58.314 --> 01:25:01.832 and it's gonna basically execute the inner loop for every
01:25:01.832 --> 01:25:05.349 sequence that it needs to do. Sorry, I meant not there. I
01:25:05.349 --> 01:25:08.928 meant in your main, like you don't have to say for each of
01:25:08.928 --> 01:25:12.385 our interval and get letters. You could just say letters
01:25:12.385 --> 01:25:15.600 equals get letters. Yeah, yeah, get letters yielded.
01:25:17.280 --> 01:25:19.290 Cool, yeah. Yeah, and.
01:25:20.490 --> 01:25:24.588 So for those that are curious, I know we're at time, so everybody
01:25:24.588 --> 01:25:28.438 can feel free to drop if they need to. And if everybody drops
01:25:28.438 --> 01:25:32.288 then I'll just speak into the into the void, gotta say it, no
01:25:32.288 --> 01:25:36.075 matter the recordings that get you. So it's good anyway. Say
01:25:36.075 --> 01:25:39.677 it, I want it. So the way that this actually works is the
01:25:39.677 --> 01:25:43.216 ienumerable type is a type that basically just defines a
01:25:43.216 --> 01:25:47.128 sequence that can be enumerated and whenever you call for each
01:25:47.128 --> 01:25:50.420 on any sort of ienumerable what it's actually doing.
01:25:50.490 --> 01:25:54.102 Because it kind of breaks down into code that vaguely looks
01:25:54.102 --> 01:25:56.450 like var and numerator equals letters.
01:25:57.550 --> 01:26:01.402 Get enumerator, I think is what it's called. Yeah. And then and
01:26:01.402 --> 01:26:05.195 then it's going to effectively do something along the lines of
01:26:05.195 --> 01:26:09.048 while enumerator dot move next or something like that. And then
01:26:09.048 --> 01:26:12.841 inside of here there's like var current, equal, and I guess to
01:26:12.841 --> 01:26:16.754 keep the terminology the same as what's up there. Enumerator dot
01:26:16.754 --> 01:26:20.366 current, I think something like that. Yeah, yeah. Iterator.
01:26:20.366 --> 01:26:23.797 Yeah. And so this is what's actually happening. And what
01:26:23.797 --> 01:26:27.048 happens in here is what's actually happening when you
01:26:27.048 --> 01:26:27.409 yield.
01:26:27.490 --> 01:26:31.077 Turn is this enumerator calling move. Next is what actually
01:26:31.077 --> 01:26:35.024 executes this piece of code down here and set in yield. Return is
01:26:35.024 --> 01:26:38.970 basically like setting what this current value is and then inside
01:26:38.970 --> 01:26:42.498 of here now interval equals current. Now you have whatever
01:26:42.498 --> 01:26:46.026 this returned, so you can actually, if you wanted to break
01:26:46.026 --> 01:26:49.793 down what's happening in this right here, it would effectively
01:26:49.793 --> 01:26:50.690 be the same as.
01:26:53.540 --> 01:26:55.630 Doing if you kind of collapse all of this down.
01:26:56.550 --> 01:26:59.316 Yield return letter and then in the foreach it would be kind of
01:26:59.316 --> 01:27:01.650 like as if that console dot writeline was right here.
01:27:02.780 --> 01:27:06.112 Instead of enumerating twice, meaning that it's enumerating
01:27:06.112 --> 01:27:09.112 the outer group and then enumerating the inner group,
01:27:09.112 --> 01:27:12.555 you're getting one enumerator that's getting populated by the
01:27:12.555 --> 01:27:15.944 inner part. Yeah, and that's the other nice thing, the other
01:27:15.944 --> 01:27:19.276 performance thing. And again, this really comes in at scale
01:27:19.276 --> 01:27:22.665 and that's why we use yield return on the product indexer is
01:27:22.665 --> 01:27:25.553 because we have clients that have thousands, 10s of
01:27:25.553 --> 01:27:28.886 thousands, even hundreds of thousands of products. And this
01:27:28.886 --> 01:27:32.163 has to be brutally fast. And honestly, surprisingly, it is
01:27:32.163 --> 01:27:32.830 pretty fast.
01:27:32.900 --> 01:27:35.618 The index I wanna say 6000 products on Rydell last night
01:27:35.618 --> 01:27:38.718 and was shocked that it was done in like 2 seconds. Two or three
01:27:38.718 --> 01:27:39.100 seconds.
01:27:39.890 --> 01:27:43.587 Um, and uh. And so on. Places where you're working with a lot
01:27:43.587 --> 01:27:47.285 of records, with a lot of data you'll return is a really nice
01:27:47.285 --> 01:27:50.804 performance tool. Now again, it's concise, so I've spent a
01:27:50.804 --> 01:27:54.442 lot of time saying this thing is really cool, you should use
01:27:54.442 --> 01:27:58.021 this. Again, concise doesn't mean readable, so in scenarios
01:27:58.021 --> 01:28:01.659 where the performance difference is negligible or not really
01:28:01.659 --> 01:28:05.476 measurable at all, it's margin of error. Don't make code harder
01:28:05.476 --> 01:28:08.876 to read for the sake of using some kind of cool feature,
01:28:08.876 --> 01:28:09.950 right? Just write.
01:28:10.020 --> 01:28:13.916 Know that somebody can read like in the example on this link pad.
01:28:13.916 --> 01:28:17.458 You could probably just return a list of list of string and
01:28:17.458 --> 01:28:20.823 you're fine. It's not, it's not. It's not gonna kill the
01:28:20.823 --> 01:28:24.365 performance of the site to return this thing with something
01:28:24.365 --> 01:28:28.143 short like this. But same thing with building a string builder.
01:28:28.143 --> 01:28:31.744 If you're only built like even if you're writing a paragraph
01:28:31.744 --> 01:28:35.227 using string builder over concatenation. Or yeah like it's
01:28:35.227 --> 01:28:39.064 not interpolation interpolation. It's not like you're not saving
01:28:39.064 --> 01:28:39.950 that much time.
01:28:40.030 --> 01:28:43.726 We're just trying to look cool. Yeah, exactly. That's, that's a
01:28:43.726 --> 01:28:47.537 very. But if you look at the two string builders that are on like
01:28:47.537 --> 01:28:51.176 the the target grouping key, we are doing string builder there
01:28:51.176 --> 01:28:54.814 with an allocated setup. Because of the specific issue of that
01:28:54.814 --> 01:28:58.395 part has to be super performant for checkout to not go crazy,
01:28:58.395 --> 01:29:02.033 take long for the analyzer and for the checkout piece, because
01:29:02.033 --> 01:29:05.152 if you have 50 orders going in, that thing was taking
01:29:05.152 --> 01:29:08.790 significantly longer to process all of those things out of the
01:29:08.790 --> 01:29:09.830 database and into.
01:29:10.130 --> 01:29:13.076 Katie just strings for comparisons, so by using string
01:29:13.076 --> 01:29:16.344 builders there and doing that stuff, we were able to improve
01:29:16.344 --> 01:29:19.665 the performance time on it. And that was a place where it's a
01:29:19.665 --> 01:29:22.612 little less readable. But I tried to make it formatted
01:29:22.612 --> 01:29:25.933 inside the screen so that you could still at least reasonably
01:29:25.933 --> 01:29:29.094 read it and understand what was going on, but require less
01:29:29.094 --> 01:29:32.630 performance cost on the CPU than a string interpolation would do.
01:29:33.430 --> 01:29:36.926 Sure. Uh, real quick before before we leave, Jeremy, I saw
01:29:36.926 --> 01:29:40.422 you unmute a little while ago and mute back again what you
01:29:40.422 --> 01:29:40.660 got?
01:29:43.560 --> 01:29:44.270 Uh.
01:29:47.330 --> 01:29:51.713 Yeah, it was. It was weird. So yeah, I misunderstood this. I
01:29:51.713 --> 01:29:55.954 maybe thought it worked more like a recursive call, but it
01:29:55.954 --> 01:29:58.110 doesn't sound like to me like.
01:29:59.500 --> 01:30:03.189 So you can slap me down if I'm wrong here, but this actually
01:30:03.189 --> 01:30:06.819 executes any faster. You just started getting return values
01:30:06.819 --> 01:30:10.448 faster. So instead of processing 1,000,000 results and then
01:30:10.448 --> 01:30:14.078 returning, you start getting results back immediately while
01:30:14.078 --> 01:30:14.320 the.
01:30:15.000 --> 01:30:18.343 While it keeps going to the end of, there's there's two
01:30:18.343 --> 01:30:22.104 significant advantages here and one of them, one of them being
01:30:22.104 --> 01:30:25.686 memory, memory characteristics and performance. But I can't
01:30:25.686 --> 01:30:29.507 really illustrate that with like this string example. It's just
01:30:29.507 --> 01:30:32.611 not enough data to show that there is a performance
01:30:32.611 --> 01:30:35.715 difference. It's hard to measure, but the other big
01:30:35.715 --> 01:30:39.118 advantage that is easier to demonstrate or to explain is
01:30:39.118 --> 01:30:42.999 with the lists approach. I have a four each here that builds the
01:30:42.999 --> 01:30:44.850 list and then I have another 4.
01:30:44.920 --> 01:30:48.681 Each year that iterates over it. That's two loops over this
01:30:48.681 --> 01:30:52.380 effectively the same data. Um. However, with yield return.
01:30:53.180 --> 01:30:57.533 It's one single for each and in the code it looks like 2, but
01:30:57.533 --> 01:31:01.746 what's actually happening is this 4H is just describing how
01:31:01.746 --> 01:31:02.800 the enumerator.
01:31:03.430 --> 01:31:05.310 Goes and gets the next value.
01:31:06.190 --> 01:31:09.223 This is the force that's actually iterating. So it's
01:31:09.223 --> 01:31:12.829 effectively for each, grab this, run this code. For each, grab
01:31:12.829 --> 01:31:16.435 this, run this code instead of for each on every single thing,
01:31:16.435 --> 01:31:20.098 build up the result, return, and then for each on everything in
01:31:20.098 --> 01:31:23.189 the result. Now that has very significant performance
01:31:23.189 --> 01:31:26.910 implications if constructing the value that's in your array is a
01:31:26.910 --> 01:31:30.401 time consuming process, because obviously in this case we're
01:31:30.401 --> 01:31:33.778 just returning a letter from a string, it doesn't. There's
01:31:33.778 --> 01:31:36.239 hardly any performance implication at all.
01:31:36.630 --> 01:31:40.873 Um, but if this was going and making a database call or doing
01:31:40.873 --> 01:31:45.049 something else instead of front loading all of that time and
01:31:45.049 --> 01:31:49.293 then post processing it all a second time, you can reduce the
01:31:49.293 --> 01:31:53.605 performance and have only one loop over that initial data set.
01:31:53.605 --> 01:31:58.123 OK, Tim described it in the chat in a way that makes sense to me.
01:31:58.123 --> 01:32:00.039 So the other is squared and.
01:32:00.920 --> 01:32:05.294 As we're doing yield return, it's ON. So I think it would be
01:32:05.294 --> 01:32:09.740 like, I think it would be O of O of two N for the 44. He says
01:32:09.740 --> 01:32:14.257 it's not a for each of a four each, which would be n ^2. Yeah.
01:32:14.257 --> 01:32:18.846 So it's doing it twice. Yeah, OK yeah, yeah. And for those that
01:32:18.846 --> 01:32:21.500 aren't familiar with big O notation.
01:32:22.450 --> 01:32:23.660 Don't bother, it's.
01:32:26.070 --> 01:32:29.563 And I don't, I don't know what the exact thing is, but there's
01:32:29.563 --> 01:32:32.612 also it also isn't just a straight like 2 inverses and
01:32:32.612 --> 01:32:35.773 it's like an in plus one or something because there is a
01:32:35.773 --> 01:32:39.266 tiny cost to it, but that cost is nowhere near the doubling of
01:32:39.266 --> 01:32:42.371 the inn right on it. And so that's where it also scales
01:32:42.371 --> 01:32:45.864 correctly, where if you're doing a minute value amount it's so
01:32:45.864 --> 01:32:49.080 negligible it's not even funny. But on a grander scale it
01:32:49.080 --> 01:32:52.295 absolutely pays for itself in that cost, like the cost of
01:32:52.295 --> 01:32:55.290 having async versus non async on very small sections.
01:32:55.380 --> 01:32:58.241 Non ace or trying to use async on it actually costs more
01:32:58.241 --> 01:33:01.454 because of the async overhead in generating those tests and you
01:33:01.454 --> 01:33:04.165 working them on the task manager. But when you get to
01:33:04.165 --> 01:33:07.177 scale it drastically pays for itself. If you wanted to just
01:33:07.177 --> 01:33:10.039 poke around and have some fun with it, go ahead and pull
01:33:10.039 --> 01:33:13.352 yourself a Ridel database backup on your local and then do like a
01:33:13.352 --> 01:33:16.214 VAR list of products equals whatever and then instead of
01:33:16.214 --> 01:33:19.427 yield returning here, just add it to the list, return the whole
01:33:19.427 --> 01:33:22.590 list and then go ahead and run your index and tell me how long
01:33:22.590 --> 01:33:24.999 it takes. I'm actually curious. Someone should.
01:33:25.390 --> 01:33:28.425 Need to do that. I might do that. Use my laptop for a
01:33:28.425 --> 01:33:31.854 heater. Figure out yeah, and figure out how. Just tell us on
01:33:31.854 --> 01:33:35.508 Monday. So do that tonight. Then tell us on Monday. I'll get rid
01:33:35.508 --> 01:33:39.161 of the text cancellation token. You're you're not gonna need it.
01:33:39.161 --> 01:33:42.084 Yeah. Tell us Monday. What fruit, what single digit
01:33:42.084 --> 01:33:43.490 percentage is that? Yeah.
01:33:45.230 --> 01:33:47.992 OK, cool. Well, if I, if you guys wanna keep going, great.
01:33:47.992 --> 01:33:50.474 Uh, I'm gonna drop. But I appreciate. Appreciate it.
01:33:50.474 --> 01:33:53.003 Brandon and James kind of walking us through that and
01:33:53.003 --> 01:33:54.970 answering a lot of questions was awesome.
01:33:55.880 --> 01:33:59.616 Yeah, those I yeah James if you can show for each async and I
01:33:59.616 --> 01:34:03.412 think next week that would be great. And then also how to like
01:34:03.412 --> 01:34:07.148 debug through a foreach async and cabana stuff that we talked
01:34:07.148 --> 01:34:10.764 about earlier that and the cabana stuff. Yeah. And then for
01:34:10.764 --> 01:34:14.439 those of you especially the new guys, so Nick and Jeremy and
01:34:14.439 --> 01:34:18.055 then also Tim, you're you've probably got some stuff you've
01:34:18.055 --> 01:34:21.791 seen and and I know team started to really get into some some
01:34:21.791 --> 01:34:25.166 some client stuff on not pay hubs. So you may have some
01:34:25.166 --> 01:34:26.069 things as well.
01:34:26.450 --> 01:34:29.319 If you guys think of anything like we've been seeing a lot of
01:34:29.319 --> 01:34:32.373 stuff with elastic, which is why you've covered it two weeks in a
01:34:32.373 --> 01:34:35.197 row now. Um, if there's other things you guys are working on
01:34:35.197 --> 01:34:37.881 that you're like, man, this doesn't make any sense or the
01:34:37.881 --> 01:34:40.843 documentation is outdated or we don't have any, whatever it is,
01:34:40.843 --> 01:34:43.527 please tell me or Eric or Brandon or somebody and we will
01:34:43.527 --> 01:34:45.240 try to do one of these kind of like.
01:34:46.400 --> 01:34:49.417 Engaging group training type things like this and we can try
01:34:49.417 --> 01:34:52.483 to everything, everywhere, all at once. Yeah, we can. We can,
01:34:52.483 --> 01:34:55.253 yeah, yeah. Or Caroline. Yeah. Caroline would also be a
01:34:55.253 --> 01:34:56.489 fantastic person to tell.
01:34:58.000 --> 01:35:01.735 And and don't worry about being like ohh, maybe I should feel
01:35:01.735 --> 01:35:05.229 like I should know this. It doesn't matter, just bring it
01:35:05.229 --> 01:35:08.964 up. Nobody. Like we make the assumption with people coming in
01:35:08.964 --> 01:35:12.218 that that you won't know anything. Not literally, not
01:35:12.218 --> 01:35:15.712 that you know nothing, but if it's separated, we're gonna
01:35:15.712 --> 01:35:19.387 assume that like, yeah, we're gonna have to show you this or
01:35:19.387 --> 01:35:23.002 document it somehow. And if you're telling us that we don't
01:35:23.002 --> 01:35:26.556 have documentation or the documentation is outdated or not
01:35:26.556 --> 01:35:27.340 useful, then.
01:35:28.010 --> 01:35:31.178 Or are you supposed to know it? That's on us. We need to fix
01:35:31.178 --> 01:35:34.398 that problem, and we can't fix it if no one's telling us what
01:35:34.398 --> 01:35:36.320 sucks. So please tell us what sucks.
01:35:37.870 --> 01:35:38.400 So.
01:35:39.150 --> 01:35:42.900 My final question, if I want to just read the code, this is the
01:35:42.900 --> 01:35:46.299 project in the solution that I should just like. Yep, for
01:35:46.299 --> 01:35:49.404 elastic search, all of the clarity dot ecommerce dot
01:35:49.404 --> 01:35:52.862 providers that searching. Yep just poke around in here. If
01:35:52.862 --> 01:35:56.495 it's on your local and you wanna just mess around with it and
01:35:56.495 --> 01:36:00.245 free time, by all means make any changes you want, poke around,
01:36:00.245 --> 01:36:04.054 see what, see what your changes, what impact they have and stuff
01:36:04.054 --> 01:36:07.570 like that. And if you if you come across a better way to do
01:36:07.570 --> 01:36:08.860 something let us know.
01:36:09.180 --> 01:36:12.585 We'll be glad to. We'll be glad to to dig into that with you and
01:36:12.585 --> 01:36:15.832 see if it's something we can get into core and and that's how
01:36:15.832 --> 01:36:19.080 that's how improvements happen, yeah. You wanna be an elastic
01:36:19.080 --> 01:36:21.700 search subject matter expert. No one will say no.
01:36:23.170 --> 01:36:26.704 In fact, I say please. Yeah, real quick to you. Interesting.
01:36:26.704 --> 01:36:30.528 I've never touched it, but yeah, yeah, fair enough. I'm done. Oh,
01:36:30.528 --> 01:36:33.946 I was gonna say I'll ask my thing after people drop, but I
01:36:33.946 --> 01:36:37.365 have something I want some feedback on. If anyone wants to
01:36:37.365 --> 01:36:40.841 stick around for like 30 seconds or a minute, yeah, you got
01:36:40.841 --> 01:36:44.259 you'll end up being like 15 minutes with Brendan and James
01:36:44.259 --> 01:36:47.620 and everyone else talking because they because we like to
01:36:47.620 --> 01:36:47.910 talk.
01:36:50.000 --> 01:36:53.268 Yeah, I was just doing some stuff for bid that was like
01:36:53.268 --> 01:36:55.720 there's some parallelism. I was doing so.
01:36:56.820 --> 01:37:00.170 I didn't want to bore anyone with it needs to leave.
01:37:04.410 --> 01:37:05.260 Let it ride, man.
01:37:07.780 --> 01:37:08.630 Can't go for it.
01:37:10.520 --> 01:37:12.460 I guess I was. What I was basically doing is giving
01:37:12.460 --> 01:37:13.580 everyone permission to escape.
01:37:14.290 --> 01:37:18.220 But um, if anybody wanna drop it is it is past the time this
01:37:18.220 --> 01:37:22.280 meeting normally ends. So get out or stick around if you want.
01:37:23.770 --> 01:37:27.147 OK, I'm only gonna do this quickly and it might be a little
01:37:27.147 --> 01:37:27.710 jank, but.
01:37:28.640 --> 01:37:33.587 It's a lot faster, so one thing we have to do unfortunately
01:37:33.587 --> 01:37:37.380 forbid is download a bunch of product images.
01:37:39.140 --> 01:37:42.283 And the reason I say unfortunately is because there's
01:37:42.283 --> 01:37:46.009 a lot of them, right? So like if you have a spreadsheet with, I
01:37:46.009 --> 01:37:48.280 don't know like this is a smaller one.
01:37:49.490 --> 01:37:53.170
of those get successfully parsed
01:37:53.170 --> 01:37:56.448 and I say only 100 because there's invalid UPC's or ones
01:37:56.448 --> 01:37:59.553 that are blank or whatever. Anyways you have like 100
01:37:59.553 --> 01:38:03.061 products, each of them might have like 5 images and then you
01:38:03.061 --> 01:38:06.569 have to download them all so that we can like store them and
01:38:06.569 --> 01:38:08.180 have them like on set right?
01:38:09.830 --> 01:38:13.955 It's a little, it's a little slow and by a little bit I mean
01:38:13.955 --> 01:38:18.081 like 3-5 minutes for like that many images like a 300 or so,
01:38:18.081 --> 01:38:22.139 I'd say like 100 images per minute basically if you're just
01:38:22.139 --> 01:38:26.264 like waiting for one image to come back at a time. So what I
01:38:26.264 --> 01:38:30.660 was trying to figure out is how can I make it happen in parallel
01:38:30.660 --> 01:38:31.539 without like.
01:38:32.980 --> 01:38:36.700 Opening a black hole, because I know that's kind of how like
01:38:36.700 --> 01:38:40.481 networking works when you when you try to do something crazy,
01:38:40.481 --> 01:38:44.079 crazy stuff happens. So you don't want to over parallelize
01:38:44.079 --> 01:38:47.616 it and yeah and have the you want to create a DDoS attack
01:38:47.616 --> 01:38:51.397 here basically. Exactly. Yeah. DDoS yourself. So that's why I
01:38:51.397 --> 01:38:54.691 have the 4H async with partitioning. Yeah, yeah. So I
01:38:54.691 --> 01:38:55.850 don't know anybody.
01:38:57.400 --> 01:39:00.594 I'll just put that out there so I could totally be over. I could
01:39:00.594 --> 01:39:03.690 be making this way more harder than it needs to be, but when I
01:39:03.690 --> 01:39:06.884 after some limited research, the first thing I'm I'm reusing the
01:39:06.884 --> 01:39:09.587 same HTTP client because apparently that's. And I know
01:39:09.587 --> 01:39:12.683 I've read this in the past and I totally forgot it, but that's
01:39:12.683 --> 01:39:15.729 apparently a good thing to do if you can is use the same one.
01:39:16.910 --> 01:39:20.063 Their performance uh, some performance implications and
01:39:20.063 --> 01:39:23.779 knowing them up and and all that there's it's nontrivial. So yeah
01:39:23.779 --> 01:39:27.270 yeah and obviously if you have if you're doing it every like.
01:39:29.040 --> 01:39:32.447 Under a second, that's for like, you know, especially in
01:39:32.447 --> 01:39:36.213 parallel. So that's why I did that. And the second thing I did
01:39:36.213 --> 01:39:39.860 was I am going to go in, I'm gonna go outside and the inside
01:39:39.860 --> 01:39:43.506 them back out. Second thing is I want to have like a list of
01:39:43.506 --> 01:39:47.332 tasks, so I could just run them all and then wait until they're
01:39:47.332 --> 01:39:48.049 done, right?
01:39:49.750 --> 01:39:53.736 And then to not DDoS myself, I was trying to figure that out. I
01:39:53.736 --> 01:39:57.598 didn't do it, so I knew it was going to happen. So I did this
01:39:57.598 --> 01:40:01.523 ahead of time. But there's this like sophomore slim class, and
01:40:01.523 --> 01:40:05.447 it basically lets you limit how many of these are running at a
01:40:05.447 --> 01:40:09.247 time. So I just set it to four because I found that to be. I
01:40:09.247 --> 01:40:13.171 mean, I didn't push the limits, let's put it that way. I think
01:40:13.171 --> 01:40:17.220 you can have like 10 connections before you start causing issues
01:40:17.220 --> 01:40:18.840 and stuff starts dropping.
01:40:18.960 --> 01:40:22.398 So I thought four was a good number, so I just kind of
01:40:22.398 --> 01:40:26.336 guessed 4. Um. So what this does is it'll it'll basically like
01:40:26.336 --> 01:40:29.900 download, it'll try to download at least four at a time.
01:40:30.590 --> 01:40:33.691 Um, until they're all done and I got it down from like 3 to
01:40:33.691 --> 01:40:36.690 however many minutes on like 3 to 5 minutes down to like.
01:40:37.460 --> 01:40:41.793 30 to 35 seconds and it it's kind of comedic to see like the
01:40:41.793 --> 01:40:45.771 images folder just go like download them all. But I was
01:40:45.771 --> 01:40:49.962 wondering is there is there anything I'm doing here that's
01:40:49.962 --> 01:40:54.366 like Oh my God, red flag. Like that's never do that. It looks
01:40:54.366 --> 01:40:58.984 OK to me. You're you're limiting the number of asynchronous ones
01:40:58.984 --> 01:41:00.759 that can run at any time.
01:41:02.020 --> 01:41:06.170 And I mean, as long as it's functional and you're not.
01:41:07.060 --> 01:41:10.328 You know, absolutely abusing the the server you're downloading
01:41:10.328 --> 01:41:13.545 the images from raising. I think it's probably fine. Now like
01:41:13.545 --> 01:41:16.813 James was saying, I think you may be able to simplify a little
01:41:16.813 --> 01:41:20.082 bit with the four each. We have an extension method built into
01:41:20.082 --> 01:41:23.247 self called for each async and you can pass in the degree of
01:41:23.247 --> 01:41:23.870 parallelism.
01:41:25.410 --> 01:41:29.383 The number of, uh, asynchronous tasks that can run, and that's
01:41:29.383 --> 01:41:33.230 not. Google the logic to yeah, and then the logic to run for
01:41:33.230 --> 01:41:35.060 each of those. So that might.
01:41:35.830 --> 01:41:39.015 Uh, have made this a little bit easier. Um, and that's something
01:41:39.015 --> 01:41:42.151 that I don't actually. It might be documented somewhere, but if
01:41:42.151 --> 01:41:45.092 it's not, we totally should document that so that you don't
01:41:45.092 --> 01:41:47.983 have to go through. We're planning on going over that next
01:41:47.983 --> 01:41:48.670 Friday, right?
01:41:50.370 --> 01:41:53.342 Yeah, sure. Yeah. Ohh I I'm sorry. I for some reason I had
01:41:53.342 --> 01:41:56.618 in my head when I thought no, I you know, I think you're totally
01:41:56.618 --> 01:41:58.180 right. I think I've yeah, yeah.
01:41:59.940 --> 01:42:03.293 That would be cool. This is it's funny because in that brand
01:42:03.293 --> 01:42:06.371 bidding project, they use this same semaphore slim as a
01:42:06.371 --> 01:42:09.669 throttler just for the same reason, right? How many threads
01:42:09.669 --> 01:42:12.638 we want to allocate for asynchronous calls, right? Or
01:42:12.638 --> 01:42:15.716 like how many? There's more than one way to skin a cat.
01:42:15.716 --> 01:42:19.070 Basically. Is is. These are just different ways of doing it.
01:42:20.330 --> 01:42:23.951 I also brought in the for each async into that project that
01:42:23.951 --> 01:42:27.693 we're talking about. Christian, I can show you where it's at.
01:42:27.693 --> 01:42:30.530 You just bring in, yeah, one method, one file.
01:42:32.040 --> 01:42:32.290 Cool.
01:42:35.490 --> 01:42:38.653 Yeah, I was just making sure because I don't do a lot of
01:42:38.653 --> 01:42:42.316 parallel stuff because generally speaking it's just not necessary
01:42:42.316 --> 01:42:45.701 and sometimes adds unnecessary complexity. But for something
01:42:45.701 --> 01:42:48.920 that might take 5 minutes and the reason. OK so the whole
01:42:48.920 --> 01:42:52.416 reason I even went through all the trouble of like how do I do
01:42:52.416 --> 01:42:55.635 this without breaking the universe was because the Amazon
01:42:55.635 --> 01:42:59.076 one this is like 300 this. Sorry this is 150 products it only
01:42:59.076 --> 01:43:02.573 could resolve and by it's not our fault like I tried my best I
01:43:02.573 --> 01:43:05.570 was like trying to figure it out but wasn't worth it.
01:43:05.640 --> 01:43:08.559 One could only like pull out like only like 100 were even
01:43:08.559 --> 01:43:11.578 valid. That could come that could come out of the sheet and
01:43:11.578 --> 01:43:14.850 then be queried by the API. But that this Amazon one dude, Oh my
01:43:14.850 --> 01:43:17.920 God, I don't know how long it would take to go through like.
01:43:18.830 --> 01:43:22.375 2023 Hundred one by one and get all the images one by one. I was
01:43:22.375 --> 01:43:25.920 like dude there's no way I have to figure this out because like.
01:43:26.540 --> 01:43:30.473 Someone's gonna be like well your code suck. I feel like
01:43:30.473 --> 01:43:35.028 you're cause it's synchronous so I guess it's it's async but it's
01:43:35.028 --> 01:43:38.410 you know, it's it's a sequential I guess rather.
01:43:39.250 --> 01:43:43.002 So I just wanted to see if there's any glaring like, Oh my
01:43:43.002 --> 01:43:46.882 gods. So yeah and yeah, we're let's take a look at that. I'm
01:43:46.882 --> 01:43:48.600 curious for the 4E chasing.
01:43:49.450 --> 01:43:51.819 If you did you say it was implemented somewhere? Ohh I see
01:43:51.819 --> 01:43:54.430 the checkout. It should be that one very all the way at the top.
01:43:55.250 --> 01:43:55.700 Yeah.
01:43:57.380 --> 01:43:59.882 Uh, in your search there you have four jsyk T this? Yeah,
01:43:59.882 --> 01:44:00.400 right there.
01:44:01.470 --> 01:44:07.212 In the e-commerce Court, yeah, I see. So that's like degrees of
01:44:07.212 --> 01:44:08.110 partition.
01:44:09.840 --> 01:44:13.548 Yep. So yeah you basically yeah and it's an extension method so
01:44:13.548 --> 01:44:16.908 you just you can call it directly on any I enumerable. So
01:44:16.908 --> 01:44:20.616 you can do like image names, dot foreach async and then in that
01:44:20.616 --> 01:44:23.340 loop have it set up how to download the image.
01:44:24.170 --> 01:44:25.640 Yeah, that's that's awesome.
01:44:27.440 --> 01:44:27.710 Yeah.
01:44:28.560 --> 01:44:32.755 I will definitely be abusing that at some point in the
01:44:32.755 --> 01:44:33.290 future.
01:44:36.430 --> 01:44:41.426 Awesome. Well, that's all I had to show, really. So that's it.
01:44:41.426 --> 01:44:42.140 But yeah.
01:44:44.100 --> 01:44:47.514 Sweet. Yeah, it looks good to me and hopefully everybody got
01:44:47.514 --> 01:44:48.970 something out of that, so.
01:44:50.810 --> 01:44:53.656 Alright, well, if everybody's good to go now, then I think
01:44:53.656 --> 01:44:56.551 we're probably good to drop in and y'all can have hopefully
01:44:56.551 --> 01:44:59.059 have a good weekend and I'll see you all next week.
01:45:00.240 --> 01:45:04.410 Happy Friday. Yeah. Thanks a lot guys. Have a good weekend.
01:45:05.880 --> 01:45:06.250