00:00:04.098 --> 00:00:08.865 It's kind of hoping that he would be in this one as well so
00:00:08.865 --> 00:00:12.758 that he could fill in whatever gaps that I have.
00:00:13.288 --> 00:00:16.498 As I say things and whatnot. Umm.
00:00:23.298 --> 00:00:28.033 We'll wait for wait for Visual Studio to finish remembering how
00:00:28.033 --> 00:00:29.068 websites work.
00:00:29.138 --> 00:00:32.598 I mean applications work and then open.
00:00:34.688 --> 00:00:38.438 You know, whatever I don't.
00:00:39.108 --> 00:00:41.368 I didn't open it cause I need to use it or anything.
00:00:44.658 --> 00:00:44.948 Uh.
00:00:44.958 --> 00:00:45.238 Loads.
00:00:45.248 --> 00:00:46.628 I'm gonna grab myself a drink.
00:01:11.098 --> 00:01:12.558 I did mention it a minute ago.
00:01:13.008 --> 00:01:13.488 Uh.
00:01:13.788 --> 00:01:17.058 During the the kudos phase of Friday training.
00:01:18.048 --> 00:01:22.928 But I'm still uh, I'm still coming off of allergies or and.
00:01:22.938 --> 00:01:23.878 Or maybe a cold?
00:01:23.888 --> 00:01:24.488 I don't know.
00:01:24.498 --> 00:01:28.708 That's been, uh, assaulting me all week.
00:01:29.768 --> 00:01:37.646 So if my if my my voice is quiet, muffled, nasally, et
00:01:37.646 --> 00:01:38.648 cetera.
00:01:39.878 --> 00:01:41.568 Uh, that is why.
00:01:42.288 --> 00:01:46.885 And if you need me to repeat something, simply ask and I
00:01:46.885 --> 00:01:47.288 will.
00:01:48.138 --> 00:01:48.738 I will repeat.
00:01:52.198 --> 00:01:56.638 Alright, waiting on things to load.
00:01:58.408 --> 00:02:00.508 Ohh growth cashing, get it out.
00:02:02.818 --> 00:02:05.568 OK, so Elasticsearch.
00:02:05.578 --> 00:02:06.368 Why are we?
00:02:06.438 --> 00:02:07.488 Why are we use it?
00:02:07.798 --> 00:02:08.538 What it does?
00:02:09.788 --> 00:02:10.668 Uh, et cetera.
00:02:12.498 --> 00:02:12.748 Uh.
00:02:12.758 --> 00:02:14.808 So umm.
00:02:16.908 --> 00:02:19.988 David gave an analogy for this recently that I thought was
00:02:19.988 --> 00:02:22.128 pretty good, so I'll kind of steal that.
00:02:23.398 --> 00:02:27.467 Uh and then James added a little more on to it that I'll I'll
00:02:27.467 --> 00:02:30.158 include in this explanation as well, so.
00:02:33.668 --> 00:02:38.328 Let's let's imagine a library full of books, right?
00:02:39.538 --> 00:02:45.884 And that library is sorted nicely and and evenly by uh by
00:02:45.884 --> 00:02:52.121 the some unique identifier for each book, like the Dewey
00:02:52.121 --> 00:02:54.418 Decimal or something.
00:02:54.468 --> 00:02:56.814 Except that's a bad example, cause the Dewey Decimal system
00:02:56.814 --> 00:02:58.378 is more like elastic than the database.
00:02:58.748 --> 00:03:01.918 So let's imagine that each book is just given a number.
00:03:01.928 --> 00:03:05.924 When it reaches the library, one for the first book, 2 for the
00:03:05.924 --> 00:03:09.603 2nd book, Etcetera, and the Books are all laid out in the
00:03:09.603 --> 00:03:13.598 line like that in the shelf, one to Infinity perfectly sorted.
00:03:14.568 --> 00:03:17.866 This is great for anybody that knows the exact number of the
00:03:17.866 --> 00:03:21.002 thing they're looking for, because they can just walk the
00:03:21.002 --> 00:03:24.138 line until they get to that number and there's your book.
00:03:24.148 --> 00:03:28.118 Perfectly easy peasy, but this is not great when you want to.
00:03:29.928 --> 00:03:33.543 Find books by some name or you maybe you don't know exactly
00:03:33.543 --> 00:03:37.098 what the name is and you, but you kind of have an inkling.
00:03:37.108 --> 00:03:40.460 Or maybe you don't know how it's spelled and you you need to just
00:03:40.460 --> 00:03:43.660 punch in something that sounds about right and have it find it
00:03:43.660 --> 00:03:45.488 for you, phonetically or et cetera.
00:03:48.018 --> 00:03:49.768 That's, that's where Lastic comes in.
00:03:50.338 --> 00:03:55.567 The library where everything is sorted by the by some specific
00:03:55.567 --> 00:03:56.148 number.
00:03:56.348 --> 00:04:00.528 That's like the database lookups are very fast on one specific
00:04:00.528 --> 00:04:04.775 system on one specific way, and they're breakneck fast in those
00:04:04.775 --> 00:04:05.438 scenarios.
00:04:05.998 --> 00:04:09.363 But if you want to start looking for more complex things, you're
00:04:09.363 --> 00:04:10.088 gonna need to.
00:04:11.308 --> 00:04:14.694 You're gonna need to implement a lot of extra kind of grouping on
00:04:14.694 --> 00:04:16.028 things you're going to be.
00:04:16.128 --> 00:04:19.150 You're gonna, you know, keeping with the analogy, you're going
00:04:19.150 --> 00:04:19.438 to be.
00:04:19.488 --> 00:04:21.198 You're gonna be checking book titles.
00:04:21.208 --> 00:04:25.676 Every single book title to see if it lines up with your search
00:04:25.676 --> 00:04:27.378 and you know, et cetera.
00:04:27.388 --> 00:04:33.448 It becomes very slow, so enter an index which takes a snapshot
00:04:33.448 --> 00:04:39.604 of all of that information and compiles it and collects it into
00:04:39.604 --> 00:04:40.758 a structure.
00:04:40.768 --> 00:04:44.751 A data structure that is built for that exact type of
00:04:44.751 --> 00:04:45.488 searching.
00:04:46.628 --> 00:04:51.058 So in essence, what it's doing is it's just analyzing all of
00:04:51.058 --> 00:04:55.052 the names, all of the descriptions, all the other data
00:04:55.052 --> 00:04:59.118 that we feed into it and generating structures that can
00:04:59.118 --> 00:05:02.168 be searched and sorted extremely quickly.
00:05:04.238 --> 00:05:09.717 So in our library slash book analogy, that's sort of like
00:05:09.717 --> 00:05:14.629 categorizing the books into sections of fiction and
00:05:14.629 --> 00:05:15.668 nonfiction.
00:05:15.678 --> 00:05:19.920 And et cetera or maybe using the Dewey Decimal System which has
00:05:19.920 --> 00:05:23.699 certain prefixes for light categories and then inside of
00:05:23.699 --> 00:05:28.073 that numerically sorting by name or things like that, where now I
00:05:28.073 --> 00:05:32.315 know that if I want nonfiction, it's gonna be 930 dot something
00:05:32.315 --> 00:05:32.978 something.
00:05:32.988 --> 00:05:33.338 I don't know.
00:05:33.348 --> 00:05:34.248 I'm just making up the numbers.
00:05:34.258 --> 00:05:35.668 I don't actually know the Dewey Decimal system.
00:05:36.628 --> 00:05:40.282 Umm, but uh, and so I can go find 930 and then here's all my
00:05:40.282 --> 00:05:42.678 relevant things to that specific piece.
00:05:42.688 --> 00:05:46.187 And then I can delve further in and find what I'm looking for in
00:05:46.187 --> 00:05:49.148 that more more, you know, refined selection of things.
00:05:50.478 --> 00:05:53.514 Umm, so that's that's sort of the analogy of what
00:05:53.514 --> 00:05:56.428 Elasticsearch does for us in a very high level.
00:05:56.678 --> 00:06:00.301 It takes a snapshot of the data from the database and snapshot
00:06:00.301 --> 00:06:01.738 part is really important.
00:06:01.748 --> 00:06:06.009 I'll talk about that more in a minute, but it takes a snapshot
00:06:06.009 --> 00:06:09.998 of the database and then kind of decomposes it into a very
00:06:09.998 --> 00:06:14.259 efficient searchable structure that allows us to query against
00:06:14.259 --> 00:06:16.828 that data with more natural language.
00:06:16.898 --> 00:06:20.521 So the way somebody might type something that's kind of close
00:06:20.521 --> 00:06:23.968 to what they're looking for synonyms or a misspelling or a
00:06:23.968 --> 00:06:27.299 word that might be in the description of the product but
00:06:27.299 --> 00:06:28.058 not its name.
00:06:28.068 --> 00:06:30.989 All that kind of stuff and it knows how to take all that and
00:06:30.989 --> 00:06:32.138 pass it into this thing.
00:06:32.588 --> 00:06:36.949 The structure that it has, which is called the index and give you
00:06:36.949 --> 00:06:40.848 back good relevant results orders of magnitude faster than
00:06:40.848 --> 00:06:44.218 trying to do all that against a database yourself.
00:06:45.148 --> 00:06:49.374 So that's the that's the the concept of an index in a
00:06:49.374 --> 00:06:50.078 nutshell.
00:06:50.638 --> 00:06:55.618 Umm, any questions on that before I before I proceed to the
00:06:55.618 --> 00:06:56.448 next part?
00:07:02.778 --> 00:07:04.478 Sounds like no.
00:07:04.648 --> 00:07:09.124 Alright, so something I mentioned in there is that the
00:07:09.124 --> 00:07:11.728 index is a snapshot of the data.
00:07:12.468 --> 00:07:16.243 That's extremely important to keep in mind, because what that
00:07:16.243 --> 00:07:19.835 means is that the index only contains the information that
00:07:19.835 --> 00:07:23.488 was in the database at the time the index was last updated.
00:07:24.678 --> 00:07:28.338 So if you have a product that's in the index with the name.
00:07:29.288 --> 00:07:35.504 Uh, cool hat and then you rename it in the database to awesome
00:07:35.504 --> 00:07:35.898 cap.
00:07:36.008 --> 00:07:36.428 I don't know.
00:07:37.958 --> 00:07:39.488 Hey, if you go in search.
00:07:39.498 --> 00:07:40.118 Cool hat.
00:07:41.168 --> 00:07:42.658 You're still gonna pull back up?
00:07:42.708 --> 00:07:45.888 Awesome cap because it's indexed as cool hat.
00:07:45.898 --> 00:07:46.358 You didn't.
00:07:46.408 --> 00:07:49.192 If you didn't update your your index, that's what the index
00:07:49.192 --> 00:07:50.908 still thinks that product is called.
00:07:52.018 --> 00:07:55.786 So keeping in mind that it's a snapshot is really important
00:07:55.786 --> 00:07:59.240 because it's gonna help you understand why indexing is
00:07:59.240 --> 00:08:03.133 necessary at certain times and also why it's not necessary at
00:08:03.133 --> 00:08:06.650 certain times a lot of people think that if they change
00:08:06.650 --> 00:08:09.978 anything ever at all, they have to reindex products.
00:08:10.588 --> 00:08:13.644 No, you only need to reindex the products if you change something
00:08:13.644 --> 00:08:15.958 that would alter the results of a catalog search.
00:08:17.368 --> 00:08:20.838 Umm, so if you're attaching new images to a product, you don't
00:08:20.838 --> 00:08:24.307 need to index to see those in the storefront, because we don't
00:08:24.307 --> 00:08:25.298 search the images.
00:08:25.308 --> 00:08:29.892 We don't analyze the images or do anything special with that,
00:08:29.892 --> 00:08:31.888 so similar stuff like that.
00:08:31.898 --> 00:08:35.375 A lot of people are a lot of people make that mistake of like
00:08:35.375 --> 00:08:38.963 ohh I I did something something and I thought I needed to index
00:08:38.963 --> 00:08:42.495 the products and so if you're ever curious, just ask yourself,
00:08:42.495 --> 00:08:45.803 would changing what I'm doing here alter the products that
00:08:45.803 --> 00:08:48.158 come back from some query on the catalog?
00:08:48.168 --> 00:08:51.068 And if the answer is no, you probably don't have to index.
00:08:51.318 --> 00:08:54.045 That being said, indexing doesn't really take that long
00:08:54.045 --> 00:08:56.918 unless you're on a project with like a bajillion products.
00:08:57.018 --> 00:08:59.428 So most and it's not gonna hurt anything to do it.
00:09:00.298 --> 00:09:03.328 So you can always err on the side of doing it anyway.
00:09:03.338 --> 00:09:05.768 Just know that there is a pattern you can learn to save
00:09:05.768 --> 00:09:08.588 yourself some time and not have to do it when you don't need to.
00:09:09.428 --> 00:09:16.535 Umm so so in index is a fast queryable snapshot of the
00:09:16.535 --> 00:09:17.698 database.
00:09:17.708 --> 00:09:23.105 It's just a a a picture of the stuff in a way that can be
00:09:23.105 --> 00:09:26.268 searched very quickly and easily.
00:09:26.338 --> 00:09:30.232 So with the the basic introductory level stuff done,
00:09:30.232 --> 00:09:34.787 we can look at where all this actually happens in Sev umm, so
00:09:34.787 --> 00:09:39.343 the the primary place the the main place that you're actually
00:09:39.343 --> 00:09:43.384 gonna see all this stuff happening is in the searching
00:09:43.384 --> 00:09:48.086 provider specifically right now the only implementation we have
00:09:48.086 --> 00:09:49.408 is elastic search.
00:09:49.558 --> 00:09:51.868 Theoretically, we could have other implementations.
00:09:53.658 --> 00:09:56.908 It's never been, to my knowledge, has never been done.
00:09:56.918 --> 00:10:00.218 It's never been needed, but theoretically you could.
00:10:00.288 --> 00:10:06.031 But umm, but the primary, the primary index that we're talking
00:10:06.031 --> 00:10:10.680 about that that 99.9% of projects are using is the
00:10:10.680 --> 00:10:12.138 product section.
00:10:12.748 --> 00:10:18.558 So we're driving results in the product catalog, umm, so.
00:10:22.698 --> 00:10:26.418 Clients, when we start with this process, there's a few steps
00:10:26.418 --> 00:10:29.298 basically to populating and querying and index.
00:10:30.048 --> 00:10:33.480 So we can kind of start by talking through these steps in
00:10:33.480 --> 00:10:35.078 the order that they happen.
00:10:36.268 --> 00:10:40.828 The first step, and I'm going to admit like kind of more granular
00:10:40.828 --> 00:10:45.387 steps like actually creating the index 1st and etcetera, uh it it
00:10:45.387 --> 00:10:49.739 you know processes that elastic does and kind of focus more on
00:10:49.739 --> 00:10:53.538 the parts that we really author and control ourselves.
00:10:54.148 --> 00:10:59.058 The first step in the process is going to be informing elastic.
00:10:59.188 --> 00:11:03.183 The shape of the data that we're going to send it and how to
00:11:03.183 --> 00:11:07.440 analyze it, and that happens in the indexer, uh, so that's gonna
00:11:07.440 --> 00:11:09.798 be this big giant thing right here.
00:11:10.158 --> 00:11:14.836 Where in essence, we're telling elastic that when we give it a
00:11:14.836 --> 00:11:19.217 product indexable model, it's gonna have a property called
00:11:19.217 --> 00:11:19.588 name.
00:11:19.678 --> 00:11:22.468 And this is the analyzer name we wanna use for that.
00:11:22.478 --> 00:11:26.758 Same with custom key brand name et cetera type name query
00:11:26.758 --> 00:11:29.488 attributes which is a nested object.
00:11:29.498 --> 00:11:32.008 It has, uh, multiple things in it.
00:11:32.438 --> 00:11:33.068 Umm.
00:11:34.848 --> 00:11:37.638 And goes in and sets up those filterable attributes.
00:11:37.648 --> 00:11:41.085 Goes and sets up the roles, brands, categories, franchises,
00:11:41.085 --> 00:11:41.658 et cetera.
00:11:41.768 --> 00:11:46.003 So we're effectively this is pretty complicated looking and
00:11:46.003 --> 00:11:50.238 most of the time no one's really gonna need to change this.
00:11:50.828 --> 00:11:54.448 But umm, but this is kind of where.
00:11:57.778 --> 00:12:01.033 Uh, this is kind of where we're basically telling elastic the
00:12:01.033 --> 00:12:04.183 structure of the data we're giving it and how to analyze it
00:12:04.183 --> 00:12:07.386 for for the best search results for the best performance and
00:12:07.386 --> 00:12:07.858 accuracy.
00:12:09.398 --> 00:12:10.678 So that's what's all going on here.
00:12:10.688 --> 00:12:12.791 And then these completions are related to suggest stuff that
00:12:12.791 --> 00:12:13.998 we'll talk about here in a minute.
00:12:14.928 --> 00:12:19.331 One of the key points about all of the functionality of talking
00:12:19.331 --> 00:12:23.390 to elastic with these the nest library, which is what this
00:12:23.390 --> 00:12:27.518 stuff is, is that it's using the query descriptor language.
00:12:28.378 --> 00:12:29.598 Umm, it's a QDL.
00:12:30.398 --> 00:12:33.258 Uh, that elastic provided.
00:12:33.428 --> 00:12:38.338 So what you're seeing here when it's doing like first or default
00:12:38.338 --> 00:12:43.248 or whatever, it's actually doing it first or default, it's doing
00:12:43.248 --> 00:12:47.931 weird stuff in the back end to build a JSON style like object
00:12:47.931 --> 00:12:51.708 graph to send to the server as part of the query.
00:12:53.678 --> 00:12:56.820 So like the first there that's on line 37, you're just using
00:12:56.820 --> 00:13:00.066 that for the sake of we have to do something in the C language
00:13:00.066 --> 00:13:00.838 to bring it up.
00:13:00.978 --> 00:13:04.793 It's not actually looking at the first of the attributes in the
00:13:04.793 --> 00:13:08.190 list, it's using it as the object reference of the type,
00:13:08.190 --> 00:13:11.588 and then afterwards you're doing your mapping individual
00:13:11.588 --> 00:13:15.343 properties like the ID key U of M value and again those things
00:13:15.343 --> 00:13:17.488 are all the keyword in the SS name.
00:13:17.678 --> 00:13:18.558 That's all stuff.
00:13:18.568 --> 00:13:20.648 That's just part of their descriptor language.
00:13:20.658 --> 00:13:23.678 To make it functional, but it's not C sharp.
00:13:23.688 --> 00:13:27.488 Language versions of like a LINQ query on how this stuff works.
00:13:27.178 --> 00:13:27.358 Yeah.
00:13:28.288 --> 00:13:33.148 So, uh, something that's kind of interesting is that.
00:13:33.388 --> 00:13:38.469 It functions somewhat similarly to the way entity framework does
00:13:38.469 --> 00:13:42.298 SQL Transpilation, where it uses uh expressions.
00:13:42.358 --> 00:13:43.558 If I can find it, there we go.
00:13:43.748 --> 00:13:48.822 System dot link dot expressions dot expression, so this is this
00:13:48.822 --> 00:13:51.358 is the C sharp expression stuff.
00:13:51.848 --> 00:13:53.998 Is effectively like code.
00:13:54.008 --> 00:13:57.306 That's not really code, it's code that gets turned into an
00:13:57.306 --> 00:14:00.884 object that you can manipulate rather than an actual executable
00:14:00.884 --> 00:14:01.778 snippet of code.
00:14:01.788 --> 00:14:05.216 And you can compile an expression to a snippet of code,
00:14:05.216 --> 00:14:09.011 but the the basically what's happening here is that it's just
00:14:09.011 --> 00:14:12.500 describing the way that it's accessing P dot filter will
00:14:12.500 --> 00:14:13.908 attributes stuff first.
00:14:13.918 --> 00:14:15.208 It's not actually doing this.
00:14:15.218 --> 00:14:19.054 It's never gets executed and the entity framework SQL queries
00:14:19.054 --> 00:14:20.168 work the same way.
00:14:20.178 --> 00:14:23.607 If you ever look at one of those, you'll see that it's in
00:14:23.607 --> 00:14:27.450 fact just getting a little head of it here, you'll see that this
00:14:27.450 --> 00:14:28.928 also takes an expression.
00:14:29.838 --> 00:14:30.648 Same thing.
00:14:30.698 --> 00:14:32.348 So this is not actually being executed.
00:14:32.358 --> 00:14:35.402 We're just telling entity framework this is how I want to
00:14:35.402 --> 00:14:38.866 read stuff for this property and it uses that to generate the SQL
00:14:38.866 --> 00:14:41.018 that actually runs against the database.
00:14:41.028 --> 00:14:42.458 So just a cool thing you can.
00:14:43.948 --> 00:14:46.301 Actually, see kind of a little bit under the hood, how they're
00:14:46.301 --> 00:14:48.728 implementing that stuff with the looking at the types that these
00:14:48.728 --> 00:14:49.288 methods accept.
00:14:50.418 --> 00:14:50.718 Umm.
00:14:51.108 --> 00:14:54.666 But anyway, so the sort of first step in the process for getting
00:14:54.666 --> 00:14:56.418 data into elastic is telling it.
00:14:56.508 --> 00:14:58.478 Here's how we're gonna give you data.
00:14:58.488 --> 00:15:02.208 Here's what this data is going to look like, so that sort of
00:15:02.208 --> 00:15:03.488 configures the index.
00:15:03.618 --> 00:15:07.173 UM and prepares it to know what data we're gonna give it and how
00:15:07.173 --> 00:15:07.938 to analyze it.
00:15:08.738 --> 00:15:13.688 The next step in that process is to actually read the data out of
00:15:13.688 --> 00:15:13.988 SEF.
00:15:14.298 --> 00:15:15.438 And yeah, what's up, Nick?
00:15:16.788 --> 00:15:20.505 I was just wondering if you said the most common use for this is
00:15:20.505 --> 00:15:21.648 for product filters.
00:15:21.658 --> 00:15:24.858 Why the product section is commented out a little bit
00:15:24.858 --> 00:15:25.628 farther down.
00:15:26.368 --> 00:15:30.511 Because this is, we're already indexing the product itself and
00:15:30.511 --> 00:15:32.878 products don't have other products.
00:15:33.148 --> 00:15:36.332 It's because this same code is more or less used on all the
00:15:36.332 --> 00:15:39.251 different indexes for like manufacturers which do have
00:15:39.251 --> 00:15:39.728 products.
00:15:39.738 --> 00:15:43.838 So if we go in here, you'll see that the manufacturer section is
00:15:43.838 --> 00:15:47.180 commented out, but the product one is not, because a
00:15:47.180 --> 00:15:49.198 manufacturer does have products.
00:15:49.208 --> 00:15:51.238 What a product doesn't have products?
00:15:49.978 --> 00:15:54.692 It's a cheat to allow like three week appear between multiple
00:15:54.692 --> 00:15:55.148 files.
00:15:55.578 --> 00:15:59.927 So like I can compare products and manufacturers and uh lots to
00:15:59.927 --> 00:16:03.665 and then visually they have to line up inside winmerge
00:16:03.665 --> 00:16:07.742 correctly because there's code there, whether it's actually
00:16:05.898 --> 00:16:06.278 Gotcha.
00:16:07.742 --> 00:16:09.848 executed or not doesn't matter.
00:16:10.258 --> 00:16:10.828 It's you.
00:16:10.838 --> 00:16:12.801 Don't do we have the right format if I didn't make it
00:16:12.801 --> 00:16:14.328 update that needs to go across the board.
00:16:14.898 --> 00:16:17.589 I know that stores and franchises and categories and
00:16:17.589 --> 00:16:20.534 auctions and like a whole bunch of these things that have
00:16:20.534 --> 00:16:23.428 products on them and then produces only one who doesn't.
00:16:23.658 --> 00:16:27.002 So let me line it all up and then I'll just let my comment
00:16:27.002 --> 00:16:30.458 blocks it start on their own line with the on the same line.
00:16:30.808 --> 00:16:33.478 Uh, be the part that's off.
00:16:33.548 --> 00:16:35.931 But where all the code in between does line up correctly
00:16:35.931 --> 00:16:36.558 and when merge.
00:16:39.418 --> 00:16:39.708 Yeah.
00:16:39.718 --> 00:16:42.391 So it's basically just to keep the files consistent and
00:16:39.788 --> 00:16:40.478 Yeah, it makes sense.
00:16:42.391 --> 00:16:42.868 structure.
00:16:44.208 --> 00:16:44.778 Umm.
00:16:45.298 --> 00:16:48.238 So yeah.
00:16:48.708 --> 00:16:51.709 So anyway, what I was saying is the next step after we've told
00:16:51.709 --> 00:16:54.806 Elastic what the data will look like is to actually get the data
00:16:54.806 --> 00:16:55.758 out of our database.
00:16:55.768 --> 00:16:59.037 So this is the part where we're taking the snapshot, and that
00:16:59.037 --> 00:17:01.778 happens in the hilariously named dump reader class.
00:17:01.868 --> 00:17:03.738 I'm looking at the manufacturer on look at the product.
00:17:03.748 --> 00:17:07.366 One umm I've been what James explained to me is that dump
00:17:07.366 --> 00:17:11.419 reader is just what they called it in their documentation and it
00:17:11.419 --> 00:17:14.288 makes sense to keep the name for consistency.
00:17:15.098 --> 00:17:19.064 But nonetheless, yeah, nonetheless, it still makes me
00:17:15.238 --> 00:17:16.548 Came right out of their example.
00:17:19.064 --> 00:17:19.578 giggle.
00:17:20.628 --> 00:17:24.255 Umm so, but what this class is actually doing is it's going and
00:17:24.255 --> 00:17:27.995 getting the raw product data out of the database and transforming
00:17:27.995 --> 00:17:31.168 it into the actual object that we're gonna give lastic.
00:17:31.178 --> 00:17:32.278 So you'll remember this name.
00:17:32.288 --> 00:17:35.849 Product indexable model is what we configured in here product
00:17:35.849 --> 00:17:36.768 indexable model.
00:17:37.648 --> 00:17:40.374 So this is where we're this is where we're telling elastic what
00:17:40.374 --> 00:17:42.418 the data will look like when we send it to him.
00:17:42.488 --> 00:17:44.624 This is where we're actually getting the data that we're
00:17:44.624 --> 00:17:45.148 going to send.
00:17:46.598 --> 00:17:47.658 So sorry.
00:17:50.358 --> 00:17:53.945 Umm, so there's a little bit of setup going on up here to build
00:17:53.945 --> 00:17:57.644 up filters and things that we're going to use in the next section
00:17:57.644 --> 00:18:00.558 and then you see this big giant foreach right here.
00:18:03.348 --> 00:18:07.634 This probably looks very confusing as to why this is in a
00:18:07.634 --> 00:18:11.698 foreach and not somewhere else or done some other way.
00:18:12.068 --> 00:18:15.749 The reason for this is to make this uh, in enumerable and yield
00:18:15.749 --> 00:18:19.258 return the results so that we can effectively return them as
00:18:19.258 --> 00:18:23.054 we read them out of the database one at a time instead of reading
00:18:23.054 --> 00:18:26.390 it all from the database all in one shot in bulk and then
00:18:26.390 --> 00:18:27.828 returning it all in bulk.
00:18:28.618 --> 00:18:32.775 So this sort of kind of allows us to stream the results and
00:18:32.775 --> 00:18:36.794 it's a little bit lighter on processing power to do that,
00:18:36.794 --> 00:18:37.348 umm, so.
00:18:37.458 --> 00:18:41.779 You'll also get your errors out of it much faster than loading
00:18:41.779 --> 00:18:45.688 200,000 products and then setting them all to elastic at
00:18:45.688 --> 00:18:50.008 once, and then elastic somewhere in there dying and not really
00:18:50.008 --> 00:18:50.968 knowing where.
00:18:51.778 --> 00:18:56.878 If you're with this streaming option, it's much more accurate
00:18:56.878 --> 00:19:00.908 to find out like 500 products into your 200,000.
00:19:01.098 --> 00:19:05.086 That's when it failed and you can stop your processing and
00:19:05.086 --> 00:19:08.939 find within the 1st 500 ± 30 products is one of the ones
00:19:08.939 --> 00:19:13.265 that's bad for stuff rather than finding it, you know two hours
00:19:13.265 --> 00:19:17.455 later because it took that long to load all those products in
00:19:17.455 --> 00:19:17.928 memory.
00:19:18.518 --> 00:19:21.798 Yeah, it it reduces the size of the haystack.
00:19:21.808 --> 00:19:25.278 You gotta look for the needle in umm.
00:19:24.578 --> 00:19:24.778 Yeah.
00:19:25.418 --> 00:19:28.788 So that's why it's directly inside this foreach.
00:19:28.798 --> 00:19:32.251 If anyone ever wondered, I've seen this in wondered, but so I
00:19:32.251 --> 00:19:35.593 mean we're doing some pretty basic stuff here, filtering by
00:19:35.593 --> 00:19:37.208 active and visible filtering.
00:19:37.218 --> 00:19:41.103 By not discontinued, we filter by these type keys which come
00:19:41.103 --> 00:19:42.058 from a setting.
00:19:42.698 --> 00:19:46.477 By default, the setting is empty, but we've used it on a
00:19:46.477 --> 00:19:50.587 bajillion different products to customize the catalog in ways
00:19:50.587 --> 00:19:54.696 the client once, for example GLV Umm GLV is a site that sells
00:19:54.696 --> 00:19:55.558 wood veneers.
00:19:56.308 --> 00:19:57.918 They wanted to.
00:19:58.128 --> 00:20:00.628 So for example, they'll have like a cut of wood that's like.
00:20:01.318 --> 00:20:03.128 Uh, I don't know.
00:20:03.198 --> 00:20:06.375 I can only think of Tonewoods from guitars, so we'll just go
00:20:06.375 --> 00:20:07.468 with Indian rosewood.
00:20:08.368 --> 00:20:12.251 So they'll sell Indian rosewood and they'll sell seven different
00:20:12.251 --> 00:20:15.238 sizes with three different backings for each one.
00:20:15.548 --> 00:20:19.127 Which means that on total there's maybe 21 different kinds
00:20:19.127 --> 00:20:21.978 of Indian rosewood products in their database.
00:20:22.348 --> 00:20:25.724 Well, they didn't want there to be 21 Indian rosewood's in the
00:20:25.724 --> 00:20:28.939 catalog that somebody has to scroll past if they don't care
00:20:28.939 --> 00:20:29.528 about that.
00:20:29.538 --> 00:20:30.188 About that.
00:20:30.498 --> 00:20:33.958 So what they wanted to do was make their catalog only show
00:20:33.958 --> 00:20:37.535 variant masters and then you click into that species of wood
00:20:37.535 --> 00:20:41.346 Indian rosewood and then on your product details page you picked
00:20:41.346 --> 00:20:44.923 the size, you pick the backing that you want and so that was
00:20:44.923 --> 00:20:48.500 that's completely supported out of the box just by using the
00:20:48.500 --> 00:20:51.138 searching product index filter by type keys.
00:20:51.768 --> 00:20:52.728 So you just give it.
00:20:52.008 --> 00:20:56.257 And contrary to that, another project absolutely wants the 21
00:20:56.257 --> 00:20:59.478 to show up, but not the master on their pages.
00:20:57.858 --> 00:20:58.038 Yeah.
00:20:59.488 --> 00:21:00.618 And that's just the way that they are.
00:21:01.148 --> 00:21:03.378 Yep, certain clients want things different ways.
00:21:03.568 --> 00:21:03.938 Uh.
00:21:03.948 --> 00:21:06.358 Rydell, the roller skate site we're working on.
00:21:06.648 --> 00:21:09.218 They want to show sets only.
00:21:09.308 --> 00:21:12.028 They don't want to show the individual sizes of the shoes.
00:21:12.038 --> 00:21:14.818 They don't wanna show the the the shoe, only variants and all
00:21:14.818 --> 00:21:16.118 that kind of variant masters.
00:21:16.408 --> 00:21:20.388 They just want to show the sets and then control all the
00:21:20.388 --> 00:21:23.878 different selections on the product details page.
00:21:24.288 --> 00:21:27.228 So again, it's all configurable with the setting.
00:21:27.508 --> 00:21:30.161 The setting is just the type key of the products you want
00:21:30.161 --> 00:21:31.258 included in the catalog.
00:21:32.218 --> 00:21:32.458 Uh.
00:21:32.738 --> 00:21:36.778 So, umm, so we can move on from that.
00:21:37.388 --> 00:21:40.727 We can see here that this big giant select is basically doing
00:21:40.727 --> 00:21:43.744 a targeted read of the properties that we actually care
00:21:43.744 --> 00:21:44.228 to index.
00:21:45.708 --> 00:21:48.549 So rather than reading out the entire product and then for
00:21:48.549 --> 00:21:51.630 everything, every brand, every category, every franchise, every
00:21:51.630 --> 00:21:54.422 manufacturer, every store, every vendor, every lot, every
00:21:54.422 --> 00:21:57.551 variant, et cetera, if we didn't do it this way, then every time
00:21:57.551 --> 00:22:00.440 we act, if we just read out the full product, every time we
00:22:00.440 --> 00:22:03.424 access one of these, we would actually be causing a lazy load
00:22:03.424 --> 00:22:06.313 to make an additional database call, which would absolutely
00:22:06.313 --> 00:22:09.105 murder the performance of this even for a small number of
00:22:09.105 --> 00:22:09.538 products.
00:22:10.928 --> 00:22:13.378 So we're sort of eager loading things.
00:22:13.388 --> 00:22:17.435 We know we're probably going to need and that kind of bundles it
00:22:17.435 --> 00:22:21.233 all into one SQL query that gets executed at once instead of
00:22:21.233 --> 00:22:24.719 breaking it out across a bajillion different calls that
00:22:24.719 --> 00:22:28.330 are happening at random times throughout the the indexing
00:22:28.330 --> 00:22:28.828 process.
00:22:30.628 --> 00:22:34.871 So what we do this select to do that targeted pull of the stuff
00:22:34.871 --> 00:22:38.980 we care about and then we map it out into an object in memory
00:22:38.980 --> 00:22:42.228 where we do a little bit of preprocessing on it.
00:22:42.238 --> 00:22:48.090 So we do some of this collapsing white space on the custom key on
00:22:48.090 --> 00:22:48.888 the name.
00:22:49.038 --> 00:22:52.555 On short description, we convert some of these into the indexable
00:22:52.555 --> 00:22:55.378 filters we're gonna store onto the objects directly.
00:22:56.468 --> 00:23:00.018 UM, we do a little bit at some of the stuff that basically do.
00:23:00.028 --> 00:23:02.714 We just can't do in the SQL side with the query, which is
00:23:02.714 --> 00:23:05.632 basically just newing up like strongly typed objects like this
00:23:05.632 --> 00:23:08.364 and doing some of these like custom methods like collapse,
00:23:08.364 --> 00:23:09.938 white space and tidying stuff up.
00:23:11.168 --> 00:23:14.208 This is a little bit of preprocessing of the data, so,
00:23:14.208 --> 00:23:17.580 but that's that's basically what this big giant SQL query is
00:23:17.580 --> 00:23:18.188 doing here.
00:23:18.458 --> 00:23:20.068 I wish I could just collapse only this.
00:23:20.078 --> 00:23:21.388 I probably collapse.
00:23:21.798 --> 00:23:21.998 No.
00:23:22.008 --> 00:23:23.028 Nope, that sucks.
00:23:23.218 --> 00:23:26.025 That's fine, but anyway, that's what this whole SQL query is
00:23:26.025 --> 00:23:28.970 doing is it's reading out just the things we need to index, and
00:23:28.970 --> 00:23:31.639 then doing a little bit of preprocessing processing on in
00:23:31.639 --> 00:23:34.492 the second select down here just to start to get it ready for
00:23:34.492 --> 00:23:34.998 sending in.
00:23:36.438 --> 00:23:39.045 So then inside the body of this, this is the part that's gonna
00:23:39.045 --> 00:23:39.748 iterate for each.
00:23:40.608 --> 00:23:45.820 Umm for each product that we're planning to index, we're going
00:23:45.820 --> 00:23:51.032 to attempt to index the first thing that happens in here is we
00:23:51.032 --> 00:23:55.168 start imposing some some early outs, if you will.
00:23:55.178 --> 00:23:58.780 So we're we're making sure that this product that we read out
00:23:58.780 --> 00:24:01.568 meets the requirements set by the app settings.
00:24:01.918 --> 00:24:04.820 So for example, if we've dictated that in order for a
00:24:04.820 --> 00:24:08.366 product to be indexed, it has to have a brand and it doesn't have
00:24:08.366 --> 00:24:10.408 a brand, then it doesn't get indexed.
00:24:10.418 --> 00:24:14.710 Same with categories if and this by the way, by default all of
00:24:14.710 --> 00:24:18.388 these are false except for category category is true.
00:24:18.758 --> 00:24:21.814 So if you have a product that you've created and it's not
00:24:21.814 --> 00:24:25.028 assigned to any category, it will not index and we'll talk a
00:24:25.028 --> 00:24:28.295 little bit about more, a little bit more about that here in a
00:24:28.295 --> 00:24:31.404 minute, because there's some useful troubleshooting tips I
00:24:31.404 --> 00:24:31.878 can give.
00:24:32.428 --> 00:24:35.581 But ultimately these will allow you to make sure that
00:24:35.581 --> 00:24:37.858 effectively bad data can't be indexed.
00:24:37.868 --> 00:24:41.847 You avoid indexing products that are missing information required
00:24:41.847 --> 00:24:44.198 for the this client's website to work.
00:24:44.328 --> 00:24:47.045 So if you have an auction website where it doesn't make
00:24:47.045 --> 00:24:49.957 sense for a product to ever show in the catalog if it's not
00:24:49.957 --> 00:24:53.062 attached to a lot, then you can enable this setting and it will
00:24:53.062 --> 00:24:55.925 completely prevent products without lots from ever showing
00:24:55.925 --> 00:24:56.798 up in the catalog.
00:24:57.498 --> 00:25:00.401 It doesn't friend them showing up in the storefront in general,
00:25:00.401 --> 00:25:03.078 just from catalog results, because again, all elastic does
00:25:03.078 --> 00:25:05.028 is dictate what comes back in the catalog.
00:25:05.038 --> 00:25:09.055 Query it doesn't drive product details, pages or custom things
00:25:09.055 --> 00:25:12.690 that are querying the database directly for like related
00:25:12.690 --> 00:25:15.368 products, carousels and things like that.
00:25:15.378 --> 00:25:18.164 Those don't go through elastic, so that's a whole other thing
00:25:18.164 --> 00:25:20.904 that I'm not going to delve into right now, but this will at
00:25:20.904 --> 00:25:23.735 least prevent those things from, not from being in the catalog
00:25:23.735 --> 00:25:26.565 itself, which is probably the main place people are gonna come
00:25:26.565 --> 00:25:27.598 across products anyway.
00:25:28.678 --> 00:25:31.858 Umm, so we do a little bit of that early out stuff.
00:25:32.178 --> 00:25:36.106 UM, and then we get into actually building it into the
00:25:36.106 --> 00:25:37.248 indexable model.
00:25:37.258 --> 00:25:41.172 So this is basically mapping those database columns onto that
00:25:41.172 --> 00:25:43.128 type that we set up in elastic.
00:25:43.138 --> 00:25:44.608 Earlier in the indexer file.
00:25:44.858 --> 00:25:46.408 So we're storing all this stuff on here.
00:25:46.418 --> 00:25:47.688 We're setting everything up.
00:25:47.828 --> 00:25:52.215 Basic stuff we handle, we have a handful of extra things down
00:25:52.215 --> 00:25:56.319 here that are basically handling more complex things like
00:25:56.319 --> 00:26:00.918 attributes, variant attributes, so the filter will attributes is
00:26:00.918 --> 00:26:01.838 kind of like.
00:26:01.848 --> 00:26:06.673 Let me see if I actually just pull up like developer right now
00:26:06.673 --> 00:26:10.348 or something and show you guys what it's doing.
00:26:13.188 --> 00:26:16.999 OK, so this is the developed demo site so those filter will
00:26:16.999 --> 00:26:20.872 attributes are things like this so I can filter by arbitrary
00:26:20.872 --> 00:26:25.000 stuff like what color it is and what material it's made of, what
00:26:25.000 --> 00:26:28.620 size and these are just arbitrary sizes, whatever things
00:26:28.620 --> 00:26:30.398 we need is digital download.
00:26:30.408 --> 00:26:34.676 All that kind of these are JSON attributes that we've dictated
00:26:34.676 --> 00:26:39.147 are filterable on the attributes dot general attribute table, umm
00:26:39.147 --> 00:26:43.211 or whatever it is a system dot general attribute, something
00:26:43.211 --> 00:26:43.888 like that.
00:26:44.438 --> 00:26:48.287 We've basically said that these ones are allowed to be used as
00:26:48.287 --> 00:26:52.135 filters in the catalog, and then we tell elastic the structure
00:26:52.135 --> 00:26:56.045 for them inside that method, and then even further from that we
00:26:56.045 --> 00:26:57.938 have handle variant attributes.
00:26:58.268 --> 00:27:01.230 Now this one is something that is related to what I was talking
00:27:01.230 --> 00:27:01.878 about earlier.
00:27:02.268 --> 00:27:06.979 Whenever we have clients that want to only display variant and
00:27:06.979 --> 00:27:08.698 masters in the catalog.
00:27:09.028 --> 00:27:12.347 When we did that for GLV, the first issue they had was like,
00:27:12.347 --> 00:27:15.340 Oh well, what if, like, sometimes we have clients that
00:27:15.340 --> 00:27:18.713 come to us and they only know that they have a 4 by 8 foot, a
00:27:18.713 --> 00:27:20.998 section of whatever that needs of veneer.
00:27:21.478 --> 00:27:24.436 But now they can't search for by 8 and find what they're looking
00:27:24.436 --> 00:27:24.618 for.
00:27:24.988 --> 00:27:28.411 So then what we ended up doing was we actually indexed the
00:27:28.411 --> 00:27:32.124 attributes of the variance on a variant master onto the variant
00:27:32.124 --> 00:27:35.605 master, so that what I can do at that point is if there's a
00:27:35.605 --> 00:27:39.434 variant master that has a 4 by 8 variant that 4 by 8 attribute is
00:27:39.434 --> 00:27:43.089 indexed with the master and I can go search by 4 by 8 and find
00:27:43.089 --> 00:27:46.628 all of the variant masters that have a four by eight option.
00:27:48.178 --> 00:27:51.212 So that's what this one is doing is it's kind of pulling some of
00:27:51.212 --> 00:27:53.872 the variants attributes up to the variant master so that
00:27:53.872 --> 00:27:56.625 queries for those still find the master itself in case the
00:27:56.625 --> 00:27:57.978 catalog is filtered that way.
00:27:59.248 --> 00:28:02.407 Umm this one handles the building out the category tree
00:28:02.407 --> 00:28:05.058 in a way that we can query against an elastic.
00:28:05.568 --> 00:28:10.678 This one handles a storing the roles so we can filter by roles.
00:28:11.638 --> 00:28:17.241 Umm these are setting up the suggestions which is basically
00:28:17.241 --> 00:28:19.668 this little text box here.
00:28:19.678 --> 00:28:20.818 If I type uh.
00:28:21.288 --> 00:28:21.838 Wood.
00:28:21.948 --> 00:28:26.090 Then it will pop me up this little drop down of words that
00:28:26.090 --> 00:28:30.513 are products that are similar or related to my search term and
00:28:30.513 --> 00:28:32.338 this is not a full search.
00:28:32.348 --> 00:28:35.333 It's a very fast kind of completion search that just
00:28:35.333 --> 00:28:38.937 pulls up very quickly some stuff that's sort of kind of closely
00:28:38.937 --> 00:28:39.838 related to this.
00:28:39.848 --> 00:28:42.670 It's not going to give back as good of results or as as
00:28:42.670 --> 00:28:45.643 accurate or results as this, but it's faster than the full
00:28:45.643 --> 00:28:46.298 catalog load.
00:28:47.608 --> 00:28:50.307 So you can see that it got like these two that have woodlore in
00:28:50.307 --> 00:28:50.518 them.
00:28:50.528 --> 00:28:51.978 This one has dark wood.
00:28:52.128 --> 00:28:55.338 As soon as light wood, this one that probably is because it was
00:28:55.338 --> 00:28:58.398 the word word work is somewhat close to wood or maybe it has
00:28:58.398 --> 00:29:01.558 wood in the description or the short description or something.
00:29:02.388 --> 00:29:08.130 Umm, but that's, uh, that's what those suggests are setting up is
00:29:08.130 --> 00:29:08.478 the.
00:29:11.828 --> 00:29:16.019 The type ahead results setup and then last is queryable
00:29:16.019 --> 00:29:20.734 attributes, which admittedly I don't really remember what that
00:29:20.734 --> 00:29:20.958 is.
00:29:20.968 --> 00:29:24.441 I think it's like making it so you can text query against
00:29:24.441 --> 00:29:25.878 certain attribute names.
00:29:25.888 --> 00:29:30.818 We needed that for projects where somebody wanted to type in
00:29:30.818 --> 00:29:31.868 20 milligram.
00:29:31.878 --> 00:29:34.633 I think it was a key source they want to be able to type like the
00:29:34.633 --> 00:29:36.888 dosage of a pill and find all all of them that had an
00:29:36.888 --> 00:29:39.058 attribute that matched that or something like that.
00:29:40.788 --> 00:29:44.398 Umm, so that's effectively what that's doing there.
00:29:45.078 --> 00:29:47.955 And then the very end of it is this yield return indexable
00:29:47.955 --> 00:29:50.930 model and that's what I was talking about earlier, how we're
00:29:50.930 --> 00:29:53.710 effectively streaming the results of these one at a time
00:29:53.710 --> 00:29:56.538 up to elastic to pass into the index as as it meets them.
00:29:57.548 --> 00:30:02.150 Umm, so if you're not familiar with the yield return syntax, I
00:30:02.150 --> 00:30:06.314 recommend looking into uh innumerables and yield returns
00:30:06.314 --> 00:30:10.696 because they're pretty cool and you can do some really neat
00:30:10.696 --> 00:30:11.938 things with them.
00:30:13.008 --> 00:30:14.338 Uh so.
00:30:15.028 --> 00:30:17.539 But I don't wanna take time going through that since it's
00:30:17.539 --> 00:30:18.708 not directly relevant here.
00:30:21.438 --> 00:30:27.907 So at this point we have told Elastic how we're gonna give it
00:30:27.907 --> 00:30:28.428 data.
00:30:29.158 --> 00:30:33.705 We've read the data out and we're giving it to elastic in
00:30:33.705 --> 00:30:35.038 the it's in here.
00:30:36.458 --> 00:30:38.478 No, I'll find it.
00:30:40.658 --> 00:30:44.308 Probably in here the bottom it's in the indexer base.
00:30:44.318 --> 00:30:47.128 Actually I think yeah.
00:30:47.678 --> 00:30:49.068 So this is where we actually do that.
00:30:49.078 --> 00:30:52.517 This is kind of some of the some of the under the hood stuff
00:30:52.517 --> 00:30:55.674 where we go in and we actually create the new index, we
00:30:55.674 --> 00:30:59.056 populate it, which is calling that dump reader that we just
00:30:59.056 --> 00:31:02.608 went through, loading all the data, streaming it in, swaps the
00:31:02.608 --> 00:31:05.990 alias and what this is actually doing is ensuring as little
00:31:05.990 --> 00:31:09.373 downtime as possible by filling the data into an index that
00:31:09.373 --> 00:31:10.838 isn't actually in use yet.
00:31:11.008 --> 00:31:15.141 And then only once the new index is fully populated and ready, we
00:31:15.141 --> 00:31:19.149 basically just swap the name of the active and the inactive one
00:31:19.149 --> 00:31:23.032 so that there's almost no down time and it goes straight from
00:31:23.032 --> 00:31:26.790 querying the current index to the new one without having to
00:31:26.790 --> 00:31:28.668 like delete the old index 1st.
00:31:28.678 --> 00:31:31.391 And then there's a bunch of downtime where the catalog might
00:31:31.391 --> 00:31:34.238 not work while we fill up a new index and then set it active or
00:31:34.238 --> 00:31:34.638 whatever.
00:31:35.858 --> 00:31:41.408 Umm so we created, populate it, swap the names and then that's.
00:31:41.488 --> 00:31:44.282 That's the process that elastic does whenever the whenever you
00:31:44.282 --> 00:31:45.168 index your products.
00:31:46.988 --> 00:31:50.114 So but at this point we've we've populated an index with data
00:31:50.114 --> 00:31:53.038 we've told last account to analyze it etcetera, etcetera.
00:31:53.348 --> 00:31:56.868 The only thing left at this point is to query against it and
00:31:56.868 --> 00:32:00.330 to use the results for whatever we need that happens in the
00:32:00.330 --> 00:32:01.138 search module.
00:32:02.248 --> 00:32:07.438 Umm, so this class is, uh, pretty large.
00:32:07.588 --> 00:32:11.692 This file is pretty large and contains basically all the logic
00:32:11.692 --> 00:32:14.558 that builds the query we send into elastic.
00:32:14.988 --> 00:32:16.688 It also uses the.
00:32:18.288 --> 00:32:20.688 Query descriptor language James was talking about earlier.
00:32:23.238 --> 00:32:25.618 Personally, I absolutely hate this syntax.
00:32:26.868 --> 00:32:27.958 I think it's terrible.
00:32:28.028 --> 00:32:29.218 It makes my head hurt.
00:32:29.508 --> 00:32:34.208 But you know something?
00:32:34.218 --> 00:32:36.038 Something life is pain.
00:32:36.048 --> 00:32:36.728 I don't know.
00:32:37.218 --> 00:32:41.884 So, but ultimately this is the nest library we're using to
00:32:41.884 --> 00:32:44.098 query against Elasticsearch.
00:32:45.328 --> 00:32:49.654 UM and we build that query in here so you can see that if we
00:32:49.654 --> 00:32:53.979 start up here, we're passing in our query, which is the text
00:32:53.979 --> 00:32:57.028 that you type into the search box in here.
00:32:59.298 --> 00:33:03.491 Or passing it into query against the name with some fuzziness and
00:33:03.491 --> 00:33:06.921 a few different ways with prefixing terms and matches
00:33:06.921 --> 00:33:10.668 which are different terminology lastic uses for different.
00:33:10.678 --> 00:33:16.868 Basically ways to uh to query against a field.
00:33:17.808 --> 00:33:19.828 Umm and match it in different ways.
00:33:20.598 --> 00:33:24.498 We do the same for custom key by brand name et cetera.
00:33:24.508 --> 00:33:27.147 All the different stuff and something that you'll see with
00:33:27.147 --> 00:33:28.668 all of these is different boosts.
00:33:29.598 --> 00:33:33.908 What these boost values do, and they're in settings now, what
00:33:33.908 --> 00:33:36.618 these do is they dictate how to apply.
00:33:40.558 --> 00:33:40.948 Uh.
00:33:40.958 --> 00:33:44.988 How to score a match on this specific thing?
00:33:45.278 --> 00:33:51.026 So obviously it's most critical whenever we're searching things
00:33:51.026 --> 00:33:54.708 that the skew or the name come up first.
00:33:54.718 --> 00:33:58.451 So if I type in would I would rather see a product with wood
00:33:58.451 --> 00:34:02.123 in its name or its skew then a product that has wood buried
00:34:02.123 --> 00:34:04.448 somewhere random and its description.
00:34:04.598 --> 00:34:06.909 We still want the one that has it in the description to come up
00:34:06.909 --> 00:34:09.111 in case somebody's looking for something and they're like, I
00:34:09.111 --> 00:34:10.338 have no idea what this is called.
00:34:10.348 --> 00:34:12.674 I just know that it's made of wood, so I type wood and
00:34:12.674 --> 00:34:14.788 eventually they'll find what they're looking for.
00:34:14.988 --> 00:34:18.034 But we're gonna prioritize things that actually directly
00:34:18.034 --> 00:34:21.400 match what they're looking for by name or SKU, and that's what
00:34:21.400 --> 00:34:24.659 these boosts do is we basically say whenever Elastic finds a
00:34:24.659 --> 00:34:27.758 match, it assigns a score for how relevant the result is.
00:34:27.968 --> 00:34:30.058 This is basically scaling that score.
00:34:30.348 --> 00:34:33.599 So we're saying that matching a name is far more important than
00:34:33.599 --> 00:34:36.798 matching a description, and we boost those results a lot more.
00:34:36.968 --> 00:34:40.748 We actually go and look in these and see that the score values
00:34:40.748 --> 00:34:44.228 are like 10,000 for name UM and for a custom key or skew.
00:34:44.238 --> 00:34:45.428 It's like even higher.
00:34:45.438 --> 00:34:47.488 It's like 15,000 for for the skew.
00:34:48.398 --> 00:34:51.962 So it gives, it makes it so that skew is the most important,
00:34:51.962 --> 00:34:55.526 followed by name, followed by, you know, et cetera, down the
00:34:55.526 --> 00:34:55.818 line.
00:34:57.408 --> 00:34:58.548 And finding more?
00:34:59.418 --> 00:34:59.908 Uh.
00:35:01.078 --> 00:35:04.771 More out there results that may still be relevant to what the
00:35:04.771 --> 00:35:08.463 person is looking for, so once we've done that, we move on to
00:35:08.463 --> 00:35:10.488 some additional kinds of filters.
00:35:10.498 --> 00:35:11.048 We do.
00:35:11.118 --> 00:35:12.928 The function score is kind of similar.
00:35:13.118 --> 00:35:15.971 It goes in and builds out a final score value based on all
00:35:15.971 --> 00:35:18.728 the different matches and different things you can query
00:35:18.728 --> 00:35:21.339 against to give you a final number that tells you how
00:35:21.339 --> 00:35:22.838 relevant this is to your query.
00:35:23.798 --> 00:35:26.118 Umm, that's kind of what's going on in here.
00:35:28.078 --> 00:35:31.554 And then we move on to some of these more complex ones where
00:35:31.554 --> 00:35:34.974 UM, for example, if you're searching by brand name, we tell
00:35:34.974 --> 00:35:36.968 that we wanna match on brand name.
00:35:37.038 --> 00:35:41.691 We wanna find those that have a field called brand name that
00:35:41.691 --> 00:35:42.988 matches this, UM.
00:35:43.038 --> 00:35:46.061 If you've passed in the flag to only find products with lots,
00:35:46.061 --> 00:35:48.158 then we find only products that have lots.
00:35:49.128 --> 00:35:49.938 Umm.
00:35:50.088 --> 00:35:51.568 And then we go into these attributes.
00:35:51.578 --> 00:35:53.998 Any and all brands, single any and all, et cetera.
00:35:54.248 --> 00:35:58.053 These are where you're going in here and picking, for example, a
00:35:58.053 --> 00:35:59.458 category from this tree.
00:35:59.968 --> 00:36:03.289 This passes into that and tells Elastic that we only want
00:36:03.289 --> 00:36:04.778 products back that are in.
00:36:04.788 --> 00:36:08.188 This breaks category and it goes and finds only those products.
00:36:08.998 --> 00:36:11.358 Uh, and I think I may have actually just crashed the site
00:36:11.358 --> 00:36:13.718 because there is a bug we're currently fixing and develop
00:36:13.718 --> 00:36:15.468 where the site crashes every now and then.
00:36:15.478 --> 00:36:18.388 So it's probably spinning up right now.
00:36:20.768 --> 00:36:23.377 Yeah, I think I think it crashed and just spun back up, but
00:36:23.377 --> 00:36:23.898 that's fine.
00:36:24.828 --> 00:36:30.072 So so you can see there are only gave me back those and we can
00:36:30.072 --> 00:36:30.488 even.
00:36:30.498 --> 00:36:32.148 I should have had my dev tools open.
00:36:32.158 --> 00:36:37.566 We'll just refresh the page and we can look at this query call
00:36:37.566 --> 00:36:41.858 and see our payload sent category as breaks pipe.
00:36:41.868 --> 00:36:42.588 Automotive breaks.
00:36:42.598 --> 00:36:47.407 This is how we set the category names and we can then look at
00:36:47.407 --> 00:36:48.338 the preview.
00:36:48.348 --> 00:36:52.076 The response that gave us back umm and it gives us a category
00:36:52.076 --> 00:36:55.503 tree that only has relevant stuff to that, so it doesn't
00:36:55.503 --> 00:36:58.990 have every category anymore because this category tree is
00:36:58.990 --> 00:37:02.718 driven only by the results that are in the catalog right now.
00:37:03.188 --> 00:37:05.855 So that just makes it so that you can't click on a category
00:37:05.855 --> 00:37:08.700 plus a query and suddenly have 0 results in the category or the
00:37:08.700 --> 00:37:11.278 catalog because that would be very weird and unintuitive.
00:37:12.678 --> 00:37:15.288 Uh, but all that stuff happens inside of here.
00:37:15.828 --> 00:37:19.507 Inside of these single any and all queries and you can see that
00:37:19.507 --> 00:37:23.129 they also accept a setting each one of these to dictate if the
00:37:23.129 --> 00:37:24.968 filter should be applied or not.
00:37:26.408 --> 00:37:30.005 So we call these methods every time and we supply the the
00:37:30.005 --> 00:37:33.293 feature flag rather than wrapping them all in a in a
00:37:33.293 --> 00:37:37.262 flag, presumably because that's this is just easier to read, so
00:37:37.262 --> 00:37:38.068 less nesting.
00:37:38.838 --> 00:37:41.640 Umm, there may be another reason, but I assume that's the
00:37:41.640 --> 00:37:41.978 reason.
00:37:42.898 --> 00:37:46.368 Umm, so you can see there's a lot of different.
00:37:45.808 --> 00:37:46.818 Like a medic complexity.
00:37:46.378 --> 00:37:47.648 Uh, yeah.
00:37:48.778 --> 00:37:49.348 Umm.
00:37:49.678 --> 00:37:53.998 So yeah, so we, we we basically passed the flag in.
00:37:50.148 --> 00:37:50.808 And reusability.
00:37:54.008 --> 00:37:57.280 If we look at anyone number of these, the last thing is setting
00:37:57.280 --> 00:37:58.098 and if it's not.
00:37:58.108 --> 00:38:00.718 If settings not enabled then we just early out we bail out
00:38:00.718 --> 00:38:03.018 because we're not gonna filter on whatever that is.
00:38:03.228 --> 00:38:05.930 So if manufacturers aren't enabled, then these methods get
00:38:05.930 --> 00:38:08.814 called, but they don't actually impact the final query we send
00:38:08.814 --> 00:38:09.638 to elastic at all.
00:38:09.948 --> 00:38:12.518 Same with franchises, categories, brands, etc.
00:38:13.028 --> 00:38:14.738 Oopsies I just rearranged a little bit.
00:38:15.308 --> 00:38:15.968 They don't.
00:38:15.978 --> 00:38:19.225 They don't impact the final query if the feature is not
00:38:19.225 --> 00:38:22.589 enabled, but we build up all these queries and things and
00:38:22.589 --> 00:38:26.300 then we send that into elastic when a client wants to customize
00:38:26.300 --> 00:38:29.953 querying and elastic, there's a handful of things that we need
00:38:29.953 --> 00:38:33.258 to address first before we assume that that's necessary.
00:38:33.528 --> 00:38:37.237 We need to 1st to determine if what they're looking for is
00:38:37.237 --> 00:38:41.323 actually a customization, or if it's just tweaking some of these
00:38:41.323 --> 00:38:44.088 mini settings we have that already control.
00:38:44.288 --> 00:38:45.038 Excuse me, control.
00:38:45.048 --> 00:38:48.380 A lot of this behavior in different ways, so if the client
00:38:48.380 --> 00:38:51.430 is just saying I only want variant masters show up my
00:38:51.430 --> 00:38:53.858 catalog, well, we have a setting for that.
00:38:54.008 --> 00:38:56.940 If the client is saying I don't want to require categories for
00:38:56.940 --> 00:38:58.708 products, we have a setting for that.
00:38:59.008 --> 00:39:02.263 If the line is saying I have to require a manufacturer on all
00:39:02.263 --> 00:39:04.258 products, we have a setting for that.
00:39:04.368 --> 00:39:07.550 All that kind of stuff can be dictated with settings, and so
00:39:07.550 --> 00:39:10.576 that's sort of the first step with these is often catalog
00:39:10.576 --> 00:39:13.601 customizations are really complicated just because as you
00:39:13.601 --> 00:39:16.418 can see these this query language is kind of tough to
00:39:16.418 --> 00:39:19.861 work with, especially for things like this where it starts to get
00:39:19.861 --> 00:39:22.678 really deep nested and kind of terrifying to look at.
00:39:24.868 --> 00:39:28.353 And so ultimately what we want is to make sure that we're not
00:39:28.353 --> 00:39:31.613 wasting time reinventing something when we already have a
00:39:31.613 --> 00:39:33.018 feature that might do it.
00:39:33.618 --> 00:39:38.085 UM, so if you're ever in a scenario where client is wanting
00:39:38.085 --> 00:39:42.403 catalog customizations or searching customizations, if at
00:39:42.403 --> 00:39:47.168 all possible, stray on the side of using existing feature flags
00:39:47.168 --> 00:39:51.783 or settings to to control the behavior and only actually come
00:39:51.783 --> 00:39:56.399 in here and add new stuff if what they need is actually going
00:39:56.399 --> 00:39:59.228 to require customization or a change.
00:39:59.718 --> 00:40:03.280 UM and most of the time, if a client does need an actual
00:40:03.280 --> 00:40:06.842 change in this file, it's probably going to be something
00:40:06.842 --> 00:40:10.717 that you're gonna need office hours to dig into anyway, so at
00:40:10.717 --> 00:40:14.216 that point we would be able to help determine does this
00:40:14.216 --> 00:40:18.278 actually need a code change, or can we just do it with settings?
00:40:19.768 --> 00:40:23.530 So, but that's kind of the main the main bulk of what's going on
00:40:23.530 --> 00:40:27.292 in this in this file, the search module is actually building all
00:40:27.292 --> 00:40:27.928 that stuff.
00:40:28.678 --> 00:40:31.798 Uh, including the sort logic right here.
00:40:32.578 --> 00:40:37.187 The aggregations which is for categories, there's basically
00:40:37.187 --> 00:40:41.873 goes in and builds up the the aggregate query for categories
00:40:41.873 --> 00:40:44.868 and you can see that that explains it.
00:40:44.878 --> 00:40:48.659 It goes and builds the category tree up to 7 levels deep, so
00:40:48.659 --> 00:40:52.501 we're kind of telling elastic how to query 7 levels deep into
00:40:52.501 --> 00:40:54.298 the category tree for things.
00:40:55.178 --> 00:40:55.468 Uh.
00:40:56.498 --> 00:41:01.446 And then the last part of that that's sort of necessary or
00:41:01.446 --> 00:41:06.729 useful to see is the search view model, additional assignments
00:41:06.729 --> 00:41:07.148 part.
00:41:07.318 --> 00:41:12.837 So this is the part where we have this I search response
00:41:12.837 --> 00:41:18.259 which is part of the nest library and we use it to read
00:41:18.259 --> 00:41:20.098 out relevant stuff.
00:41:20.208 --> 00:41:23.395 So for example, what I was talking about earlier is like
00:41:23.395 --> 00:41:26.078 all these categories that show up in this tree.
00:41:26.408 --> 00:41:31.232 And again, if I click breaks, it's only the two of them that's
00:41:31.232 --> 00:41:35.979 actually part of the response from elastic in the search tree
00:41:35.979 --> 00:41:36.438 model.
00:41:36.448 --> 00:41:37.898 Additional assignments for categories.
00:41:38.148 --> 00:41:42.968 So we dig into these, UM, uh, these aggregations that came out
00:41:42.968 --> 00:41:46.258 of elastic and it actually tells us which.
00:41:47.548 --> 00:41:51.406 Categories are relevant to the results that we read out from
00:41:51.406 --> 00:41:55.517 elastic that our query returned, so we'll only get back products
00:41:55.517 --> 00:41:58.996 that are actually in or sorry categories that are that
00:41:58.996 --> 00:42:01.968 actually have products in our catalog results.
00:42:02.718 --> 00:42:07.503 Umm, which is again how we how we avoid giving you a 0 product
00:42:07.503 --> 00:42:09.098 match on the catalog.
00:42:11.018 --> 00:42:13.518 So that's happening for all the different things.
00:42:13.528 --> 00:42:16.708 So that's happening for brand names.
00:42:16.718 --> 00:42:20.330 So all the different brand names that products have lands down
00:42:20.330 --> 00:42:23.655 here, which most of them apparently well actually do that
00:42:23.655 --> 00:42:27.152 show again, you see all the different things that are in the
00:42:27.152 --> 00:42:30.248 brand name field on the property, all land in here as
00:42:30.248 --> 00:42:33.688 something I can query by and that's all coming from elastic
00:42:33.688 --> 00:42:36.898 because elastic is saying uh, the brand names for every
00:42:36.898 --> 00:42:40.280 product in these results for every page, not just the page
00:42:40.280 --> 00:42:42.688 I'm on, but every page, all 179 products.
00:42:43.288 --> 00:42:47.012 UM, these are all the matching brand names for those and I can
00:42:47.012 --> 00:42:50.321 filter by any one of those and go look at find one that
00:42:50.321 --> 00:42:52.508 actually is a decent number of them.
00:42:52.518 --> 00:42:54.338 Think John Deere is 18?
00:42:54.348 --> 00:42:54.718 Cool.
00:42:54.868 --> 00:42:58.701 So we click on that we got 18 products now that are John Deere
00:42:58.701 --> 00:42:59.188 related.
00:42:59.928 --> 00:43:00.608 Umm.
00:43:00.938 --> 00:43:03.186 And these are all products that have that and we go look at
00:43:03.186 --> 00:43:05.471 categories and see that these are the relevant categories to
00:43:05.471 --> 00:43:05.658 that.
00:43:05.668 --> 00:43:07.428 That's all coming out of that same thing.
00:43:07.438 --> 00:43:10.361 It's reading them out of the buckets that elastic gives us
00:43:10.361 --> 00:43:10.608 back.
00:43:10.718 --> 00:43:13.608 Elastic is telling us you search for products.
00:43:13.618 --> 00:43:17.521 Here are the additional aggregations you configured that
00:43:17.521 --> 00:43:21.698 are relevant to the results we just provided you, and we use
00:43:21.698 --> 00:43:25.601 that to help basically drill into the results you get as
00:43:25.601 --> 00:43:28.408 you're searching for things and elastic.
00:43:28.418 --> 00:43:32.930 So I'm looking for a John Deere and I am looking for a crawler,
00:43:32.930 --> 00:43:34.128 whatever that is.
00:43:35.118 --> 00:43:35.478 Cool.
00:43:35.488 --> 00:43:37.238 That's this is the product I'm looking for.
00:43:37.248 --> 00:43:38.278 Great, I found what I need.
00:43:38.958 --> 00:43:43.035 Umm, so that kind of thing where we're trying to make it it sort
00:43:43.035 --> 00:43:46.986 of intuitive natural thing that somebody can click through and
00:43:46.986 --> 00:43:50.748 and refine their results just like any other, just like any
00:43:50.748 --> 00:43:51.438 other site.
00:43:51.958 --> 00:43:53.278 Umm, so.
00:43:54.098 --> 00:43:58.516 So that's the that's the majority of the the important
00:43:58.516 --> 00:43:58.998 stuff.
00:43:59.788 --> 00:44:00.748 Uh. Here?
00:44:02.148 --> 00:44:02.658 Umm.
00:44:03.918 --> 00:44:07.449 As a quick side note, suggest module is the part that actually
00:44:07.449 --> 00:44:10.868 goes in and does the uh the quick type ahead results for the
00:44:10.868 --> 00:44:11.428 drop down?
00:44:12.828 --> 00:44:13.638 Same deal here.
00:44:13.648 --> 00:44:17.556 If a client wants this customized, it's probably a
00:44:17.556 --> 00:44:19.088 matter of adjusting.
00:44:19.218 --> 00:44:21.667 This may involve code changes, since there's not as many
00:44:21.667 --> 00:44:23.728 settings in here, but it may involve adjusting.
00:44:23.738 --> 00:44:27.058 Maybe some of these removing some of these fuzzy ones if they
00:44:27.058 --> 00:44:30.163 want less fuzziness, but sometimes refining this can be a
00:44:30.163 --> 00:44:30.698 huge pain.
00:44:30.948 --> 00:44:33.411 It's kind of tough to get through exact result you're
00:44:33.411 --> 00:44:33.958 looking for.
00:44:34.628 --> 00:44:38.364 So again, just making sure that this is something that you
00:44:38.364 --> 00:44:42.417 actually need to do and getting time with with uh, you know me,
00:44:42.417 --> 00:44:46.026 James or somebody who has experience with this, that can
00:44:46.026 --> 00:44:49.889 help you make sure that you're going down the right path and
00:44:49.889 --> 00:44:53.308 and that we're we're you know we're not wasting time.
00:44:53.318 --> 00:44:56.428 We're doing this in a way that actually is necessary.
00:44:57.488 --> 00:44:57.918 Umm.
00:44:58.028 --> 00:45:02.241 So on that note, that's a majority of the back end stuff I
00:45:02.241 --> 00:45:03.598 wanted to dig into.
00:45:04.488 --> 00:45:08.893 Umm and I want to first make sure that nobody has questions
00:45:08.893 --> 00:45:10.728 and then after that once.
00:45:10.738 --> 00:45:11.768 Any questions are answered.
00:45:11.778 --> 00:45:16.678 I wanted to spend some time digging into the dos and don'ts
00:45:16.678 --> 00:45:22.068 of querying with the plastic and using the catalog in general and
00:45:22.068 --> 00:45:24.518 some tips on useful debugging.
00:45:26.168 --> 00:45:30.034 You know, day-to-day stuff for whenever you're we've all had
00:45:30.034 --> 00:45:33.836 times where on a project you're like, I don't know why this
00:45:33.836 --> 00:45:37.765 product it's not showing up in the catalog when we go through
00:45:37.765 --> 00:45:41.694 some of the the tricks you can use to try to help narrow down
00:45:41.694 --> 00:45:42.708 stuff like that.
00:45:42.718 --> 00:45:45.443 So before we go down that way though, it just make sure it
00:45:45.443 --> 00:45:48.353 does anybody have any questions on anything, any, anything you
00:45:48.353 --> 00:45:50.478 want me to double back on or add more detail?
00:46:00.638 --> 00:46:02.048 Sound it's like no.
00:46:01.988 --> 00:46:04.578 He ohh you doing good man.
00:46:02.058 --> 00:46:07.108 Again, ooh, sweet alrighty.
00:46:04.588 --> 00:46:05.138 Keep rolling.
00:46:07.818 --> 00:46:12.220 So I wanna start with a couple of the dos and don'ts and I wish
00:46:12.220 --> 00:46:16.552 we had front Enders here because I've seen a lot more of these
00:46:14.038 --> 00:46:14.298 She.
00:46:16.552 --> 00:46:20.404 Donuts from new front end developers that don't realize
00:46:18.038 --> 00:46:20.398 I'm so then I'm gonna fall.
00:46:20.404 --> 00:46:22.948 that the you're unmuted, by the way.
00:46:22.958 --> 00:46:25.738 Tim, I know if you meant to mute again or not, you're good.
00:46:24.218 --> 00:46:24.928 I'm sorry.
00:46:24.998 --> 00:46:25.648 No, sorry.
00:46:26.268 --> 00:46:29.281 It's all good, but I've seen a lot more of this from new for
00:46:29.281 --> 00:46:32.097 newer front end developers who don't realize the kind of
00:46:32.097 --> 00:46:34.418 possible negative ramifications it could have.
00:46:34.428 --> 00:46:37.745 But I still want to talk through these anyway and hopefully new
00:46:37.745 --> 00:46:38.418 front Enders.
00:46:38.428 --> 00:46:40.438 Will they eventually also see this recording?
00:46:41.048 --> 00:46:44.822 But one of the big things that I've seen in a handful of
00:46:44.822 --> 00:46:48.728 projects recently is front Enders applying filtering to or
00:46:48.728 --> 00:46:49.588 even backend.
00:46:49.598 --> 00:46:52.334 There's applying filtering to the product IDs that get
00:46:52.334 --> 00:46:55.218 returned after they've already come out of Elasticsearch.
00:46:56.258 --> 00:46:59.829 That's a big no, because what that ends up doing at best is
00:46:59.829 --> 00:47:02.983 you get a very bizarre, unintuitive experience where
00:47:02.983 --> 00:47:06.614 you've got six products on this page and the bottom says one
00:47:06.614 --> 00:47:10.422 through 9 of whatever, and the product page size keeps changing
00:47:10.422 --> 00:47:12.148 as you go to different pages.
00:47:12.158 --> 00:47:13.498 You're stripping out random results.
00:47:14.798 --> 00:47:16.968 At best you get unintuitive crap like that.
00:47:17.238 --> 00:47:20.753 At worst, you filter out every product on a page and the code
00:47:20.753 --> 00:47:24.268 gets very confused on the front end because it assumes you're
00:47:24.268 --> 00:47:25.968 gonna have at least something.
00:47:26.218 --> 00:47:28.826 So then you have a blank page and when the page is blank you
00:47:28.826 --> 00:47:29.168 get the.
00:47:29.178 --> 00:47:31.816 Sorry, no results that your pagination disappears and you've
00:47:31.816 --> 00:47:34.108 soft locked the catalog and it doesn't work anymore.
00:47:34.638 --> 00:47:38.555 So if you're going to filter products for catalog results, it
00:47:38.555 --> 00:47:40.008 needs to be in elastic.
00:47:40.018 --> 00:47:42.794 You can't just pick random products to not display after
00:47:42.794 --> 00:47:44.158 they've come out of elastic.
00:47:44.928 --> 00:47:48.153 Not without a hell of a lot of other work to make that happen,
00:47:48.153 --> 00:47:50.098 and it's not worth the time to do it.
00:47:50.108 --> 00:47:51.398 So just don't do it.
00:47:52.108 --> 00:47:55.116 Don't don't arbitrarily remove things from the array of
00:47:55.116 --> 00:47:58.553 products to render on the front end or back end after it's come
00:47:58.553 --> 00:47:59.358 out of elastic.
00:47:59.368 --> 00:48:01.673 If it's, if you need to filter by it, it needs to be put into
00:48:01.673 --> 00:48:02.268 elastic somehow.
00:48:04.368 --> 00:48:06.778 And yes, that's a giant pain in the ***.
00:48:06.788 --> 00:48:10.055 It's unfortunate, but that's that's just the reality of
00:48:10.055 --> 00:48:10.638 situation.
00:48:10.648 --> 00:48:13.229 If you want to not break the website you have to, you have to
00:48:13.229 --> 00:48:14.728 follow the way that it does things.
00:48:15.568 --> 00:48:15.858 No.
00:48:15.918 --> 00:48:17.998 So big thing that we want to avoid.
00:48:18.008 --> 00:48:24.969 There another thing that we want to avoid on catalog stuff that I
00:48:24.969 --> 00:48:31.191 haven't seen as much in some time, but still worth calling
00:48:31.191 --> 00:48:37.729 out is umm, with uh projects that want to like index by stock
00:48:37.729 --> 00:48:39.838 and stuff like that.
00:48:39.888 --> 00:48:42.238 Or by pricing UM.
00:48:42.308 --> 00:48:45.260 And just assuming that it's going to be the same for
00:48:45.260 --> 00:48:48.435 everybody, assuming that it's gonna like, there's always
00:48:48.435 --> 00:48:51.609 something that will happen, especially on projects where
00:48:51.609 --> 00:48:54.338 pricing varies per user, we can sort by pricing.
00:48:54.348 --> 00:48:57.157 We do have that built in, but it just goes by the I believe it
00:48:57.157 --> 00:48:59.698 just goes off the flat price on the product that doesn't
00:48:59.698 --> 00:49:01.748 actually calculate the price to the provider.
00:49:01.758 --> 00:49:02.038 Correct.
00:49:02.558 --> 00:49:05.912 Umm, so that's something to keep in mind is that this is not
00:49:05.912 --> 00:49:09.431 always going to be accurate, and it's extremely computationally
00:49:09.431 --> 00:49:12.950 difficult to make it accurate, because in essence, if you think
00:49:12.950 --> 00:49:16.304 about it, in order to know if any product has a higher price
00:49:16.304 --> 00:49:19.714 than every other product, you have to calculate the price for
00:49:19.714 --> 00:49:20.868 every single product.
00:49:21.878 --> 00:49:25.410 And I don't know if you've ever looked at pricing calculations,
00:49:25.410 --> 00:49:28.391 but they range from extremely extremely simple single
00:49:28.391 --> 00:49:32.033 properties like the flat pricing field all the way up to querying
00:49:32.033 --> 00:49:35.234 across dozens of tables because we have extremely complex
00:49:35.234 --> 00:49:38.656 pricing logic in certain places that allow you to dictate the
00:49:38.656 --> 00:49:40.698 rules in a bajillion different ways.
00:49:42.408 --> 00:49:46.105 So we can't exactly just blanket support crying for every single
00:49:46.105 --> 00:49:49.744 product price all at one time to figure out which one should be
00:49:49.744 --> 00:49:50.938 first in the results.
00:49:51.158 --> 00:49:54.524 So we go with the simple route of filtering on the flat price
00:49:54.524 --> 00:49:55.338 on the product.
00:49:55.408 --> 00:49:58.179 Now, what that might mean is you might get things that look out
00:49:58.179 --> 00:49:59.088 of order to a person.
00:49:59.098 --> 00:50:02.189 They're gonna be probably generally correct, flowing from
00:50:02.189 --> 00:50:05.494 generally highest to generally lowest or vice versa, but they
00:50:05.494 --> 00:50:08.478 each individual one might be transposed here and there.
00:50:08.708 --> 00:50:09.798 That's as good as it's gonna get.
00:50:11.318 --> 00:50:13.945 Similarly with inventory, this is something that comes up a lot
00:50:13.945 --> 00:50:16.449 and the the initial thought process is always ohh just index
00:50:16.449 --> 00:50:17.188 inventory move on.
00:50:17.638 --> 00:50:19.408 The problem with that is what I mentioned earlier.
00:50:19.418 --> 00:50:21.348 The index is a snapshot.
00:50:21.618 --> 00:50:24.699 It's not a live call, so if there's one instance of
00:50:24.699 --> 00:50:28.254 something in stock, you index your product and then someone
00:50:28.254 --> 00:50:30.328 orders it for the next 15 minutes.
00:50:30.338 --> 00:50:34.116 Somebody can search by in stock products and see an out of stock
00:50:34.116 --> 00:50:35.918 product until the index reruns.
00:50:36.858 --> 00:50:41.152 UM, so that's it's something that we don't support out of box
00:50:41.152 --> 00:50:45.377 for that reason, there's been instances of clients that have
00:50:45.377 --> 00:50:49.325 said that's fine and it's something that we end up doing
00:50:49.325 --> 00:50:49.948 for them.
00:50:50.208 --> 00:50:54.368 You know, after giving them that sort of caveat of like, hey,
00:50:54.368 --> 00:50:58.191 just so you know, this is this does have the significant
00:50:58.191 --> 00:51:00.338 drawback, it may be unintuitive.
00:51:00.588 --> 00:51:01.578 It's, you know it.
00:51:01.588 --> 00:51:03.318 That's why we don't support it out of box.
00:51:03.328 --> 00:51:05.278 If it's something you want, we can do it, et cetera.
00:51:05.448 --> 00:51:06.378 But that's a big call.
00:51:06.388 --> 00:51:08.598 Out is that it's not as easy as just.
00:51:08.608 --> 00:51:09.988 Ohh, just index the stock and move on.
00:51:11.348 --> 00:51:12.598 There's more to it than that.
00:51:12.608 --> 00:51:15.515 That has to be considered, and if you're on a project where
00:51:15.515 --> 00:51:18.615 that comes up, there's there's those sort of foot guns you have
00:51:18.615 --> 00:51:21.618 to call out to the client to make sure they're aware of that.
00:51:22.138 --> 00:51:25.947 And if and obviously when I say you have to call it out, I mean
00:51:25.947 --> 00:51:29.576 call it out to the PM, they'll handle the communication with
00:51:29.576 --> 00:51:33.087 the client, but ultimately somebody has to be the one that
00:51:33.087 --> 00:51:35.348 throws up that, that warning sign of.
00:51:35.538 --> 00:51:39.504 There's a reason this isn't already there, and make sure
00:51:39.504 --> 00:51:43.957 that the client is well aware of that and that they don't, that
00:51:43.957 --> 00:51:46.948 they are comfortable with that limitation.
00:51:48.288 --> 00:51:52.622 So uh, but that is if anybody again, if anyone ever wondered
00:51:52.622 --> 00:51:55.818 why we don't filter by inventory by default.
00:51:55.828 --> 00:51:57.898 That's why because it's not consistent.
00:51:57.908 --> 00:51:58.728 It's not stable.
00:52:01.298 --> 00:52:03.964 And again, computationally, potentially computationally
00:52:03.964 --> 00:52:06.963 expensive, to determine how much of a product is in stock when
00:52:06.963 --> 00:52:10.010 you're talking about spreading it across multiple warehouses or
00:52:10.010 --> 00:52:12.485 multiple bins in a single warehouse across multiple
00:52:12.485 --> 00:52:15.437 warehouses across the entire country, and some users may only
00:52:15.437 --> 00:52:17.388 be able to ship from certain warehouses.
00:52:17.398 --> 00:52:20.177 So once again, inventory then becomes dependent on who's
00:52:20.177 --> 00:52:23.346 logged in, and you can't really index a general inventory number
00:52:23.346 --> 00:52:24.028 for a product.
00:52:25.198 --> 00:52:25.768 Umm.
00:52:25.778 --> 00:52:30.790 In that scenario, so again, like most things, it's not as simple
00:52:30.790 --> 00:52:32.948 as it seems at first glance.
00:52:32.958 --> 00:52:36.668 It might be for certain clients who are willing to accept that
00:52:36.668 --> 00:52:40.319 limitation, but because of all the sort of possible foot guns
00:52:40.319 --> 00:52:43.558 there, that's why we don't have it at a box right now.
00:52:44.468 --> 00:52:47.178 So those are a couple of this sort of basic.
00:52:48.438 --> 00:52:50.248 Catalog don'ts?
00:52:50.538 --> 00:52:55.507 Umm, I can't really think of any duos besides just do stuff that
00:52:55.507 --> 00:52:59.864 is sane and logical and don't do stuff that is insane or
00:52:59.864 --> 00:53:00.628 illogical.
00:53:01.378 --> 00:53:02.628 Umm so.
00:53:03.238 --> 00:53:06.198 But any questions on any of that?
00:53:15.068 --> 00:53:16.838 Sounds like no again.
00:53:18.078 --> 00:53:21.402 And no questions, the filtering on the front end though, is a
00:53:21.402 --> 00:53:22.688 definite no no for sure.
00:53:22.188 --> 00:53:22.428 Yep.
00:53:22.698 --> 00:53:24.728 I found that out first hand, it'll break the paging.
00:53:24.738 --> 00:53:26.018 It breaks the whole thing.
00:53:26.358 --> 00:53:27.398 Yep, Yep.
00:53:27.438 --> 00:53:31.151 And obviously we could probably make the front end a little bit
00:53:31.151 --> 00:53:34.748 more robust against breaking if the results is somehow empty.
00:53:34.998 --> 00:53:38.106 But the ultimate the ultimate bottom line for that is we
00:53:38.106 --> 00:53:41.541 shouldn't have to because the results are rate should never be
00:53:41.541 --> 00:53:41.868 empty.
00:53:41.878 --> 00:53:44.919 There shouldn't be a scenario where that happens in the 1st
00:53:44.919 --> 00:53:47.859 place and so we making the catalog more robust against it
00:53:47.859 --> 00:53:50.748 is basically coding against people making bad decisions.
00:53:51.608 --> 00:53:53.938 Uh, and we shouldn't have to do that.
00:53:53.948 --> 00:53:56.377 We should just enforce that those bad decisions don't get
00:53:56.377 --> 00:53:57.298 made in the 1st place.
00:53:58.578 --> 00:53:59.018 Umm.
00:53:59.428 --> 00:54:01.558 So, uh yeah.
00:54:01.608 --> 00:54:06.027 So moving on from the dos and don'ts, some of the things I
00:54:06.027 --> 00:54:10.671 wanted to talk about next were related to troubleshooting and
00:54:10.671 --> 00:54:13.068 understanding what's going on U.
00:54:13.108 --> 00:54:17.288 So I'm going to purposely break ohh man, this isn't on my local.
00:54:17.428 --> 00:54:17.778 I don't know.
00:54:17.788 --> 00:54:18.698 Have my local spun up.
00:54:21.658 --> 00:54:22.248 And it's off.
00:54:22.258 --> 00:54:22.768 Alright, hold on.
00:54:33.908 --> 00:54:37.195 Spent my local up and I'm going to go purposely break elastic
00:54:37.195 --> 00:54:37.778 real quick.
00:54:38.028 --> 00:54:41.157 My turning the service off on my local and showing what that
00:54:41.157 --> 00:54:43.568 might look like if there's a connection issue.
00:54:48.908 --> 00:54:53.548 OK, I'll wait for my uh local to spin up real quick.
00:55:09.178 --> 00:55:09.688 There we go.
00:55:09.938 --> 00:55:14.756 Alright, so I'm gonna jump over the catalog and we're going to
00:55:14.756 --> 00:55:19.268 see that whenever I go to the catalog, it's going to fail.
00:55:22.918 --> 00:55:25.408 You see that this query call is hanging.
00:55:25.538 --> 00:55:26.468 It's still pending.
00:55:26.478 --> 00:55:27.998 It's taking a while to come back.
00:55:29.418 --> 00:55:30.998 Umm, suspect.
00:55:31.108 --> 00:55:31.708 Why could that be?
00:55:37.068 --> 00:55:37.428 Come on.
00:55:39.648 --> 00:55:42.641 OK, funny that it gave me a 200, but there will be an error in
00:55:42.641 --> 00:55:42.878 here.
00:55:42.948 --> 00:55:44.398 Yeah, on the debug information.
00:55:44.568 --> 00:55:46.708 So you can see that there's an invalid nest response.
00:55:45.378 --> 00:55:48.838 Wonderful overlap on the buttons at the top with the message
00:55:48.838 --> 00:55:49.178 there.
00:55:49.598 --> 00:55:51.808 Yeah, we need to add some padding on to this thing.
00:55:51.978 --> 00:55:57.486 It's weird that that they've been having because I looked at
00:55:57.486 --> 00:56:03.265 the the the stuff for this and it's oops, it's a call 12 inside
00:56:03.265 --> 00:56:07.238 a row inside the grid body inside a column.
00:56:07.248 --> 00:56:10.275 So there's a row, a row and then a grid body with a row and
00:56:10.275 --> 00:56:13.302 another row, and I'm assuming this grid body thing has like
00:56:13.302 --> 00:56:15.068 negative top padding or something.
00:56:14.808 --> 00:56:18.758 That that grid Brooke body is supposed to be a row, that that
00:56:18.758 --> 00:56:20.478 should be where the row is.
00:56:20.488 --> 00:56:22.228 I don't know why there's a grid body in the way.
00:56:24.708 --> 00:56:24.888 Yeah.
00:56:24.768 --> 00:56:27.255 That's what's causing the problem, because the CSS for
00:56:27.255 --> 00:56:30.103 bootstrap is gonna be looking for a row followed by a row, not
00:56:30.103 --> 00:56:31.278 row followed by grid body.
00:56:31.408 --> 00:56:32.648 But it has a row inside it.
00:56:31.658 --> 00:56:32.438 That's what I figured.
00:56:34.278 --> 00:56:35.288 Yeah, that fixes it.
00:56:34.648 --> 00:56:38.942 You could see that the row but the the calls that are in there
00:56:35.298 --> 00:56:36.678 Starting row onto that, yeah.
00:56:38.942 --> 00:56:43.236 are what need to be on that grid body and then strip the inner
00:56:41.718 --> 00:56:41.878 Yeah.
00:56:43.236 --> 00:56:43.508 row.
00:56:43.518 --> 00:56:44.358 Yeah, it gets.
00:56:44.368 --> 00:56:46.398 It's gonna be ugly until it's fully fixed, but.
00:56:45.838 --> 00:56:46.528 Yeah.
00:56:46.738 --> 00:56:48.148 Anyway, yeah.
00:56:48.158 --> 00:56:51.650 Anyway, looking at the error, I can see that I got an invalid
00:56:51.650 --> 00:56:52.438 nest response.
00:56:53.598 --> 00:56:54.968 Uh, we can scroll through this.
00:56:54.978 --> 00:56:57.702 It's kind of annoying that it, like there's no word wrap or
00:56:57.702 --> 00:56:59.608 anything, but just keep rolling this way.
00:57:00.118 --> 00:57:01.998 Bad request local hosts.
00:57:02.968 --> 00:57:04.398 You know, that's the path that tried to hit.
00:57:04.478 --> 00:57:07.334 We keep rolling, eventually see unable to connect to remote
00:57:07.334 --> 00:57:08.238 server status code.
00:57:08.788 --> 00:57:09.508 Unknown.
00:57:10.738 --> 00:57:14.032 Uh, no connection could be made because the target machine
00:57:14.032 --> 00:57:15.148 actively refused it.
00:57:15.458 --> 00:57:19.327 So you might see this on maybe on a QA site where nobody
00:57:19.327 --> 00:57:23.535 pointed Elasticsearch to the right box or on a new production
00:57:23.535 --> 00:57:27.879 site where it's not configured to point to the right, the right
00:57:27.879 --> 00:57:29.508 IP or port or something.
00:57:30.178 --> 00:57:34.358 Umm, but this is the message you'll see somewhere in here.
00:57:34.368 --> 00:57:37.410 If you're pointing to the wrong thing or elastic search is off
00:57:37.410 --> 00:57:38.858 on the box you're pointing at.
00:57:39.088 --> 00:57:43.340 Obviously for for this sample I just turned Elasticsearch
00:57:43.340 --> 00:57:47.299 service off completely in my uh services here just to
00:57:47.299 --> 00:57:48.398 demonstrate UM.
00:57:48.648 --> 00:57:50.648 And so we'll turn it back on.
00:57:52.198 --> 00:57:53.268 Takes a minute usually.
00:57:57.038 --> 00:57:59.048 And then refresh this page once it's done.
00:57:59.458 --> 00:58:00.568 Should be very soon.
00:58:01.178 --> 00:58:04.238 Probably man, it takes a long time.
00:58:04.248 --> 00:58:04.708 There we go.
00:58:04.948 --> 00:58:09.444 Now I should be able to just refresh this page and the query
00:58:09.444 --> 00:58:10.918 call no longer dies.
00:58:11.068 --> 00:58:12.378 Gives me back actual data.
00:58:12.488 --> 00:58:16.121 There's my stuff so that no connection could be made error
00:58:16.121 --> 00:58:18.338 or something you might come across.
00:58:18.348 --> 00:58:21.915 If you're standing up a new QA stage or production site and you
00:58:21.915 --> 00:58:25.426 haven't correctly configured the IP and the port that the that
00:58:25.426 --> 00:58:28.826 elastic is supposed to connect to, so something to watch out
00:58:28.826 --> 00:58:32.114 for if you're setting up new sites or instances, or if you
00:58:32.114 --> 00:58:35.625 have any issues on your local and you're seeing that, it might
00:58:35.625 --> 00:58:37.408 mean that it lastic is just off.
00:58:38.818 --> 00:58:42.342 So you can check that in the services snap in here quick way
00:58:42.342 --> 00:58:45.981 to get to the services dot MSC and it opens up this little guy
00:58:45.981 --> 00:58:49.158 and you can just find Elasticsearch and make sure that
00:58:49.158 --> 00:58:52.450 it started something you might see if you have the wrong
00:58:52.450 --> 00:58:55.916 version of Java installed is if you start the Elasticsearch
00:58:55.916 --> 00:58:58.688 service and it instantly stops after you start.
00:58:58.698 --> 00:59:02.693 It means you probably have the wrong version of Java installed
00:59:02.693 --> 00:59:06.370 and at that point just ask somebody and they can help you
00:59:06.370 --> 00:59:07.828 find the right version.
00:59:08.698 --> 00:59:09.508 I forget what it is.
00:59:09.518 --> 00:59:11.798 It's like 1.8 point one or something like that.
00:59:12.158 --> 00:59:15.138 But there's a specific version we have to use of Java to
00:59:15.138 --> 00:59:18.379 support this older version of Elastic Search we're on, and if
00:59:18.379 --> 00:59:21.724 you have the wrong like a newer version installed, it can throw
00:59:21.724 --> 00:59:22.508 out for a loop.
00:59:22.558 --> 00:59:25.558 So and by throw it for a loop, I mean crash it completely.
00:59:26.658 --> 00:59:28.808 So just another thing to watch out for.
00:59:30.568 --> 00:59:33.311 So those are kind of just some of the basic environment or
00:59:33.311 --> 00:59:33.868 setup stuff.
00:59:34.538 --> 00:59:38.698 Uh, then moving on to actual querying.
00:59:40.078 --> 00:59:42.268 And indexing.
00:59:42.598 --> 00:59:49.858 So let's say that I've added some products onto QA site.
00:59:49.868 --> 00:59:51.538 Or maybe I synced?
00:59:51.548 --> 00:59:54.689 Or maybe a connect developer sync them into a QA site and the
00:59:54.689 --> 00:59:57.778 client or where like hey, we can't see these on the catalog.
00:59:57.828 --> 00:59:58.548 What's going on?
01:00:00.258 --> 01:00:04.038 Uh, so when we were looking at the I don't Reeder earlier, one
01:00:04.038 --> 01:00:07.578 of the things that I pointed out in here is that we have a
01:00:07.578 --> 01:00:09.258 handful of built-in filters.
01:00:10.308 --> 01:00:13.301 We filter by products that are active, visible and not
01:00:13.301 --> 01:00:14.008 discontinued.
01:00:14.828 --> 01:00:19.112 I also mentioned that by default the requiring a category is
01:00:19.112 --> 01:00:23.185 true, so if we go and look at how it determines if it has
01:00:23.185 --> 01:00:27.398 categories, we want to go up here and look and see and find
01:00:27.398 --> 01:00:31.822 has categories up here it finds any category where the product
01:00:31.822 --> 01:00:33.648 category record is active.
01:00:33.758 --> 01:00:37.116 The category itself is active and visible and has the included
01:00:37.116 --> 01:00:38.288 menu flag set to true.
01:00:39.168 --> 01:00:41.008 So what?
01:00:41.018 --> 01:00:44.803 That all boils down to is that for a product to display in the
01:00:44.803 --> 01:00:48.707 catalog, the product itself has to be active and visible and not
01:00:48.707 --> 01:00:49.488 discontinued.
01:00:49.818 --> 01:00:53.823 It also has to be assigned to a product or sorry to a category
01:00:53.823 --> 01:00:57.637 where the product category record that joins them is active
01:00:57.637 --> 01:01:01.450 and the category itself is active is visible and include in
01:01:01.450 --> 01:01:01.768 menu.
01:01:02.158 --> 01:01:05.228 Those are the out of box default requirements for a product to
01:01:05.228 --> 01:01:06.348 show up in the catalog.
01:01:06.398 --> 01:01:09.966 If you're on a custom, if you're on a client project, there may
01:01:09.966 --> 01:01:13.200 be other ones that have been added, for example, like the
01:01:13.200 --> 01:01:15.318 requiring a lot for auction products.
01:01:15.328 --> 01:01:18.276 Or maybe you're on a brand brand site and requires a brand as
01:01:18.276 --> 01:01:19.988 enabled and all that kind of stuff.
01:01:20.528 --> 01:01:21.028 Umm.
01:01:22.138 --> 01:01:23.328 And then those are additional things.
01:01:23.338 --> 01:01:23.808 You'll wanna.
01:01:23.818 --> 01:01:27.736 You'll wanna keep in mind and then same with like the the type
01:01:27.736 --> 01:01:31.716 key filter that we talked about, but those those things are the
01:01:31.716 --> 01:01:35.696 bottom level base requirements for a for a complete vanilla CEF
01:01:35.696 --> 01:01:38.308 site to display a product in the catalog.
01:01:38.478 --> 01:01:42.484 And what you can do is, I mean obviously you could attach if
01:01:42.484 --> 01:01:46.556 you wanted to and look at what comes back in the dump reader,
01:01:46.556 --> 01:01:50.694 you can just put a break point in there if you wanted to do it
01:01:50.694 --> 01:01:54.700 a sort of uh brute force, see slow way or maybe it's on a QA
01:01:54.700 --> 01:01:57.918 site and you don't have the luxury of attaching.
01:01:58.628 --> 01:02:01.862 UM, in that case, what you can do is you can go and look at
01:02:01.862 --> 01:02:02.508 where is it.
01:02:05.358 --> 01:02:08.836 This one, the one I'm using right now, you can actually go
01:02:08.836 --> 01:02:10.368 in here and just query it.
01:02:10.378 --> 01:02:13.100 Basically, recreate the query that elastic is going to use to
01:02:13.100 --> 01:02:15.910 determine which ones are sorry and if framework is going to use
01:02:15.910 --> 01:02:17.798 to determine which ones come back on that.
01:02:18.628 --> 01:02:23.063 So for now I'll just do P dot star from products dot product P
01:02:23.063 --> 01:02:27.498 so this is just selecting every product in my database, all of
01:02:27.498 --> 01:02:29.258 them, any number of them.
01:02:29.308 --> 01:02:30.718 Everything that's in here. Cool.
01:02:33.138 --> 01:02:36.148 So let's start by just imposing the product specific filters.
01:02:37.378 --> 01:02:39.198 So it has to be active.
01:02:40.468 --> 01:02:44.968 It has to be is visible and it cannot be discontinued.
01:02:46.158 --> 01:02:50.431 So I'm starting out with 175 records, applying that already
01:02:50.431 --> 01:02:52.068 filtered something out.
01:02:52.078 --> 01:02:55.154 There was something that was either not active, not visible,
01:02:55.154 --> 01:02:56.868 or was discontinued in that list.
01:02:57.078 --> 01:03:00.684 So that's already one product that's not going to display.
01:03:00.684 --> 01:03:00.928 Umm.
01:03:03.418 --> 01:03:04.028 OK.
01:03:04.078 --> 01:03:07.528 Well, the next thing was it has to be part of a category.
01:03:07.628 --> 01:03:13.094 So we're gonna go ahead and add an inner join for products
01:03:13.094 --> 01:03:18.837 product category on PC's master ID equal to E dot ID and then
01:03:18.837 --> 01:03:21.338 the actual category itself.
01:03:22.118 --> 01:03:27.233 The queries that category and PC's anybody equals and I think
01:03:27.233 --> 01:03:28.388 that an alias.
01:03:30.478 --> 01:03:35.054 OK, so now we also want to find where the product category
01:03:35.054 --> 01:03:40.019 record the join record is active and the category is active and
01:03:40.019 --> 01:03:44.827 the category is visible and the category has included in menu
01:03:44.827 --> 01:03:45.758 set to true.
01:03:46.568 --> 01:03:51.513 This will be the the default products that should be indexed
01:03:51.513 --> 01:03:56.619 in the catalog on a standard set site, and I don't think it'll
01:03:56.619 --> 01:03:58.078 index, uh hang on.
01:04:01.328 --> 01:04:01.738 Yeah.
01:04:01.778 --> 01:04:04.088 So yeah, so it trimmed out a little bit more.
01:04:04.248 --> 01:04:07.383 So that means that I had a handful of products that were
01:04:07.383 --> 01:04:10.683 valid on the product table but were not assigned to a valid
01:04:10.683 --> 01:04:11.178 category.
01:04:11.188 --> 01:04:14.121 Here they weren't assigned to a category at all, or the category
01:04:14.121 --> 01:04:16.828 they're in was inactive, not visible, et cetera or whatever
01:04:16.828 --> 01:04:17.098 it is.
01:04:18.548 --> 01:04:20.968 And so ultimately that will tell me what I need right here.
01:04:21.708 --> 01:04:26.201 Umm, I'm too lazy to do it right now, but theoretically you could
01:04:26.201 --> 01:04:30.148 turn this query on its head and and you know invert every
01:04:30.148 --> 01:04:34.232 condition here to look for like not active, not visible and
01:04:34.232 --> 01:04:38.588 change all these to or instead of and and find specifically the
01:04:38.588 --> 01:04:40.698 products that won't be indexed.
01:04:41.808 --> 01:04:42.388 Umm.
01:04:43.338 --> 01:04:46.398 And then you can kind of start to dig into why those are why
01:04:46.398 --> 01:04:49.507 those are missing from the query or missing from the from the
01:04:49.507 --> 01:04:49.808 index.
01:04:49.818 --> 01:04:50.068 What?
01:04:50.078 --> 01:04:51.978 What flag needs to be adjusted or whatever?
01:04:52.618 --> 01:04:56.485 Umm, I'll paste this little query in the chat for anybody
01:04:56.485 --> 01:04:57.618 that wants to uh.
01:04:57.928 --> 01:04:58.428 Save that.
01:05:02.438 --> 01:05:02.928 Yeah.
01:05:02.938 --> 01:05:06.228 Find the codes. Never.
01:05:06.238 --> 01:05:09.298 There we go and it is squeakquel.
01:05:11.988 --> 01:05:15.147 Just in case anyone's ever troubleshooting and wants to
01:05:15.147 --> 01:05:18.418 just copy paste that and again like I said, you can also.
01:05:18.768 --> 01:05:21.258 You can also expand this to support other things.
01:05:21.268 --> 01:05:24.278 So if yeah, maybe you have to type filtering on.
01:05:24.288 --> 01:05:32.049 You can enter join products dot product type PT on dot paper ID
01:05:32.049 --> 01:05:39.204 equals PT dot ID and you can learn here and say and PT dot
01:05:39.204 --> 01:05:40.538 name in uh.
01:05:41.188 --> 01:05:44.879 Variant master or general and that would make it so that
01:05:44.879 --> 01:05:48.958 you're also only filtering by products that are variant master
01:05:48.958 --> 01:05:52.649 or general, and so again that would filter out even more
01:05:52.649 --> 01:05:54.008 products on my local.
01:05:54.758 --> 01:05:57.281 So if you had that type filtering on and and these were
01:05:57.281 --> 01:05:59.985 the two that you set up, then that might tell you even more
01:05:59.985 --> 01:06:02.778 reason why certain things are not showing up in your catalog.
01:06:02.818 --> 01:06:07.057 So you can kind of build on to this query additional filters
01:06:07.057 --> 01:06:11.227 that are being applied by the logic and try to deduce or or
01:06:11.227 --> 01:06:13.728 find why your products are missing.
01:06:13.738 --> 01:06:18.749 You can comment out individual little conditions one at a time
01:06:18.749 --> 01:06:20.578 on this and figure out.
01:06:20.628 --> 01:06:23.338 Ohh, it's because the category has is visible.
01:06:23.348 --> 01:06:26.507 Set defaults for some reason or whatever it is, so that can help
01:06:26.507 --> 01:06:29.325 you kind of narrow down why you're not seeing the product
01:06:29.325 --> 01:06:30.248 you're looking for.
01:06:31.308 --> 01:06:35.543 Umm, another trick you can use is if your query seems to be
01:06:35.543 --> 01:06:38.648 working OK, but you're still not seeing it.
01:06:38.838 --> 01:06:41.878 Something I like to do just at that point, since I've I've you
01:06:41.878 --> 01:06:45.062 know, there's a lot of data here to be hard to build a query that
01:06:45.062 --> 01:06:46.558 supports absolutely everything.
01:06:47.538 --> 01:06:49.208 Umm is I'll.
01:06:49.258 --> 01:06:50.648 I'll throw a breakpoint inside the.
01:06:50.658 --> 01:06:53.688 Skip this early out right here, because if it gets past this,
01:06:53.688 --> 01:06:54.958 there's nothing else that.
01:06:55.058 --> 01:06:57.886 Well, there is technically this that earlies out, but that it's
01:06:57.886 --> 01:06:58.858 a very specific thing.
01:06:58.868 --> 01:07:02.409 You would probably be aware of if you were on the project and
01:07:02.409 --> 01:07:05.951 hasn't really been used in a long time that I've seen, but if
01:07:05.951 --> 01:07:09.377 it gets past this early out, generally speaking the product
01:07:09.377 --> 01:07:12.976 gets sent into elastic at that point, not including this other
01:07:12.976 --> 01:07:13.718 specific one.
01:07:14.528 --> 01:07:18.021 So, uh, I like to put the break point right here inside this if
01:07:18.021 --> 01:07:21.568 we can put it on the continue or on the open bracket either way.
01:07:22.088 --> 01:07:22.598 Umm.
01:07:23.008 --> 01:07:26.390 And that breakpoint will only be hit if a product was loaded from
01:07:26.390 --> 01:07:29.208 the database, but didn't pass one of these conditions.
01:07:30.228 --> 01:07:30.738 Umm.
01:07:30.808 --> 01:07:33.954 And then what you can do is you can attach to if you're clicking
01:07:33.954 --> 01:07:36.858 the index button in the admin, you can attach to the admin.
01:07:37.068 --> 01:07:40.337 If you're running the scheduled task in the scheduler, you
01:07:40.337 --> 01:07:43.882 detach the scheduler at pool and then you run it, and when this
01:07:43.882 --> 01:07:46.818 breakpoint gets hit, you can mouse over the version.
01:07:46.828 --> 01:07:49.586 Here you can add it to your watch list down there in the
01:07:49.586 --> 01:07:52.682 debugger and you can go see what product ID it is and which you
01:07:52.682 --> 01:07:55.584 can kind of mouse over each of these and see which of these
01:07:55.584 --> 01:07:56.358 doesn't line up.
01:07:56.368 --> 01:08:00.597 What's missing that's causing it to skip so that can be if
01:08:00.597 --> 01:08:01.098 you're.
01:08:01.168 --> 01:08:03.748 If you're dealing with an issue and you can't seem to figure it
01:08:03.748 --> 01:08:06.409 out through SQL, that's a little bit more of a brute force way to
01:08:06.409 --> 01:08:08.828 jump into the actual code and see if you can figure it out.
01:08:09.758 --> 01:08:10.218 Umm.
01:08:11.258 --> 01:08:14.788 So another another useful tip there? Umm.
01:08:17.848 --> 01:08:21.746 I uh uh, I don't know if I can think of anything else off the
01:08:21.746 --> 01:08:23.318 top of my head right now.
01:08:23.398 --> 01:08:24.588 That's probably more stuff.
01:08:24.688 --> 01:08:26.478 Anybody else have other elastic stuff?
01:08:26.488 --> 01:08:29.928 They've Elasticsearch shaped Rakes.
01:08:29.938 --> 01:08:32.438 They've stepped on that, have flew up and hit him in the face.
01:08:33.768 --> 01:08:35.108 Scooby Doo villain style.
01:08:39.188 --> 01:08:42.618 I had some elastic updates to do the other day.
01:08:42.928 --> 01:08:46.448 Actually, we just finished it up earlier this morning, but it was
01:08:46.448 --> 01:08:49.968 mostly due to signal R not being set up right, which isn't really
01:08:49.968 --> 01:08:51.408 specific to elastic search.
01:08:53.048 --> 01:08:55.568 Yeah, that's interesting.
01:08:55.638 --> 01:08:58.198 I I'm not sure why you'd have to change anything in elastic for
01:08:58.198 --> 01:08:58.838 signalr to work.
01:08:59.578 --> 01:09:00.098 Umm.
01:09:00.188 --> 01:09:01.018 Is that on tin bids?
01:09:02.028 --> 01:09:02.528 Yeah.
01:09:02.538 --> 01:09:05.298 The products in the catalog weren't loading properly.
01:09:07.068 --> 01:09:10.027 In Ohh so not actually related to anything in elastic but just
01:09:10.027 --> 01:09:12.658 related probably to like the bidding factory failing to
01:09:12.658 --> 01:09:15.429 connect to signalr and then refusing to render products or
01:09:15.429 --> 01:09:18.295 something because they were waiting for the thing to come up
01:09:18.295 --> 01:09:18.858 or whatever.
01:09:19.358 --> 01:09:19.698 Yeah.
01:09:20.208 --> 01:09:22.108 Most probably that alright.
01:09:22.118 --> 01:09:24.990 So it's not actually related to like any elastic search,
01:09:24.990 --> 01:09:27.408 querying or filtering or anything like that so.
01:09:32.078 --> 01:09:32.298 Cool.
01:09:39.998 --> 01:09:41.458 Well, uh.
01:09:43.158 --> 01:09:43.668 Livana.
01:09:45.318 --> 01:09:47.528 Uh, yeah, I never use Kibana.
01:09:48.058 --> 01:09:52.067 Uh, I'm I'm normally if I get to a point where I've determined
01:09:52.067 --> 01:09:56.013 stuff is sending something into elastic and it's not correct,
01:09:56.013 --> 01:09:59.258 I'm just going to try to find it in the safe side.
01:10:00.558 --> 01:10:04.368 But you you can in fact use Kibana.
01:10:04.378 --> 01:10:06.298 I think I have it downloaded somewhere in here.
01:10:08.248 --> 01:10:11.878 Data tools come on, that's installer.
01:10:13.918 --> 01:10:14.658 Uh, cabanas.
01:10:14.668 --> 01:10:21.264 A tool you can use to inspect the data that actually gets put
01:10:21.264 --> 01:10:23.498 into uh, elastic and.
01:10:26.378 --> 01:10:29.328 And the shape of the index and all that different stuff.
01:10:29.898 --> 01:10:32.788 Uh, give me a moment to remember how to ruin it.
01:10:32.898 --> 01:10:34.758 I guess it's probably just cabana.bat.
01:10:48.578 --> 01:10:49.818 It sure is thinking.
01:10:52.598 --> 01:10:53.718 It takes a while to start up.
01:10:54.318 --> 01:10:54.498 Yeah.
01:10:55.058 --> 01:10:59.101 Yeah, there's also the error you get from the catalog saying that
01:10:59.101 --> 01:10:59.958 the parameter.
01:11:01.688 --> 01:11:04.218 S is null or whatever it's talking about.
01:11:04.228 --> 01:11:08.933 The search index in the app settings not being correct or
01:11:08.933 --> 01:11:12.258 missing the pool, and I'm talking about.
01:11:12.268 --> 01:11:14.708 We're still like parameter ***** and all.
01:11:13.448 --> 01:11:13.688 Yeah.
01:11:17.758 --> 01:11:21.018 Umm, that's a that's a good one to check.
01:11:21.168 --> 01:11:24.186 Uh app settings wise, whenever you're setting stuff up is just
01:11:24.186 --> 01:11:26.868 making sure that your product index is correctly named.
01:11:27.888 --> 01:11:32.434 Umm every any index but the indexes are all formatted at a
01:11:32.434 --> 01:11:37.288 specific way, which is that for example product indexes always
01:11:37.288 --> 01:11:40.678 have to start with product search dash umm.
01:11:41.148 --> 01:11:45.525 And generally speaking, they're formatted such that they're
01:11:45.525 --> 01:11:50.121 pulled by file real quick and show you I should probably be in
01:11:50.121 --> 01:11:52.018 the environment one, yeah.
01:11:52.108 --> 01:11:55.673 So for example, let's product dash, search, Dash, site
01:11:55.673 --> 01:11:57.358 acronym, Dash Environment.
01:11:57.468 --> 01:12:02.026 So for example, tin bids, QA is going to be product dash, search
01:12:02.026 --> 01:12:06.023 Dash 10, dash, QA all lower case, no spaces, just dashes
01:12:06.023 --> 01:12:08.758 between the things and the part right.
01:12:08.828 --> 01:12:13.019 Here is the most important because this is how it actually
01:12:13.019 --> 01:12:16.358 knows which one to do on the the indexer side.
01:12:16.888 --> 01:12:20.489 Whenever you tell it to do to index products, it as far as I
01:12:20.489 --> 01:12:24.327 remember it just passes in like product dash search and goes and
01:12:24.327 --> 01:12:27.278 finds that matching index and does all the stuff.
01:12:28.568 --> 01:12:31.664 So important to keep in mind that this isn't just a
01:12:31.664 --> 01:12:34.878 convention that we follow for the for the heck of it.
01:12:34.888 --> 01:12:38.251 There's actual code enforcement behind this, so keep keep that
01:12:38.251 --> 01:12:38.678 in mind.
01:12:39.088 --> 01:12:42.428 It needs to follow this convention or stuff won't work.
01:12:46.818 --> 01:12:48.548 As it could good reminder.
01:12:48.928 --> 01:12:49.358 Thanks Tim.
01:12:53.388 --> 01:12:53.888 Let's see.
01:12:55.528 --> 01:12:56.458 And they could spun up.
01:12:56.908 --> 01:13:00.278 Yep, and it's running localhost 56.
01:13:00.288 --> 01:13:03.608 01 so we go to local hosts 5601.
01:13:05.398 --> 01:13:08.388 And here we are.
01:13:09.258 --> 01:13:15.031 And in here now this sort of allows us to look at different
01:13:15.031 --> 01:13:15.608 stuff.
01:13:17.038 --> 01:13:21.594 Like I said, I rarely use this, so I never really remember what
01:13:19.388 --> 01:13:21.848 The only thing I ever go to is dev tools.
01:13:21.594 --> 01:13:22.448 to do. Yeah.
01:13:26.238 --> 01:13:27.548 And then collapse the site.
01:13:27.558 --> 01:13:30.162 The blue side bar and then you never look at any of the ones
01:13:30.162 --> 01:13:30.418 again.
01:13:32.618 --> 01:13:36.556 Yeah, I like my my product indexes, product search, dev,
01:13:34.448 --> 01:13:34.778 Alright.
01:13:36.556 --> 01:13:40.493 local and look at that it even it even autocompletes the
01:13:40.493 --> 01:13:41.598 indices in here.
01:13:43.238 --> 01:13:43.848 Uh.
01:13:43.858 --> 01:13:45.858 And then I guess it probably product index for model.
01:13:44.058 --> 01:13:46.308 He Seth configs repo.
01:13:46.368 --> 01:13:49.815 One of the environment setup files is Kibana query dot text
01:13:49.815 --> 01:13:53.377 to give you the the default set of stuff that you would wanna
01:13:53.377 --> 01:13:57.054 put in here, including the top two queries which you run at the
01:13:57.054 --> 01:13:57.628 same time.
01:13:57.858 --> 01:14:01.638 Tell you what, all the indexes are and all the aliases are.
01:14:05.758 --> 01:14:06.348 Look at that.
01:14:06.758 --> 01:14:12.788 So I I just searched the index with no queries, nothing passed
01:14:12.788 --> 01:14:18.147 in and it gave me back some stuff give me back hits for
01:14:18.147 --> 01:14:19.008 whatnots.
01:14:21.028 --> 01:14:21.568 Umm.
01:14:22.498 --> 01:14:24.258 166 hits in fact.
01:14:25.338 --> 01:14:25.798 Umm.
01:14:27.288 --> 01:14:30.457 And each one of these is one of the products that it managed to
01:14:30.457 --> 01:14:31.298 read back for me.
01:14:32.498 --> 01:14:37.443 And I can go and see the short description for it and some of
01:14:37.443 --> 01:14:41.988 the suggest stuff that's passed in, it's ID 7, etcetera.
01:14:42.128 --> 01:14:45.967 So this will actually show me what got put into elastic when
01:14:45.967 --> 01:14:48.798 all is said and done after the index is run.
01:14:49.088 --> 01:14:51.698 Just this single simple little query right here.
01:14:51.808 --> 01:14:55.048 Just basically dumps everything you passed in more or less.
01:14:56.268 --> 01:14:56.778 Umm.
01:14:57.188 --> 01:15:00.901 And you could see what what you're working with, so you can
01:15:00.901 --> 01:15:01.458 actually.
01:15:01.468 --> 01:15:04.607 It's kinda you can see how it does some of the like analysis
01:15:04.607 --> 01:15:05.018 on this.
01:15:05.318 --> 01:15:08.609 The on this thing right here is it like splits on spaces and
01:15:08.609 --> 01:15:11.791 stuff like that and kind of builds up this set of relevant
01:15:11.791 --> 01:15:14.758 words and then at the end has the full relevant thing.
01:15:16.228 --> 01:15:21.149 Umm, with the full string to search inside of as kind of a
01:15:21.149 --> 01:15:24.818 last resort I guess, but that's kinda cool.
01:15:24.828 --> 01:15:28.033 You can see that it's like basically split it all up and
01:15:28.033 --> 01:15:29.438 gave all the basic words.
01:15:30.738 --> 01:15:34.242 Umm, we used to also have or maybe something that we just
01:15:34.242 --> 01:15:38.109 need to configure, but you can also configure it to ignore what
01:15:38.109 --> 01:15:39.438 are called stop words.
01:15:39.448 --> 01:15:46.960 So that stuff like and and the and of words that don't like
01:15:46.960 --> 01:15:49.088 proper searching.
01:15:49.338 --> 01:15:50.048 Yeah.
01:15:50.258 --> 01:15:53.588 Etiquette as to omit those words from searches anyway, because
01:15:53.588 --> 01:15:57.024 they're they're gonna come up in everything, everywhere, all the
01:15:57.024 --> 01:15:57.288 time.
01:15:57.338 --> 01:16:00.288 It's just noise in your search and so.
01:16:02.028 --> 01:16:06.557 This kind of strips that out so that you don't get stuff coming
01:16:06.557 --> 01:16:07.618 up as matching.
01:16:08.578 --> 01:16:10.268 Maybe you searched the word theme.
01:16:10.738 --> 01:16:14.011 Uh, if you strip out all the instances of the, then you're
01:16:14.011 --> 01:16:17.339 not going to get as much overlap on the relevant stuff just
01:16:17.339 --> 01:16:20.668 because it has the word the somewhere in its description or
01:16:20.668 --> 01:16:23.108 its name or whatever, or similar with like.
01:16:23.118 --> 01:16:24.538 If you're searching for like Ant.
01:16:25.718 --> 01:16:30.465 And you strip out the word, and uh, then you're not gonna get as
01:16:30.465 --> 01:16:32.218 much overlap for things.
01:16:32.218 --> 01:16:33.448 To just have that word in them.
01:16:33.878 --> 01:16:38.367 So it's words that are likely to cause overlap or have little
01:16:38.367 --> 01:16:39.598 issues like that.
01:16:39.898 --> 01:16:43.027 They call them stop words and you can actually configure
01:16:43.027 --> 01:16:46.541 elastic to automatically strip them out as part of the analysis
01:16:46.541 --> 01:16:49.834 process, which can help your results be a little bit better
01:16:49.834 --> 01:16:50.438 one second.
01:16:58.288 --> 01:16:59.018 Uh, yeah.
01:16:59.128 --> 01:17:05.030 So if anybody's curious of where you can get this cabana thing
01:17:05.030 --> 01:17:11.026 from, it is in the Elasticsearch zip that we that you should be
01:17:11.026 --> 01:17:15.148 using to install elastic on your locals it.
01:17:15.198 --> 01:17:16.718 And if you don't know where that is.
01:17:15.918 --> 01:17:18.568 And it's in the uh self configs.
01:17:19.148 --> 01:17:20.748 Yeah, that is.
01:17:20.788 --> 01:17:22.138 Ohh it's in self config now, that's.
01:17:23.358 --> 01:17:27.078 Yeah, it's been there for a while under environment setup.
01:17:25.028 --> 01:17:25.278 Yeah.
01:17:28.038 --> 01:17:29.358 There's a cabana query text.
01:17:30.128 --> 01:17:34.022 As the Kibana query text, uh talking about the the actual
01:17:34.022 --> 01:17:36.908 like full elastic search setup 7 zip file.
01:17:38.168 --> 01:17:38.768 Oh that.
01:17:38.778 --> 01:17:39.558 Oh yeah, I don't.
01:17:39.568 --> 01:17:41.568 I don't have that in there cause it's a path, a gig.
01:17:41.508 --> 01:17:41.918 It's.
01:17:41.928 --> 01:17:42.998 Yeah, it's gigantic.
01:17:43.008 --> 01:17:45.168 Yeah, I was like, wow, I'm surprised that's in there.
01:17:45.388 --> 01:17:45.838 And you like?
01:17:45.848 --> 01:17:46.568 It's in there for a while.
01:17:46.578 --> 01:17:48.438 I was like, wow, have I not noticed that?
01:17:49.228 --> 01:17:55.213 OK, so anyway, that file I believe is the link to it in the
01:17:55.213 --> 01:18:01.298 self configs is and I'm sorry I'm getting called on discord.
01:18:01.308 --> 01:18:03.428 Let me kill that real quick one moment.
01:18:13.298 --> 01:18:13.668 OK.
01:18:13.748 --> 01:18:18.375 They they decided to just do it long enough to to get on my
01:18:18.375 --> 01:18:20.688 nerves and then kill the call.
01:18:20.748 --> 01:18:21.848 That's fantastic.
01:18:22.328 --> 01:18:28.247 Anyway, so if we go and look at the CEF configs repo in the
01:18:28.247 --> 01:18:34.461 documentation, the Windows Setup guide has a link to that file
01:18:34.461 --> 01:18:36.828 that I think is correct.
01:18:39.008 --> 01:18:39.878 It's got to find it.
01:18:40.158 --> 01:18:41.028 It's toward the bottom.
01:18:42.908 --> 01:18:43.288 There it is.
01:18:44.158 --> 01:18:44.698 Umm.
01:18:44.958 --> 01:18:45.928 If this link, yeah.
01:18:45.938 --> 01:18:50.407 If this link doesn't work, absolute worst case you can
01:18:50.407 --> 01:18:55.608 reach out to to any any one of us and we can probably get you a
01:18:55.608 --> 01:18:57.558 link and then update it.
01:18:57.568 --> 01:18:58.938 But I'm gonna check and see if this link works.
01:19:02.438 --> 01:19:05.131 Yep, looks like it doesn't click that download button, so that's
01:19:05.131 --> 01:19:05.338 good.
01:19:09.008 --> 01:19:16.358 Uh, so yeah, that's that's pretty much pretty much it for.
01:19:19.798 --> 01:19:20.468 What I had.
01:19:25.708 --> 01:19:26.208 Than any other.
01:19:27.308 --> 01:19:29.598 Uh, thoughts.
01:19:30.118 --> 01:19:33.028 You will you show that the top two queries in the cabana text.
01:19:34.588 --> 01:19:36.648 Oh, the Kibana query dot TXT, yeah.
01:19:36.368 --> 01:19:38.558 Yeah, like run them together.
01:19:38.648 --> 01:19:41.591 And so that you can see the results at the same time and the
01:19:41.591 --> 01:19:42.218 results were.
01:19:43.008 --> 01:19:44.988 I mean these two or these two?
01:19:46.218 --> 01:19:47.698 The the top two gits.
01:19:48.058 --> 01:19:48.278 OK.
01:19:49.408 --> 01:19:52.352 Obviously you have to swipe out UG with a name that actually
01:19:50.298 --> 01:19:50.568 Yeah.
01:19:50.578 --> 01:19:51.488 Changed UAG to.
01:19:51.538 --> 01:19:51.718 Yeah.
01:19:52.352 --> 01:19:52.738 matters.
01:19:53.468 --> 01:19:53.628 Yeah.
01:19:54.738 --> 01:19:58.020 Umm, change that to just my develop local since it's the one
01:19:58.020 --> 01:19:58.988 I've been showing.
01:19:59.298 --> 01:19:59.768 OK.
01:20:00.018 --> 01:20:01.678 Can I just control and run it?
01:20:01.818 --> 01:20:02.248 OK, yeah.
01:20:02.258 --> 01:20:02.368 Cool.
01:20:02.878 --> 01:20:03.408 UM.
01:20:03.558 --> 01:20:05.238 Yeah. So.
01:20:03.938 --> 01:20:08.085 Ohh yeah, so basically what this is doing is it's pulling out the
01:20:08.085 --> 01:20:12.106 the indices the the names of the indices and their aliases that
01:20:12.106 --> 01:20:14.368 are indexed that match this search.
01:20:14.518 --> 01:20:16.238 It's product search, dev, something.
01:20:17.388 --> 01:20:19.948 Umm, that's cool.
01:20:21.828 --> 01:20:26.686 And then you can see the the index names are all dated as you
01:20:26.686 --> 01:20:29.898 can see, they've all got a date on them.
01:20:29.908 --> 01:20:33.591 They're actual name with the date and time showing when it
01:20:33.591 --> 01:20:37.462 was created, and then the second one is the aliases, which is
01:20:37.462 --> 01:20:40.208 what they're actually being connected to a.
01:20:40.218 --> 01:20:45.139 So product search, Dev, local the alias points to the most
01:20:45.139 --> 01:20:48.808 recent one product search, Dev Local 51122.
01:20:48.818 --> 01:20:49.948 Wow, I haven't indexed my.
01:20:49.998 --> 01:20:52.208 No way, I haven't indexed my products in over a year.
01:20:52.258 --> 01:20:52.838 That's insane.
01:20:54.088 --> 01:20:55.238 That seems definitely wrong.
01:20:54.638 --> 01:20:54.888 What?
01:20:54.898 --> 01:20:55.048 No.
01:20:55.058 --> 01:21:01.318 Let's let's twenty 23511 at 2200 hours.
01:20:58.218 --> 01:20:59.028 Ohh yeah, 20.
01:20:59.038 --> 01:20:59.788 Yeah, I saw.
01:20:59.798 --> 01:21:01.048 Yeah, 51122.
01:21:01.118 --> 01:21:02.988 Yeah, I was gonna say no way.
01:21:02.998 --> 01:21:06.838 I have an index on my products over a year, but I can't read so
01:21:06.838 --> 01:21:07.978 I was right anyway.
01:21:09.098 --> 01:21:12.585 Umm but yeah, so you can see that uh it's tell it's saying
01:21:12.585 --> 01:21:16.189 that the actual name products search, dev local the one that
01:21:16.189 --> 01:21:19.557 Seth is gonna connect to is pointing you the most recent
01:21:19.557 --> 01:21:23.339 index and then these other two indices appear are given aliases
01:21:23.339 --> 01:21:24.048 of products.
01:21:24.058 --> 01:21:28.831 Search Dev local old and that's to make sure that this doesn't
01:21:28.831 --> 01:21:33.832 the actual live site connects to the most recent uh index and not
01:21:33.832 --> 01:21:34.968 to the old one.
01:21:36.308 --> 01:21:40.492 And if and if there's something wrong with your new index that
01:21:40.492 --> 01:21:44.808 that's broken like you've got 3 indexes there, the 1 from 420 at
01:21:44.808 --> 01:21:47.398 1953, that one only has a count of 99.
01:21:47.408 --> 01:21:49.906 Let's say that was the newest one that would tell you
01:21:49.906 --> 01:21:50.738 something's wrong.
01:21:50.748 --> 01:21:56.005 Like you went from 1000 and over 1100 to under 100, something's
01:21:56.005 --> 01:21:56.498 wrong.
01:21:56.688 --> 01:21:59.868 So so for a production instance, you might immediately swap the
01:21:59.868 --> 01:22:03.047 alias back using Cabana so that you're talking to a correct one
01:22:03.047 --> 01:22:05.779 of your customers aren't impacted, and then you can go
01:22:05.779 --> 01:22:08.660 debug and figure out what's wrong on your back and figure
01:22:08.660 --> 01:22:10.498 out what the with the alias process.
01:22:10.948 --> 01:22:13.018 Or I'm sorry with the the indexing process.
01:22:13.228 --> 01:22:16.087 It's also like if nothing's getting into the index, your
01:22:13.468 --> 01:22:13.638 Yeah.
01:22:16.087 --> 01:22:19.246 index size is gonna be like a couple of KB, and it'll say like
01:22:19.246 --> 01:22:20.098 a dog count of 0.
01:22:20.288 --> 01:22:23.796 So like that tells you that it's just not indexing anything in
01:22:23.796 --> 01:22:27.359 there, so that will let you know when it happened and then what
01:22:27.359 --> 01:22:28.528 actually got into it.
01:22:28.648 --> 01:22:31.224 And then you can go run like generic queries to tell you what
01:22:31.224 --> 01:22:33.758 products got in there like this part isn't getting in there.
01:22:33.858 --> 01:22:37.084 Let me run an unfiltered search by doing a get on that
01:22:37.084 --> 01:22:40.662 particular alias or on that particular index and get all the
01:22:40.662 --> 01:22:44.181 results back in a giant Jason BLOB and do control F in that
01:22:44.181 --> 01:22:47.759 window and and see if you can find the skew for that product
01:22:47.759 --> 01:22:51.571 and if it doesn't work well, you know that it never got in there
01:22:51.571 --> 01:22:52.568 in the 1st place.
01:22:52.798 --> 01:22:54.328 Then you start backtracking.
01:22:54.338 --> 01:22:55.698 Where did I get?
01:22:55.708 --> 01:22:58.618 Where did I lose the product from that?
01:23:03.968 --> 01:23:05.138 Indeed.
01:23:05.928 --> 01:23:09.508 So yeah, Kiwana does things it.
01:23:09.518 --> 01:23:12.328 It I should probably put time into.
01:23:13.748 --> 01:23:17.721 A getting better at it and it would probably be good for
01:23:17.721 --> 01:23:21.903 everybody to have at least a decent understanding of what's
01:23:21.903 --> 01:23:23.088 going on in here.
01:23:23.478 --> 01:23:25.428 It's a good it's a good debugging tool.
01:23:25.438 --> 01:23:27.758 I have successfully used it to find issues before.
01:23:29.178 --> 01:23:31.519 Faster than I probably would have found them just poking
01:23:31.519 --> 01:23:34.229 through the the code or you know throwing break points around and
01:23:34.229 --> 01:23:35.748 hoping to find what I'm looking for.
01:23:35.898 --> 01:23:40.468 So yeah, learn it.
01:23:41.018 --> 01:23:45.716 Learn to use it as with all tools, it exists to make your
01:23:45.716 --> 01:23:46.688 life easier.
01:23:51.288 --> 01:23:56.569 And on the note of making life easier, my head hurts and it's
01:23:56.569 --> 01:23:58.528 almost 4:00 o'clock so.
01:24:01.158 --> 01:24:01.458 Uh.
01:24:01.718 --> 01:24:05.367 Unless anyone has any more pressing questions, I think we
01:24:05.367 --> 01:24:05.618 can.
01:24:05.878 --> 01:24:10.548 Yeah, I think we can call it a, call it a day.
01:24:12.298 --> 01:24:16.398 Is there any way to edit the elastic settings from the admin
01:24:16.398 --> 01:24:16.868 portal?
01:24:16.878 --> 01:24:18.958 If you were like a admin of a website.
01:24:19.468 --> 01:24:19.598 Yes.
01:24:21.648 --> 01:24:26.032 Depending on what you mean by elastic settings, but I'll uh, I
01:24:26.032 --> 01:24:29.858 mean actually just found develops on spin up the oven.
01:24:30.068 --> 01:24:33.028 My local, but assuming you're talking about.
01:24:36.538 --> 01:24:40.338 Cefs elastic settings and not like actual Elasticsearch
01:24:40.338 --> 01:24:41.288 configuration.
01:24:41.678 --> 01:24:41.838 Yes.
01:24:41.778 --> 01:24:45.680 UM, yeah, so all those settings live in the same settings
01:24:45.680 --> 01:24:48.908 administration thing that everything else does.
01:24:49.198 --> 01:24:53.912 So as long as you're working on a Sev site that is 2022.2 or
01:24:53.912 --> 01:24:58.239 newer, you can edit those settings in the admin and I'm
01:24:58.239 --> 01:25:00.248 waiting for it to LogMeIn.
01:25:08.308 --> 01:25:08.688 There we go.
01:25:09.168 --> 01:25:13.903 Umm that settings editor is under site maintenance settings
01:25:13.903 --> 01:25:17.138 for here and you can search for elastic.
01:25:18.098 --> 01:25:23.602 Umm to find the actual specific stuff about like the actual port
01:25:23.602 --> 01:25:24.618 and whatnot.
01:25:25.148 --> 01:25:29.426 Or you can search clarity dot searching to get to kind of the
01:25:29.426 --> 01:25:33.842 broader stuff like index names, auction index requiring, vendor
01:25:33.842 --> 01:25:35.498 store product, etcetera.
01:25:35.828 --> 01:25:38.508 You can thumb through here and find what you're looking for.
01:25:38.518 --> 01:25:41.158 So for example, I'm like that product index type keys.
01:25:41.548 --> 01:25:44.849 You can enable editing and come in here and say actually I do
01:25:44.849 --> 01:25:46.818 need this so I Oh yeah, it's bugged.
01:25:47.768 --> 01:25:51.258 I gotta fix that since it's mapping out undefined.
01:25:51.268 --> 01:25:54.088 There's no array to add to, just needs to null coalesce that.
01:25:54.098 --> 01:25:57.896 But anyway, once that's fixed, you can click this little plus
01:25:57.896 --> 01:26:01.755 and add variant master and add general and add kit or whatever
01:26:01.755 --> 01:26:03.838 they want and build out whatever.
01:26:05.318 --> 01:26:09.166 Whatever settings they want in here, you can you can adjust the
01:26:09.166 --> 01:26:10.548 rest of these settings.
01:26:10.558 --> 01:26:12.608 You can make it not require a category.
01:26:12.618 --> 01:26:16.161 You could make it, yes, require a store or a vendor or whatever
01:26:16.161 --> 01:26:17.378 the the project needs.
01:26:19.358 --> 01:26:24.769 So, uh as an additional note, if you click on any of these, it
01:26:24.769 --> 01:26:28.548 pops up the dependencies of those settings.
01:26:28.798 --> 01:26:32.426 So for example, this one depends on manufacturers being enabled
01:26:32.426 --> 01:26:35.430 and that means that if I try to set this to true and
01:26:35.430 --> 01:26:38.717 manufacturers aren't enabled, it's still going to show as
01:26:38.717 --> 01:26:41.891 false because this is not enabled and it's dependent on
01:26:41.891 --> 01:26:42.628 this setting.
01:26:42.708 --> 01:26:48.198 So just as a the pro tip on that one.
01:26:48.308 --> 01:26:52.428 So, uh yeah. Cool.
01:26:57.278 --> 01:27:00.310 And you can use this by the way, for anyone watching this later
01:27:00.310 --> 01:27:03.247 or who is currently listening, you can use this for more than
01:27:03.247 --> 01:27:04.478 just the elastic settings.
01:27:04.488 --> 01:27:07.862 You can change any setting that's going through the the app
01:27:07.862 --> 01:27:08.368 settings.
01:27:08.378 --> 01:27:12.571 Key default value generates app settings et cetera attributes
01:27:12.571 --> 01:27:13.788 that we've set up.
01:27:15.228 --> 01:27:19.375 So if you're working on trying to get a payment provider to
01:27:19.375 --> 01:27:23.522 work like authorize, you can type, authorize and go in here
01:27:23.522 --> 01:27:27.461 and change these on the fly, save it and it reloads that
01:27:27.461 --> 01:27:31.677 provider plugging in the new credentials and then you can go
01:27:31.677 --> 01:27:33.128 in and rerun payment.
01:27:33.178 --> 01:27:36.748 If it works and that's a lot faster than changing it in the
01:27:36.748 --> 01:27:40.437 app settings file, killing the app pools and then spinning it
01:27:40.437 --> 01:27:44.185 back up and trying again, it's a little bit quicker to come in
01:27:44.185 --> 01:27:47.933 here and just have it open and and you know if for some reason
01:27:47.933 --> 01:27:51.503 you're iteratively testing different sets of credentials or
01:27:51.503 --> 01:27:52.098 something.
01:27:52.108 --> 01:27:55.556 I've definitely had clients that are like, I don't know, maybe
01:27:55.556 --> 01:27:56.158 it's these.
01:27:56.168 --> 01:27:56.658 I don't know.
01:27:56.668 --> 01:27:57.608 Maybe it's these.
01:27:57.648 --> 01:28:01.971 So if you're ever in that scenario, it's a lot faster to
01:28:01.971 --> 01:28:06.218 test that right here than it is to stop your app pools.
01:28:06.228 --> 01:28:07.228 Change your settings file.
01:28:07.238 --> 01:28:08.708 Start the app pools et cetera.
01:28:09.058 --> 01:28:12.576 So keep in mind that you can change all almost all the
01:28:12.576 --> 01:28:16.668 settings only a couple of them are intentionally marked as read
01:28:16.668 --> 01:28:16.988 only.
01:28:16.998 --> 01:28:19.998 For example, like site root URL because you can break your shirt
01:28:19.998 --> 01:28:20.828 if you changed it.
01:28:21.928 --> 01:28:24.508 Ohh, so uh.
01:28:24.518 --> 01:28:24.848 I didn't.
01:28:24.858 --> 01:28:27.694 I didn't want to have that something that a client could go
01:28:27.694 --> 01:28:30.718 in here and just change and then maybe like I broke my website.
01:28:31.968 --> 01:28:32.408 Umm.
01:28:32.638 --> 01:28:37.198 So anyway, yeah, cool.
01:28:44.878 --> 01:28:46.478 I want to go ahead and kill the recording now.
01:28:49.288 --> 01:28:50.308 That was great, Brendan.