| Topic | Presenter | Summary | Duration |
| ------------------------------ | ------------ | ---------------------------------------------------------------------- | -------- |
| C4: How to Work with Pipelines | Johnathan L. | This video gives a programming overview of how to work with pipelines. | 8:48 |00:00:08.470 --> 00:00:12.089 Video will attempt to give a mostly00:00:12.089 --> 00:00:13.640 programming knowledge agnostic00:00:13.723 --> 00:00:16.775 overview of how to work with pipelines.00:00:16.780 --> 00:00:18.614 There will be a separate video after00:00:18.614 --> 00:00:20.565 this which will go a little bit deeper00:00:20.565 --> 00:00:22.419 into the code and how things work00:00:22.419 --> 00:00:24.149 that's intended more for developers.00:00:24.150 --> 00:00:26.614 However, this video is intended to also00:00:26.614 --> 00:00:29.948 serve as an introduction for non developers.00:00:29.950 --> 00:00:32.904 So pipelines are designed to solve the00:00:32.904 --> 00:00:36.617 problem of needing to make changes to it00:00:36.617 --> 00:00:38.987 and integration without necessarily having00:00:39.063 --> 00:00:41.730 to dive into the actual code itself.00:00:41.730 --> 00:00:44.859 This could be because the developer doesn't00:00:44.859 --> 00:00:48.205 have the time to go in and make that00:00:48.205 --> 00:00:51.217 large changes to a hard large code base.00:00:51.220 --> 00:00:53.985 Or just that scene on developer doesn't00:00:53.985 --> 00:00:56.607 want to have to pay a developer00:00:56.607 --> 00:00:58.297 to build and do that.00:00:58.300 --> 00:01:01.005 When where another the solution00:01:01.005 --> 00:01:04.400 of Irada was a pipeline graph.00:01:04.400 --> 00:01:06.776 So a graph is composed of many nodes,00:01:06.780 --> 00:01:10.227 each node as you can see from the top00:01:10.227 --> 00:01:14.770 part of each node. Has a particular?00:01:14.770 --> 00:01:16.898 If there are C Class associated with it,00:01:16.900 --> 00:01:19.028 or in the case of an embedded pipeline,00:01:19.030 --> 00:01:19.560 another pipeline.00:01:21.970 --> 00:01:24.040 So the other part is the00:01:24.040 --> 00:01:25.420 edges between the nodes.00:01:25.420 --> 00:01:27.265 These edges connect to different00:01:27.265 --> 00:01:29.480 ports and we'll go into what00:01:29.480 --> 00:01:31.286 ports are in just a minute.00:01:33.500 --> 00:01:36.086 So as you can see here,00:01:36.090 --> 00:01:39.114 there are two main distinctions of nodes.00:01:39.120 --> 00:01:43.216 There are the just simple classes or scripts.00:01:43.220 --> 00:01:44.882 And there are pipelines which can00:01:44.882 --> 00:01:46.750 be either embedded or sub pipelines,00:01:46.750 --> 00:01:48.498 for example. This one.00:01:48.498 --> 00:01:50.683 Here is a sub pipeline.00:01:50.690 --> 00:01:53.315 The distinction here is that a sub00:01:53.315 --> 00:01:55.402 pipeline will execute exactly once00:01:55.402 --> 00:01:58.066 every time its parent pipeline executes,00:01:58.070 --> 00:02:00.374 whereas an embedded pipeline00:02:00.374 --> 00:02:03.830 will execute all the way through.00:02:03.830 --> 00:02:06.390 Now when I say all the way through,00:02:06.390 --> 00:02:08.273 I mean until one of the nodes00:02:08.273 --> 00:02:10.272 that returns a control flow of00:02:10.272 --> 00:02:12.147 halts or similar against control.00:02:12.150 --> 00:02:15.390 But controls are in a minute.00:02:15.390 --> 00:02:17.553 So the way that the pipeline graph00:02:17.553 --> 00:02:19.966 executes is it looks for all the00:02:19.966 --> 00:02:22.090 nodes which have no imports connected.00:02:22.090 --> 00:02:23.491 In their case.00:02:23.491 --> 00:02:27.909 This is only the sport or only this node.00:02:27.910 --> 00:02:29.286 Then starting from them,00:02:29.286 --> 00:02:30.318 it will execute.00:02:30.320 --> 00:02:32.756 As long as the control parts does00:02:32.756 --> 00:02:35.334 not return any errors or any other00:02:35.334 --> 00:02:37.512 things in which would cause it00:02:37.589 --> 00:02:39.766 to skip the rest of the graph.00:02:39.770 --> 00:02:41.842 It will then find all the notes00:02:41.842 --> 00:02:43.815 is connected to and haven't list00:02:43.815 --> 00:02:45.855 two execute then it just repeats00:02:45.855 --> 00:02:48.086 that over and over and over again.00:02:48.090 --> 00:02:49.150 It's the next load.00:02:49.150 --> 00:02:51.149 Execute it as long as it doesn't00:02:51.149 --> 00:02:52.859 make us quit the pipeline.00:02:52.860 --> 00:02:54.230 We add everything that's connected00:02:54.230 --> 00:02:55.999 to and then that keep executing00:02:55.999 --> 00:02:57.028 through that list.00:02:59.920 --> 00:03:01.666 So I've been mentioning ports and00:03:01.666 --> 00:03:03.439 then control flows quite a bit here,00:03:03.440 --> 00:03:05.066 so let's take a look at.00:03:05.070 --> 00:03:07.695 Those are over here on the left00:03:07.695 --> 00:03:10.327 hand side you can see it ports.00:03:10.330 --> 00:03:12.060 Then on the right side,00:03:12.060 --> 00:03:15.460 the bottom right side has the output ports.00:03:15.460 --> 00:03:16.880 Times, although there is00:03:16.880 --> 00:03:18.300 currently no visual distinction,00:03:18.300 --> 00:03:22.314 the top right side is the air control parts.00:03:22.320 --> 00:03:24.528 Now some nodes don't have control points that00:03:24.528 --> 00:03:27.198 it is intended to have a visual distinction.00:03:27.200 --> 00:03:30.416 Eventually it's just not truly implemented.00:03:30.420 --> 00:03:33.939 Final type of port is a node level part.00:03:33.940 --> 00:03:35.885 So that's what every node00:03:35.885 --> 00:03:38.719 has up here at the top left.00:03:38.720 --> 00:03:40.705 These can only be connected00:03:40.705 --> 00:03:42.690 to through a control port.00:03:42.690 --> 00:03:44.594 It is also intended to eventually have00:03:44.594 --> 00:03:46.700 a node level port on the right side,00:03:46.700 --> 00:03:48.555 which would allow you to do this.00:03:48.560 --> 00:03:49.082 For example,00:03:49.082 --> 00:03:50.909 kind of this mode which has no00:03:50.909 --> 00:03:52.300 control port to another node.00:03:52.300 --> 00:03:52.567 However,00:03:52.567 --> 00:03:54.703 that is not implementing the UI just yet.00:03:57.750 --> 00:03:59.256 So I've been talking about these00:03:59.256 --> 00:04:00.513 control flows. Let's take a00:04:00.513 --> 00:04:02.019 look at what those are exactly.00:04:04.420 --> 00:04:06.583 So it flow is a special type00:04:06.583 --> 00:04:09.160 of object which can be one of00:04:09.160 --> 00:04:10.740 these possible possible values.00:04:10.740 --> 00:04:12.840 It can be a continued flow,00:04:12.840 --> 00:04:14.600 in which case nothing happens.00:04:14.600 --> 00:04:16.820 It just keeps moving on00:04:16.820 --> 00:04:19.040 through the pipeline as normal.00:04:19.040 --> 00:04:21.160 Make me a skip pathlow.00:04:21.160 --> 00:04:21.904 So for instance,00:04:21.904 --> 00:04:24.053 in this case out here we have control00:04:24.053 --> 00:04:25.848 flow from the insured customers.00:04:25.850 --> 00:04:27.018 Any serious sub pipeline00:04:27.018 --> 00:04:28.186 connected to be pushed.00:04:28.190 --> 00:04:29.630 Acrs node over here.00:04:29.630 --> 00:04:31.430 If this port right here00:04:31.430 --> 00:04:33.339 with returning Skip path,00:04:33.340 --> 00:04:35.762 he will skip this node and now00:04:35.762 --> 00:04:37.770 any any descendants of this node,00:04:37.770 --> 00:04:40.087 meaning any end nodes that this node00:04:40.087 --> 00:04:42.548 is connected to from its output ports.00:04:45.390 --> 00:04:47.357 Now that is distinguished from a skip00:04:47.357 --> 00:04:49.479 cycle flow in that if this words00:04:49.479 --> 00:04:51.009 were returning skip cycle flow,00:04:51.010 --> 00:04:53.817 this one would not be executed either.00:04:53.820 --> 00:04:55.190 So under a skip path,00:04:55.190 --> 00:04:57.074 if this error skip out, there's a00:04:57.074 --> 00:04:59.550 trend from here this one is in a skip.00:04:59.550 --> 00:05:01.950 This one is not.00:05:01.950 --> 00:05:03.760 If a skip cycles returned,00:05:03.760 --> 00:05:05.570 the entire pipeline will immediately00:05:05.570 --> 00:05:07.380 go back to the beginning.00:05:10.380 --> 00:05:11.908 A hopeful will completely00:05:11.908 --> 00:05:13.436 stop the pipeline altogether.00:05:13.440 --> 00:05:15.738 It's bad, it's what I mean.00:05:15.740 --> 00:05:17.655 Earlier when I say that00:05:17.655 --> 00:05:19.187 embedded pipeline will plummet,00:05:19.190 --> 00:05:21.100 all it's dry or stopped.00:05:24.160 --> 00:05:26.624 And then error flows are just a special00:05:26.624 --> 00:05:29.595 case of halts as well as skip cycle errors.00:05:34.040 --> 00:05:36.686 So each node can have its own00:05:36.686 --> 00:05:38.120 parameters assigned to it.00:05:38.120 --> 00:05:40.862 For example, on this one you00:05:40.862 --> 00:05:44.060 have or chat for example. Uh.00:05:47.120 --> 00:05:48.915 For example, on the node00:05:48.915 --> 00:05:50.710 which goes out and finds00:05:50.787 --> 00:05:52.837 all the orders from Shopify.00:05:52.840 --> 00:05:54.946 We see that we have in00:05:54.946 --> 00:05:56.350 the saved parameter set.00:05:58.550 --> 00:06:02.316 The same parameters set shop by singers.00:06:02.320 --> 00:06:05.576 Well, there are two types of node arguments.00:06:05.580 --> 00:06:07.498 There are immediate sand saved, which00:06:07.498 --> 00:06:10.042 sensitive is just a special case of saved.00:06:12.290 --> 00:06:15.674 Immediate arguments allow you to specify00:06:15.674 --> 00:06:18.800 properties on that class directly.00:06:18.800 --> 00:06:22.000 In other words, configuration values.00:06:22.000 --> 00:06:25.600 So for example, this get note over here.00:06:25.600 --> 00:06:28.849 In general I get node and it takes its00:06:28.849 --> 00:06:31.777 inputs and pulls some field out of it.00:06:31.780 --> 00:06:34.600 So in this case we can look at its immediates00:06:34.665 --> 00:06:37.295 and see the path is set to be customer ID.00:06:39.480 --> 00:06:41.461 So in this case we are sending00:06:41.461 --> 00:06:42.900 an order into the git.00:06:42.900 --> 00:06:44.895 It will pull off the order property,00:06:44.900 --> 00:06:48.588 then it will pull off the ID property.00:06:48.590 --> 00:06:49.850 Then it will take that and just00:06:49.850 --> 00:06:51.270 send it along through the pipeline.00:06:54.450 --> 00:06:56.255 It is intended to eventually00:06:56.255 --> 00:06:58.060 have that be more streamlined,00:06:58.060 --> 00:06:59.865 where you can actually pull00:06:59.865 --> 00:07:01.670 properties directly off of imports.00:07:01.670 --> 00:07:03.836 However, that is not implemented yet.00:07:07.370 --> 00:07:09.920 Those saved argument sets are basically00:07:09.920 --> 00:07:12.710 just a special case of immediates.00:07:12.710 --> 00:07:14.942 This will go out to the00:07:14.942 --> 00:07:17.299 database and try to find any.00:07:17.300 --> 00:07:21.308 Any set of immediates names this.00:07:21.310 --> 00:07:23.860 So it will then just immediately00:07:23.860 --> 00:07:26.060 apply them to this node.00:07:26.060 --> 00:07:28.258 This is just so you don't have00:07:28.258 --> 00:07:30.586 to write them in the same node00:07:30.586 --> 00:07:32.554 over and over and over again.00:07:32.560 --> 00:07:34.190 So for example this shop,00:07:34.190 --> 00:07:35.810 fine Shopify orders node uses,00:07:35.810 --> 00:07:38.330 uses the same as any other00:07:38.330 --> 00:07:41.225 Shopify node which we can go out00:07:41.225 --> 00:07:43.857 say to the customer in the sink.00:07:43.860 --> 00:07:46.720 You look over here.00:07:46.720 --> 00:07:49.224 You'll see this also using the same set.00:07:49.230 --> 00:07:49.992 So this node,00:07:49.992 --> 00:07:52.137 and we know we saw previously we used00:07:52.137 --> 00:07:54.265 these same set of arguments at runtime.00:07:54.270 --> 00:07:55.455 Plus any overrides you may00:07:55.455 --> 00:07:56.403 put in the immediates.00:08:03.220 --> 00:08:05.495 These dollar sign special nodes00:08:05.495 --> 00:08:07.770 are special end nodes which00:08:07.850 --> 00:08:10.320 which expose the inner workings.00:08:10.320 --> 00:08:14.250 Net two of the pipeline itself.00:08:14.250 --> 00:08:16.406 So in this case, they're being used00:08:16.406 --> 00:08:18.422 to expose the inputs and outputs00:08:18.422 --> 00:08:20.474 of the pipeline for the whole,00:08:20.480 --> 00:08:22.352 so this pipeline is capable of00:08:22.352 --> 00:08:24.420 taking in a Shopify customer ID,00:08:24.420 --> 00:08:26.388 outputting ECS customer and Shopify customer.00:08:26.390 --> 00:08:28.898 We saw this being used back00:08:28.898 --> 00:08:30.570 here in the orders.00:08:30.570 --> 00:08:32.078 Where we are affinion00:08:32.078 --> 00:08:33.586 the Shopify customer ID.00:08:33.590 --> 00:08:35.576 You need this.00:08:35.576 --> 00:08:36.900 Extraneous link.00:08:41.680 --> 00:08:43.248 And then we are taking out the00:08:43.248 --> 00:08:44.779 ECS customer and Shopify customer.00:08:44.780 --> 00:08:46.607 Although in this case we don't actually00:08:46.607 --> 00:08:48.368 have to use them for anything.