Most navigation is accomplished via the navbar on the top of the screen:
- Environment: Not yet implemented. Will be a Hangfire-esque overview of which/how many
pipelines are failing, which are succeeding, etc.
- Pipelines: The landing page - come here to see all data transformation pipelines present in the system
- Pipeline Editor: Reached by clicking on a pipeline from the main page. Allows editing pipelines in a
graph-like manner.
- RunSets: Combination landing page and editor - allows combining pickers/pipelines/pushers together to form
a runset which may then be scheduled.
- Mappings: A landing page for seeing two lists - either all pipelines or all Remote Systems.
- Mappings Editor: Reached either by selecting two Remote Systems (TODO) or a mapping.
- TODO: The editor will automatically filter itself down to the appropriate types for the selection.
- Schedules: Combination landing page and editor - allows scheduling when certain RunSets should be activated,
and with what parameters.
- Logs: A filterable log dump of everything happening inside C4 - come here to track down issues with pipelines
or loaded modules.
- User Icon/Username: Not yet implemented. Will eventually be used for a users page.
As this page is more complex than most of the rest of the UI, it has been given its own documentation.
Please see Working With Pipelines.
The Runsets page allows listing, creating, and editing runsets, which are used to tie together pickers,
pipelines, and pushers into cohesively executing units.
From here, you may edit the following properties of a Runset:
- Name: A friendly name for the Runset which is also used by the rest of the system for identification.
This is the name provided to the scheduler later.
- Intended Working Set: The working set we expect this runset to be scheduled with.
- Pickers: An ordered list of scripts which are to be run prior to the pipelines, and which shall
sync all of a particular entity class's data into R0 so the pipelines don't need to make as many network calls.
- Pipelines: An ordered list of pipelines to run to transform the data which was synced into R0.
- Pushers: An ordered list of script which are to be run after the pipelines, and which shall
find any changes made by the pipelines to entities of each pusher's respective systems and send those
changes back to those systems in bulk.
Note: Presently, only half of this page (viewing individual mapping pairings) is built out.
The ability to select two ERPs will come in a future update, but for now may be accessed manually
via the URL /mappings/from/<System1>/to/<System2>, where <System1> is the name of the system
to be mapped from, and <System2> is the name of the system to be mapped to.
The editor's view is composed of 3 columns:
- A listing of all available types on the lefthand side, organized by general categories.
- Click a category to reveal the types inside, click a type to reveal the available fields inside.
- A listing of all currently relevant mappings
- In the main
/mappings/from/... URL, this is all mappings between any pairing of types from the two systems.
- In the focused per-mapping view, this will only be the focused mapping. (TODO)
- In the same manner as the first section, a column with all categories/types/fields from the destination system.
Note: Due to the high number of types and fields which can be present on the page, it can sometimes
take a second or two for the boxes to register that they were dropped on.
- Scroll to the bottom of the middle column
- Drag a type from the source column into the box marked Drag a source type here.
- Drag a type from the destination column into the box marked Drag a destination type here.
- Click
+ ADD MAPPING
See below for an example:
Note: Mappings don't save themselves. Use the <img src="./img/SaveIcon.png" height=15 /> to save yourself.
From the destination column, drag a field you wish to map into the appropriate section for that mapping.
You may drop it either on top of the heading for that pairing, or anywhere in the background for that mapping.
From the destination card (right side of the mappings column), you may select whether to enable remappings. These act
as a static lookup table. For example, system A may store the credit card type as a string ("VISA", "MasterCard", etc),
while System B may store it as a hardcoded ID (5, 6, etc). This could be represented in the mappings UI like so:
<img src="./img/Remappings.png" height=500 style="margin:auto; display:block;" />
Next, perform the following in any order:
- Drag one or more source fields into the empty card on the left.
- Add constant values using the
< > icon on the source card.
- Re-order mapping parts (source fields or constant values) by clicking and dragging the 3 horizontal
bars on that part.
- Enable or disable regex mode for any given part.
Explanation of mapping semantics:
- "Regex mode" allows you to take the stringified value of that source field, apply a regex to it, and
re-format it as some other string.
- Example: System A stores names as a single
Name field in the format "First Last", but system B
wants them as FirstName and LastName. This may be achieved as follows:
-
[^\s] means anything which is not whitespace (spaces, tabs, etc).+ means one or more.- Putting things inside
() parenthesis makes them available as $1, $2, etc in order of declaration
- Thus,
([^\s]+) means "one or more non-space characters grouped together", and ([^\s]+) ([^\s+]) means
"one or more non-space characters grouped together (a word) followed by a space and another of the same word"
- For better documentation on regexes, see Wikipedia
and Regexr.
- If only one source field is attached to a destination field, it's a "straight mapping", meaning that the
value (date/etc) is passed on without converting to/from a string. This is important for things like arrays
of objects.
- Note that if there are remappings attached to that destination, any value will still get stringified.
- If there are multiple parts (fields/constants) feeding into a single destination field, all parts will be
stringified and concatenated together with no spaces in between in the order they are encountered.
-
- Note that although the constant appears blank, it actually has a single space inside.
The schedules page allows defining certain rules for automatically running RunSets at certain times, as well as adding
parameters that apply on top of runsets'/pipelines' normal parameters.
The shedules view has the following columns:
- Runset: The name of the runset scheduled.
- All schedules will be grouped by their runset; each runset's name will only appear once on the page.
When this column is blank, it means it's the same as the previous runset.
- Name: The human-friendly name given this schedule, if any. Example: "Nightly orders sync".
- Scheduled: A human-friendly description of the cronstring on this schedule.
- Note: Presently, this description is jammed into a single box. The plan, however, is to have this spread into
multiple boxes for better visual clarity.
- Next Run: The next time the schedule should be or have been run, taking into account the last time the pipeline
actually was run and the attached cronstring.
- Status: Whether this schedule is active or inactive. If inactive, the schedule will never be run.
-
- Name: As discussed.
- Cron String: A string in the cron format.
- See crontab.guru for an easy way to understand a given expression.
- Enabled: As discussed.
... - A context menu that allows deleting the schedule or triggering it early (TODO).
Although Where is shown as a JSON array (["Foo","Bar","Baz"]), here we shall show it in path format
(Foo/Bar/Baz) to make it friendlier.
Log entries are composed of 4 parts:
- Kind: one of {Audit,Info,Debug,Error}, optionally combined with one of {Upsert,Delete}.
- When: The date and time the entry was recorded.
- Where: A path to where in the code's execution an entry was recorded. Think of it like a lite stacktrace.
The backend is free to add new elements to the path as it goes, but there are some major patterns to be aware of:
/Watson/session <ID>/...: The entry happened as a result of a user on a particular session making a
request via the Web API./Det/: The entry happened as a result of some normal proccess in the backend.../Pump Pipeline <Name>/...: The entry happened while pumping the named pipeline.../Node <Name>/...: The entry happened while executing the named node<No path at all>: The entry happened in the earliest stages of C4's boot process, before we had the
logger initialized.-
At time of writing, the logs view works thusly:
- At time of page load, the most recent 1,000 log entries are loaded into the browser
- Every 5 seconds, the page will request all new entries from the backend
- Note that currently, this will eventually overload the browser. Reloading the page from time
to time is recommended. This is a known flaw and will be fixed.
- Log entries are colored:
- Audits and Info are blue
- Errors are red
- Debugs are yellow
- Log entries may be filtered out using the Exclude regex field at the top. Exceptions to the exclusion
may be specified using the Include regex field.
- Example: An exclude of "foo" will exclude any log containing "foo", including those containing "foobar". The
latter can be re-included by specifying an include of "foobar".