Pre/Post-Build Plugins

Pre and post-build plugins enable you to apply code before or after the build process to prepare, verify, and test data.

Before a build begins, you can:

1. Bring in files/prepare data
2. Validate your data and cancel a build if there’s a problem

After a build is complete, you can:

1. Validate your data

2. Initiate activity in other systems

This page describes how to install and configure both pre and post-build add-ons and provides a sample post-build add-on for exporting tables in your data model to a CSV file.

How Pre/Post-Build Add-ons Work

Pre/post build add-ons are bash scripts that run commands you define. For Sisense to execute your script, you must give the bash script a specific name and place it in a directory that Sisense recognizes. For pre-build add-ons, when you begin a build, Sisense checks the directory for any scripts and when a script exists, runs the script synchronously as part of the build. For post-build add-ons, Sisense runs your add-ons while finalizing the build.

Sisense runs one script per ElastiCube. For each ElastiCube you should place your script in a directory with the name of the ElastiCube’s farm. For example, for the Sample Ecommerce ElastiCube, the directory should be called aSampleIAAaECommerce. This directory should be located under the following path:

/opt/sisense/storage/build-plugins/aSampleIAAaECommerce

You can find the name of your ElastiCube’s farm under the following path:

/opt/sisense/storage/farms

Within this directory, your bash script for pre-build add-ons should be called prebuild.sh and for post-build add-ons postbuild.sh. If these files are not located under a directory with the relevant Elasticube farm’s name, and if the file name is not called prebuild.sh or postbuild.sh, Sisense will not run the script. Your scripts must also have execution permission.

Sisense does not currently support libraries such as JavaScript and Python out of the box, but you can leverage JavaScript or Python libraries from your shared storage. In addition, you can run standard and MClient commands in addition to commands supported by the Sisense CLI. Before using the Sisense CLI commands, you must be logged in, for example:

login_sisense 10.11.12.13:30845 user@sisense.com Passw0rd

Arguments

The scripts can take the following as arguments:

1. Pod Name : Instance host name.

2. Build ID: GUID unique for each build.

3. Data Source Display Name: The name of the source as its displayed n the dashboard, for example, Sample ECommerce.

4. ElastiCube Physical Name: The name of the ElastiCube as define in the farm /opt/sisense/storage/farms, for example, aSampleIAAaECommerce

5. Build Originator: Where was the build initiated, REST, NextECM, Scheduler, or the CLI.

6. Build Intent Type: What type of build, first/replace build, or "by table"/"changes only"), one of: NEW/EXISTS.

7. Temp Directory: See Frameworks.

8. ECM Model Path: (Only for pre-build add-ons) The ECM model JSON file.

9. Queries Index Path: (Only for post-build add-ons), an index of all tables (base and custom) with their physical queries.

Framework

Sisense provides a framework that includes a set of interfaces and utilities that simplifies the implementation and maintenance of your add-ons.

Sisense recommends you use the build_plugin_template.sh and plugin_utils.sh when creating your add-ons.

To use the framework, come the plugin_utils.sh in the build-plugins home directory. Next, rename the build_plugin_template.sh to prebuild.sh/postbuild.sh depending on which type of add-on you want to create. Then, modify the file by adding your commands as needed.

The content of the framework includes:

Category	Interface	Template/Utils Usage
Globals	Arguments and constants.	The arguments and constants are read into global variables and printed in the INFO level at the beginning of the add-on template.
Logging	DEBUG prints: stdout lines (including any external tools).	Not needed (all stdout is intercepted by the service in DEBUG level by default), however for readability log_debug function is available.
	INFO prints: stdout lines starting with "[INFO]:"	log_info does it for you. Also the template prints start and end messages.
	WARN prints: stderr lines. Error prints are not ERROR level because they don't necessarily mean the add-on failed. However all errors are collected and, if the add-on fails, the errors are sent to the user as part of the failure message.	log_error redirects the print to stderr.
Temp dir	All files in the directory .../build-plugins/tmp starting with the build ID are automatically deleted at the end of the plugin runtime.	To create a full temp path for a simple filename like file.txt, send it to the function temp_filename.
Exit value	The exit value must be 0 to succeed. Other values cancel the build and are reflected to the user (if extracted). Note that if the exit value is not extracted (including when caused by a timeout), the add-on may not stop.	When executing a command in the script, its success can be validated using validate_last_call getting the error message and exit value to exit with in case the command failed.
JSON	The models saved for the add-ons usage are represented as JSON.	To get an array of the JSON keys, use parse_json_keys, getting the JSON file path. To get a field content by name, use parse_json_field, getting the JSON content and the field path (JQ syntax).
CLI	The CLI functionality is available in .../build-plugins/sisense.sh - copied on the first plugin execution. See the CLI section for details and examples.	The template imports the CLI source.
Other		A function to replace special characters with underscores in a string, simplify_name is useful for making names file system compatible.

CSV Export Plugin

The CSV Export add-on is an example provided by Sisense that exports the data of all or specific tables of an ElastiCube, including custom tables, into CSV files.

To set up the add-on and see instructions for how to create a post-build add-on, see Export Tables To CSV Post Plugin. After the build is complete, a CSV file is exported to the relevant directory.

Possible Error Messages for the CSV Export Add-on

When an error occurs, your users might receive one of the following errors:

Exit value 101 - {export target} wasn't created or is not a directory. This error is caused by a permission or parameter issue.

Exit value 102 - The table {table} has no query defined. Only if configured. This is caused by first build after changes or in a long while, or the table name is misspelled or changed.

Exit value 103 - Failed to save query file {query path}. This error is caused by a permission issue.

Exit value 104 - MClient runtime returned {code} . Should include the error outputs.

Exit value 105 - Failed to save CSV of {table}. This error is caused by a permission issue.

Troubleshooting

Below are common failures that might occur and how to resolve them when working with add-ons:

Problem	Troubleshooting
Build succeeds without any indication that the plugin was executed.	If the plugin leaves no "trace" in form of a log file, export files or other, the only way to make sure it runs (other than making it fail or watching the build log in UI) is to find the relevant log prints in management.log: BE#855626 Executing {scriptId} (scriptId = prebuild.sh/postbuild.sh with the build ID). If instead there is a matching print: BE#982374 No pre-build plugin available, or BE#850934 No post-build plugin available Then indeed the plugin wasn't found. Make sure you put it in the right location and the physical name of the cube is correct. If taken from farms, check again - some imported old cubes might change their physical name after the first build. Monitoring is planned - see Open Issues section.
The add-on fails with an exit code but no message	Some exit values might mean an internal failure (depends on the OS), usually 2, 126 or 127. This might mean: The bash lib cannot be found or a command is not executable - might be caused by the file not being formatted for UNIX EOL (if copied to the server from Windows for example) - make sure you format the EOL to UNIX in Notepad++ or otherwise.The script not having the execution permissions for the user - add the permission.The file is busy - try closing the session. You should get more information by executing the script manually. To avoid confusion it is recommended to print errors and return meaningful exit codes (0-255).
The add-on fails with compilation error in an empty line	Validate the UNIX EOL format.This can also be caused by invisible characters when copy-pasting script lines from elsewhere.
The plugin succeeds although the operation failed and there are error messages.	Error messages are allowed as long as the exit value is 0. To make sure the plugin fails, implement a validation and/or call validate_last_call to make sure the right exit value is returned.
Not all my error prints are reflected to the user in case of failure	Try looking for the missing message interception in the log: BE#883471 {scriptId} stderr: ...
I can't cancel an add-on execution	Add-ons can't be canceled. You can try to cancel the build before it gets to the plugin.
The build is taking significantly longer with the add-ons	Add-ons run synchronously, and execution time depends on the implementation. Working with data (for instance CSV export) is linearly dependent on the data size and might add significant time to the build, without the option to cancel.
It looks like a file the add-on is working on has been tampered with	The plugin can execute simultaneously from several places by several builds. Make sure you use the temp_filename utility for temporary files and use the scriptDir (or other cube-unique path) to save outputs. Don't open/modify the files while a build is running. Also, note that timeouts don't guarantee a termination of the script, so it's possible that a previous execution is still working. Consider increasing the timeout.
All queries in the query index are empty	This happens on the first build, because the SQL translation service doesn't yet recognize the data source.

.r.