Pre/Post-Build Plugins

Pre and post-build plugins enable you to apply code before or after the build process to prepare, verify, and test data.

Before a build begins, you can:

1. Bring in files/prepare data.

2. Validate your data and cancel a build if there is a problem.

After a build is complete, you can:

1. Validate your data.

2. Initiate activity in other systems.

The information provided here details how to install and configure both pre and post-build add-ons and provides a sample post-build add-on for exporting tables in your data model to a CSV file.

How Pre/Post-Build Add-ons Work

Pre/post build add-ons are bash scripts that run commands you define. For Sisense to execute your script, you must give the bash script a specific name and place it in a directory that Sisense recognizes. For pre-build add-ons, when you begin a build, Sisense checks the directory for any scripts and when a script exists, runs the script synchronously as part of the build. For post-build add-ons, Sisense runs your add-ons while finalizing the build.

Sisense runs one script per ElastiCube. For each ElastiCube you should place your script in a directory with the name of the ElastiCube's farm. For example, for the Sample Ecommerce ElastiCube, the directory should be called aSampleIAAaECommerce. This directory should be located under the following path:

/opt/sisense/storage/build-plugins/aSampleIAAaECommerce

You can find the name of your ElastiCube's farm under the following path:

/opt/sisense/storage/farms

Within this directory, your bash script for pre-build add-ons should be called prebuild.sh and for post-build add-ons postbuild.sh. If these files are not located under a directory with the relevant Elasticube farm's name, and if the file name is not called prebuild.sh or postbuild.sh, Sisense will not run the script. Your scripts must also have execution permission.

Sisense does not currently support libraries such as JavaScript and Python out of the box, but you can leverage JavaScript or Python libraries from your shared storage. In addition, you can run standard and MClient commands in addition to commands supported by the Sisense CLI. Before using the Sisense CLI commands, you must be logged in, for example:

login_sisense 10.11.12.13:30845 user@sisense.com Passw0rd

Arguments

The scripts can take the following as arguments:

1. Pod Name: Instance host name.

2. Build ID: GUID unique for each build.

3. Data Source Display Name: The name of the source, as displayed in the dashboard, for example, Sample ECommerce.

4. ElastiCube Physical Name: The name of the ElastiCube, as defined in the /opt/sisense/storage/farms farm, for example, aSampleIAAaECommerce.

5. Build Originator: Where the build was initiated - REST, NextECM, Scheduler, or the CLI.

6. Build Intent Type: The type of build, first/replace build, or "by table"/"changes only"). Valid values: NEW/EXISTS.

7. Temp Directory: See Framework.

8. ECM Model Path (only for pre-build add-ons): The ECM model JSON file.

9. Queries Index Path (only for post-build add-ons): An index of all tables (base and custom) with their physical queries.

Framework

Sisense provides a framework that includes a set of interfaces and utilities that simplifies the implementation and maintenance of your add-ons.

Sisense recommends you use the build_plugin_template.sh and plugin_utils.sh when creating your add-ons.

To use the framework:

1. Place plugin_utils.sh in the build-plugins home directory.
2. Rename build_plugin_template.sh to prebuild.sh or postbuild.sh, depending on the type of add-on you want to create.
3. Modify the file, by adding your commands as needed.

The content of the framework includes:

Category	Interface	Template/Utils Usage
Globals	Arguments and constants.	The arguments and constants are read into global variables and printed in the INFO level at the beginning of the add-on template.
Logging	DEBUG prints: stdout lines (including any external tools).	Not needed (all stdout is intercepted by the service in DEBUG level by default). However, for readability, the log_debug function is available.
	INFO prints: stdout lines starting with "[INFO]:"	log_info does it for you. Also the template prints start and end messages.
	WARN prints: stderr lines. Error prints are not ERROR level because they do not necessarily mean that the add-on failed. However, all errors are collected. If the add-on fails, the errors are sent to the user as part of the failure message.	log_error redirects the print to stderr.
Temp dir	All files in the .../build-plugins/tmp directory that start with the build ID are automatically deleted at the end of the plugin runtime.	To create a full temp path for a simple filename like file.txt, send it to the temp_filename function.
Exit value	The exit value must be 0 to succeed. Other values cancel the build and are reflected to the user (if extracted). Note that if the exit value is not extracted (including when caused by a timeout), the add-on may not stop.	When executing a command in the script, its success can be validated using validate_last_call, getting the error message and exit value to exit with if the command failed.
JSON	The models saved for the add-ons usage are in JSON format.	To get an array of the JSON keys, use parse_json_keys, getting the JSON file path. To get a field's content by field name, use parse_json_field, getting the JSON content and the field path (JQ syntax).
CLI	The CLI functionality is available in .../build-plugins/sisense.sh - copied on the first plugin execution. See the CLI section for details and examples.	The template imports the CLI source.
Other		A function to replace special characters with underscores in a string, simplify_name is useful for making names file system compatible.

CSV Export Plug-in

The CSV Export add-on is an example provided by Sisense that exports the data of all or specific tables of an ElastiCube, including custom tables, into CSV files.

The steps below describe how to set up the add-on and provide an example of how to create a post-build add-on. After the build is complete, a CSV file is exported to the relevant directory. The CSV Export add-on might take a long time for larger ElastiCubes. In these cases, you can modify the post-build script timeout from the default time of 60 seconds. You can modify the timeout via the PreBuildScriptTimeout and PostBuildScriptTimeout properties.
When the add-on is executed, it will only succeed if all tables found in the queries index are successfully exported. Tables not found in the index for whatever reason will either fail the build or be quietly skipped, according to the failIfNoQuery parameter. If undefined queries fail the build, disable the add-on, rebuild the ElastiCube, re-enable the plugin, and build changes only. If undefined queries are skipped due to a change in the model, build changes only.

The exported CSV files are named by their table, without any special characters except '', for instance My table: copy(1) will become My table copy_1_.csv. The CSVs' content is a header-less, comma separated, UTF-8 encoded "preview-like" data representation.

To install the CSV Export add-on:

1. Add the plugin_utils.sh and the sisense.sh files to the build-plugins/ directory.

2. Add the postbuild.sh file to the build-plugins/{cube physical name} directory.

3. Modify the parameters to the postbuild.sh script as needed.

4. Leave the tables list empty to export all available tables. To export only specific tables, list their exact display names, for example: =("My Table 1" "Some Custom").

5. If you want to ignore the "no query defined" error, set failIfNoQuery to false.

6. The default CSV's export directory is build-plugins/{cube}/export. If necessary, change it to any accessible (or creatable) directory, as long as its ElastiCube or build is unique.

Possible Error Messages for the CSV Export Add-on

When an error occurs, your users might receive one of the following errors:

Exit value 101 - {export target} wasn't created or is not a directory . This error is caused by a permission or parameter issue.

Exit value 102 - The table {table} has no query defined. Only if configured. This is caused by the first build after changes or in a long while, or the table name is misspelled or changed.

Exit value 103 - Failed to save query file {query path}. This error is caused by a permission issue.

Exit value 104 - MClient runtime returned {code}. Should include the error outputs.

Exit value 105 - Failed to save CSV of {table}. This error is caused by a permission issue.

Troubleshooting

Below are common failures that might occur and how to resolve them when working with add-ons:

Problem	Troubleshooting
The build succeeds without any indication that the plugin was executed.	If the plugin leaves no "trace" in the form of a log file, export files or other, the only way to make sure it runs (other than making it fail or watching the build log in the UI) is to find the relevant log prints in management.log: BE#855626 Executing {scriptId} (scriptId = prebuild.sh/postbuild.sh with the build ID). If instead there is a matching print: BE#982374 No pre-build plugin available, or BE#850934 No post-build plugin available then, indeed, the plugin was not found. Make sure you put it in the right location and the physical name of the cube is correct. If taken from farms, check again - some imported old cubes might change their physical name after the first build. Monitoring is planned - see the Open Issues section.
The add-on fails with an exit code but no message	Some exit values might mean an internal failure (depending on the OS), usually 2, 126 or 127. This might mean: The bash lib file cannot be found or a command is not executable. This might be caused by the file not being formatted for UNIX EOL. Make sure you format the EOL to UNIX in Notepad++ or a similar editor. The script may not have execution permissions for the user - add the permission. The file is busy - try closing the session. You should get more information by executing the script manually. To avoid confusion, it is recommended to print errors and return meaningful exit codes (0-255).
The add-on fails with a compilation error in an empty line	Validate the UNIX EOL format. This can also be caused by invisible characters when copying and pasting script lines from elsewhere.
The plugin succeeds although the operation failed and there are error messages	Error messages are allowed as long as the exit value is 0. To make sure the plugin fails, implement a validation and/or call validate_last_call to make sure the right exit value is returned.
Not all my error prints are reflected to the user in case of failure	Look for the missing message interception in the log: BE#883471 {scriptId} stderr: ...
I cannot cancel an add-on execution	Add-ons cannot be canceled. You can try to cancel the build before it gets to the plugin.
The build is taking significantly longer with the add-ons	Add-ons run synchronously, and execution time depends on the implementation. Working with data (for instance a CSV export) is linearly dependent on the data size and might add significant time to the build, without the option to cancel it.
It looks like a file the add-on is working on has been tampered with	The plugin can execute simultaneously from several places by several builds. Make sure you use the temp_filename utility for temporary files and use the scriptDir (or other cube-unique path) to save outputs. Do not open or modify the files while a build is running. Also, note that timeouts do not guarantee termination of the script, so it is possible that a previous execution is still working. Consider increasing the timeout.
All queries in the query index are empty	This happens on the first build, because the SQL translation service does not yet recognize the data source.