Connectors Plug-ins

The Connectors Plug-ins feature allows you to apply custom logic to modify how Sisense connects and works with a data source using Java code. There are four flows that you can modify:

  • Connection Flow: Modify how Sisense connects to a data source, for example, by adding parameters specific to the data source.
  • Discovery Flow: Modify which metadata is discovered by filtering tables or columns.
  • Query Execution Flow: Modify the query before it is executed.
  • Data Extraction Flow: Modify the extracted data during a build or a live connection to a data source.

Plug-ins are one (or more) Java classes in the connector folder that implement one or more interfaces that they are related to. You develop your plug-ins for some or all of the flows according to its scope. The scope of your plug-in controls the flow in which the plug-in is applied. For example, connection plug-ins can be applied to both Query Execution and Discovery flows. Defining the scope of your plug-in is optional. If you do not define a scope, the plug-in is automatically applied to all scopes. The scope is defined on the plug-in level.

Plug-in Life Cycle

There are four stages to the life cycle of a plug-in:

  • Initialization: You prepare a tar.gz file containing all the relevant files for the plug-in. To introduce the plug-in to the system, you execute a CLI command to install the plug-in. This command copies and extracts the tar.gz content into the specified connector's directory, compiles the Java files and uses the classLoader to load the plug-in classes. Then, you need to restart the Query service.
  • Creation: The plug-in is initiated with the connector it belongs to and the plug-in classes are loaded.
  • Execution: The plug-in’s methods are executed according to the defined scope.
  • Destruction: The plug-in is destroyed with the connector it belongs to and the onDestroyPlugin method is executed.

Enabling the Connectors Plug-in

By default, the Connectors Plug-in feature is disabled, which prevents it from being applied to your builds or Live connections.

To enable the Connectors Plug-in feature:

  1. In Sisense, go to Admin > search for and select System Management (located under Server & Hardware) > click Configuration.

  2. In the top-left corner, click the Sisense logo five times.

  3. Open the Connectors section and toggle EnableConnectorsPlugins to Enabled.
    Or
    In the Sisense CLI, run the following command:
    si config set -key connectors.enableConnectorsPlugins -new-value true
    Restart the Query service.

To access the Sisense CLI, see Accessing the Sisense CLI in Using Sisense CLI Commands.

Creating Plug-ins

Sisense supports four Connector plug-ins types based on the four possible flows you can modify:

  • Connection extension
  • Discovery extension
  • ExecuteQuery extension
  • DataExtraction extension

This section describes how to implement each type and provides examples of how you can use them.

Plug-in Packaging
You should package your plug-ins as a tar.gz file. Inside the tar.gz file you should include the following files:

  • Sisense Connectors SDK: This is the SDK JAR file provided by Sisense. Contact your CSM for the latest version.
  • manifest.yaml file: A manifest file that you create, which contains two key-value pairs for each plug-in. The keys are name and version. The value of name is the name of your plug-in, which you define. The version is the version of the plug-in, which you define. These values are displayed in the Name and Version columns when you run the list plug-ins command.
  • Java Plug-in files: The Java files that contain the logic of your extensions. To implement a plug-in, you need to extend one or more extension classes (specified below), and override methods to implement your custom logic. The Java classes should not reside in a package.

Example:

A typical project's structure:

Example:

manifest.yaml:

Example:

tar.gz content:

Connection Extensions
You can implement Connection extensions to modify how Sisense connects to your data source by overriding the following methods:

  • ConnectionParameters beforeConnectionCreated(ConnectionParameters connectionParameters, ConnectionUiParameters connectionUiParameters, ConnectorConfig connnectorConfig)

    • ConnectionParameters contains:

      • String connectionString
      • Map<String, Object> connectionArguments

    • Allows you to modify connection parameters, for example adding a connection string parameter.

  • void afterConnectionCreated(Connection connection): Allows you to configure additional connection parameters.

      @ConnectorPlugin(extensionScopes={PluginExtensionScope.LIVE, PluginExtensionScope.DISCOVERY})
      class MyConnectionPlugin extends ConnectionExtension {
      ....
      @Override
      public void afterConnectionCreated(Connection connection) {
      doSomethingWithConnection(connection);
      }
      }

Discovery Extensions
You can use the Discovery extension to modify data found in the data source.

Discovery extensions can override the following methods:

  • List< String > getDatabases(List< String > databases): Allows you to modify the database list.
  • List< Table > getTables(List< Table > tables): Allows you to modify the list of returned tables.
  • Table getTableDetails(Table table): Allows you to modify the list of returned table details.
      interface DiscoveryPlugin extends ConnectorPlugin {
      default List<String> getDatabases(List<String> databases) {
      return databases;
      }
      default List<Table> getTables(List<Table> tables) {
      return tables;
      }
      default Table getTableDetails(Table table) {
      return table;
      }
  
      }    

The example below filters tables that start with the letter "A" on the data source level.

      @ConnectorPlugin
      class FilterTablesPlugin extends DiscoveryExtension {
      ....
      @Override
      public List<Table> getTables(List<Table> tables) {
      return tables
      .stream()
      .filter(table -> table.getName().startWith("A"))
      .collect(Collectors.toList());
      }
      }  

ExecuteQuery Extension

Execute query extensions can be used to modify execute query behavior.

  • String beforeExecuteQuery(QueryContext queryContext, String query): Allows you to modify a query before its executed.
      @ConnectorPlugin
      class OrderByQueryPlugin extends ExecuteQueryExtension {
      ....
      @Override
      public String beforeExecutQuery(QueryContext queryContext, String query) {
      // sort the results by some column
      return query + " ORDER BY " + someColumn;
      }
      }

DataExtraction Extension
With the Data extraction extension, you can modify or filter data that has been extracted from your data source.

  • Object getObject(int columnIndex): Allows you to modify the returned value for a given column.
  • Boolean moveToNextRow(): Allows you to skip/filter certain rows.
  • int getColumnCount(): Allows you to change the number of columns in the response (e.g. add a column).
      @ConnectorPlugin(extensionScopes=PluginExtensionScope.BUILD)
      class PasswordRemover extends DataExtractionPlugin {
      ....
      @Override
      public Object getObject(int columnIndex) {
      if (columnIndex == PASSWORD_COLUMN) {
      return "";
      }
      return super.getObject(columnIndex);
      }
      }

The example below adds a new column that is a sum of the other two columns.

      @ConnectorPlugin(name="AddCalculatedColumnPlugin", scope=PluginExtensionScope.BUILD)
      class AddCalculatedColumnPlugin extends DataExtractionExtension {
      private int calculatedColumn;
      ....
  
      @Override
      public int getColumnCount() {
      calculatedColumn = resultSet.getMetadata().getColumnCount() + 1;
      return calculatedColumn;
      }
  
      @Override
      public Object getObject(int columnIndex) {
      if (columnIndex == calculatedColumn) {
      return resultSet.getInt(3) + resultSet.getInt(5);
      }
      return super.getObject(columnIndex)
      }
      }

Managing Connector Plug-ins

This section provides a list of commands you can run to install and manage your connector plug-ins. For information accessing the Sisense CLI, see Using Sisense CLI Commands.

Installing Plug-ins 

      si connectors plugins install -connector-id<connectorID> -file<full path to plugin [.tar.gz]>

This command extracts the tar.gz file into the "plugins" folder under the connectors folder. Sisense compiles the plug-in’s Java files, and if it succeeds, Sisense activates the plug-in according to the extensions it implements and its scope.

The tar.gz file must be located on a shared partition /opt/sisense/storage

      > si connectors plugins install -connector-id sql -file /opt/sisense/storage/myPlugins/myConnectionPlugin.tar.gz
      Extract myConnectionPlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/sql/plugins
      and compile it

Viewing a List of Installed Plugins 

      si connectors plugins list

This command lists all plug-ins for all connectors and displays a table with the following columns: Connector ID | Name | Type | Scope | Plugin Version | SDK Version

  • Connector ID: The connector ID the plug-in belongs to.
  • Name: The plug-in name is taken from the name annotation.
  • Type: The interface plug-in implements.
  • Scope: Query, build which are taken from the scope annotation. Defined as a combination of a plugin scope and its methods scopes.
  • Plugin Version: The version of the plug-in that is defined in the manifest.
  • SDK Version: The version of the SDK the plug-in is compatible with that is included in plug-in’s tar.gz file.

Viewing a List of Plug-ins for a Connector 

      si connectors plugins list -connector-id <connectorID> -

This command lists all plug-ins for a specific connector and displays a table with the following columns: Name | Type | Scope | Plugin Version | SDK Version

      > si connectors plugins list -id sql
      Connector ID| Name | Type | Scope | Plugin Version | SDK Version
      sql | myConnectionPlugin | Connection | query | 1.0.0 | 1.0.3

Uninstalling a Plug-in 

      si connectors plugins uninstall -connector-id <connectorID>

This command uninstalls all the plug-ins in the "plugins" folder and removes all the files from the "plugins" folder.

Adding a Plug-in and Uninstalling It 

      > si connectors plugins install -id sql -file /opt/sisense/storage/myPlugins/myConnectionPlugin.tar.gz
      Extract myConnectionPlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/sql/plugins
      and compile it
  
      > si connectors plugins list -connector-id sql
      Connector ID| Name | Type | Scope | Plugin Version | SDK Version
      sql | myConnectionPlugin | Connection | query | 1.0.0 | 1.0.3
  
      > si connectors plugins uninstall -connector-id sql
      unload myConnectionPlugin class and delete the files from the "plugins" folder
  
      > si connectors plugins list -connector-id sql
      Connector ID| Name | Type | Scope | Plugin Version | SDK Version

Replacing a Plug-in 

      > si connectors plugins install -connector-id sql -file /opt/sisense/storage/myPlugins/myConnectionPlugin.tar.gz
      Extract myConnectionPlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/sql/plugins
      and compile it
  
      > si connectors plugins uninstall -connector-id sql
      unload myConnectionPlugin class and delete the files from the "plugins" folder
  
      > si connectors plugins install -connector-id sql -file
      /opt/sisense/storage/myPlugins/myNewConnectionPlugin.tar.gz
      Extract myNewConnectionPlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/sql/plugins
      and compile it
  
      > si connectors plugins list -connector-id sql
      Connector ID| Name | Type | Scope | Plugin Version | SDK Version
      sql | myNewConnectionPlugin | Connection | query | 1.0.0 | 1.0.3

Installing Plug-ins for Multiple Connectors 

      > si connectors plugins install -connector-id sql -file /opt/sisense/storage/myPlugins/mySqlPlugin.tar.gz
      Extract mySqlPlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/sql/plugins and
      compile it
  
      > si connectors plugins install -connector-id oracle -file /opt/sisense/storage/myPlugins/myOraclePlugin.tar.gz
      Extract myOraclePlugin.tar.gz and copy its content into /opt/sisense/storage/connectors/framework/oracle/plugins and
      compile it
  
      > si connectors plugins list
      Connector ID| Name | Type | Scope
      sql | mySqlPlugin | Connection | query
      sql | mySqlPlugin | Discovery | query
      oracle | myOraclePlugin | DataExtraction| build
      oracle | myOraclePlugin | ExecuteQuery | query