Distributing ElastiCube Builds to Query Nodes

The Sisense Orchestrator Service is an automated service that distributes builds and balances queries across an Setting Up ElastiCube Sets. The Sisense Orchestrator Service is installed during the default installation of Sisense. To distribute ElastiCube builds across all query nodes, you must configure the Sisense Orchestrator Service. When an ElastiCube is building, or receiving a build from the build node, the Sisense Orchestrator Service redirects requests to another available ElastiCube in the set.

After you have set up all of your nodes, you can verify that each query node is working by stopping the Sisense.ECMS service. When this service is stopped, queries should be redirected to the other query nodes in your configuration.

Note :

After configuring the Sisense Orchestrator Service, the directory C:\ProgramData\SiSense\PrismServer\ElastiCubeData\ and the relevant ElastiCube folders within it must be shared with permissions for ‘Everyone’ so Sisense can build or update the ElastiCube s in those folders.

Keep in mind that each of your nodes must have sufficient disk space to support the build node.

In addition, the user of the build node must have Administrator access to the service Sisense.Orchestrator on each of the query nodes, so the build node can write to them. You can set this in the properties of the Sisense.Orchestrator service.

The Sisense Orchestrator Service is defined in a JSON file called config.json in the Sisense.Orchestration Config folder (C:\ProgramData\Sisense\Sisense.Orchestrator\config.json) of your Sisense Installation folder of your build node.

To distribute your ElastiCube builds through the Sisense Orchestrator Service, you define and save the config.json file. After saving the file, the Sisense Orchestrator Service automatically begins to build ElastiCubes in your ElastiCube Set. You can review the progress of the build in the C:\ProgramData\Sisense\application-logs\orchestrator-service folder in a log file whose file name you define in the JSON file.

The default config.json file has a basic structure with no values.

To learn more about how to configure the config.json file, see the following tutorial:

Sisense supports a wide range of objects and values and several examples and their descriptions are provided in the table below:

Key	Example	Value
general	<="" td=""> `{ "general":{ "logLevel":"INFO", "emails":[ { "email":"john.test@sisense.com", "name":"Example" } ]`	The General object contains two objects, logLevel and emails that define what types of logs Sisense generates and where to send them to.
logLevel	"logLevel":"INFO",	The type of info returned in the log. The possible values you can enter are Info, debug, and Error.
emails	`"emails":[ { "email":"john.test@sisense.com", "name":"Example" } ]`	This object defines who receives an email and their email address when an email is triggered according to the events you define in the mail object.
email	`"emails":[ { "email":"john.test@sisense.com", "name":"Example" } ]`	The email address where emails are to be sent depending on the value of the When key.
name	`"emails":[ { "email":"john.test@sisense.com", "name":"Example" } ]`	The name of the recipient of the email.
cubes	`"cubes":{ "HA_QueryWeb1":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.91\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" }, "HA_QueryWeb2":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.93\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" } },`	The cubes object contains all your ElastiCubes in your configuration and their location. As the Sisense Orchestrator Service is installed on the build node, the build cubes are local, so you must define the name of the ElastiCube. For ElastiCubes on a query node, you must define the ElastiCube name, URL of the remote server, and its directory.
build ElastiCubes	`"HA_QueryWeb1":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.91\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" },`	The Build ElastiCube object defines the ElastiCubes that will be used as Build ElastiCubes nodes.
ecube	`"HA_QueryWeb1":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.91\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" },`	The name of the ElastiCube.
url	`"HA_QueryWeb1":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.91\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" },`	The URL address of the ElastiCube server. If the URL is to a secure address, the value should include the username and password credentials. For example, “ssh://username:password@10.50.1.128:/C/ecubes”. See Scenario 1 for an example.
localPath	`"HA_QueryWeb1":{ "ecube":"Sample ECommerce", "url":"\\\\127.128.129.91\\C:\\ProgramData\\ Sisense\\PrismServer\\ElastiCubeData\\", "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\" },`	Directory of the ElastiCubes.
tasks	`"tasks":{ "task1":[ { "build":{ "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ], "queue":[ "entire" ] } },`	The Tasks object contains a task array that defines which ElastiCube should be built, the type of build, and to which ElastiCubes the build should be distributed to. In high availability scenarios, running multiple concurrent builds is not recommended and can result in stability issues. Sisense recommends that tasks are spaced out so each build is complete before the next one begins.
task	`"tasks":{ "task1":[ { "build":{ "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ], "queue":[ "entire" ] } },`	The name of the task array. Currently, you can define only one task. Defining multiple tasks in the config.json file may cause builds to fail. In high availability scenarios, running multiple concurrent builds is not recommended and can result in stability issues. Sisense recommends that tasks are spaced out so each build is complete before the next one begins.
build	`"build":{ "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ],`	The build object defines the ElastiCube to be built and distributed.
cube	`"build":{ "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ],`	The name of the cube to be built.
queue	`"queue":[ "accumulate", "entire" ]`	The value of this key is the type of build Sisense should attempt and in what order. There are two possible values delimited by a comma: accumulate: Attempts an accumulative build. entire: Attempts an entire build. schemaChanges: Attempts to update the build only if changes were made to the schema since the previous build. Sisense recommends the following value: "accumulate","entire" In this example, Sisense attempts to do an accumulative build first, and if it fails, Sisense attempts an entire build. If the build fails for all builds, Sisense retries again according to the next scheduled build.
distribute	`{ "distribute":[ "HA_QueryWeb1", "HA_QueryWeb2" ] },`	Determines which cubes the latest build should be distributed to. The value should be the ElastiCube name for your ElastiCubes on your query nodes. For example, [“cube1, cube2, cube3”]. You can add multiple ElastiCubes delimited by a comma separator.
reattach	`"reattach":{ "deleteOldDbfarm":true, "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ] } },`	Reattach is an object that contains two objects, DeleteOldDbfarm and Cube.
deleteOldDbfarm	`"reattach":{ "deleteOldDbfarm":true, "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ] } },`	A boolean value that determines if an old ElastiCube is to be deleted after it has been updated.
cube	`"reattach":{ "deleteOldDbfarm":true, "cube":[ "HA_QueryWeb1", "HA_QueryWeb2" ] } },`	The name of the ElastiCube to be reattached or not.
mail	`{ "mail":"build" }`	Indicates when an email alert is triggered. Email alerts can be triggered for the following scenarios: None: No email is ever sent. Build: An email is sent when a build is complete regardless if the build was successful or not. Fail: An email is sent when a build fails.
scheduler	`"scheduler":[ { "task":"task1", "schedule":"15 14 * * *", "enabled":true } ]`	The Scheduler object contains an array of tasks that define when a task is to be task is to be initiated.
task	`"scheduler":[ { "task":"task1", "schedule":"15 14 * * *", "enabled":true } ]`	Defines which task to perform and the order. Currently, Sisense only supports one task.
schedule	`"scheduler":[ { "task":"task1", "schedule":"15 14 * * *", "enabled":true } ]`	The time that a task is to be initiated in Cron format. Some examples: -To run a build each night at midnight, enter the value "0 0 * * * " -To run a build every hour, enter the value "0 * * * *" See the full Cron format reference here.
enabled	`"scheduler":[ { "task":"task1", "schedule":"15 14 * * *", "enabled":true } ]`	A boolean value that indicates if the task is to be executed or not.

Use Cases

The following section presents two use cases and an example config.json file that demonstrates how to support these use cases. For assistance with additional HA configurations, contact your Customer Service Manager.

Scenario 1 – Single Stack

In this example, there is one build node and two query nodes. Each query node is hosted on a separate machine while the components that make up the query node are hosted on the same machine as a single application stack.

In this example, the build cube is called “Sample ECommerce”. This ElastiCube is distributed to two ElastiCube servers defined in the cube1 and cube2 objects. The build cube is distributed first to cube1 and then to cube2 as defined in the tasks object. The first time the Sisense Orchestrator Service tries to build the BuildTest1 ElastiCube , it attempts an accumulative build and if that fails, an entire build. After each build is replicated, the previous version of the ElastiCube is deleted as specified in the reattach object. This task takes place hourly. If the distribution fails, Sisense sends an email to johndoe@Sisense .com with a debug log attached.

{  
   "general":{  
      "logLevel":"INFO",     //Determines type of log events recorded
      "emails":[  
         {  
            "email":"john.test@sisense.com",    //Who to send reports
            "name":"Example1"  
         }
      ]
   },
   "cubes":{  
      "localCubeBuildTest1":{  
         "ecube":"Sample ECommerce"      //Name of the ecube to be distributed
      },  
      "cube1":{  
         "ecube":"Sample ECommerce",    //Name of the ecube on the query node
         "url":"\\\\127.128.129.91\\C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce",
         "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce"
      },
      "cube2":{  
         "ecube":"Sample ECommerce",   //Name of the ecube on the query node
         "url":"\\\\127.128.129.92\\C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce",
         "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce"
      }
   },
   "tasks":{  
      "task1":[  
         {  
            "build":{       
               "cube":[     
                  "cube1",
                  "cube2"
               ],
               "queue":[        
                  "accumulate",   //The first type of build to be attempted
                   "entire"   //The second type of build to perform if the first fails
               ]
            }
         },
         {  
            "distribute":[   //The order of how ecubes on the build node is distributed to query nodes
               "cube1",    
               "cube2"
            ]
         },
         {  
            "reattach":{  
               "deleteOldDbfarm":true,
               "cube":[  
                  "cube1",
                  "cube2"
               ]
            }
         },
         {  
            "mail":"build"   //What event triggers an email
         }
      ]
   },
   "scheduler":[  
      {  
         "task":"task1",   //The name of the task object that is to be executed
         "schedule":"0 * * * *",   //How often to perform a task in Cron format
         "enabled": true
      }
   ]
}

Scenario 2 – Distributed Stack

In this example, there are three query nodes in which each component is located in a different remote location. This scenario, while costly to implement, demonstrates a distributed application stack configuration where each component is replicated and located on a separate server independent of the other components. If any component fails, the rest of the architecture remains intact. For example, the web server is on one server, the ElastiCube s are stored on another server, and the MongoDB is located on another server.

The build cube in this example is called “BuildTest1”. This ElastiCube is distributed to three remote ElastiCube servers defined in the cube1, cube2, and cube3 objects. The build cube is distributed first to cube1, then cube2, and then cube3 as defined in the tasks object. The URLs are standard URLs for cube1 and cube2 while the URL for cube3 is secured through SSH. When the URL is secured, you must provide the URL and the required credentials to access it.

The first time the Sisense Orchestrator Service tries to build the BuildTest1 ElastiCube , it attempts an accumulative build and if that fails, an entire build. If both builds fail, Sisense attempts to update the schema only. After each build is replicated, the previous version of the ElastiCube is deleted as specified in the reattach object. This task takes place At 14:15 on the 1st of every month. After a build is successful, Sisense sends an email to johndoe@Sisense .com.

{  
   "general":{  
      "logLevel":"DEBUG",
      "emails":[  
         {  
            "email":"john.test@sisense.com",
            "name":"Example1"
         }
      ]
   },
   "cubes":{  
      "localCubeBuildTest1":{  
         "ecube":"Sample ECommerce"
      },  
      "cube1":{  
         "ecube":"Sample ECommerce",
         "url":"\\\\127.128.129.91\\C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce",
         "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce"
      },
      "cube2":{  
         "ecube":"Sample ECommerce",
         "url":"\\\\127.128.129.92\\C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce",
         "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce"
      },
        "cube3":{  
         "ecube":"Sample ECommerce",
         "url":"\\\\127.128.129.93\\C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce",
         "localPath":"C:\\ProgramData\\Sisense\\PrismServer\\ElastiCubeData\\Sample ECommerce"
      }

   },
   "tasks":{  
      "task1":[  
         {  
            "build":{  
               "cube":[  
                  "cube1",
                  "cube2", 
                   "cube3"
               ],
               "queue":[  
                  "accumulate",
                   "entire",
                   "schemachanges"
               ]
            }
         },
         {  
            "distribute":[  
               "cube1",
               "cube2",
                "cube3"
            ]
         },
         {  
            "reattach":{  
               "deleteOldDbfarm":true,
               "cube":[  
                  "cube1",
                  "cube2",
                   "cube3"
               ]
            }
         },
         {  
            "mail":"fail"
         }
      ]
   },
   "scheduler":[  
      {  
         "task":"task1",
         "schedule":"15 14 1 * *",
         "enabled": true
      }
   ]
}

Add Query Nodes

In some configurations, replicating components can improve performance by scaling out and preventing any potential bottlenecks. Each component you replicate must be added to the ElastiCube Set and the config.json file. You can continue to add query nodes according to your requirement so long as the nodes are included in the config.json file.
To add a query node:

Stop the Sisense Orchestrator Service. Open Windows Services, select Sisense.Orchestrator, and click .
In the Sisense , click Admin and select the Data Sources tab on the left.
Hover over the ElastiCube you want to add to the ElastiCube Set and click on the menu that appears. Click Add to ElastiCube Set and select the set that you want to add the ElastiCube to.
Edit the config.json located at Sisense/Sisense.Orchestration/Config/ and add the new ElastiCube to the Cubes object.
Save the config.json file.
Restart the Sisense Orchestrator Service.

Remove Query Nodes

You can remove query nodes by removing them from the ElastiCube Set and the config.json file.

To remove a query node:

Stop the Sisense Orchestrator Service. Open Windows Services, select Sisense.Orchestrator, and click .
In the Sisense , click Admin and select the Data Sources tab on the left.
In the ElastiCube Set table, click and clear the checkboxes of the ElastiCube s to be removed. Click Save after you have selected the relevant ElastiCubes.
Edit the config.json located at Sisense/Sisense.Orchestration/Config/ and delete the relevant ElastiCube s from the Cubes object.
Save the config.json file.
Restart the Sisense Orchestrator Service.

Next Steps

Setting Up ElastiCube Sets

.r.