Activating the Dashboard and Widget Deltas MongoDB Structure Change - BETA

To achieve a significant storage space reduction, reduce resource consumption, and increase stability in MongoDB, original dashboards and widgets will be stored in a base structure within the database, and each user's changes to the base dashboards or widgets will be stored in a deltas structure within the database. In this way, each user will continue to see dashboards or widgets with their personal changes applied but the database does not need to store a full copy of the dashboard or widget for each change.

Note

  • This feature is currently in Beta. As such, it should be tested in a non-production environment before the migration is performed.

  • Recovery snapshots with a single collection must not be used for a rollback when the deltas feature is activated.
  • Ensure that there is a lot of free space in the database prior to migrating to this structure, as the structure migration process may temporarily require up to 150% of the current database space requirements for dashboards and widgets. After the migration, testing, and deleting the old widgets collection, the database size will be significantly smaller than it was before the migration.
  • The migration process may take less than 30 minutes, or for a large DB it may take a lot longer. For example, for a 20Gb DB, it may take 2-4 hours. Ensure that it will not be interrupted with a timeout error. The migration time also depends on the number of CPUs you have available. This job utilizes all available CPU cores, so increasing CPU cores can speed up the migration process. It is recommended to perform the installation or upgrade, including the migration process, at a time that least impacts users.

  • After this process, the widgets collection will not be automatically deleted. It is kept in the DB as a backup. However, Sisense will no longer use this collection, so it can be deleted once the new approach's functionality is confirmed.

Enabling Widget Deltas

  1. Backup your MongoDB database.
  2. In the config.yaml file, set the enable_widget_deltas parameter to true.
  3. Run the installation or upgrade.

Upgrading

The widget deltas upgrade is similar to the standard Sisense version upgrade and can happen as part of it. There are two possible approaches, either of which are acceptable:

  • Upgrade to the new major Sisense version first and use the standard configuration (with deltas disabled) and then run another upgrade to the same version once the version functionality is confirmed.

  • Upgrade the new major Sisense version already with new approach and deltas enabled. In this case, it is still possible to disable deltas, but use the standard upgrade later if needed.

Once the upgrade process is initiated, the migration job is started automatically. The main purpose of this migration job is to convert widgets from a 'widgets' collection to 'widgetsBases' and 'widgetsDeltas' collections.

Checking Logs During/After Migration

The migration runs inside a 'migration' job in k8s. In order to verify the logs created, the migration pod name should be used and then the logs can be checked using the following command:

kubectl -n sisense logs 'pod-name' -f