Steps to Upgrade

To upgrade your data store, first install the latest software to upgrade the Admin CLI. See Installation for more details. Before proceeding with the upgrade, see General Upgrade Notes.

Then, use the upgraded CLI to verify that all the Storage Nodes do not violate any basic requirements.

kv-> verify prerequisite
...
Verification complete, no violations.  

Once all violations (if any) are corrected, install the latest software on a Storage Node that is running an admin service.

Do the following:

  1. On a Storage Node running an admin service:

    1. Place the updated software in a new $KVHOME directory. If Storage Nodes share this directory using NFS, you only need to do this once for each shared directory.

    2. Stop the Storage Node using the CLI. Stopping the Storage Node shuts down the admin service on that Storage Node.

      If you have configured the Storage Node to automatically start the Storage Node Agent when the system reboots using /etc/init.d, or some other mechanism, then you need to modify your system startup script to point to $NEW_KVHOME. This is needed so that $KVHOME points to the correct directory after any system reboot.

      Once you have modified the system startup script, shutdown the Storage Node:

      java -Xmx64m -Xms64m \
      -jar $KVHOME/lib/kvstore.jar stop -root $kvroot
    3. Restart the Storage Node using the updated software release:

      nohup java -Xmx64m -Xms64m \
      -jar $NEW_KVHOME/lib/kvstore.jar start -root $kvroot & 

      (If the system is configured to automatically restart the Storage Node Agent, this step may not be necessary.)

    4. Use the CLI to connect to the Storage Node which is now running the updated software release. See

      java -Xmx64m -Xms64m \
      -jar $NEW_KVHOME/lib/kvstore.jar runadmin -port 5100 -host node1 \
      -security $KVROOT/security/root.login
    5. Verify that all the Storage Nodes in the data store are running the proper software level required to upgrade to the new release.

      kv-> verify prerequisite
      Verify: starting verification of store mystore 
      based upon topology sequence #315
      ...
      Verification complete, no violations.

      The most likely reason for a violation is if you are attempting a release level downgrade. For example, it is illegal to downgrade from a higher minor release to a lower minor release.

      Note:

      It is legal to downgrade from a higher major release to a lower major release. For example downgrading from 22.3.15 to 22.3.14 would be legal, while downgrading from 22.3.15 to 22.2.15 would not be legal.

      In any case, if the verify prerequisite command shows violations, resolve the situation before you attempt to upgrade the identified Storage Nodes.

    6. Obtain an ordered list of the Storage Nodes to upgrade.

      kv-> show upgrade-order
      Calculating upgrade order, target version: 22.3.15, prerequisite: 20.1.5
      sn3 sn4
      sn2 sn5
      sn6
      This command displays one or more Storage Nodes on a line. Multiple Storage Nodes on a line are separated by a space. If multiple Storage Nodes appear on a single line, then those Storage Nodes can be safely upgraded simultaneously, if desired. When multiple Storage Nodes are upgraded simultaneously, the upgrade must be completed on all Storage Nodes before the Storage Nodes next on the list can be upgraded. If at some point you lose track of which group of Storage Nodes should be upgraded next, you can always run the show upgrade-order command again. In the above example, you can upgrade sn3 and sn4 simultaneously. Then upgrade sn2 and sn5 simultaneously. And, finally, upgrade sn6.

      Note:

      You must completely upgrade a group of Storage Nodes before continuing to the next group. That is, upgrade sn3 and sn4 before you proceed to upgrade sn2, sn5, or sn6.
  2. For each of the Storage Nodes in the first group of Storage Nodes to upgrade (sn3 and sn4, in this example):

    1. Place the new release software in a new $KVHOME directory. If Storage Nodes share this directory using NFS, you need to do this only once for each shared directory.

    2. Stop the Storage Node using the CLI utility.

      If you have configured the Storage Node to automatically start the Storage Node Agent when the system reboots using /etc/init.d, or some other mechanism, then you need to modify your system startup script to point to $NEW_KVHOME. This is needed so that $KVHOME points to the correct directory after any system reboot.

      Once you have modified the system startup script, shutdown the Storage Node:

      java -Xmx64m -Xms64m \
      -jar $KVHOME/lib/kvstore.jar stop -root $kvroot
    3. Restart the Storage Node using the updated software release:

      nohup java -Xmx64m -Xms64m \
      -jar $NEW_KVHOME/lib/kvstore.jar start -root $kvroot & 

      (If the system is configured to automatically restart the Storage Node Agent, this step may not be necessary.)

  3. Verify the upgrade before upgrading your next set of Storage Nodes. This command shows which Storage Nodes have been successfully upgraded, and which Storage Nodes still need to be upgraded.

    kv-> verify upgrade
    Verify: starting verification of store mystore 
    based upon topology sequence #315
    ...
    Verification complete, 0 violations, 3 notes found.
    Verification note: [sn2]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer
    Verification note: [sn5]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer
    Verification note: [sn6]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer

    Those Storage Nodes which have been upgraded are identified with a verification message that includes the current software version number:

    Verify upgrade: Storage Node [sn3] on node03:5000
    Zone: [name=Boston id=zn1 type=PRIMARY allowArbiters=false 
    masterAffinity=false] Status: RUNNING   Ver: 22.3.15 2022-12-24 09:50:01 UTC 
    Build id: c8998e4a8aa5 Edition: Enterprise

    Those Storage Nodes which still need to be upgraded are identified in two different ways. First, the verification message for the Storage Node indicates that an upgrade is still necessary.

    Verification note: [sn2]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer

    Second, the very end of the verification output identifies all the Storage Nodes that still need to be upgraded.

    Verification complete, 0 violations, 3 notes found.
    Verification note: [sn2]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer
    Verification note: [sn5]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer
    Verification note: [sn6]    Node needs to be upgraded from
    21.2.15 to version 22.3.15 or newer

    Note:

    If the verification shows Storage Nodes you thought were upgraded as being still in need of an upgrade, you must resolve that problem before upgrading the other Storage Nodes in your data store. As a sanity check, you can verify just those Storage Nodes that you finished upgrading.

    kv-> verify upgrade -sn sn3 -sn sn4
    Verify: starting verification of store mystore 
    based upon topology sequence #315
    ...
     
    Verification complete, no violations.
  4. You can continue upgrading groups of Storage Nodes, as identified by the show upgrade-order command. Follow the procedure outlined above. Stop the existing release Storage Node using the existing release stop command, then restart the Storage Node using the new release start command. Continue doing this until all Storage Nodes have been upgraded.

    If at some point you lose track of which group of nodes should be upgraded next, you can always run the show upgrade-order command again:

    kv-> show upgrade-order
    Calculating upgrade order, target version: 22.3.15, prerequisite: 20.1.5
    sn2 sn5
    sn6
  5. When you are all done upgrading your Storage Nodes, the verify upgrade command will show no verification notes at the end of its output:

    kv-> verify upgrade
    Verify: starting verification of store mystore 
    based upon topology sequence #315
    ...
     
    Verification complete, no violations.