Our story on upgrading Graylog version 4.3.15 to 5.0

Gepost door op


Graylog is a open log centralization application that is easy to use for teams to digest huge amounts of logs from a diversity of systems and applications.  The enterprise version has a broad usability for security in a wide range of IT landscapes.


We needed to upgrade Graylog to version 5.X coming from the latest 4.3.15 and wanted to do this as good as it gets en wanted to share our proces to get there. Our environment exist of one server with Graylog-open and Elasticsearch, and production is one Graylog-Operations server and the backend has a cluster of three Elasticsearch servers. Everything in a virtual environment.

First one starts with investigating if there are breaking changes to get to Graylog 5.0. The only omission is that we need te upgrade mongodb to version 5.0 and need to check our version of mongodb. Checking mongodb most systems could be on version 4.2, guidelines state that it must be updated to version 4.4 first before upgrading to version 5.0. Graylog 4.3.15 is compatible with mongodb 4.4.

The second dependency is Elasticsearch 7.10.2 but we where already there.

There is a different approach on upgrading Graylog-Open and Graylog-Operations/Enterprise so take care on the difference on that. Our Graylog systems resides on Rocky Linux 8 so the approach here is CentOS/RHEL based.

Step one:  Upgrading mongodb from version 4.2 >> 4.4

  • Create a repo file: /etc/yum.repos.d/mongodb-org-4.4.repo
[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc
  • Stop the graylog service ( # systemctl stop graylog-server.service )
  • backup mongodb ( # mongodump )
  • Stop the mongodb service ( # systemctl stop mongod.service )
  • Check if there are updates for mongodb version 4.4 ( # dnf check-update )
  • Update mongodb to  ( # dnf update mongodb-org* ) and accept dependencies that can occur
  • Start the mongodb service ( # systemctl start mongod.service )
  • Check if the mongod service startend correctly (  # systemctl status mongod.service )
  • Start the Graylog Service ( # systemctl start graylog-server.service )
  • And check if Graylog is starting correct ( # less /var/log/graylog-server/server.log )
mongodb upgrade source: docs/upcoming/release-notes/4.4-upgrade-standalone
mongodb installation source: mogodb.com/docs/v4.4/tutorial

Step two in upgrading Graylog to version 5.0
(Open and Operations/Enterprise)

This is a two step operation, first mongodb must be upgraded to version 5.0 to resolve the dependency for Graylog 5.0

Step one: upgrade MongoDB  to from version 4.4 >> 5.0

  • create a repo file /etc/yum.repos.d/mongodb-org-5.0.repo
[mongodb-org-5.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/5.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc
  • Stop the graylog service ( # systemctl stop graylog-server.service )
  • Adjust the compatiblilty version of mongodb
    # mongosh
    >     db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
    > db.adminCommand( { setFeatureCompatibilityVersion: "4.4" } )
  • backup mongodb ( # mongodump )
  • Stop the mongodb service ( # systemctl stop mongod.service )
  • Check if there are updates for mongodb version 5.0 ( # dnf check-update )
  • Update mongodb to  ( # dnf update mongodb-org* ) and accept dependencies that can occur
  • Check if the mongod service startend correctly (  # systemctl status mongod.service )
  • Check if the mongod service startend correctly (  # systemctl status mongod.service )
mongodb installation source: mongodb.com/docs/v5.0/tutorial

Upgrade Graylog-Open from version 4.3.15 >> 5.0

Take care on updating Graylog-Open (free) and Graylog-Operations/Enterprise (licensed) as described below in the upgrade proces!

Update  Graylog-open

  • Check if Graylog is not running ( # systemctl status graylog-server.service )
  • Install the Graylog 5.0 repo file, this shall be an update to the current one and update Graylog-Open
sudo rpm -Uvh https://downloads.graylog.org/repo/packages/graylog-5.0-repository_latest.rpm
sudo yum clean all
sudo yum install graylog-server
sudo systemctl deamon-reload
  • Start the Graylog Service ( # systemctl start graylog-server.service )
  • Check if Graylog is starting correct ( # less /var/log/graylog-server/server.log )

After Graylog is working correct there is one option that can be removed in /etc/graylog/server/server.conf for not being supported anymore named
mongodb_threads_allowed_to_block_multiplier

If Graylog is working well for any version of graylog working with mongdb 5.0, FeatureCompatibilityVersion can be set at the latest level.
  • Adjust the compatiblilty version of mongodb
    # mongosh
    >     db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
    > db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )
You could check for differences between your current  server.conf file and the new one installed bij Graylog server.conf.rpmnew and correct entries to your own configuration or preferences. ( # sdiff server.conf server.conf.rpmnew)


Upgrade Graylog-Operations/Enterprise from version 4.3.15 >> 5.0

This proces is a little different, before 5.0 Graylog had several packages to come to the so called enterprise version of it. The additions should be removed before updating.
  • Check if Graylog is not running ( # systemctl status graylog-server.service )
  • Remove the plugin packages, install the Graylog 5.0 repo file, this shall be an update to the current one and update Graylog-Operations to 5.0
sudo yum remove graylog-integrations-plugins graylog-enterprise-plugins graylog-enterprise-integrations-plugins
sudo rpm -Uvh https://downloads.graylog.org/repo/packages/graylog-5.0-repository_latest.rpm
sudo yum clean all
sudo yum install swab graylog-server  graylog-enterprise
  • Start the Graylog Service ( # systemctl start graylog-server.service )
  • And check if Graylog is starting correct ( # less /var/log/graylog-server/server.log )
After Graylog is working correct there is one option that can be removed in /etc/graylo/server/server.conf for not being supported anymore
  • mongodb_threads_allowed_to_block_multiplier 
You could check for differences between your current  server.conf file and the new one installed bij Graylog server.conf.rpmnew
and correct entries to your own configuration or preferences.
  ( # sdiff server.conf server.conf.rpmnew)

Sources on upgrading Graylog 4.3 >> 5.0

Graylog documents: graylog.org/docs/5-0/upgrading
Graylog Releases:    graylog.org/releases
Graylog-5.0 blog:    graylog.org/post/graylog-5-0

Youtube: Whats new in Graylog 5.0




Reacties

Een reactie posten