August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

Categories

August 2025
M T W T F S S
 123
45678910
11121314151617
18192021222324
25262728293031

DSO load failed: SSL0166E: Failure attempting to load GSK library (libgsk7ssl.so) Configuration Failed

The following error will be displayed in the error_log while starting the Apache webserver after enabling the SSL module.

Error: [Tue Dec 09 00:00:18 2014] [notice] (2019)DSO load failed: SSL0166E: Failure attempting to load GSK library (libgsk7ssl.so) Configuration Failed

OS – Linux

download (4)
SSL0166E: Failure attempting to load GSK library (libgsk7ssl.so) Configuration Failed

To check if the GSK library or the Key kdb file is corrupted, Use the following commands.

/opt/HTTPServer/bin/gsk7capicmd -cert -list -db key.kdb

If the following error is seen, Then it is due to the bit conflict issue between the server and the OS.

error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

ie, If you have a 32-bit IHS installed and you don’t have the supportive libraries installed in the os 32 bit libraries, (/usr/lib). The above error is seen.

To check the bit of IHS, Use the following command:

/opt/HTTPServer/bin/httpd -V

Solution: To resolve this issue, You have to find out the missing library using the following command.

yum whatprovides libstdc++.so.5

Command’s output:

Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is receiving updates from RHN Classic or RHN Satellite.
compat-libstdc++-33-3.2.3-69.el6.i686 : Compatibility standard C++ libraries

Now install the library that is required for libstdc++.so.5.

yum install compat-libstdc++-33-3.2.3-69.el6.i686

check the /usr/lib folder, The libstdc++.so.5 is listed.

Restart the IHS server, The ssl issue will be resolved.

A+ on apache2.4 ssl

Here’s my config for apache2.4:

sslas

1) 4096 bit key:
You will need to generate a 4096 bit key instead of the default 2048 bit key to get the key exchange to 100%. To do this, run letsencrypt-auto with this flag: --rsa-key-size 4096

./letsencrypt-auto --agree-dev-preview --server \
https://acme-v01.api.letsencrypt.org/directory auth --rsa-key-size 4096

2) SSL Settings:
Add these directives to your apache2 config in the vhost section:

SSLEngine on
SSLCompression off
SSLCipherSuite "HIGH:!aNULL:!MD5:!3DES:!CAMELLIA:!AES128"
SSLHonorCipherOrder on
SSLProtocol TLSv1.2
SSLUseStapling on

3) SSL Certificate and Key File Settings:
Also add these directives to your apache2 config in the same vhost section. Keep in mind you need to update the path to your specific certificate and key that relate to your domain (<<< YOUR DOMAIN HERE >>>):

SSLCertificateFile "/etc/letsencrypt/live/<<< YOUR DOMAIN HERE >>>/fullchain.pem"
SSLCertificateKeyFile "/etc/letsencrypt/live/<<< YOUR DOMAIN HERE >>>/privkey.pem"

4) DH parameters:
Generate >=4096 dhparams using your openssl binary. This will take some time:

openssl dhparam -out /etc/ssl/private/dhparams_4096.pem 4096

If you have openssl >= 1.0.2d installed (type openssl version to find out), you can use the following line in your apache2 vhost config:

SSLOpenSSLConfCmd DHParameters "/etc/ssl/private/dhparams_4096.pem"

If you do not have openss >= 1.0.2d, you will need to append your dhparameters to the bottom of your certificate file:

cat /etc/letsencrypt/live/<<< YOUR DOMAIN HERE >>>/fullchain.pem \
    /etc/ssl/private/dhparams_4096.pem > \
    /etc/letsencrypt/archive/<<< YOUR DOMAIN HERE >>>/fullchain_dhparams_4096.pem

Then you will use this file in place of your SSLCertificateFile above:

SSLCertificateFile "/etc/letsencrypt/archive/<<< YOUR DOMAIN HERE >>>/fullchain_dhparams_4096.pem"

Keep in mind that if you manipulate the certificate (issue a new one, etc), you will need to repeat this step as the dhparams will not be added to that certificate!

5) Headers:
You need to set a Public-Key-Pin header. Generate the first pin against the letsencrypt chain cert with the following code (read more here: https://raymii.org/s/articles/HTTP_Public_Key_Pinning_Extension_HPKP.html88):

openssl x509 -noout -in /etc/letsencrypt/live/<<<< YOUR DOMAIN HERE >>>>/chain.pem -pubkey | \
openssl asn1parse -noout -inform pem -out /tmp/fingerprint.key;
openssl dgst -sha256 -binary /tmp/fingerprint.key | openssl enc -base64

The next pin comes from your private key. You should keep this key in a safe place:

openssl x509 -noout -in /etc/letsencrypt/live/<<<< YOUR DOMAIN HERE >>>>/privkey.pem -pubkey | \
openssl asn1parse -noout -inform pem -out /tmp/fingerprint.key;
openssl dgst -sha256 -binary /tmp/fingerprint.key | openssl enc -base64

You will use these fingerprints in the section below (<<< YOUR CUSTOM PIN HERE >>>).

I haven’t figured out a good programmatic way to get the chain cert keys, but I have found that you can get them with https://dev.ssllabs.com/ssltest/analyze.html70 by analyzing your own site once it’s already running.

Also add these directives to your apache2 config in the same vhost section:

Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure
Header always set Public-Key-Pins "pin-sha256=\"<<< YOUR CHAIN PIN HERE >>>=\"; pin-sha256=\"<<< YOUR PRIVATE KEY PIN HERE >>>\"; max-age=31536000; includeSubDomains"
Header always set X-Frame-Options SAMEORIGIN
Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
Header always set X-Content-Type-Options nosniff

6) SSL Stapling Cache:
Add the following line to your ssl.conf /etc/apache2/conf-available/ssl.conf file:

echo 'SSLStaplingCache shmcb:/tmp/stapling_cache(2097152)' >> \
/etc/apache2/conf-available/ssl.conf`

Enable your ssl.conf file:

a2enconf ssl

7) Restart Apache:

service apache2 restart

8) Test your site at https://www.ssllabs.com/ssltest/analyze.html

keystores and truststores for websphere

This post will guide you on how to create and configure custom keystores and truststores instead of using the default keystores and truststores that are created during the profile creation.

Imagine if you have two or more applications running on the same application server profile, Then we can use this feature if any application needs specific certificates. The keystores and truststores can be configured at server/cluster level.

-> Default Keystores and Truststores are created and configured by default at cell level during the profile creation.

-> Custom Keystores and Truststores can be created and configured for at server/application/cluster/node level. This will ensure the keystores are independent of other applications.

Incase of the following error:

Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

The following steps can be used to fix this error.

Creation of Keystore and Truststores from console:

Step 1:  Navigate to Security >SSL certificate and key management > Manage endpoint security configurations > Inbound > NodeName > ServerName >  Key stores and certificates > New

Step 2: Fill the relevant fields.

Name : CustomKeystore
Description : Keystore specific to an application
Management scope: This is grayed out, This shows the scope it is currently in, (Cell)(Node)(Server) -> It belongs to server scope.
Path : The Path Where you want the keystore.p12 to be created.
Password : Defualt will be WebAS
Type : I have choosen p12, You can choose what ever you want to use.
Remotely Managed : In case if the path of the key file is present in a remote server, You have to check this option and provide the alias name of that server which is present in serverindex.xml or else error will be displayed.

Step 3: Once you click on apply, a file with the name that you have given in the path will be created.

Step 4: The same process can be followed to create a trust store.

With the above steps, The keystore and truststores are created.

Now to configure the websphere application server to identify these keystores and truststores and override the default keystore and truststore properties, The following steps are required to be followed.

Step 1: Login to the websphere application server console,

Step 2: Navigate to Servers > Server Types > WebSphere application Servers > Server (Select the server that you want these customized keystores to be used) > Java and Process Management > Process Definition > Java Virtual Machine

Step 3:  Add these following properties to the Generic arguments.

-server -Djavax.net.ssl.trustStore=Path to truststore/trust.p12 -Djavax.net.ssl.trustStorePassword=WebAS -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.keyStore=Path to Keystore/key.p12 -Djavax.net.ssl.keyStorePassword=WebAS -Djavax.net.ssl.keyStoreType=PKCS12

Step 4:  Apply the settings and restart the JVM and check the process information

ps -eaf | grep java | grep <servername>

This should reflect the parameters added in the Generic arguments.

That’s it. You are now using the customized keystores and certificates.

gskcmd command

The following instructions will guide you to create csr and import the signed crt in the default kdb of IHS.

Step 1: Finding the Kdb

The default Kdb will be present in the IHS installation path. Find the key.kdb and take a back up before executing the commands.

Step 2: Creation of Csr

Use the following command to create the csr.
gskcmd will be present in the bin folder of IHS. gsk7cmd can be used based on the version of IHS.
./gskcmd -certreq -create -db /opt/IHS/SSL/key.kdb -pw WebAS -label lablename -dn “CN=hostname, OU=ou name, O=organization, L=Location, ST=state, C=country code” -size 2048 -file filename.csr
-db is the key.kdb path
-pw is the kdb password, The defualt password is WebAS
-label is any name that you choose the certificated to be labled.
-dn is the domain information
CN = Common name (dns name of the name with which you are trying to use https://hotname protocol)
OU = Organization unit name (ex: IT services)
O = Organization that you are working in, or to whom you are configuring the certificate.
L = Location (ex: Germany)
C = 2 digit country code (ex: GE)
This command will create a filename.csr with the information that you have provided.

Step 3: Certificate request

The csr has to be sent to the certificate signing authorities like verisign, Thwate or Geotrust. They will create a crt file and send the server certificate along with the root and intermediate certificates.

Step 4: Importing the server certificate

Following command will import the server certificate to the kdb file.

Save the certificate file received from the signing authority as server.crt and execute the following command.

gskcmd -cert -receive -file server.crt -db /opt/IHS/SSL/key.kdb -pw WebAS -label servercertificate -format ascii

This command will import the server certificate.

Step 5: Set the server certificate to default

Following command will make the server certificate as default. When you access the https protocol of the website, The default certificate will be choosen.

gskcmd -cert -setdefault -label servercertificate -db /opt/IBM/HTTPServer/SSL/key.kdb

Step  6: Importing the CA/Intermediate certificate

Along with the server certificate, The signing authority will send the intermediate certificate, This certificate has to be imported to prevent the ssl handshake errors.

Save the Intermediate certificate as rootCAcertificate.crt and execute the following command.

gskcmd -cert -add -file rootCAcertificate.crt -db /opt/IHS/SSL/key.kdb -label primaryCA -pw WebAS -format ascii

Step 7: Populate the certificates

gskcmd -cert -populate -db /opt/IHS/SSL/key.kdb -pw WebAS
 

Now add the kdb path in the httpd.conf in ssl module tags and restart the apache server with -DSSL option.

Basic commands of gskcmd to check the certificates:

List the certificates : ./gskcmd -cert -list -db /opt/IHS/SSL/key.kdb 

Check the details of the certificates:  

./gskcmd -certreq -details -label labelname -db /opt/IHS/SSL/key.kdb
./gskcmd -cert -details -label labelname -db /opt/IHS/SSL/key.kdb

Delete a certificate : ./gskcmd -cert -delete -label labelname -db /opt/IHS/SSL/key.kdb

Extract a certificate : ./gskcmd -cert -extract -db certificate.jks -label labelname -target test_myne.cer -type jks

fix pack on WebSphere Application Server ND 7 and IBM HTTP Server (IHS)

Steps to apply fix pack on WebSphere Application Server ND 7 and IBM HTTP Server (IHS)

 

 

ollowing steps will guide you to upgrade the WAS ND from 7.0.0.9 to 7.0.0.25 version and IHS server using silent installation on Solaris environment.

Following files are required for the upgrade and they can be downloaded from IBM Site :

-> 7.0.0-WS-IHS-SolarisSparc64-FP0000025.pak – Fix pack for IHS.

-> 7.0.0-WS-PLG-SolarisSparc64-FP0000025.pak – Fix pack for IHS Plugin.

-> 7.0.0-WS-WASSDK-SolarisSparc64-FP0000025.pak – Fix pack for WAS SDK.

-> 7.0.0.25-WS-UPDI-SolarisSparc64.tar.gz – Update installer.

-> 7.0.0-WS-WAS-SolarisSparc64-FP0000025.pak – Fix pack for WAS Application server.

Step 1 :

Installation of update installer :  

Update installer tool is used to install and uninstall fix pack on WAS software.

->Extract the 7.0.0.25-WS-UPDI-SolarisSparc64.tar.gz and navigate to UpdateInstaller directory.

->Create a response file with the following format and save the file as update.txt.

Update.txt :

-OPT disableOSPrereqChecking=”true”

-OPT silentInstallLicenseAcceptance=”true”

-OPT allowNonRootSilentInstall=”true”

-OPT installLocation=”/WebSphere/UpdateInstaller” – Location where the update installer has to be installed.

Command : 

./install -silent -options update.txt

Check the logs :

/WebSphere/UpdateInstaller/logs/install/log.txt

After a successful installation, Following message will be logged in log.txt.

 

 

download (1)

 

Step 2 :

Installation of WAS Fix : 

Following response file can be created with name wasfix.txt.

Wasfix.txt :

-W maintenance.package=”7.0.0-WS-WAS-SolarisSparc64-FP0000025.pak”

-OPT disableNonBlockingPrereqChecking=”true”

-OPT allowNonRootSilentInstall=”true”

-W product.location=”/WebSphere/AppServer”

-W update.type=”install”

Command : 

Cd /WebSphere/UpdateInstaller

./update.sh -silent -options wasfix.txt

Check the logs: 

/WebSphere/AppServer/logs/update/7.0.0-WS-WAS-SolarisSparc64-FP0000025.install/Update.log

 

download (2)

 

Step 3 :

Installation of JDK Fix : 

Following response file can be created with name wasjdkfix.txt.

wasjdkfix.txt :

-W maintenance.package=”7.0.0-WS-WASSDK-SolarisSparc64-FP0000025.pak”

-OPT disableNonBlockingPrereqChecking=”true”

-OPT allowNonRootSilentInstall=”true”

-W product.location=”/WebSphere/AppServer”

-W update.type=”install”

Command :

Cd /WebSphere/UpdateInstaller

./update.sh –silent -options wasjdkfix.txt

Check the logs:

/WebSphere/AppServer/logs/update/7.0.0-WS-WASSDK-SolarisSparc64-FP0000025.install

updatelog.txt

download (3)

 

Step 4 :

Installation of IHS Fix :  

Following response file can be created with name wasihsfix.txt.

Wasihsfix.txt :

-W maintenance.package=”/fixpack25/7.0.0-WS-IHS-SolarisSparc64-FP0000025.pak”

-OPT disableNonBlockingPrereqChecking=”true”

-OPT allowNonRootSilentInstall=”true”

-W product.location=”/IBMIHS”

-W update.type=”install”

Command :

./update.sh -silent -options wasihsfix.txt

Step 5 :

Installation of IHS Plugin Fix :  

Following response file can be created with name ihsplugin.txt.

Ihsplugin.txt :

-W maintenance.package=”/fixpack25/7.0.0-WS-PLG-SolarisSparc64-FP0000025.pak”

-OPT disableNonBlockingPrereqChecking=”true”

-OPT allowNonRootSilentInstall=”true”

-W product.location=”/IBMIHS/Plugins”

-W update.type=”install”

Command :

./update.sh -silent -options ihsplugin.txt

 

There was an error while opening the file store file “WLS_DIAGNOSTICS000000.DAT

Error Message : 

<Critical> <WebLogicServer> <marsvca1> <AdminServer> <main> <<WLS Kernel>> <> <> <1352125428677> <BEA-000362> <Server failed. Reason:
weblogic.diagnostics.lifecycle.DiagnosticComponentLifecycleException: weblogic.store.PersistentStoreException: java.io.IOException: [Store:280021]There was an error whi
le opening the file store file “WLS_DIAGNOSTICS000000.DAT”
[Store:280021]There was an error while opening the file store file “WLS_DIAGNOSTICS000000.DAT”
Error from fcntl() for file locking, Resource temporarily unavailable, errno=11

Why is this error displayed : 

 
This error is displayed when the NFS system does not release the lock on the weblogic stores. Weblogic Application server maintains locks on the files that are used for storing the data and transaction logs to protect the files from data corruption in case of two instances of the same weblogic applicaiton server are started accidentally. If the server is shutdown abruptly or is crashed, Many times the locks are no released, As a result when the weblogic application server is restarted, The server will fail to acquire locks that are previously locked.

Solution : 

Delete the .Dat and .lok files from the following respective directories and restart the server.

Paths for Admin Server : 

 
$DOMAIN_HOME/servers/<server name>/tmp/Adminserver.lok
$DOMAIN_HOME/servers/<server name>/data/ldap/ldapfiles/Embedded.lok
$DOMAIN_HOME/servers/<server name>/data/store/default/_WLS_ADMINSERVER000000.DAT
$DOMAIN_HOME/servers/<server name>/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT

Path for Managed Servers : 

 
$DOMAIN_HOME/servers/<server name>/tmp/<servername>.lok
$DOMAIN_HOME/servers/<server name>/data/ldap/ldapfiles/Embedded.lok
$DOMAIN_HOME/servers/<server name>/data/ldap/store/default/_WLS_<servername>000000.DAT
$DOMAIN_HOME/servers/<server name>/data/ldap/store/diagnostics/WLS_DIAGNOSTICS000000.DAT

Caused by : java.lang.NoSuch Method Error:javax.xml.ws.WebFault.MessageName()Ljava/lang/string;

Issue:

This error is seen when the JAX-WS libraries are missing.

Solution:

When deploying webservice project to the tomcat server, Tomcat do not implement the entire java EE sepcifications and it needs the JAX-WS libraries deployed to support the project.

add the jars in the endorsed directory.

By default the endorsed directory will be taken as $CATALINA_HOME/endorsed.

If you want to force use a different path, Use the following in the setenv.sh or the catalina.sh file.

-Djava.endorsed.dirs=$JAVA_ENDORSED_DIR;

sticky sessions between Apache and tomcat app

Why sticky sessions?

Imagine you have a Apache server which load balances between two tomcat servers(tomcat1 and tomcat2) in the backend, With the use of sticky sessions, When the user sends the first request, If apache routes it to tomcat1, It will keep on sending the other requests made by the same user to the same server. If the tomcat1 is down, Then the user will be forwared to tomcat2 and will be asked to login again as the session information is lost in the tomcat1.

When you use sticky session with session replication, This issue will be solved. Considering the above scenario, When the tomcat1 is down, The session will be carried over to the tomcat2 and the user will still be able to continue the work with out any disruption.

Implementation :

Sticky sessions can be implemented in many ways, out of which I am listing the configuration with proxy and configuration with worker.properties.

Configuration with mod_proxy:

After the installation of apache, Open the http.conf file, Enable the mod_proxy.so, mod_proxy_http.so and proxy_balancer_module.

LoadModule proxy_module mod_proxy.so
LoadModule proxy_http_module mod_proxy_http.so
LoadModule proxy_balancer_module

<IfModule mod_proxy_balancer.c>
ProxyPass /context balancer://tomcatcluster/context stickysession=JSESSIONID|jsessionid
<Proxy balancer://tomcatcluster>
BalancerMember http://tomcatlocalhost:9080 route=tomcatroute1
BalancerMember http://tomcatlocalhost:9081 route=tomcatroute2
</Proxy></IfModule>

Once this configuration is done in apache, Save the httpd.conf and restart the server.

After tomcat server installation, open the server.xml and ensure http connector port is same as the port that is defined in the cluster properties of apache configuration file. For sticky sessions, Ensure that the jvm route value is same as the route value in the httpd.conf file.

tomcatroute1 worker:

<Engine defaultHost=”localhost” name=”Catalina” jvmRoute=”tomcatroute1″>
</Engine><Connector port=”9080″ protocol=”org.apache.coyote.http11.Http11Protocol” … />

tomcatroute2 worker:

<Engine defaultHost=”localhost” name=”Catalina” jvmRoute=”tomcatroute2″>
</Engine>
<Connector port=”9081″ protocol=”org.apache.coyote.http11.Http11Protocol” … />

Save the configuration, Now the sticky sessions are configured.

Configuration with worker.properties:

The tomcat configuration will be same as above, Only the apache configuration would differ.

add the following configuration in httpd.conf.

LoadModule jk_module mod_jk.so
<IfModule mod_jk.c>
JkLogFile logs/mod_jk.log
JkLogLevel debug
JkShmFile logs/jk-runtime-status
JkWorkersFile conf/workers.properties

JkMount /context loadbalancer
JkMount /context/* loadbalancer
</IfModule>

In the worker.properties file, Add the following configuration:

worker.list=loadbalancerworker.tomcatroute1.port=9080worker.tomcatroute1.host=localhostworker.tomcatroute1.type=ajp13worker.tomcatroute1.lbfactor=1worker.tomcatroute2.port=9081worker.tomcatroute2.host=localhostworker.tomcatroute2.type=ajp13worker.tomcatroute2.lbfactor=1worker.loadbalancer.type=lbworker.loadbalancer.balanced_workers=tomcatroute1,tomcatroute2

GSK_ERROR_BAD_CERT (gsk rc = 414)

GSK_ERROR_BAD_CERT (gsk rc = 414)

When an HTTPS request is sent to a IBM WebSphere Application Server V6.1 server, from a web server, the web server plug-in log shows the error Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414)

Cause: Not correct signer certificate in plugin…kdb file

The certificate sent by WebSphere Application Server to the plug-in cannot be authenticated by the plug-in key file. The reason is that the plug-in key file does not have the adequate signer to authenticate the certificate sent by WebSphere Application Server.

Resolving the problem

1.        In the administrative console, go to Security > SSL certificate and key management.

Before doing any changes, put select Dynamically update the runtime when changes occur on this page. This option makes sure that changes are propagated to runtime immediately after they are saved. This option requires a restart to become active after it is selected. If this option is enabled, make sure that you make SSL configuration changes when the system does not have a high burden on it to prevent performance impacts.

2.        Click the Manage endpoint security configurations link.

3.        Expand Inbound or Outbound, expand the cell name to see the list of nodes.

For all the nodes that appear in the list:

Opening an empty text file will help you through the process.

4.        Go to Key stores and certificates which is under Related Items.

5.        Click on the NodeDefaultKeyStore. Under Additional Properties, click on Personal Certificates.

6.        Note down the serial number of the default certificate. Select the box near the default certificate. Click Extract.

7.        Write the file name to be extracted with the full path, leave the data type as it is, note down the file path after the serial number. ClickOK.

If you chose to create a cell profile after your initial WebSphere Application Server installation, the cell manager node and the stand alone node you have created that time might have the same certificate with the same serial number. Do not let it confuse you.

After the previous instructions are done for all nodes, continue with the following steps.

8.        Come to the Manage endpoint security configurations page where you see the node list again (instructions 1-3).

9.        Expand the node which includes the web server.

10.        Click on the web server, then click on Key stores and certificates.

11.        Click on the CMSKeyStore.

12.        Click on the Signer certificates. You can either add here all the certificates you have extracted, or you can click on default certificates in this page, if there are any, and compare their serial numbers with the numbers that you have taken note of to determine which default certificates are missing.

For all the certificates or just the missing ones apply the instructions below.

13.        Click Add on the current page.

14.        Enter the certificate file path, an alias as you wish, and leave the data type as it is. Click OK.

When you are sure that you have the complete set of default certificates added as signer certificates, save the changes, and synchronize.

Dynamic Clusters change the master configuration on websphere

In the dynamic cluster environment, You should not change the configuration like java parameters to the individual cluster members, If it is changed in individual server level, The master configuration will overwrite the server level configuration. Hence to avoid this, For any configuration related changes on the dynamic clusters need to changed @ Master configuration.

Step 1: Login to the Dmgr console.

Step 2: Navigate to the dynamic clusters as shown below and click on the corresponding dynamic cluster.

 

dynamicluster (1)

Step 3: On the left hand side the server template is seen. Click on that.

dynamicluster (2)

 

Step 4: Then you can click on the JVM process definition or any other configuration required.

 

dynamiclusterStep 5: Save the changes and restart the cluster members. This should replicate the changes at cluster member level.