Problem:

Due to our disk space having filled up, I shutdown the Oracle WebLogic AdminServer 11g (11.1.1.3), cleared some logs, and restarted it.

However, when I start it up, it just stops at the entry below. No movement in the log whatsoever, and nothing is happening on the server.

oracle@oradev:/u01/app/oracle/middleware/user_projects/domains/soa_domain/bin> ./startWebLogic.sh
*****************************************************
** Setting up SOA specific environment…
*****************************************************
EXTRA_JAVA_PROPERTIES= -da:org.apache.xmlbeans…
.
LD_LIBRARY_PATH=/u01/app/oracle/middleware/patch_wls1033/profiles/default/native:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64:/u01/app/oracle/middleware/wlserver_10.3/server/native/linux/x86_64/oci920_8:/u01/app/oracle/middleware/Oracle_SOA1/soa/thirdparty/edifecs/XEngine/bin
.
USER_MEM_ARGS=-Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
*****************************************************
** End SOA specific environment setup
*****************************************************
.
.
JAVA Memory arguments: -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
.
WLS Start Mode=Production
.
CLASSPATH=/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/user-patch.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/soa-startup.jar::/u01/app/oracle/middleware/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/oracle/jdk1.6.0_21/lib/tools.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules_10.3.3.0.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/webservices.jar:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/oracle/middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.common.adapters_11.1.1/oracle.soa.common.adapters.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/commons-cli-1.1.jar:/u01/app/oracle/middleware/oracle_common/soa/modules/oracle.soa.mgmt_11.1.1/soa-infra-mgmt.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.xdk_11.1.0/xsu12.jar:/u01/app/oracle/middleware/modules/features/weblogic.server.modules.xquery_10.3.1.0.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/db2jcc4.jar:/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/soa-infra:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/fabric-url-handler_11.1.1.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/quartz-all-1.6.5.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.fabric_11.1.1/oracle.soa.fabric.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.adapter_11.1.1/oracle.soa.adapter.jar:/u01/app/oracle/middleware/Oracle_SOA1/soa/modules/oracle.soa.b2b_11.1.1/oracle.soa.b2b.jar:/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrf.jar:/u01/app/oracle/middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/u01/app/oracle/middleware/wlserver_10.3/server/lib/xqrl.jar:/u01/app/oracle/middleware/Oracle_SOA1/lib:/u01/app/oracle/middleware/jdk160_18/lib
.
PATH=/u01/app/oracle/middleware/wlserver_10.3/server/bin:/u01/app/oracle/middleware/modules/org.apache.ant_1.7.1/bin:/u01/app/oracle/jdk1.6.0_21/jre/bin:/u01/app/oracle/jdk1.6.0_21/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:.
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version “1.6.0_21”
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
Starting WLS with line:
/u01/app/oracle/jdk1.6.0_21/bin/java -server   -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true   -da -Dplatform.home=/u01/app/oracle/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/middleware/wlserver_10.3/server  -Ddomain.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dcommon.components.home=/u01/app/oracle/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig -Doracle.server.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol  -Digf.arisidbeans.carmlloc=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/carml  -Digf.arisidstack.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.oamprovider_11.1.1  -Dweblogic.jdbc.remoteEnabled=true -Doracle.security.jps.policy.migration.validate.principal=false  -da:org.apache.xmlbeans… -Dsoa.archives.dir=/u01/app/oracle/middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dtangosol.coherence.clusteraddress=227.7.7.8 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dcom.sun.management.jmxremote -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/app/oracle/middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/app/oracle/middleware/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dums.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/middleware/patch_wls1033/profiles/default/sysext_manifest_classpath  weblogic.Server
<Dec 22, 2010 2:46:25 PM EST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 17.0-b16 from Sun Microsystems Inc.>
<Dec 22, 2010 2:46:26 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401 >
<Dec 22, 2010 2:46:30 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Dec 22, 2010 2:46:30 PM EST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Dec 22, 2010 2:46:30 PM EST> <Notice> <Log Management> <BEA-170019> <The server log file /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/logs/AdminServer.log is opened. All server side log events will be written to this file.>

Solution:

I worked an Oracle Support analyst and we reset some passwords and were able to bring up the server again. I am somewhat convinced that this was related to our disk being full.

This is what we proceeded doing:

1. Shutdown the Admin Server

2. Remove the “cache”, “date”, and “tmp” folders from the server AdminServer location

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/cache

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data

rm -rf /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/tmp

This did not solve our problem.

3. Reset the weblogic password by running this command:

java weblogic.security.utils.AdminAccount weblogic welcome1 .

… and resetting the password in:

./servers/AdminServer/security/boot.properties

This partially resolved the problem.

4. We noticed that the “system-jazn-data.xml” file was zero bytes in size:

ll /u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig

-rw-r—– 1 oracle ldba      0 Dec 21 20:49 system-jazn-data.xml

I copied the file from a separate server, and the Admin Server now started up.

The question remains… why was the “system-jazn-data.xml” file zero bytes?

Most likely when the disk was full, some process was probably trying to save to it but was unable to, thus blanking out the file. This is my only explanation for this.

Also, since the “system-jazn-data.xml” file that I copied over was not exactly identical, I had to recreate some user accounts.

Applicable Versions:

• Oracle WebLogic Server 11g (10.3.3)
• Oracle SOA Suite 11g (11.1.1.3)

 

 

Reset the AdminServer Password in WebLogic 11g and 12c

If you forget the AdminServer password for your WebLogic 11g domain, you can reset it from the command line using the following process.

• Set up the following environment variables. They are not necessary for the process itself, but will help you navigate. In this case my domain is called “ClassicDomain”. Remember to change the value to match your domain.

  export MW_HOME=/u01/app/oracle/middleware
  export DOMAIN_HOME=$MW_HOME/user_projects/domains/ClassicDomain

• Shut down the WebLogic domain.

  $ $DOMAIN_HOME/bin/stopWebLogic.sh

• Rename the data folder.

  $ mv $DOMAIN_HOME/servers/AdminServer/data $DOMAIN_HOME/servers/AdminServer/data-old

• Set the environment variables.

  $ . $DOMAIN_HOME/bin/setDomainEnv.sh

• Reset the password using the following command. Remember to substitute the appropriate username and password.

  $ cd $DOMAIN_HOME/security
  $ java weblogic.security.utils.AdminAccount <username> <password> .

• Update the “$DOMAIN_HOME/servers/AdminServer/security/boot.properties” file with the new username and password. The file format is shown below.

  username=<username>
  password=<password>

• Start the WebLogic domain.

  $ $DOMAIN_HOME/bin/startWebLogic.sh