Troubleshooting ORDS-22001 After Migration to Oracle 21c

We run multiple database Apex instances configurations on a single Tomcat. After migrating from Apex 18c to 21c we couldn’t access Apex anymore.

The error was misleading:

https://server/ords21/DBNAME

404 Not Found
2021-10-29T17:12:42.556Z | HyrcATg4C7e87ph3eRaCig | ORDS-22001

The procedure named DBNAME could not be accessed, it may not be declared, or the user executing this request may not have been granted execute privilege on the procedure, or a function specified by security.requestValidationFunction configuration property has prevented access. Check the spelling of the procedure, check that the execute privilege has been granted to the caller and check the configured security.requestValidationFunction function.

Most articles on the Internet suggest checking the security request validation function. But there wasn’t anything wrong with it.

The XML configuration files for the the pool were there:

ls /u00/oracle/orabase/product/apache-tomcat-9.0.13/webapps/ords21/ords21/conf/dbname*
/u00/oracle/orabase/product/apache-tomcat-9.0.13/webapps/ords21/ords21/conf/dbname_al.xml
/u00/oracle/orabase/product/apache-tomcat-9.0.13/webapps/ords21/ords21/conf/dbname_pu.xml
/u00/oracle/orabase/product/apache-tomcat-9.0.13/webapps/ords21/ords21/conf/dbname_rt.xml
/u00/oracle/orabase/product/apache-tomcat-9.0.13/webapps/ords21/ords21/conf/dbname.xml

I traced the system calls while accessing the application, but Tomcat didn’t read any of the XML configuration files. This means, it had to read them at at startup and cache them.

So I checked the tomcat log (/u00/oracle/orabase/product/apache-tomcat-9.0.13/logs/catalina.out) from the startup and found the culprit:

catalina.out:2021-10-29T17:14:25.175Z WARNING     The pool named: |dbname|| is invalid and will be ignored: The username or password for the connection pool named |dbname||, are invalid, expired, or the account is locked

The password information in the XML files didn’t match the password in the database anymore.

Resetting the passwords and resyncing with XMLs resolved the issue.

In conclusion, ORDS error messages in the browser might be cryptic. Since the ORDS proces caches the configuration, you need to check the Tomcat logs from the startup for the root cause.