wiki1568: Home page


1.) Information required while reporting Teamforge UI defects:[#]#

  • The Subject of the support ticket needs to have a clear description of the issue that the users are facing.
  • The Description of the ticket needs to ideally have the following details and structure (This is applicable while reporting any issues to the Support).

                          i). Is the issue specific to a single user/ repository/ project or if it is happening for all users and across all projects in the site.

                         ii). When was the last time/ date the application worked fine without showing the current issue?

                         iii). Are there any recent application/ Network/ Hardware configuration changes by other teams.

                         iv). Is the issue existing in Prod/ UAT/ Stage? Or all the instances?

                         v). Debug logs will only be helpful if they are captured from the time when the issue/ error is recreated.

                         vi). A detailed description of all the steps that lead to reproducing the issue.

  • Upload all the application logs after reproducing the issue in DEBUG log level with proper Time Stamp. Application logs can be obtained from this path: /opt/collabnet/teamforge/log/apps/ (specifically vamessages log and server log), other logs from /opt/collabnet/teamforge/log/

Steps to enable DEBUG: Steps to Change LOG LEVEL: In Teamforge > Click 'Admin' > System Tools > Configure Logging > Log Level > Debug > Save.

Note: Make sure to enable INFO level logging once the required logs are captured. Having the DEBUG log level for a long time will cause a high server load and may result in application performance issues.


2.) Information required while reporting Teamforge Site Outages or Slow performance issues:[#]#

  • Teamforge services status (Provide the output of command 'teamforge status')
  • Provide the output (screen shots) of below 2 commands.

                           1. top

                           2. df -h

  • Provide all the application logs (in DEBUG) covering at least an hour's time when the application was up before the outage/ issue actually happened. Provide logs from the following paths below:

            /opt/collabnet/teamforge/log/apps/

           /opt/collabnet/teamforge/log/runtime/

  • Attach following file: /opt/collabnet/teamforge/runtime/conf/runtime-options.conf

3.) Information required while reporting Teamforge mail service not functional:[#]#

  • Provide the screenshot of “Mail server” status from Teamforge admin > System Tools > Server status Check James service status from below command:

                   teamforge status -s mail

                   teamforge status -s phoenix

  • Attach all the logs from locations:

                   /opt/collabnet/teamforge/log/phoenix/

                   /opt/collabnet/teamforge/log/apps/

  • Provide the output of the below database query from the Teamforge database.

                  select distinct(queue), count(*) from stored_message group by queue;


4.) Information required while reporting Teamforge search service not functional:[#]#

  • Provide the screenshot of “Search server” status from Teamforge admin > System Tools > Server status 
  • Check ‘rmisearch’ service status from below command:

                     teamforge status -s codesearch-elasticsearch

  • Attach all the logs from following locations:

                   /opt/collabnet/teamforge/log/phoenix/

                  /opt/collabnet/teamforge/log/apps/

  • Output of below database query

                  select distinct(queue), count(*) from stored_message group by queue;


5.) Information required while reporting LDAP related issues:[#]#

  • Is the LDAP configured using 'ctf_auth_manager' add-on or Teamforge Native LDAP?
  • Is there any user login issues presently? For all users or single user?
  • Screenshot of 'Manage Existing Profiles' page (Admin > 'Look' project > Auth manager > Manage Existing Profile).
  • Try validating a username in LDAP profile page and share the result if validation successful or ending up with any error message.
  • Was there any recent change in LDAP profile and LDAP server settings?
  • Is the SSL certificate updated for LDAP server recently?
  • Screenshot of 'Federation' and 'LDAP' pages in Admin > Identity
  • Share outputs of below commands from Teamforge application server:
telnet  <LDAP server hostname>  <port>
ldapsearch -D '<bind DN>' -W -p <port> -h '<LDAP hostname>' -b <base DN> -s sub "cn=<canonical name>"

Note: If 'telnet' and 'ldapsearch' packages not installed on the server, above commands would give 'command not found' error. Install telnet using 'yum install telnet' and ldapsearch using 'yum install openldap-clients'.


6.) Information required on artifact display/ Markdown editor related issues:[#]#

  • Is the display issue experienced in all browsers (IE, Chrome, Firefox)?
  • Attach the screenshot of the display rendering issue.
  • Is artifact created before 19.0 upgrade or after upgrade?
  • Share a copy of expected text rendering in txt format.
  • Is the same issue occur in all the artifacts and in all projects?
  • Is the rendering text display same when 'Code' option is used?

...

7.) BCP process in NSE:[#]#

 i). Verify the hostnames of BCP servers(DB, App, SVN and eventq) & their respective IP address in /etc/hosts file of all the server during the IP swap.
ii). Make sure all the host names of BCP server is properly configured in the site-options.conf during the hostname swap.
iii). Provision servers in this order: DB > App > eventq & SCM
iv). Finally make sure all the services are up and running on all the servers. Then try to access the PROD Teamforge URL on the browser and verify the login.

Refer:

Issues encountered in Step iii Provisioning

Check List before proceeding with BCP

...


...

8.) Teamforge Stop and Start Process:[#]#

While using multi-host setup of Teamforge application, the stop and start process has to be followed in a specified order. There is a possibility that services may go doomed if the order if not followed properly.
---
Stopping a running Teamforge Server:
SCM server > EventQ server > Application Server > DB server
....
Starting a Teamforge Server:
DB server > Application Server > EventQ server > SCM server

...


...

9.) SSL certificate renewal:[#]#

1. Open /opt/collabnet/teamforge/etc/site-options.conf and verify current certificate locations by below tokens and take backup of the same:
SSL_CERT_FILE=
SSL_KEY_FILE=
SSL_CHAIN_FILE=

           (Reviewing site-options.conf of NSE, the certs are in /opt/collabnet/cert/ location)

2. Generate new certificate with Certificate in .crt format, Key in .key format and Chain certificate in .crt format.
3. Replace the certificate, key and chain certificate in /opt/collabnet/cert/ with new one. Perform this action in App server and SCM server. Assuming NSE is generating wildcard certificate with CN = *.nse.co.in, you can use same certificate for App and SCM servers, if not separate certificates are required with appropriate domain names.
4. Reload apache in App server and SCM server: systemctl reload httpd
5. Verify if the new certificate is reflecting in UI.

...


10.) Teamforge Logs for appropriate information:[#]#

/opt/collabnet/teamforge/log/apps/vamessages.log --> Application level log entries

/opt/collabnet/teamforge/log/apps/server.log --> Server level log entries (JBOSS)

/opt/collabnet/teamforge/log/apps/vasession.log --> User login/logout

/opt/collabnet/teamforge/log/apps/vadaemon.log --> Event logs and search logs

/opt/collabnet/teamforge/log/apps/jboss_console.log --> JBOSS dumps

/opt/collabnet/teamforge/log/apps/service.log --> JBOSS service log

/opt/collabnet/teamforge/log/httpd/access --> soap level logs

/opt/collabnet/teamforge/log/httpd/error --> apache errors

/opt/collabnet/teamforge/log/phoenix/mailet.log --> outgoing emails

/opt/collabnet/teamforge/log/phoenix/smtpserver.log --> incoming emails

/opt/collabnet/teamforge/log/phoenix/spoolmanager.log --> mail spooling

/opt/collabnet/teamforge/log/phoenix/default-serach.log --> search index

/opt/collabnet/teamforge/log/integration --> logs on tomcat integrations

/opt/collabnet/teamforge/log/iaf --> logs on binary

/opt/collabnet/teamforge/log/etl --> logs on ETL

/opt/collabnet/teamforge/log/runtime --> logs on runtime recreation, post-install and templates

/opt/collabnet/teamforge/log/add-ons --> logs on add-ons installed in the serve