Wiki

Case Status
Log In

Wiki

 
Data Recovery Policy
  • RSS Feed

Last modified on 7/14/2014 2:13 PM by User.

Tags:

Data Recovery Policy


Overview

Protection against data loss is the primary goal. In an ideal world, a backup is always available and the server never crashes. In the real world, assume that minor actions could cause the database to be permanently damaged. Only the actions below are authorized.

Recovery of SQL server

Although SQL server is supported by Ingen Software, any recovery effort of SQL server by Ingen Software staff is not supported. Attempts to recover an SQL server database is not authorized by Ingen Software staff.

Failure Recovery Steps

Identify the type of error being returned by OASIS and the database server. Then use the grid below to determine the recovery process.

OASIS

Sybase

Recovery Process

Disc Full

Service Running or Not Running

Disc full recovery process

Connection terminated

Service Running

Debugging Connection Errors

Connection Terminated

Service Not Running

Sybase Service Crash

Assertion Error

Service Running or Not Running

Assertion Error Recovery Process

-

-

If the customer EVER requires data recovery for any reason, the process is billable – follow the SQL file recovery process below.

 

Disc Full Recovery Process

NOTICE: it is possible to lose the database on a disc full scenario. Follow the steps below. If an assertion error is returned by Sybase after the process is completed, then proceed to Assertion Error Recovery Process.

Steps

  1. Stop the “Adaptive Server Anywhere – oasis” service
  2. Clear disc space on attached discs – recommend at least 1GB free on all drives.
  3. Restart the ASA Service
  4. Attempt to login

Any errors are likely Assertion errors – go to that process next.

Debugging Connection Errors

Most errors are due to network hardware or software being reconfigured (including DHCP reassignments). The steps below are not complete. If these steps are completed and the issue is not resolved, then a PC support person is required to resolve the issues.

Steps

  1. On the server, launch OASIS and attempt to login. If the login fails, stop and start the Sybase Service. If login succeeds after service restart – go to Sybase Service Crash
  2. Use IPCONFIG on the server to determine the server’s IP address
  3. Open the ASA console to see the port number used by Sybase (may have to restart the service to see the red SQL icon for Sybase).
  4. Verify all firewalls on the server are off.
  5. Attempt to login from a client PC. If successful, then the error was server side.
  6. Attempt to access the internet from the PC. If not successful, the PC is not connected to the network.
  7. Ensure all firewalls on the PC are off. Attempt OASIS login again. If successful, the CUSTOMER must contact their PC support person or the makers of the firewall to add an exception for the IP address and port number from above.
  8. Switch to the Sybase driver in OASIS. Use the IP address and port number from the server. If successful, add these parameters to the ODBC connection. If this issue reoccurs with the customer and the IP address changes – then the server does not have a static address. Try using the server name or a static IP address
  9. In some rare cases, it is possible for server (or other) software bug to not allow connections. Upgrade the client operating system, the server operating system and Sybase to the latest service packs of the software (service pack, EBF, etc.). Retry OASIS connection.

 

At this point, the issue is most likely network related. Access to trained PC support personnel on site is now required. Opening a case with Sybase will likely also be required. Only 2 hours of Ingen Software support time is authorized with a valid maintenance agreement. From thence, the time is chargeable to the customer.

 

Sybase Service Crash

A Sybase Service Crash should be considered a critical condition as it is possible that the crash could lead to the Assertion Error scenario.

Steps

  1. Restart the service and login to OASIS. If the home screen appears, then attempt to repeat the crash (will need to query the customer for what was happening during the time of the crash).
  2. Determine if the service is being started manually. If so, install the standard OASIS / Sybase service.
  3. Identify when the server was last started. If the “server” is actually a user’s PC, then the issue is likely due to the user shutting down the PC for another reason.
  4. Check the server logs to see if there are any other issues with the server.
  5. Check for the amount of memory used versus the amount of total memory.
  6. Check the amount of space available on ALL hard drives. Recommend freeing space on all drive to a minimum of 1GB free before continuing. (Remember, temporary files are created on the C: drive on the server by Sybase. The database will hopefully be on another drive.)
  7. Move the database to another computer and attempt to reproduce. If resolved, then the problem is likely the server hardware or operating system.
  8. If the crash is reproducible, upgrade to the latest EBF from Sybase and attempt to repeat the crash. If reproducible, obtain a copy of the database and open a case with Sybase.

 

Sybase Service Crashes are rare, and are likely due to a software bug in Sybase, an operating system bug or a hardware problem.

 

Assertion Error Recovery Process

90% of the time, this is due to a damaged database. The other 10% of the time it is due the disc being full or a configuration error.

 

Steps

  1. Stop the Sybase database
  2. Ensure that ALL hard drives have a minimum of 1GB of disc space free.
  3. Upgrade to the latest Ingen Software approved build of Sybase.
  4. Start the database. If you get an assertion error – then see the data recovery process.
  5. Log into OASIS and attempt to access a PO and a Quote – including attachments. If the assertion error occurs – then see the data recovery process.

 

Data Recovery Process

Data recovery is not covered by the Ingen Software maintenance agreement. All time spent during the recovery is billable. Billable hours are only collected when Ingen Software support staff is remotely connected to the clients server, actively monitoring the recovery process, researching the incident (likely with Sybase), meeting about the recovery process or otherwise working with the recovery process. Hours will not necessarily be tied to the amount of time required to “spin” data through the recovery process.

 

Every 10 hours of billable time will require a review with the customer. Ingen Software must provide information about what actions have been taken. Time to completion may not be identifiable.

 

There are three forms of recovery:

  • Log file recovery
  • SQL file recovery
  • Sybase recovery

To save the customer money, recovery should be attempted in the above order.

 

BEFORE BEGINNING ANY RECOVERY PROCESS, THE CUSTOMER MUST PURCHASE A 100GB (OR BETTER) USB ATTACHABLE HARD DRIVE. THIS IS FOR THEIR BENEFIT AS IT WILL SPEED THE OVERALL RECOVERY PROCESS AND PROTECT AGAINST FURTHER DATA LOSS.

 

Log file recovery

1.       Copy the current oasis.db, oasis.log and any older log files in the oasis database directory to the USB hard drive. DISCONNECT THE USB DRIVE FROM THE SERVER. STEP #2 NOT AUTHORIZED BEFORE THIS STEP IS CONFIRMED COMPLETE!!!!!

2.       Upgrade Sybase to the latest EBF approved by Ingen Software.

3.       Create a sub directory in the oasis database directory named “damaged”. Move the oasis.db file to the damaged directory.

4.       Locate a backup copy of the oasis.db file. The newer the better. Copy to the oasis database directory and attempt to MANUALLY start the database. The Sybase splash screen may “freeze” for hours (this is potentially good news). Confirm that the dbsrv9.exe process is getting CPU time and wait. REBOOTING THE SERVER OR STOPPING THE SERVICE FOR ANY REASON WILL LIKELY DAMAGE THE OASIS.DB FILE.

5.       If the database starts, attempt an OASIS connection and review the data – the customer should see all information from the current day (no data loss).

6.       If the database fails with another assertion error, identify an oasis.db file prior to the one recovered from backup. It is possible to backup a damaged database file! IMPORTANT, IF THIS STEP IS TAKEN, YOU MUST CREATE A NEW DATABASE ID FOR THE DATABASE – ZERO EXCEPTIONS!

 

If an undamaged oasis.db file is not available, then an SQL file recovery or even a Sybase recovery may be required.

SQL file recovery

  1. Overnight mail the USB drive to Ingen Software.
  2. Rename the oasis database directory OASIS_old.
  3. If an older undamaged oasis.db file with matching log file is available, consider making that database available to the customer during the recovery process – it may take weeks for the recovery.
  4. Identify any available oasis.db files that are not damage (if any). Create a blank oasis.db file if necessary.
  5. Use dbtran to convert all the log files to SQL and make not of the starting and ending sequence values.
  6. Use dbisql to connect to the oasis.db and execute the sql files – in order of log file creation – until the database is recovered.
  7. If any data is missing – consider the Sybase recovery method.
  8. When complete, copy the new oasis.db and oasis.log files to the USB drive and return to the customer.
  9. When the customer is ready, all the transactions in the database created after the failure must be exported from any temporary database in .oasisz format.
  10. If a temporary database is in use, stop the sybase service and rename the database directory OASIS_temp
  11. Copy the database from the USB drive and start the Sybase service.
  12. Have the customer review to determine if the database is to be used. If so, any transactions created after the failure must be copied to the recovered database.

Sybase recovery

This process is not guaranteed and is very expensive. The process will be chargeable by both Sybase and Ingen Software.

 

  1. Open a case with Sybase and have the client sign the required work order with Sybase. INGEN SOFTWARE WILL NEVER SIGN A WORKORDER FOR THE RECOVERY OF A CUSTOMER’S DATABASE!
  2. Collect the required information for Sybase and mail the USB drive to Sybase.
  3. Most likely, Ingen Software will receive a set of “write files” from Sybase that must be used to build a new database.
  4. In some cases, it might be required for individual tables to be exported from the log files an applied to the recovered database.
  5. When complete, copy the new oasis.db and oasis.log files to the USB drive and return to the customer.
  6. When the customer is ready, all the transactions in the database created after the failure must be exported from any temporary database in .oasisz format.
  7. If a temporary database is in use, stop the sybase service and rename the database directory OASIS_temp
  8. Copy the database from the USB drive and start the Sybase service.
  9. Have the customer review to determine if the database is to be used. If so, any transactions created after the failure must be copied to the recovered database.