Fallback Server load testing benchmark to lower hardware

Scenarios Qs:

I have one prod server with 96gb ram and 92 CPU with 3 databases and i have another server with 32gb ram and 16CPU's. can we use this server as a fall back for prod with this configuration? if not please suggests me how to provide evidence that server is not able to handle the production database.

Solution:

Using a fallback server with significantly lower specifications than the production server can be challenging. To determine whether the fallback server can handle the production workload, it's important to assess the differences in hardware and the workload requirements of your production databases. Here’s a step-by-step guide to evaluate the situation and gather evidence.

Step-by-Step Evaluation

1. Analyze Current Production Workload

1. Monitor CPU Usage:

• Use tools like top, htop, or vmstat to monitor the CPU usage on the production server.

• Record the average, peak, and sustained CPU usage.

1. Monitor Memory Usage:

• Use tools like free, vmstat, or sar to monitor memory usage.

• Record the average, peak, and sustained memory usage.

1. Disk I/O Performance:

• Use tools like iostat or vmstat to monitor disk I/O performance.

• Record the average and peak I/O operations per second (IOPS).

1. Database Performance Metrics:

• Use Oracle’s Automatic Workload Repository (AWR) reports to gather detailed performance metrics.

• Look at CPU time, I/O wait times, and memory usage in these reports.

2. Compare Server Specifications

1. Production Server:

• RAM: 96GB

• CPU: 92 Cores

1. Fallback Server:

• RAM: 32GB

• CPU: 16 Cores

3. Assess Resource Requirements

1. CPU Utilization:

• Calculate the average CPU usage on the production server.

• Check if the fallback server’s 16 cores can handle the peak and average CPU load.

1. Memory Utilization:

• Compare the production server’s memory usage against the fallback server’s 32GB RAM.

1. Disk I/O:

• Ensure the fallback server’s storage can handle the IOPS and throughput requirements.

4. Performance Testing

1. Benchmarking:

• Run benchmarking tests on both servers using tools like sysbench or Oracle Real Application Testing.

• Compare the results to see how the fallback server performs under simulated production loads.

1. Load Testing:

• Perform load testing on the fallback server using a copy of the production database.

• Measure response times, throughput, and resource utilization.

5. Documentation and Evidence

1. AWR Reports:

• Collect AWR reports from the production environment.

• Highlight high CPU usage, memory usage, and I/O wait times.

1. Resource Usage Reports:

• Generate reports using monitoring tools to show average and peak resource usage on the production server.

1. Benchmarking and Load Test Results:

• Document the results of benchmarking and load tests on the fallback server.

• Compare these results with the production server’s performance.

Conclusion

If the fallback server cannot match the production server's performance, the evidence gathered will support your case. Here’s a summary of what you can present:

1. CPU Comparison:

• Production server: Average CPU utilization is X% on 92 cores.

• Fallback server: X% CPU utilization on 16 cores exceeds the available capacity.

1. Memory Comparison:

• Production server: Average memory usage is Y GB out of 96GB.

• Fallback server: Y GB out of 32GB is insufficient.

1. I/O Performance:

• Production server: Average IOPS is Z.

• Fallback server: Benchmark results show IOPS capabilities below required Z.

1. Performance Testing Results:

• Documented results showing degraded performance on the fallback server under similar loads.

Sysbench Benchmarking Tool

By presenting this detailed comparison and testing evidence, you can clearly show that the fallback server is not capable of handling the production database workload.