Testing Guide

Two-Tier Testing Architecture

Our testing approach eliminates the chicken-and-egg problem in OpenAPI development by separating static specification validation from dynamic API behavior testing.

Why Two-Tier Testing Matters

Testing Workflow

Running Tests

⚡ Tier 1: OpenAPI Validation (Spectral)

Lightning-fast validation of specification quality:

# Professional-grade OpenAPI validation
npm run docs:validate

# Validate Mermaid diagrams in documentation
npm run docs:validate-mermaid

# Run both validations
npm run docs:validate-all

What Tier 1 Validates:

• ✅ OpenAPI 3.1.1 syntax compliance
• ✅ Security scheme definitions
• ✅ Response schema consistency
• ✅ Parameter validation rules
• ✅ Documentation completeness

🧪 Tier 2: Ruby Testing

Real API behavior validation with live integration:

# Core gem functionality (fast - default)
bundle exec rake test           

# Complete testing including live API
bundle exec rake test:all       

# Live API integration tests only
bundle exec rake test:live   

# Run specific test file
bundle exec ruby test/cyber_trackr_helper_test.rb

What Tier 2 Validates:

• ✅ Helper method functionality
• ✅ Error handling and recovery
• ✅ Real API response parsing
• ✅ Rate limiting behavior
• ✅ Integration patterns

CI/CD Pipeline Architecture

Our cross-platform CI/CD pipeline ensures reliability across all environments:

Cross-Platform Success

Why Our Tests Pass Everywhere:

Test Structure

Directory Layout

test/
├── cyber_trackr_helper_test.rb  # Core gem functionality
├── live_api_validation_test.rb  # Live API integration  
├── test_helper.rb              # Shared test utilities
└── fixtures/                   # Test data and mocks

Test Categories

Test Type	Purpose	Speed	Dependencies
Unit Tests	Helper method validation	⚡ < 1 second	Mock responses only
Integration	Live API behavior	🐌 ~30 seconds	cyber.trackr.live API
Spectral	OpenAPI validation	⚡ < 1 second	Node.js + OpenAPI spec

Writing Tests

Unit Test Example

class HelperTest < Minitest::Test
  def test_list_stigs_filters_srgs
    mock_documents_list  # Use test helpers
    
    stigs = @client.list_stigs
    
    assert_equal 2, stigs.size
    refute stigs.key?(:Application_Security_Requirements_Guide)
  end
end

Using Test Helpers

Always use the test helpers from test/test_helper.rb:

# Good - uses proper mock structure
mock_documents_list(sample_document_list)

# Bad - incomplete mock missing required fields
stub_request(:get, url).to_return(body: {}.to_json)

CI/CD Testing

Our CI/CD pipeline tests across multiple platforms:

• Platforms: Ubuntu, Windows, macOS
• Ruby Versions: 3.2, 3.3, 3.4
• Test Types: Unit tests, live API integration, OpenAPI validation

Platform-Specific Notes

Windows Compatibility:

• Uses Faraday (pure Ruby) instead of typhoeus (requires libcurl.dll)
• No external dependencies required
• All tests pass on Windows GitHub Actions runners

Coverage

Run tests with coverage reporting:

bundle exec rake test
# Coverage report generated in coverage/index.html

Integration Testing

Live API Tests

# Test against live cyber.trackr.live API
bundle exec rake test:stage2b

Note: Live API tests may skip some endpoints due to:

• JSON control character issues in responses
• Rate limiting
• Network timeouts

These skips are expected and don't indicate test failures.

Debugging Tests

WebMock Issues

If you see WebMock conflicts:

# Disable WebMock for specific tests
WebMock.allow_net_connect!
# ... run test
WebMock.disable_net_connect!

Test Data

Test fixtures are stored as Ruby hashes in test files rather than separate fixture files for simplicity.

Best Practices

1. Mock API responses for unit tests
2. Use live API sparingly for integration tests
3. Handle expected failures gracefully (JSON parsing issues)
4. Test error conditions not just success cases
5. Keep tests fast - use mocks for core functionality tests