Testing Framework

Purpose-built testing tools for AI applications. Semantic matching, statistical assertions, cost tracking, and regression testing.

npm install @rana/testing

AI Test Runner

Purpose-built test runner for AI applications with special matchers

import { aiTest, expect } from '@rana/testing';

aiTest('summarizes articles correctly', async () => {
  const result = await summarize(article);

  // Semantic matching - not exact string comparison
  await expect(result).toSemanticMatch('A brief overview of AI trends');

  // Check it mentions key topics
  await expect(result).toContainConcepts(['machine learning', 'neural networks']);
});

Semantic Matching

Compare outputs by meaning, not exact strings

import { semanticMatch, semanticSimilarity } from '@rana/testing';

// Check if two texts have the same meaning
const match = await semanticMatch(
  "The weather is nice today",
  "It's a beautiful day outside"
);
// match.isMatch: true, match.similarity: 0.87

// Get similarity score
const score = await semanticSimilarity(text1, text2);
// Returns 0-1 similarity score

Statistical Assertions

Handle non-deterministic AI outputs with statistical testing

import { aiTest, expect } from '@rana/testing';

aiTest('classifier is mostly accurate', async () => {
  const results = await runMultiple(100, () =>
    classify(email, ['spam', 'ham'])
  );

  // Pass if 90%+ are correct
  await expect(results).toMostlyBe('ham', { threshold: 0.9 });

  // Check distribution
  await expect(results).toHaveDistribution({
    'ham': { min: 0.85, max: 0.95 },
    'spam': { min: 0.05, max: 0.15 }
  });
});

Latency Assertions

Ensure responses meet performance requirements

import { aiTest, expect } from '@rana/testing';

aiTest('responds within SLA', async () => {
  const result = await timed(() => chat('Hello'));

  // Assert latency
  await expect(result).toRespondWithin(2000); // 2 seconds

  // P95 latency over multiple runs
  const results = await runMultiple(50, () => chat('Hello'));
  await expect(results).toHaveP95LatencyUnder(3000);
});

Cost Assertions

Control costs with per-test budget limits

import { aiTest, expect } from '@rana/testing';

aiTest('stays within budget', async () => {
  const result = await tracked(() =>
    summarize(longDocument)
  );

  // Assert cost
  await expect(result).toCostLessThan(0.05); // $0.05

  // Assert token usage
  await expect(result).toUseTokensLessThan({
    input: 1000,
    output: 500
  });
});

Snapshot Testing

Catch regressions with semantic snapshots

import { aiTest, expect } from '@rana/testing';

aiTest('prompt output is stable', async () => {
  const result = await generate(prompt);

  // Semantic snapshot - allows minor wording changes
  await expect(result).toMatchSemanticSnapshot();

  // Regression testing against baseline
  await expect(result).toPassRegression({
    baseline: 'baseline-v1',
    tolerance: 0.1 // Allow 10% variation
  });
});

Configuration

// rana.test.config.ts
import { defineConfig } from '@rana/testing';

export default defineConfig({
  // Run tests in parallel
  parallel: true,

  // Global timeout
  timeout: 30000,

  // Cost limits
  maxCostPerTest: 0.10,
  maxCostPerSuite: 1.00,

  // Semantic matching model
  semanticModel: 'text-embedding-3-small',

  // Snapshot directory
  snapshotDir: '__snapshots__',

  // Reporter
  reporter: ['default', 'html'],
});

Jest Integration

Use RANA's AI matchers with your existing Jest setup:

// jest.setup.ts
import '@rana/testing/jest';

// Now use in any Jest test
test('AI output is correct', async () => {
  const result = await myAIFunction();
  await expect(result).toSemanticMatch('expected meaning');
});