Classification, Extraction, and Summarization
These common patterns need different evaluation criteria. Classification should be measured with per-class precision and recall, not only overall accuracy when classes are imbalanced. Extraction should track field-level correctness, missing-field behavior, and normalization errors. Summarization should preserve key facts and avoid adding unsupported claims.
For bulk jobs, use bounded concurrency, retry only transient failures, and store job status so the batch can resume. Keep raw inputs and outputs only when retention policy permits; redact sensitive fields and establish deletion behavior.
Batch classify tickets with bounded concurrency
async function processBatch(items, classify, concurrency = 5) {
const results = new Array(items.length);
let next = 0;
async function worker() {
while (true) {
const index = next++;
if (index >= items.length) return;
results[index] = await classify(items[index]);
}
}
await Promise.all(
Array.from({ length: Math.min(concurrency, items.length) }, worker)
);
return results;
}💡 This small worker pool caps simultaneous calls and preserves input order in the result array. Production code should also implement per-item retries, cancellation, rate-limit handling, and durable job checkpoints.
Key Takeaways
- Evaluate each task type with suitable metrics.
- Use bounded concurrency for bulk requests.
- Design retention and redaction before logging raw AI data.
Knowledge Check
Which metric set is especially useful for an imbalanced classification task?