25. Missing Fields, Nulls, Invalid Types, and Data Quality

A missing field, a null value, a malformed string, and an empty array are different data states. Expressions may handle them differently. Use $ifNull when missing and null values should share a fallback; use explicit validation when the distinction matters.

For imported numeric data, $convert can help normalize values, but failed conversions should be counted or quarantined where accuracy matters. Replacing every invalid amount with zero may conceal errors and distort reports. Validate data at write time and handle empty arrays deliberately.

A fallback is a business decision

Distinguish missing, null, malformed, and out-of-range values when they require different remediation.

Code Example
db.importedOrders.aggregate([
  { $set: { convertedAmount: { $convert: { input: "$amount", to: "decimal", onError: null, onNull: null } } } },
  {
    $group: {
      _id: null,
      totalDocuments: { $sum: 1 },
      validAmountDocuments: { $sum: { $cond: [{ $ne: ["$convertedAmount", null] }, 1, 0] } },
      invalidOrMissingAmountDocuments: { $sum: { $cond: [{ $eq: ["$convertedAmount", null] }, 1, 0] } }
    }
  }
]);
Key Takeaways
  • Missing and null are not always equivalent.
  • Use conversion and validation intentionally.
  • Handle empty arrays explicitly.
  • Expose invalid-data counts instead of hiding failures.
Knowledge Check

Why is replacing every invalid amount with zero dangerous?