Time Series And Aggregation Framework (mongo)

I'm trying to synchronise two functions I run in my app. First one checks the count of the documents I save to MongoDB every time block (e.g. every 10 seconds) in the real time: va

Solution 1:

Your error is how you're calculating _id for $group operator, specifically its second part:

second: { $subtract: [
    { $second: "$time" },
    { $mod: [
        { $second: "$time" },
        timeBlock / 1000
    ]}
]}

So, instead of splitting all your data into 10 timeBlock milliseconds long chunks starting from new Date(end - 10 * timeBlock), you're splitting it into 11 chunks starting from from the nearest divisor of timeBlock.

To fix it you should first calculate delta = end - $time and then use it instead of the original $time to build your _id.

Here is an example of what I mean:

Document.aggregate({
    $match: {
        time: {
            $gte: new Date(end - 10 * timeBlock),
            $lt: new Date(end)
        }
    }
}, {
    $project: {
        time: 1,
        delta: { $subtract: [
            new Date(end),
            "$time"
        ]}
    }
}, {
    $project: {
        time: 1,
        delta: { $subtract: [
            "$delta",
            { $mod: [
                "$delta",
                timeBlock
            ]}
        ]}
    }
}, {
    $group: {
        _id: { $subtract: [
            new Date(end),
            "$delta"
        ]},
        count: { $sum: 1 }
    }
}, {
    $project: {
        time: "$_id",
        count: 1,
        _id: 0
    }
}, {
    $sort: {
        time: 1
    }
}, function(err, result) {
    // ...
})

I also recommend you to use raw time values (in milliseconds), because it's much easier and because it'll keep you from making a mistake. You could cast time into timeParts after $group using $project operator.