`Added to Fleet` set to `Never` when enrolling manually on MDM after installing fleetd first

[lucasmrod]

`Added to Fleet` set to `Never` when enrolling manually on MDM after installing fleetd first

[jacobshandling]

@zayhanlon got it. Sharon is out today, but we saw the P2 label and have prioritized accordingly.

[lucasmrod]

Sorry, I incorrectly added the :product label. This is a bug. Let us know how big of a priority is fixing this (vs other bugs/features).

[sharon-fdm]

Thanks @zayhanlon.
I bumped it the top of our bug list and will get this looked at asap.

[lucasmrod]

Hi folks!

Do we know why the date is "Never" even tho the agent is installed ?

The bug we introduced in v4.51.0 caused MDM enrolls/re-enrolls to set the Added to Fleet field to Never. (We are fixing this in the next release.)

Looking at the bug description, my understanding is that customer-preston is doing the opposite. Their workflow:
Turn on MDM manually (installed enrollment profile)
Install fleetd

Gotcha. AFAICS the Never would happen in these scenarios:
A. fleetd installed first, manual MDM enroll after.
B. MDM re-enroll (e.g. if the user unenrolls from MDM manually by uninstalling the enrollment profile and then re-enrolling to MDM by installing the enrollment profile again)
C. During DEP migration flow (where fleetd is installed first then MDM turned on in Fleet).

I'm guessing for customer-preston these "Never" happen because of either (B) or (C)?

[lucasmrod]

Don't we have a "created at" timestamp in the hosts table in the Fleet DB that gets created when fleetd is installed and host enrolls? Maybe this the bug?

created_at timestamp is set when the host is enrolled through DEP (so it's different than last_enrolled_at, which is set when fleetd enrolls via orbit/osquery)

[lucasmrod]

if the host_disks.created_at fix is going to apply only for device that has "Never" date, I think it's acceptable

Yes, that would be the plan.

[lucasmrod]

@xpkoala @PezHub I've added QA notes to the issue.

[JoStableford]

Related to a Slack conversation

[noahtalerman]

Hey @lucasmrod and @sharon-fdm!

I'm not sure why this ended up on the drafting board...is the expected behavior unclear? Something else?

The "Added to Fleet" timestamp is when the host enrolls to Fleet. When a host shows up in Fleet for the first time. More context here: https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-use-mdm-on-off-instead-of-mdm-enrolled-unenrolled

[sharon-fdm]

Thanks @lucasmrod.
As always, let's keep on our board and estimate soon.

[sharon-fdm]

Hey team! Please add your planning poker estimate with Zenhub @getvictor @jacobshandling @lucasmrod @mostlikelee @RachelElysia

[zayhanlon]

@sharon-fdm adding a p2 label here because the 'workaround' option of deleting the host to get this updated with the correct date does not work for customer-preston. they're unable to delete the hosts. their entire enrollment status and workflow is based on this 'enrolled date' field

[zayhanlon]

Thank you all!

[lucasmrod]

adding a p2 label here because the 'workaround' option of deleting the host to get this updated with the correct date does not work for customer-preston. they're unable to delete the hosts. their entire enrollment status and workflow is based on this 'enrolled date' field

@zayhanlon @noahtalerman
I have a PR in review that solves the issue moving forward (for new devices added to Fleet).
Let's discuss what we should do with hosts that already are in this state (Added to Fleet = Never). What enroll date do we set on them?

[zayhanlon]

@lucasmrod guessing that there's no way for us to figure out the actual enroll date?

[lucasmrod]

I'll be taking a look at where we can get the enrolled date from. AFAICS it won't be exact, most likely an approximation (deduced from other MySQL table row timestamps).

[lucasmrod]

@zayhanlon @noahtalerman

Added to Fleet is the date the osquery agent enrolled (different than the MDM enroll time or the time that the host was ingested via DEP).

I did some digging and we can use the host_disks.created_at which most of the cases is close to the hosts.last_enrolled_at. In some cases it's many days apart (e.g. if you uninstall+install fleetd again on a device, then last_enrolled_at will be updated and will be different than host_disks.created_at)

Here's a dump from dogdood (46 hosts have 0 day difference, and 14 have a non-0 difference):

select datediff(hd.created_at, h.last_enrolled_at) as days_between_last_enrolled_at_and_host_disks from host_disks hd join hosts h on hd.host_id=h.id join host_mdm hmdm on hmdm.host_id=h.id where h.platform = 'darwin' and h.last_enrolled_at !;
+----------------------------------------------+
| days_between_last_enrolled_at_and_host_disks |
+----------------------------------------------+
|                                           -8 |
|                                            0 |
|                                            0 |
|                                          -92 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                           -2 |
|                                         -231 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                         -147 |
|                                         -130 |
|                                          -91 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                           -6 |
|                                            0 |
|                                            0 |
|                                          -70 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                          -52 |
|                                          -96 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                          -12 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                           -9 |
|                                          -10 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
|                                            0 |
+----------------------------------------------+
60 rows in set (0.20 sec)

Let me know if we are a-ok going this route. (Via a migration in 4.54.0 to fix these "Never" hosts using host_disks.created_at)

[zayhanlon]

Seems like it makes sense but I'll wait for @noahtalerman

[zayhanlon]

@lucasmrod @noahtalerman is there NO other way for us to capture the actual enrollment date vs this option^ ? The diff on some of these is really high

[lucasmrod]

is there NO other way for us to capture the actual enrollment date vs this option^ ? The diff on some of these is really high

I ran out of ideas. Let me check with the team.

[valentinpezon-primo]

@lucasmrod - thanks for looking into this !

Do we know why the date is "Never" even tho the agent is installed ? 🤔

We have some internal flow that triggers if the enrollment date change, in our minde this date should never changed, and if it is, it means the device was re-enrolled

if the host_disks.created_at fix is going to apply only for device that has "Never" date, I think it's acceptable

[noahtalerman]

Hey @lucasmrod thanks for digging into this!

Installing fleetd on macOS and then enrolling to MDM manually causes the host's Added to Fleet in the host details to be Never.

Looking at the bug description, my understanding is that customer-preston is doing the opposite. Their workflow:

1. Turn on MDM manually (installed enrollment profile)
2. Install fleetd

@valentinpezon-primo please correct me if I'm wrong.

Do we know why the date is "Never" even tho the agent is installed ?

I had the same question...

Don't we have a "created at" timestamp in the hosts table in the Fleet DB that gets created when fleetd is installed and host enrolls? Maybe this the bug?

Meaning, assuming I'm right about customer-preston's workflow, we're not populating that "created at" timestamp for hosts that turn on MDM manually and then have fleetd installed.

I did some digging and we can use the host_disks.created_at which most of the cases is close to the hosts.last_enrolled_at. In some cases it's many days apart (e.g. if you uninstall+install fleetd again on a device, then last_enrolled_at will be updated and will be different than host_disks.created_at)

@lucasmrod, are there other cases that we know of in which the dates are far apart?

If I'm understanding correctly, customer-prestons hosts that have "Never" haven't had fleetd uninstalled and then re-installed yet.

@valentinpezon-primo is that right?

If that's right, and assuming we don't have a "created at" timestamp for hosts that turn on MDM manually and then have fleetd installed, it sounds like host_disks.created_at will be accurate.

[valentinpezon-primo]

Yes @noahtalerman , we do this (profile then fleetd is installed by fleet)

But this behavior is new (the "never" date), was properly populated before !

[PezHub]

Thanks @lucasmrod and good timing. I literally encountered this yesterday after enrolling my device via DEP, removing the profile then manually re-enrolling. I'll be sure to test the other scenarios listed above as well.

[PezHub]

I had to test scenario B for another mdm ticket today and confirm this fixes the issue (screenshot I posted above)

[fleet-release]

Enrolling manually,
Fleet's date now accurate,
Calm as cloud city's view.