Datadog Issue with s3 get integration

[rayzhou-bit]

In exploring issues with orphaned span (Request #1877349), we discovered that s3.getobject spans in our services are orphaned.

• This is happening specifically with s3.getobject and not other resources.
• This is happening to our edxapp services (service: edx-edx-app-*) for both django webapps and celery workers.
• The expected behavior is that almost all of these spans should be children to django.request.

This suggests that there may be an issue with Datadog s3 integration in 2U.

Datadog request ticket: https://help.datadoghq.com/hc/en-us/requests/2077640

[rayzhou-bit]

Next step is to apply manual instrumentation by inserting a tracer.trace via our function_trace. Datadog has asked us to place this call in the SoftwareSecurePhotoVerification model on this line of code.

function_trace defined here: https://github.com/openedx/edx-django-utils/blob/1871db85cd98f3986c7848ee20954f2f1bbd3845/edx_django_utils/monitoring/internal/utils.py#L98