Workflow Eviction

[dabercro]

Right now, the console loops through all workflows in its database while updating errors seen. Since workflows with errors add up quickly, this now takes forever.

It would be good to have some way of looping through a subset of workflows. This can either be looking at workflows that are newer than some age, or only looking at workflows that appear with status "manual" in http://cms-unified.web.cern.ch/cms-unified/public/statuses.json (what the old console does).

@vlimant may be able to suggest a table or query that can be directly made to get the workflows that make the statuses.json page.

[dabercro]

@vargasa I would assign this to you, but it looks like you need to be a collaborator first. (Invite was just sent.) Let me know if you have any other questions on this issue.

[vlimant]

the json file on the web is not used, the console reads directly in oracle

https://github.com/CMSCompOps/wtc-console/blob/master/src/unified/models.py

[vlimant]

there seems to be something here

wtc-console/src/workflows/tasks.py

Line 149 in 1179441

# TODO old data cleanup here

as a hook for cleaning

[dabercro]

Ah, I guess I meant, the statuses.json can still be used as a way to determine what the loop should be. How are workflows evicted from that JSON file?

[vlimant]

do not tie to the json file. evicting anything that is not in "manual" anymore is an ok way.

[vlimant]

although, we might want to keep much more, if we are going to use the new console as the new unified report (which was one of the goal here)

[dabercro]

Okay @vargasa I think that means that you'll need to connect to the Oracle database to fill the tasks table. Also, we'll need to add a filter (status like "%manual%" for a first pass) that isn't there right now.

[vlimant]

wtc-console/src/workflows/tasks.py

Line 30 in 1179441

.filter(status__icontains='manual') \

maybe

[sharad1126]

assistance-manual, assistance-manual-recovered, assistance-announced-manual, assistance-announced-manual-recovered are the categories that should have workflows in WTC console because these are the ones that I need to take action on.

[vargasa]

I'll keep this here for reference: https://cms-unified.web.cern.ch/cms-unified/assistance.html#assistance-manual

[vargasa]

Hi @sharad1126 I can confirm that the following status are being fetched from Unified:

<QuerySet [u'assistance-agentfilemismatch-manual', u'assistance-agentfilemismatch-manual-recovered', u'assistance-announced-manual-recovered', u'assistance-manual', u'assistance-manual-recovered']>

This was done by logging:

UnifiedWorkflow.objects.filter(status__icontains='manual').values_list('status',flat=True).distinct()

Which provides a set of unique workflows status from unified that contains the string 'manual'. So maybe the problem is not here (tasks.py) but in the front-end

[vargasa]

@sharad1126 I took a couple of workflows from here and they are shown also on wtc-console as it is in my dev instance. How are you accessing wtc-console?

[sharad1126]

@vargasa I am generally accessing the WTC console through the unified error reports as it directly sends me to the console page of that particular workflow to take an action.

[vargasa]

@sharad Can you send me a link and/or the part of the report to check? Please