Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generic javascript template support [WIP] #8

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Generic javascript template support [WIP] #8

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ee32c69
style
EntilZha Apr 24, 2015
db68e1e
js file
EntilZha Apr 24, 2015
f9dec35
added django models for Task and Resources along with tests. Created …
Apr 25, 2015
045a280
added placeholder for requirements function
Apr 28, 2015
957db07
deleted old dashboard code and replaced it with a placeholder boostra…
EntilZha Apr 28, 2015
e855286
did some more work on reactjs dashboard
Apr 29, 2015
693cff3
ignore "wreckless" changes in this commit, trying to get a working de…
Apr 29, 2015
12ad933
better tracking
Apr 29, 2015
dc08649
trimming down requirements.txt
EntilZha Apr 29, 2015
97f0259
removed very samll numpy dependency. Adds a lot of overhead on builds…
EntilZha Apr 29, 2015
bd398c1
reversions and light refactor
EntilZha May 1, 2015
3da6d64
refactored where models are, renamed task fields for clarity
EntilZha May 1, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions ampcrowd/basecrowd/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ def __unicode__(self):
class Meta:
abstract = True

@staticmethod
def to_json(task_group):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these methods aren't actually outputting JSON. Either have the method return json.dumps(return_dict), or rename it to_dict().

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, the json_view decorator can automatically convert the dict so will do to_dict

return {
'group_id': task_group.group_id,
'tasks_finished': task_group.tasks_finished,
}


# Model for an individual task.
class AbstractCrowdTask(models.Model):
Expand Down Expand Up @@ -68,6 +75,20 @@ def __unicode__(self):
class Meta:
abstract = True

@staticmethod
def to_json(task):
return {
'task_type': task.task_type,
'data': task.data,
'creation_time': task.create_time,
'task_id': task.task_id,
'num_assignments': task.num_assignments,
'mv_answer': task.mv_answer,
'em_answer': task.em_answer,
'is_complete': task.is_complete,
'completed_assignments': len(task.responses.all())
}


# Model for workers
class AbstractCrowdWorker(models.Model):
Expand Down Expand Up @@ -150,3 +171,43 @@ def add_model_rels(self):
# responses pertain to a task
self.add_rel(self.response_model, self.task_model, models.ForeignKey,
'task', 'responses')


class TemplateResource(models.Model):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good if these models lived in a different file from the abstract models that should be subclassed to implement new crowds. It might make sense to restructure into a basecrowd/models/ directory containing a abstract_crowd_models.py and a task_interface_models.py (or somesuch).

name = models.CharField(max_length=200)
content = models.TextField()
direct_dependencies = models.ManyToManyField('self', symmetrical=False, related_name="upstream_dependencies")
direct_requirements = models.ManyToManyField('self', symmetrical=False, related_name="upstream_requirements")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick, but if I'm interpreting this right, shouldn't the related_name be downstream_requirements? In general, dependencies are 'upstream' in the sense that they're required earlier.

Also remind me again the difference between dependencies and requirements?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The names I chose for these is very poor. The idea is that a template might have

  1. Dependencies it will import
  2. A set of requirements in order to run

For example, suppose I have two js templates. In order for one to work, it requires the other. This would be one way to enforce this and have a way to show which templates are related to each other. The other case would be a way to include stuff like jquery/react within the system instead within the template.

Upon further thought, I think it would help if I renamed those to something like imports and requirements. Thoughts on name?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not totally understanding. Why would an import not also be a requirement? Are you trying to support the use case where templates gracefully degrade if some (non-required) imports are not available? Or are you trying to differentiate between template inheritance and including external libraries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second one is closer to the intent. Dependencies (or imports) would be akin to import statements in python, or script tags in html. Ideally, these should only be imported once, even if multiple sections of the template will import them.

Requirements are for enforcing template inheritance/composition.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then imports and requirements sound like better names.

But I'm wondering if it wouldn't be cleaner to place these in separate models, one model for TemplateResource (TR) objects that represent interface HTML templates, and one model for RenderingResource (RR) objects that represent an importable script tag. Then TRs could have one field pointing to required TRs, and one field pointing to required RRs. The RRs for a hierarchy of TRs could be marshaled easily with code like RR.objects.filter(tr__in=[list of TRs being rendered])

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea, it cleans things up a bit and makes it clearer. This week is a bit busy at work, but hopefully I can find some time to work on it on evenings now that I finished a refactor on another project.


@property
def dependencies(self):
query_set = list(self.direct_dependencies.all())
all_dependencies = set(query_set)
current_dependencies = list(query_set)
while True:
prior_size = len(all_dependencies)
new_dependencies = set()
for dep in current_dependencies:
deps = set(dep.direct_dependencies.all())
new_dependencies = new_dependencies.union(deps)
all_dependencies = all_dependencies.union(deps)
if prior_size == len(all_dependencies):
break
else:
current_dependencies = list(new_dependencies)
return all_dependencies

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty painfully inefficient in terms of DB queries. I'll put a little thought into it, but there should be an easier way to do this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

@property
def requirements(self):
pass


def __str__(self):
return self.name


class TaskType(models.Model):
name = models.CharField(max_length=200)
iterator_template = models.ForeignKey(TemplateResource, related_name='iterator_task')
point_template = models.ForeignKey(TemplateResource, related_name='point_task')
renderer = models.ForeignKey(TemplateResource, related_name='renderer_task')
10 changes: 10 additions & 0 deletions ampcrowd/basecrowd/templates/basecrowd/javascript.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!DOCTYPE html>
<html>
<head lang="en">
<meta charset="UTF-8">
<title></title>
</head>
<body>

</body>
</html>
3 changes: 0 additions & 3 deletions ampcrowd/basecrowd/tests.py
View file
Open in desktop

This file was deleted.

63 changes: 63 additions & 0 deletions ampcrowd/basecrowd/tests/test_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
from django.test import TestCase
from basecrowd.models import TaskType, TemplateResource


class TasksTestCase(TestCase):
def setUp(self):
html = TemplateResource.objects.create(name="html")
js = TemplateResource.objects.create(name="js")
css = TemplateResource.objects.create(name="css")
bootstrap = TemplateResource.objects.create(name="bootstrap")
bootstrap.direct_dependencies.add(html, js, css)

jquery = TemplateResource.objects.create(name="jquery")
jquery.direct_dependencies.add(html, js, css)

reactjs = TemplateResource.objects.create(name="reactjs")
reactjs.direct_dependencies.add(js)

jqueryui = TemplateResource.objects.create(name="jqueryui")
jqueryui.direct_dependencies.add(jquery)

bootstrap_react_gallery = TemplateResource.objects.create(name="bootstrap react gallery")
bootstrap_react_gallery.direct_dependencies.add(reactjs, bootstrap)

monolithic_app = TemplateResource.objects.create(name="monolithic app")
monolithic_app.direct_dependencies.add(
bootstrap_react_gallery, jqueryui, jquery, js, css, html, reactjs
)

def test_template_dependencies(self):
html = TemplateResource.objects.get(name="html")
css = TemplateResource.objects.get(name="css")
js = TemplateResource.objects.get(name="js")
jquery = TemplateResource.objects.get(name="jquery")
bootstrap = TemplateResource.objects.get(name="bootstrap")
reactjs = TemplateResource.objects.get(name="reactjs")
jqueryui = TemplateResource.objects.get(name="jqueryui")
bootstrap_react_gallery = TemplateResource.objects.get(name="bootstrap react gallery")
monolithic_app = TemplateResource.objects.get(name="monolithic app")
self.assertSetEqual(
jquery.dependencies,
{html, css, js}
)
self.assertSetEqual(
html.dependencies,
set()
)
self.assertSetEqual(
reactjs.dependencies,
{js}
)
self.assertSetEqual(
jqueryui.dependencies,
{jquery, html, js, css}
)
self.assertSetEqual(
bootstrap_react_gallery.dependencies,
{html, js, css, reactjs, bootstrap}
)
self.assertSetEqual(
monolithic_app.dependencies,
{html, js, css, reactjs, jquery, jqueryui, bootstrap, bootstrap_react_gallery}
)
2 changes: 2 additions & 0 deletions ampcrowd/basecrowd/urls.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@
url(r'^(\w+)/responses/$', views.post_response, name='post_response'),
url(r'^(\w+)/tasks/$', views.create_task_group, name='create_tasks'),
url(r'^(\w+)/purge_tasks/$', views.purge_tasks, name='purge_tasks'),
url(r'^(\w+)/summary/task_groups', views.get_task_groups, name='get_task_groups'),
url(r'^(\w+)/summary/tasks', views.get_tasks, name="get_tasks")
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is related to your demo dashboard, not the template stuff, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, its dashboard. Would it be better to make that a different pull request, or include that? Those are basically poorly urled, json endpoints for tasks and task groups.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's too much hassle, I'm fine keeping it as one PR. But then we could definitely clean up the urls 😄

I'm not sure why the /summary prefix is necessary. Anything wrong with /crowds/{crowd_name}/task_groups/ and /crowds/{crowd_name}/tasks/? Note the trailing slash, so we can support /task_groups/{id} when we want to click into one in the dashboard.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

summary is not, I just didn't have much time to spend on making sure there were no url collisions (got it ready 30 minutes before presentation). I can clean this up.

23 changes: 19 additions & 4 deletions ampcrowd/basecrowd/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from django.views.decorators.http import require_GET, require_POST
from django.http import HttpResponse
from datetime import datetime
from jsonview.decorators import json_view
import pytz
import json
import os
Expand All @@ -16,6 +17,21 @@
logger = logging.getLogger('crowd_server')


@require_GET
@json_view
def get_task_groups(request, crowd_name):
interface, model_spec = CrowdRegistry.get_registry_entry(crowd_name)
logger.debug(type(interface))
logger.debug(type(model_spec))
return interface.get_task_groups()

@require_GET
@json_view
def get_tasks(request, crowd_name):
interface, model_spec = CrowdRegistry.get_registry_entry(crowd_name)
return interface.get_tasks()


@require_POST
@csrf_exempt
def create_task_group(request, crowd_name):
Expand Down Expand Up @@ -159,7 +175,7 @@ def get_assignment(request, crowd_name):

# Load the template and render it.
template = get_scoped_template(crowd_name, current_task.task_type + '.html',
context=context)
context=context)
return HttpResponse(template.render(RequestContext(request, context)))


Expand All @@ -182,7 +198,6 @@ def get_scoped_template(crowd_name, template_name, context=None):
@require_POST
@csrf_exempt
def post_response(request, crowd_name):

# get the interface implementation from the crowd name.
interface, model_spec = CrowdRegistry.get_registry_entry(crowd_name)

Expand Down Expand Up @@ -216,9 +231,9 @@ def post_response(request, crowd_name):
# Check if this task has been finished
# If we've gotten too many responses, ignore.
if (not current_task.is_complete
and current_task.responses.count() >= current_task.num_assignments):
and current_task.responses.count() >= current_task.num_assignments):
current_task.is_complete = True
current_task.save()
gather_answer.delay(current_task.task_id, model_spec)

return HttpResponse('ok') # AJAX call succeded.
return HttpResponse('ok') # AJAX call succeeded.
22 changes: 12 additions & 10 deletions ampcrowd/docker-entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,25 @@ SSL=0
FOREGROUND=0
while getopts "dsf" OPTION
do
case $OPTION in
d)
DEVELOP=1
;;
s)
SSL=1
;;
f)
FOREGROUND=1
;;
case "$OPTION" in
d) DEVELOP=1;;
s) SSL=1;;
f) FOREGROUND=1;;
esac
done

export DEVELOP
export SSL
export FOREGROUND

ARG1=${@:OPTIND:1}

if [ "$ARG1" = "test" ]
then
(cd ampcrowd && python manage.py test **/tests/)
exit
fi

if [ "$DEVELOP" -eq "1" ]
then
echo "Celery launched in debug mode"
Expand Down
10 changes: 9 additions & 1 deletion ampcrowd/internal/interface.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from django.db.models import Count, F, Q

from basecrowd.interface import CrowdInterface
from models import CrowdTask
from models import CrowdTask, CrowdTaskGroup

SLACK = 2
ORDER_WEIGHT = 0.3
Expand Down Expand Up @@ -64,4 +64,12 @@ def get_eligible_tasks(worker_id):
# No duplicates
.distinct())

@staticmethod
def get_task_groups():
return CrowdTaskGroup.objects.all()

@staticmethod
def get_tasks():
return map(CrowdTask.to_json, CrowdTask.objects.all())

INTERNAL_CROWD_INTERFACE = InternalCrowdInterface('internal')
105 changes: 105 additions & 0 deletions ampcrowd/results_dashboard/static/dashboard.css
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
/*
* Base structure
*/

/* Move down content because we have a fixed navbar that is 50px tall */
body {
padding-top: 50px;
}


/*
* Global add-ons
*/

.sub-header {
padding-bottom: 10px;
border-bottom: 1px solid #eee;
}

/*
* Top navigation
* Hide default border to remove 1px line.
*/
.navbar-fixed-top {
border: 0;
}

/*
* Sidebar
*/

/* Hide for mobile, show later */
.sidebar {
display: none;
}
@media (min-width: 768px) {
.sidebar {
position: fixed;
top: 51px;
bottom: 0;
left: 0;
z-index: 1000;
display: block;
padding: 20px;
overflow-x: hidden;
overflow-y: auto; /* Scrollable contents if viewport is shorter than content. */
background-color: #f5f5f5;
border-right: 1px solid #eee;
}
}

/* Sidebar navigation */
.nav-sidebar {
margin-right: -21px; /* 20px padding + 1px border */
margin-bottom: 20px;
margin-left: -20px;
}
.nav-sidebar > li > a {
padding-right: 20px;
padding-left: 20px;
}
.nav-sidebar > .active > a,
.nav-sidebar > .active > a:hover,
.nav-sidebar > .active > a:focus {
color: #fff;
background-color: #428bca;
}


/*
* Main content
*/

.main {
padding: 20px;
}
@media (min-width: 768px) {
.main {
padding-right: 40px;
padding-left: 40px;
}
}
.main .page-header {
margin-top: 0;
}


/*
* Placeholder dashboard ideas
*/

.placeholders {
margin-bottom: 30px;
text-align: center;
}
.placeholders h4 {
margin-bottom: 0;
}
.placeholder {
margin-bottom: 20px;
}
.placeholder img {
display: inline-block;
border-radius: 50%;
}
Loading