Thoughts on how to divvy-up validators

[michaelcueno]

Types of Validations (Let me know if I'm missing something)

1. Trivial syntax ( We can handle these, if, else, then , unless .. )
2. Non-trivial syntax
   -Contain a loop
   -Thoses using nil?
   -Anything else
   These will be broken into categories for sexp* keyword* or function*, each of which will have as many columns as new unknowns found and will be listed in order of frequency
   *A keyword being anything in the first argument of a :call
   *A function being anything in the second argument of a :call
   Example AST: s(call, s(Keyword, Keyword), function, arglist)
   *An sexp would be the :call in the above example
3. call an external method (and fails)
4. Interact with the database (still working on how to detect these)
5. Use something we havent' seen yet (Unknown case, basically if the translator fails, but the log contains none of the above)

Order of Ops (Assuming we have all apps in a single directory) This will be automated in a script
o For each app

1. Install app  (I don't think this should be a problem as long as we are dealing with rails 3 apps.. Not too sure though)

2. run rake translate ( this will perform the tasks listed below *this is something I wrote just to be clear*)
  o For each model
    - Run get_validations
    - Divvy into Rails built-ins, Custom model defined validations, Custom Validators, Assiciations or something else
    o Run mimic_server_side_validations on model (This will only analyze Custom Validators and Custom model defined validations)

3. run divvy*

*divvy is what I'm calling the external program to parse the log files and create some human readable data (Piecharts!)

[michaelcueno]

This is less of an issue and more of something I wanted documented. Basically just my plan of attack, and I wanted to share it with you guys in case anyone had a better idea or saw a problem.

Let's discuss this at the meeting.

Tim

On Fri, Jun 22, 2012 at 1:09 PM, mcueno2
reply@reply.github.com
wrote:

Types of Validations (Let me know if I'm missing something)

1. Trivial syntax ( We can handle these, if, else, then , unless .. )
2. Non-trivial syntax
       -Contain a loop
       -Thoses using nil?
       -Anything else
          These will be broken into categories for keyword* or function_, each of which will have as many columns as new unknowns found and will be listed in order of frequency
          _A keyword being anything in the first argument of a :call
          *A function being anything in the second argument of a :call
              Example AST: s(call, s(Keyword, Keyword), function, arglist)
3. call an external method (and fails)
4. Interact with the database (still working on how to detect these)
5. Use something we havent' seen yet (Unknown case, basically if the translator fails, but the log contains none of the above)

Order of Ops (Assuming we have all apps in a single directory) This will be automated in a script
 o For each app

   1. Install app  (I don't think this should be a problem as long as we are dealing with rails 3 apps.. Not too sure though)

   2. run rake translate ( this will perform the tasks listed below this is something I wrote just to be clear)
     o For each model
       - Run get_validations
       - Divvy into Rails built-ins, Custom model defined validations, Custom Validators, Assiciations or something else
       o Run mimic_server_side_validations on model (This will only analyze Custom Validators and Custom model defined validations)

   3. run divvy*

*divvy is what I'm calling the external program to parse the log files and create some human readable data (Piecharts!)

---

Reply to this email directly or view it on GitHub:
#4

[michaelcueno]

mike@sisl-ibm-lt:/Research/candidates$ ls
divvy.rb Insoshi-Rails-3 piecharts.rb rails-prelaunch-signup redmine
mike@sisl-ibm-lt:/Research/candidates$ ruby piecharts.rb
crunching data...
=========== Overview statistics =============
Number of rails applications: 3
Numer of apps with custom validations 2
Total validation methods: 121
Total number of custom defined validation methods: 13 or 10%

=========== Non-custom validations breakdown ============
Total association validations (Those that occur when a has_many is used): 66 or 54%
Total builtin validations: 37 or 30%
Total unknown validations: 5 or 4%

=========== Custom defined validations breakdown ===========
Total validations defined as Validators: 0 or 0%
Total validations defined in model: 13 or 10%

=========== Breakdown of custom validator types =============
Number of successfully translated validators: 1 (7%)
Number of unsuccessfully translated validators: 12
Number of validators with errors related to...

1. Keyword: 4
2. Function: 4
3. S-Expression: 4
4. External Method: 9
5. Database: 0
6. Unknown: 0

=========== Most common S-Expression failures =============
iter ----- 6
ivar ----- 4
lasgn ----- 4
lvar ----- 3
op_asgn_or ----- 4
rescue ----- 1

=========== Most common Keyword failures =============
colon3 ------ 1
const ------ 1

=========== Most common Function failures =============
<< ------ 3
[] ------ 4
all_dependent_issues ------ 2
attachment_max_size ------ 1
cross_project_issue_relations? ------ 1
demodulize ------ 1
empty? ------ 2
enabled_scm ------ 1
has_key? ------ 2
include? ------ 3
mike@sisl-ibm-lt:~/Research/candidates$

Cool stuff, Mike.

Here's a brain dump of what I think we agreed was the information we
want to collect. Let me know if this doesn't jive with what you're
thinking.

• Builtins
  o Number of NUMERICALITY
  o Number of REGULAR EXPRESSION
  o ... [other simple built-in validations]
  o Number of HAS_MANY validators
  o ... [other validations run because of associations]
• Custom: We want to analyze each BLOCK of a validation, record which
  validator that block belongs to, and record which application that
  validator belongs to. For each block, we want to know which syntactic
  fragment of Ruby it utilizes, i.e. which of the following syntactic
  constructs it uses.
  o if-then-else, case, unless
  o loops
  o Ruby's built-in function calls (record which ones)
  o developer-written, recursive function calls (non-recursive are inlined)
  o other keywords
  We also want to check for each block whether the ERRORS array is being
  modified in a way that we're sure it's signaling an error. All other
  blocks are signaled as having UNKNOWN-SEMANTICS.

Additionally, there are some meta properties we want to collect about
each block/validator we analyze.
o The file/line number of the block, for easy manual reference.
o Whether a validator is defined in the model or as a Validator
o Enough info so that we can run an improved version of our
translator on just those validators it failed on originally.

Conceptually, we want to think about the data we collect as being a
database that we can query after the analysis finishes. Ideally
we'll be able to answer questions we don't yet even know are
important.

Tim

On Thu, Jul 5, 2012 at 2:31 PM, mcueno2
reply@reply.github.com
wrote:

mike@sisl-ibm-lt:/Research/candidates$ ls
divvy.rb Insoshi-Rails-3 piecharts.rb rails-prelaunch-signup redmine
mike@sisl-ibm-lt:/Research/candidates$ ruby piecharts.rb
crunching data...
=========== Overview statistics =============
Number of rails applications: 3
Numer of apps with custom validations 2
Total validation methods: 121
Total number of custom defined validation methods: 13 or 10%

=========== Non-custom validations breakdown ============
Total association validations (Those that occur when a has_many is used): 66 or 54%
Total builtin validations: 37 or 30%
Total unknown validations: 5 or 4%

=========== Custom defined validations breakdown ===========
Total validations defined as Validators: 0 or 0%
Total validations defined in model: 13 or 10%

=========== Breakdown of custom validator types =============
Number of successfully translated validators: 1 (7%)
Number of unsuccessfully translated validators: 12
Number of validators with errors related to...

1. Keyword: 4
2. Function: 4
3. S-Expression: 4
4. External Method: 9
5. Database: 0
6. Unknown: 0

=========== Most common S-Expression failures =============
iter ----- 6
ivar ----- 4
lasgn ----- 4
lvar ----- 3
op_asgn_or ----- 4
rescue ----- 1

=========== Most common Keyword failures =============
colon3 ------ 1
const ------ 1

=========== Most common Function failures =============
<< ------ 3
[] ------ 4
all_dependent_issues ------ 2
attachment_max_size ------ 1
cross_project_issue_relations? ------ 1
demodulize ------ 1
empty? ------ 2
enabled_scm ------ 1
has_key? ------ 2
include? ------ 3
mike@sisl-ibm-lt:~/Research/candidates$

---

Reply to this email directly or view it on GitHub:
#4 (comment)