Do we want to quantify how much human effort vs machine effort and if so HOW??

[lwasser]

I think this should have a multiplier in it. Maybe 3x?

Originally posted by @MicahGale in #371

[sneakers-the-rat]

I don't think this is really possible to quantify or specify, this is intended more as a values statement: "please consider the amount of labor you are asking someone to do - if your style of working asks someone else to do significantly more labor than you did when creating something, please do not submit it here." there isn't really a way for anyone to estimate how much time something will take to review, and people who are using AI tooling will certainly underestimate it anyway (because the generated code is already perfect, right? so simple. i just typed "make code good" and it appeared, so how hard could it be to review?)

we could give some heuristics like "say 100 lines of code takes an hour to review properly, if you did not spend an hour of your own time per every 100 lines of code that you committed, then you are likely asking reviewers to spend more time reviewing your code than you did."

[MicahGale]

Given how hard this is to quantify would it be ok to take a wait and see approach to see if it necessary to quantify this?

[sneakers-the-rat]

Just spot checked a JOSS submission that was driven by claude (i.e. "claude get this ready for JOSS"), and it included a "docs for reviewers" markdown file that gave an estimated review time of 15 minutes for a 33.6ksloc package, which is... laughable... so i am now leaning a bit towards needing to provide some kind of guidance here.

the most problematic submissions i've seen have been from people (almost as a rule, two people in the contributors list, solo author and claude) who maybe have never written code before in a serious way (e.g. outside of analysis scripts) before AI. So maybe they have never done a code review before, let alone a whole package review, and they might not have any idea how much time it actually takes. Especially since if they have submitted code to a traditional journal before, it's probably unlikely that the reviewers seriously evaluated the code for quality and just "ran the code, see if it did stuff, call it a day" - the claude-generated reviewer doc assumed that reviewers would not actually be reading the code and just be running the top-level entrypoint.

since we ask reviewers to estimate how much time it took them to review the package, we could do some quick back of the envelope estimation of time / lines and say something like "package review takes more time than you think, particularly with LLM-driven packages, which tend to be more code intensive than human-written packages. We estimate that review takes {n hours} per thousand lines, and we expect that authors will have spent many times that reviewing (not generating) their own code. If you have not performed that level of internal review, please spend your own time carefully and critically evaluating your own code before submitting. Ask yourself "is the code in a state where I feel good about asking someone to volunteer {n hours} of their time to review this""

idk that's just loose draft text, but that would be the kind of thing that might be helpful: a heuristic rule that gets the purpose across without needing to have some firm quantification of time spent.

[lwasser]

Hi Y'all - just circling back on this issue. I don't yet see a TODO that someone could tackle but we have made some updates to our policy. Where would you like to see some language about how much effort the AUTHOR has put into reviewing their package if it's generated in any way?

I like the idea of being a bit more specific around how much time someone has invested into creating the package, testing it, and reviewing the structure, usability, etc before submitting, particularly if it was co-developed with a model. Is that what you are thinking?

[MicahGale]

Personally after reviewing the existing policy I think the intent here is well captured by it. I personally don't think any hard numbers need to be made now, unless it becomes a problem in the future.