Project status

[cocreature]

First just to make sure this is not taken the wrong way, let me say that I am really thankful for all the hard work you put into llvm-general. It’s a great project and I like working with it.

Now to my actual point: It seems like you are quite busy and can’t invest a lot of time into llvm-general. This is obviously completely fine, nobody is obligated to put time into open source.

However it would be a shame if llvm-general had to suffer from that. Having a busy maintainer means that people have to wait longer for responses to issues and prs which makes them less likely to contribute more and also makes it harder to move the project forward.

The only solution that I see apart from you suddenly gaining more time which seems (sadly) unlikely, is giving more people commit access and thereby making this more of a community project. Cabal has done this recently in the style of The Pull Request Hack.

This obviously means that you’re no longer the benevolent dictator and things don’t necessarily go the way you want them to. But I think expecting people to be sensible and still run bigger changes via PRs to find some consensus on whether they are a good idea is not too risky (at least not if they have already positively contributed in the form of a PR in the past).

Since I guess I’m kinda asking for commit rights for myself with this even though I hate doing so, let me get one thing that I am a bit unhappy with and that is the support of different LLVM versions. It is not clear which ones are actively maintained, they get out of sync for changes that make sense for all of them and versioning llvm-general by LLVM doesn’t fit into hackage where the versions typically increase monotonically. I propose to only actively support one LLVM version (people who need support for older LLVM versions can just check older llvm-general releases), maybe throw some CPP in if the effort of supporting multiple LLVM versions in one release is not too much effort and decouple the versioning scheme from the LLVM scheme.

[bscarlet]

I'm ultimately not just open to but interested in sharing commit access and ultimately hackage release access, but only after I'm fairly sure of where it's going to take the project. Please continue to discuss where you want to make progress, and try to make PRs that apply cleanly. We can start a mailing list or use bugs to discuss individual issues. I can try to respond to individual PRs with a better turnaround time (especially if they're individually smaller). Once you've got a pile of rubber-stamp-only PR acceptances under your belt, it would make sense to give you commit access to take out the double-check.

I don't think your proposal for actively supporting only one LLVM version is a good idea. Users of LLVM are often constrained as to the version of LLVM they must use - e.g. by GPU-vendor supplied parts of an overall toolchain when using LLVM for GPGPU stuff. Dropping older versions can leave such users high-and-dry. I would like to support some reasonable "last few" LLVM releases. I was doing so up to 3.5, supporting 3.4 and 3.5, then got behind. I'm hoping to pick up with your work on 3.8 and start dropping the old ones. Certainly clearer documentation as to the support state of each LLVM version would be a good idea.

[cocreature]

Thanks for your response!

Your argument about GPUs is convincing I haven’t thought of that one. What I am not sure is, if it really makes sense to maintain more or less separate code bases for different LLVM versions. The reason for this is that this makes it quite awkward to contribute changes that affect all branches, e.g. the recent GHC-8 PR should probably be in every branch. Furthermore as you already mentioned it makes it unclear which LLVM versions are supported. While CPP and/or cabal flags to support different LLVM versions are definitely not great I feel like it’s still less awkward and makes it easier to both maintain and contribute for others. Personally I think supporting the last 3 major LLVM releases sounds like a reasonable number of different versions to maintain.

Next it should be clear which GHC versions are supported. I would just go with the last 3 major releases as it is common in the Haskell ecosystem. (Which means that 7.6 support would be dropped in favor of 8.0)

Now to the other changes that I would like to see and help implementing

• Obviously and most importantly bug fixes and support for more LLVM features.
• Cleanup the codebase. llvm-general has huge chunks of fairly brittle TH code in it which mostly consists of abbreviated variable names that are not obvious and is thereby fairly hard to read and understand. By refactoring some of that into smaller portions with easier to read names and comments, it gets easier to maintain, easier to figure out why an LLVM update broke something and easier for contributors. A further step would be to enable -Wall to make it easier to spot bugs (I already have a patch making the 3.8 branch -Wall clean lying around locally). It might also make sense to define a (nonformal) styleguide to make sure that the codebase is somewhat consistent.
• Cleanup the flag selection process. This includes selecting between libstdc++ and libc++ (@acowley has already implemented that on my llvm-3.8 branch) as well as figuring out which flags provided by llvm-config are actually necessary (see http://lists.llvm.org/pipermail/llvm-dev/2015-August/088922.html and our previous discussions) vs simply selecting all of them. IMHO the best solution here is to be conservative and filter out only the flags that we know are necessary as it is quite easy to add more when people run into issues.
• Be able to load llvm-general into stack ghci. This would also fix the usage of tools like intero. -fshared-llvm is not enough for that (it only helps for projects that depend on llvm-general. I have a patch but it doesn’t yet take into account the different libc++ implementations.

Can’t think of anything else right now.

[tinco]

Just my 2cts. If someone is tied to an older version of LLVM, why not let them use an older version of llvm-general? I think with a small project like this having only a single person as core committer it is too ambitious to be supporting multiple compilation platforms and GHC versions. Especially if like you say LLVM-General is not exactly where you want it to be. Also, it's not like older versions of llvm-general are unusable.

I'd say if no one is paying you to offer that support, why bend over backwards to offer it? I can't presume to speak for the majority, but from my perspective it seems like most users of llvm-general enjoy nice up to date dependencies, as well as probably your sanity. Too many open source maintainers burn out because of trying to satisfy everyone in their communities. Unless of course they are paying you. In which case, carry on :P

[bscarlet]

cocreature:

• On multiple LLVM versions:
  • The reason I'm using branches rather than CPP for different LLVM versions is that LLVM rearranges itself between versions. For purely additive and subtractive changes, CPP would be fine, but down in the details features get restructured as the LLVM team decides to change how things work. I believe git and diff add significant value to the hard task of porting changes between different LLVM versions.
  • I think supporting 3 LLVM versions would be fine. Right now ideally that would be 3.6, 3.7 and 3.8, but given I (we?) are behind, just getting from the current 3.3, 3.4, 3.5 to 3.4, 3.5, 3.8 would be progress.
• I also agree aiming for the last 3 GHC releases is a reasonable goal.
• Please open bugs for any particular LLVM features you'd like to support.
• I think concrete, small PRs would be a much more productive format for discussing cleanup for improved readability than any broad statements. I use so much TH to avoid the reams of boilerplate which not using it would require, but if you see ways to make it clearer, so much the better. I've actually found it not to be particularly burdensome, but I agree it's opaque to the uninitiated.
• I think you're oversimplifying flag selection, but as usual PRs are welcome, especially small ones.
  • The most immediate issue is figuring out what changed in Cabal that broke the code to filter flags differently for hsc C code and llvm-general C++ code.
  • Supporting using a different compiler than was used to build LLVM would be good, but is a feature addition
  • It's not as easy to "add more when people run into issues" as you say, because which flags are required are a function of the users environment - there isn't one right answer, there's a right answer for each compiler, OS, etc.
• I am not yet a fan of stack.
  • I have yet to see any explanation of how they intend both to maintain version compatibility and advance package versions as the number of included packages increases.
  • Building stack requires a pre-built stack, last I checked. Thus using it requires trusting that their binaries correspond to their source. That's a very bad property for a low-level build infrastructure tool.

[bscarlet]

tinco:
Your suggestion would make llvm-general unusable for a large range of use cases, my own included.

[cocreature]

I am not yet a fan of stack.

The problem is not specific to stack at least cabal new-repl can’t load llvm-general either (don’t have the time to setup a sandbox to try normal cabal repl right now).

[bscarlet]

What problem is not specific to stack?

[cocreature]

Not being able to load llvm-general into ghci during development. Although taking a closer look it seems like the error for cabal new-repl is slightly different, but this is getting off topic. I’ll open separate issues for these problems.