DDEV install fails, because web folder is owned by root

[gitressa]

I am not sure when this started, I first noticed it a few days ago. Hoping it was an isolated event, which would go away next time, I just chowned web-folder to my user, and ran the command again. But it happened again ...

To reproduce, install Starshot in DDEV with the README method, and see that it fails, since web folder is not writable. Excerpts from the command line:

[...]
- Installing drupal/core (10.3.0-beta1): Extracting archive
  Install of drupal/core failed
[...]
Unhandled promise rejection with RuntimeException: /var/www/html/web/core does not exist and could not be created:  in phar:///usr/local/bin/composer/src/Composer/Util/Filesystem.php:260

web folder is owned by root:

$ ls
total 560K
-rw-r--r--  1 user  user 6,8K 23 maj 11:53 composer.json
-rw-r--r--  1 user  user 510K 23 maj 11:54 composer.lock
drwxr-xr-x  2 user  user 4,0K 23 maj 11:53 drush
drwxr-xr-x  4 user  user 4,0K 23 maj 11:53 patches
-rw-r--r--  1 user  user 1,6K 23 maj 11:54 patches.lock.json
-rw-r--r--  1 user  user 2,9K 23 maj 11:53 phpunit.xml
-rw-r--r--  1 user  user 6,4K 23 maj 11:53 README.md
drwxr-xr-x 16 user  user 4,0K 23 maj 11:53 recipes
drwxr-xr-x  3 user  user 4,0K 23 maj 11:53 tests
drwxr-xr-x 12 user  user 4,0K 23 maj 11:54 vendor
drwxr-xr-x  2 root  root 4,0K 23 maj 11:54 web

[phenaproxima]

Fun times. Maybe something broke upstream in DDEV? What version are you on?

[gitressa]

$ ddev version
 ITEM             VALUE                                    
 DDEV version     v1.23.1                                  
 architecture     amd64                                    
 cgo_enabled      0                                        
 db               ddev/ddev-dbserver-mariadb-10.11:v1.23.1 
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.23.1              
 docker           26.1.3                                   
 docker-api       1.45                                     
 docker-compose   v2.27.0                                  
 docker-platform  linux-docker                             
 mutagen          0.17.2                                   
 os               linux                                    
 router           ddev/ddev-traefik-router:v1.23.1         
 web              ddev/ddev-webserver:v1.23.1              

Host machine is Debian 12. If I create a standard DDEV Drupal 10 instance, everything works as usual.

[isalmanhaider]

Try changing the ownership of the web folder to match the other folders. Ensure the web folder is owned by your user instead of root:

sudo chown -R user:user web
chmod -R 755 web

[gitressa]

Thanks for the suggestion, and that's also what I did as a workaround. But it shouldn't be necessary in the first place ... An observation: The web-folder is created by the ddev install command.

Also, if I ddev delete and then try to rm the folder, I get this message:

$  rm starshot/ -rf
rm: cannot remove 'starshot/web/sites/default/default.settings.php': Permission denied
rm: cannot remove 'starshot/web/sites/default/default.services.yml': Permission denied
rm: cannot remove 'starshot/web/sites/default/settings.php': Permission denied
rm: cannot remove 'starshot/web/sites/default/files': Permission denied

This also doesn't happen in standard DDEV/Drupal installations, so something is off with the permissions in Starshot ...

[phenaproxima]

I agree with @gitressa, it should not necessary to change the permissions yourself as a workaround.

If you look at .ddev/commands/host/install, though, it's doing all set-up in the context of the web container. The only command the host needs is open, which doesn't touch anything.

So I'm a bit stumped as to why this would be happening.

[gitressa]

Yea, it's strange. I wonder if the issue below and its PR are connected to this? It may have started since the release of DDEV 1.23.1 a week ago ...

• vscode - mkdir: cannot create directory ‘/workspaces/d9simple’: Permission denied ddev/ddev#6187
• build: fix devcontainers postCreateCommand permissions, fixes #6187 ddev/ddev#6194.

[roromedia]

It is strange on my side as well. I am on an Apple Silicon Mac and upgraded to the latest DDEV and installed Starshot without a problem. As soon as I launched it, it works, but soon after the database errors hit me like explained here: #59
I switched to "gRPC Fuse" as Randy Fay mentioned that VirtioFS might still be buggy. Then reinstalled Starshot and everything worked.
Afterwards I worked on other D10 projects and realized that I run into strange issues with the filesystem between host and guest system (saving on host does not update in Docker - when saved inside Docker it immediately responded on host IDE). I had to switch back to VirtioFS in order to proceed which makes now Starshot unreliable buggy again...
Tried a few other tweaks (like changing permissions, ...) and googled for solutions but haven't found any definitive solution to it yet.
My reasoning currently: It could maybe work with "gRPC Fuse" and enabled mutagen as I got the feeling that any file changes inside Docker always work in the direction to the host, but the other way round it doesn't - mutagen might eventually trigger the filesystem to sync more reliable? Not tested yet as I normally don't use mutagen because of the slight delays in sync.
Before Starshot I had zero issues with VirtioFS on DDEV 1.22.3 and DDEV 1.22.7 so I am not 100 % sure what combination of versions trigger the issues with VirtioFS - it is not Starshot itself. The only real difference between Starshot and my other projects I can see: Starshot is a single image in Docker Desktop and my other projects have a separate mariadb-image.
Maybe you could try to branch the Starshot database out into a separate image to see if the database corruption issue could be mitigated by that and then we could use VirtioFS which was at least for me flawless until now. I also got another D10 installation without DDEV spread over several Docker images (one for web, one for solr, one for db and so on), also there no VirtioFS-issues so far the last months.

[thegbomb]

I think it is an issue with the ddev start command. When no webroot exists, ddev start creates the folder.
It seems to be doing so with root ownership. Not sure if this is new or existed previously, though, as I usually composer create-project before doing ddev config and as such the webroot folder is usually already in place.

My workaround has been to edit the host > command > install file

Add a step to:
mkdir -p web
This creates the folder with the correct permissions before ddev does so.
composer install then populates it as expected.

[gitressa]

Thanks for sharing @roromedia and @thegbomb. My hunch is that "something" changed with DDEV 1.23.1 and instead of creating the web-folder as owned by the user, it is now owned by root. I reopened ddev/ddev#6158 with a link to this issue, and hope the DDEV team can take a look at it at some point.

In the meantime, as a workaround, Starshot could ship with an empty web-folder, so that it is owned by the user? Like so: https://github.com/phenaproxima/starshot-prototype/tree/main/web

[stasadev]

It is because of the change in .ddev/config.yaml made in:

• Install Project Browser with recipe support #74

working_dir:
  web: /var/www/html/web

web doesn't exist in this repo because of disable_settings_management: true.

And the working_dir is created by Docker itself, if it doesn't exist, we can't control this behavior:

• Set owner of items created by Docker on the host docker/cli#4582
• Add ability to mount volume as user other than root moby/moby#2259

---

I think you should commit the web directory by putting some file in it.

[phenaproxima]

I think that sounds like a good workaround. Let's create web/.gitkeep and commit that.

[thegbomb]

#95

[phenaproxima]

Merged #95, so tentatively closing this, but please reopen if the trouble persists.

[rfay]

Just a note: If you're having ownership of files changed to root ownership, something is wrong with your Docker setup. I think @gitressa is a linux user, I'd be happy to debug this with you, please come over to the DDEV Discord and let's look. DDEV goes to great lengths to make sure that this never happens, and runs all of its containers (especially the web container) as the normal user (not as root). Maybe you're running an additional container that runs as root in this environment?

[stasadev]

Opened

• fix: create working_dir for web if it doesn't exist, fixes #6158 ddev/ddev#6242

[gitressa]

Thanks for the analysis and solutions. As I commented earlier, I also need to chmod folders in Starshot to be able to delete them, which is not ordinarily necessary ... Maybe the newly present web will take care of this as well? I'll follow along in ddev/ddev#6242. Thanks!

[rfay]

Yes, it seems that this is a longstanding Docker bug on Linux, but for various reasons it rarely shows up with DDEV, but in this case it's a result of the working_dir being set.

[gitressa]

With web-folder now present from the get-go, the install completes, and all is well.

I do still need to chmod 755 web/sites/default before being allowed to delete the Starshot project folder. This does not happen in standard DDEV/Drupal installations, because web/sites/default folder has drwxr-xr-x not dr-xr-xr-x, so I created #97.

[rfay]

@gitressa your umask must be set very restrictively. Please try mkdir junk and ls -ld junk and see what the result is.

[gitressa]

If the problem is my set up, shouldn't a standard Drupal 10 installation have the same problem then?

$ mkdir junk
$ ls -ld junk
drwxr-xr-x 2 user user 4096 25 maj 14:32 junk

[rfay]

Please ls -ld web on a fresh checkout of the repo. If it's permissions 555, it's how your git is set up. On my system it's 755 permissions.

[gitressa]

Sure!

$ ls -ld web
drwxr-xr-x 2 user user 4096 25 maj 15:35 web

[rfay]

Oh, sorry, I misread your original set of issues. You said

I do still need to chmod 755 web/sites/default before being allowed to delete the Starshot project folder.

But I read "I have to chmod web".

Your issue is a result of the thing Drupal has done from the beginning of the universe of making sites/default read-only. It's a crazy thing and not something an application should do IMO.

DDEV prevents this problem by adding to settings.ddev.php the line
$settings['skip_permissions_hardening'] = TRUE;
to turn off that behavior, which is not useful for local development (and I would maintain never useful)

[gitressa]

Thanks for the clarification @rfay. So, since settings.ddev.php is not present in Starshot, is #97 a pragmatic solution to this problem?

[rfay]

No, I don't think so. Drupal is always trying to set that directory read-only. Just use the Drupal settings technique with skip_permissions_hardening.

Note that this has nothing to do with Drupal 10, Drupal 11, or Starshot. It's all about how Drupal has behaved since maybe Drupal 5 or something.

[gitressa]

My aim is to make the Starshot folder deletable out-of-the-box, without the user having to do anything, just like a standard DDEV Drupal 10 installation ...

Do you think adding skip_permissions_hardening in settings.php as a last step of ddev install instead would work better, or are you thinking of something else?

[rfay]

Do you think adding skip_permissions_hardening in settings.php as a last step of ddev install instead would work better, or are you thinking of something else?

I don't understand why it wouldn't just be set in settings.php to begin with (as it is with DDEV). So if this project is creating a settings.php, add it. Or add a settings.local.php.

My preference of course would be to just use DDEV for local dev, which has these things dialed in, and which also uses "real" database types.

[gitressa]

I would definitely be ready to review it, if the DDEV team were to make an MR, to improve DDEV integration in Starshot ...

[rfay]

We already have so much trouble with scope... I don't know why there should be a problem with DDEV integration with Starshot. If there are problems, you know we're here to help. (Early on, it looked like Starshot for some reason was unwilling to use drush or standard database types, and was using the very crippled and mostly unmaintained in-core installer. That may be the issue, not sure.)

[phenaproxima]

Okay, folks...

This isn't DDEV's responsibility to fix. I'm guessing this happens because I disabled DDEV's settings management in config.yml. Why did I do that? Because DDEV tries to insert credentials for its database container, and that doesn't work with the core/scripts/drupal install command, which is currently hard-coded to use a SQLite database.

To me, this is a case of several reasonable assumptions (from Drupal and from DDEV) turning bad when they converge.

The proper fix here would be to allow DDEV to manage the site settings -- and, indeed, to use DDEV's built-in database by default, rather than SQLite -- but I'm pretty sure that's not going to fly until core/scripts/drupal install can accept user-supplied DB credentials.

I'm not sure if there's an issue/patch for that already; I feel like I saw @mglaman working on that, but I might have imagined it. If there IS such a patch, and it's likely to be accepted by the core committers, then Starshot could go ahead and apply that patch and take advantage of it to fix this.

[phenaproxima]

@rfay We're using the core installer because it's not unmaintained, and currently it's the only installer I know of that can install directly from a recipe. If Drush added support for that (that is, drush si should be able to take either a recipe path or a profile name), we could switch happily to Drush and this problem would be solved.

[rfay]

The core installer can only do sqlite3 and hasn't been touched since 2018. I spent some time looking at it for

• fix: Install drush without affecting git index justafish/ddev-drupal-core-dev#33

which had the same class of problem, ddev-drupal-core-dev was tethered to the core installer. We're going to have it untethered soon, but it's a standard problem. Sqlite3 is not "normal" or representative of real situations, and the core installer is also not capable of doing what "normal" drupal does.

[roromedia]

Getting rid of SQLite in favour of any mySQL flavour would be very much appreciated!

[roromedia]

Can't we switch away from quick-start feature to regular composer installation with "--no-install" flag and then run "drush site:install" and the needed recipes via drush?
With DDEV it would be very simple to add more commands for a completely unattended setup, for a composer-only based installation it might involve some more manual steps or maybe we could use "post-install-cmd"
"https://getcomposer.org/doc/articles/scripts.md#command-events" to start drush site:install afterwards?

[phenaproxima]

Getting rid of SQLite is a non-starter, IMHO, because that would kill the "zero to Drupal in one command" functionality.

But being able to override it is an existing feature request. We do ultimately need to be able to install into MySQL or Postgres.

[phenaproxima]

@roromedia drush si introduces some additional problems, because then recipes have to undo stuff that the install profile did. It's not great. We really want to be able to install directly from a recipe.

The core installer already supports that, but Drush would need to as well before we could switch to using it. Not sure if Drush already has an open PR for that...I know they already added a recipe command, but that's not what we need. We need install from a recipe. For example:

drush si recipes/starshot --db-url=mysql://user:pass@dbhost/db

[roromedia]

Thanks @phenaproxima for clarifying, makes sense. Seems overriding database in the quick-start feature would be the best approach for this.

[rfay]

Getting rid of SQLite is a non-starter, IMHO, because that would kill the "zero to Drupal in one command" functionality.

It wouldn't make sense to "get rid of" Drupal's longstanding support of SQLite. Supporting the "normal" Drupal usage of other databases makes lots of sense.

However, if "Zero to Drupal in one command" were to fundamentally mean installing with sqlite, it will quickly be considered a fraud by users, because once they do it they have no place to go forward, and they can't even use the database they export in a real situation. It may have value, but super important not to over-promise. For example the built-in webserver in PHP is a useful tool, but if it were promoted as a full webserver, it would way out-promise its reality.

[roromedia]

I completely agree @rfay . Also all the benefits of a quick-start are gone immediately if you end up with SQLite after the installation. I would also hope for mySQL from day one as Starshot should cater to a different focus group which should be less developer-oriented and therefore very easy to deploy on any "regular" (IMHO any LEMP/LAMP/MAMP stack) hosting. The most common stack where any mySQL-flavour is pre-configured would be the easiest for new Starshot-users.

[thegbomb]

I think there are 2 things going on the quick-start one click installer is one thing, but this prototype is a different scenario in itself.

By adding ddev, we have already moved away from the one click installer as we need docker and ddev installed to use this.
We may as well use mysql or mariadb if using ddev.
I have it running locally on mariadb, I removed the
-omit_containers: [db]
and
-disable_settings_management: true
from the ddev config file and then ran the install command. Everything seems fine. My only worry would be that drupal:install is removing the settings.php file, which other commands want to manipulate as well e.g. ddev adds its include lines when they are missing. So some work may need to be done there to make that bit more robust.

[roromedia]

The first option for many users might still be the composer-based one-liner which facilitates composer create-project without any DDEV as stated in the README.md here: https://github.com/phenaproxima/starshot-prototype/?tab=readme-ov-file#installation
I personally prefer DDEV over composer create-project as nearly zero configuration is needed on a typical machine. So for the DDEV-way I would also suggest to adapt the installation routine to make it mariadb-based like you already did.

[greg-1-anderson]

See drush-ops/drush#6026

@roromedia drush si introduces some additional problems, because then recipes have to undo stuff that the install profile did. It's not great. We really want to be able to install directly from a recipe.

...

drush si recipes/starshot --db-url=mysql://user:pass@dbhost/db