Scoped storage implementation of Android 11+ and also bump older vers…

[Xavron]

…ions onto it if they have issues with the old code.

All final tests:

[x] == passed
[] == not tested; not looked into
[*] == couldn't connect on this device for unknown reason and not debugging this; not tested

Continual Backups 1GB-81GB:
Android 12 internal [x]
Android 12 sd card  [x]
-
Android 8 internal  [x]
Android 8 sd card   [x]
-
Android 7 internal  [*]
Android 7 sd card   [*]
-
Android 6 internal  [x]
Android 6 sd card   []

Full restore of multiple backups 1GB-81GB:
Android 12 internal [x]
Android 12 sd card  [x]
-
Android 8 internal  [x]
Android 8 sd card   [x]
-
Android 7 internal  [*]
Android 7 sd card   [*]
-
Android 6 internal  []
Android 6 sd card   []

Filezilla:
Android 12 internal [x] 
Android 12 sd card  [x] 
-
Android 8 internal  [x]
Android 8 sd card   [x]
-
Android 7 internal  [*]
Android 7 sd card   [*]
-
Android 6 internal  [x]
Android 6 sd card   []

Other:
Test backup and FileZilla after app reinstall [x] <- write external storage is a problem here (see known issues).
Test older File code execution is still working + clean install [x] <- tested on Android 6 device.

Successful tests include:
movement
creations
deletions
transfer up
transfer down
change of target location without restart of app
target in sub folders
target at picker location
multi-clients (running at the same time and at separate times)
multi-clients supplied different sub folders
user added chroot
picker
sd card
internal
various folders & sub folder tests (eg duplicate looking but original sub folder paths)
chroot
files going to the correct folders
multi file transfer
etc

Known issues:

o Some classes such as CmdRNFR and couple or so of others are not implemented with this code so certain uses could fail to work properly.

o Android 12 is not allowing files to be created in dirs with eg "?" in the name but allows this with "#" and also fine with creating dir with "?" in the name.

o At least on Android 12 with the new code, should eg FileZilla have a set remote path in site settings that is different from the app, then the new code currently
  doesn't deal with this where the input param is blatantly incorrect.

o When reinstalling app after having used write external storage, the app suggests its fine via the storage path, but it is not. Write external storage must be
  used again or the code will fail to work. Hardened new code here against hard crashes but that's it.

Random notes:

o Older storage code mostly kept the same and so older Android was tested for correct code execution against the new code.
    Two exceptions to this are:
    	o LIST code improvement is notable change with it otherwise seen as working fine.
    	o Added code for older Android versions to use the new storage where possible to get the app working where it would otherwise fail to work at all.

o Android 7.1.1 sd card didn't need the new code as it could read and write to the sd card with the old code.

o FileZilla has a setting to increase its timeout so eg LIST failure due to amount of files can be further fixed by increasing that (is up to client application).
    But, the new LIST optimizations greatly decreases the time so at least its much better overall.

o Noting the device battery optimization to be complete. Eg Samsung needs Unrestricted to be set or connection failures will happen as the app gets killed (it will
    happen even in the middle of use making it seem like the app code has a problem when it doesn't).

[Xavron]

To note. I have some raw use in there for minimizing duplication. I'm sure that will totally freak out some people rofl. Its easy and quick enough to create duplication from it anyway if its not palatable. :)

Just briefly tested clean build of the code pull so its all there and working.

Possible someone has a device where some issues show up that I've not seen.

Also did not run Firebase auto testing, monkey test, or other auto tests on this which I normally do for weeks. (Though of course I was, have been, and still am running automated backups using it.)

[Xavron]

One more thing I forgot to add. I intentionally "separated" the scoped code from what you had to not introduce any new issues and maintain backward compatibility as it is currently. Could then work on either as needed instead of dealing with even more conflicts and regressions.

Since File isn't compatible with DocumentFile and scoped storage, I felt this was best. Didn't want outright full duplication so eliminated some of the worst spots for that.

I don't plan on working on this or the app further. Might be possible that I will for a very short time. The code is fully working and no issues on my end. So I mostly leave this code in your hands to do as you like with it :)

[ppareit]

Great! Will look into getting this pulled in.

[ppareit]

One more thing I forgot to add. I intentionally "separated" the scoped code from what you had to not introduce any new issues and maintain backward compatibility as it is currently. Could then work on either as needed instead of dealing with even more conflicts and regressions.

Glad you did that!

[Xavron]

Now also briefly tested as good on Android 13 as well. It should be all good there anyway but now I got some actual use in :)

[Xavron]

Running new different application and seeing an app crash. Will likely look into it and tack that on.

Edit: Fixed. Will send along soon-ish. There's something else I need to look into.

[Xavron]

swiftp/app/src/main/java/be/ppareit/swiftp/server/CmdMKD.java

Line 72 in 6bb7a94

sessionThread.writeString("250 Directory created\r\n");

I didn't expect that. Got a FTP client application rejecting the 250 but accepts the 257. The 257 is also used by a domain name registrar. Although, I see some seem to suggest that 250 can be used or both even lol.

Kind of something like this: https://stackoverflow.com/questions/46201096/handling-babyftp-mkd-250-response-with-indy

Edit: Looks like correct thing would be to adjust it to 257 which is of course supposed to be spec. But, it still is iffy as in 2008, Microsoft FTP server was sending MKD 250 also https://bugs.launchpad.net/bzr/+bug/224373 :)

Any thoughts on that if you're not too busy?

[eakteam]

@Xavron It seems not to be working on Android 13...
I just forked from this repo -> https://github.com/Xavron/swiftp/tree/Scoped_storage and tried to test it.
Installed on my phone Samsung Note 20 but it fails to select starting directory if writing external storage is enabled. Also it fails to list folder and files sometimes and shows empty.

Tested with FileZilla

[Xavron]

Added in new commit 8fcae43 for various fixes.

Edit - Forgot to add:

[All tested and checks while using target of 33.]

Fixed various path related issues plus checked all for conflicts and re-tested:

  o Possible hard crashes seen when using new FTP client application.

  o Issue seen with new FTP client application where root wasn't sent back as "/".

  o Occasional not returning to root with FileZilla.

  o New code using old code on Android 8.0 sd card that was not creating folders with FileZilla.

Noted issues not fixed:

o Android 8.0: Not using write external storage with sd card results in the old code running and in a hard crash.

o Write external storage at least on Android 8.0 with target 33 isn't working right until the app is force ended then started again.

[Xavron]

@Xavron It seems not to be working on Android 13... I just forked from this repo -> https://github.com/Xavron/swiftp/tree/Scoped_storage and tried to test it. Installed on my phone Samsung Note 20 but it fails to select starting directory if writing external storage is enabled. Also it fails to list folder and files sometimes and shows empty.

Tested with FileZilla

I've been primarily testing on multiple Samsung devices and including Android 12 & 13 [local and sd card] and not sure what's happening there on your device.

While you're using "write external storage", that is fully the OS there. It simply gives access and that access is obtained and the path is then automatically entered into the apps "manage users..." section since that can't be used with scoped storage. [That path wouldn't be doing anything different from Samsung device to Samsung device so it shouldn't be that.]

I've not seen the other issues you mentioned either on Samsung devices using FileZilla. If you're not using "writing external storage" than that may be a reason. The scoped storage code I added only works when "writing external storage" is in use as that is scoped storage and what is required for Android 11+. The older code has various issues I've fixed on scoped storage only.

If you have some specific error that can be traced back to the new code here then I can attempt to fix it. [Since there's too many possibilities of failure not involving the app, am not able to fix anything based on what you said since I can't reproduce it.]

Edits:

I do see scoped storage use retains access and still uses it at least on Android 13 after its disabled. May or may not be an issue on older devices. Not an issue on Android 13 anyway as it has to be used.

FileZilla can fail to list folders where the file counts are too high verses its time out which can be set in FileZilla settings. LIST is fully working so am unsure what else is the issue there.

Oh, this one doesn't have the notifications fix. That I'm not including with it. So there is that.

The app does have an issue with the old storage perm check code on clean start with Android now denying that causing the app to exit (also app code doing that). I didn't test a clean start on Android 13 until now I guess. That's now fixed and not yet committed. [Oh, that's probably from the target of Android 13 so I shouldn't include it lol.]

Not using or failing to use "write external storage" on clean app install will result in it failing to use starting directory since the app has no perm to use it. So that's correct.

I do see an issue when sending 9k files. Something is happening in there. Am looking into it.

[eakteam]

@Xavron It seems not to be working on Android 13... I just forked from this repo -> https://github.com/Xavron/swiftp/tree/Scoped_storage and tried to test it. Installed on my phone Samsung Note 20 but it fails to select starting directory if writing external storage is enabled. Also it fails to list folder and files sometimes and shows empty.
Tested with FileZilla

I've been primarily testing on multiple Samsung devices and including Android 12 & 13 [local and sd card] and not sure what's happening there on your device.

While you're using "write external storage", that is fully the OS there. It simply gives access and that access is obtained and the path is then automatically entered into the apps "manage users..." section since that can't be used with scoped storage. [That path wouldn't be doing anything different from Samsung device to Samsung device so it shouldn't be that.]

I've not seen the other issues you mentioned either on Samsung devices using FileZilla. If you're not using "writing external storage" than that may be a reason. The scoped storage code I added only works when "writing external storage" is in use as that is scoped storage and what is required for Android 11+. The older code has various issues I've fixed on scoped storage only.

If you have some specific error that can be traced back to the new code here then I can attempt to fix it. [Since there's too many possibilities of failure not involving the app, am not able to fix anything based on what you said since I can't reproduce it.]

Edits:

I do see scoped storage use retains access and still uses it at least on Android 13 after its disabled. May or may not be an issue on older devices. Not an issue on Android 13 anyway as it has to be used.

FileZilla can fail to list folders where the file counts are too high verses its time out which can be set in FileZilla settings. LIST is fully working so am unsure what else is the issue there.

Oh, this one doesn't have the notifications fix. That I'm not including with it. So there is that.

The app does have an issue with the old storage perm check code on clean start with Android now denying that causing the app to exit (also app code doing that). I didn't test a clean start on Android 13 until now I guess. That's now fixed and not yet committed. [Oh, that's probably from the target of Android 13 so I shouldn't include it lol.]

Not using or failing to use "write external storage" on clean app install will result in it failing to use starting directory since the app has no perm to use it. So that's correct.

I do see an issue when sending 9k files. Something is happening in there. Am looking into it.

Thank you so much for your detailed explanation.
Just in case i am using targetSdkVersion 33
Regards and thanks for your great work!

[Xavron]

Thank you so much for your detailed explanation.
Just in case i am using targetSdkVersion 33
Regards and thanks for your great work!

You welcome.

Same here. Had it set to target 33 for a month or two now :)

Its good timing though since I just finished fixing and testing some things today. Might as well fix another and another lol.

But, still churning through the files trying to locate the cause of the last issue found. Looks like its going to take some time since its not happening often. 58 [known] times out of 8k.

Edit - Found the issue. I'm not expecting any further ones in there but won't know for sure just yet. Will do extensive testing.

But, apparently, my scoped storage code path traversal for getting to the proper URI location faster than Google's turtle one, kept going on the same dir once a match was found... but if that dir still contained a dir in the next sub then it would enter in there (in the wrong one) instead lol. D'oh!

Guess you had a lot of those. I had none during testing of it.

Should be fixed now. Have to fully test it, and if good, then probably within 24 hours sometime will commit it.

[Xavron]

Commit 7d3424a

Fixed:

o Files were misplaced or failed to transfer under a specific condition that was missed by previous testing

Tests done [all using Android 13, sd card, & target of 33]:

o FileZilla transfer up of 9,828 files with 3,432 folders [SUCCESS]

o FileZilla transfer down of 9,828 files with 3,432 folders. All there and no failures [SUCCESS]

o All files correct and in the right place? Files transferred were source code files of a project. Downloaded project opened and build successfully [SUCCESS]

o Two Android backups using two different client applications and a small restore test in the one [SUCCESS]

o FileZilla transfer up of swiftp source [SUCCESS]

o FileZilla transfer down of swiftp source [SUCCESS]

o Remove project swiftp and move over swiftp transferred from FileZilla. Open, clean, build, and install, and run app [SUCCESS]

o FileZilla full deletion of the swiftp project files that were transferred up [SUCCESS]

[Xavron]

Commit ffbab8e

Override wasn't being fully applied resulting in path related issues. Android 11+ were not affected and work fine on first use.

Now Fixed:

Getting a weird issue. Tested on Android 8.0 but it might affect others. Need to look into it further.

First use of write external storage (until app is force ended) results in:

IOException: Permission denied

IllegalArgumentException: Failed to determine if 6419-B678:backupTest/Sub/OnebackupTest/Sub/One is child of 6419-B678:backupTest/Sub/One

Might be a path related issue or regression since /Sub/OnebackupTest should be not that lol.

Affecting:

Only older Android versions that were bumped onto scoped storage due to failure to otherwise read and write.

[Xavron]

NOT AN ISSUE. I probably didn't wait for it to all download as a second attempt was fine. HOWEVER. certain files do see as eg 240 bytes but downloaded are 246 bytes. Although they function fine, I don't like that and will look into it.

Possible issue (needing to be looked into):

The upload and download of swiftp application source code using FileZilla test that I did above resulted in project having no knowledge of code changes, git log, etc.

May be a specific issue with file bytes or very specific files in there somewhere.

Further testing done:

o Test of just the .git folder [SUCCESS] other than the fact that the hidden attribute isn't being added or is being lost.

[Xavron]

Issue found:

Android 13 target of 13 (not necessarily being from the target; not tested) FileZilla transfer down slows down to an incredible slow crawl with the screen off (even with swiftp CPU keep awake setting enabled and OS unrestricted battery enabled for the app).

[eakteam]

Issue found:

Android 13 target of 13 (not necessarily being from the target; not tested) FileZilla transfer down slows down to an incredible slow crawl with the screen off (even with swiftp CPU keep awake setting enabled and OS unrestricted battery enabled for the app).

Maybe wake locks can do the job?

val wakeLock: PowerManager.WakeLock =
        (getSystemService(Context.POWER_SERVICE) as PowerManager).run {
            newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "MyApp::MyWakelockTag").apply {
                acquire()
            }
        }

Or

PowerManager powerManager = (PowerManager) getSystemService(POWER_SERVICE);
WakeLock wakeLock = powerManager.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "MyApp::MyWakelockTag");
wakeLock.acquire();

[Xavron]

Maybe wake locks can do the job?

I thought pre-Android 13 that unrestricted battery solved that so not really sure what's going on.

However, the app already has the CPU wake lock and its still slow:

swiftp/app/src/main/java/be/ppareit/swiftp/FsService.java

Line 321 in 6bb7a94

private void takeWakeLock() {

[eakteam]

Maybe wake locks can do the job?

I thought pre-Android 13 that unrestricted battery solved that so not really sure what's going on.

However, the app already has the CPU wake lock and its still slow:

swiftp/app/src/main/java/be/ppareit/swiftp/FsService.java

Line 321 in 6bb7a94

private void takeWakeLock() {

hmmmm, unrestricted battery or bettery optimization??

<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>

fun isIgnoringBatteryOptimizations(context: Context): Boolean {
        val pwrm = context.applicationContext.getSystemService(Context.POWER_SERVICE) as PowerManager
        val name = context.applicationContext.packageName
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            return pwrm.isIgnoringBatteryOptimizations(name)
        }
        return true
    }

fun checkBattery() {
        if (!isIgnoringBatteryOptimizations() && Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            val name = resources.getString(R.string.app_name)
            Toast.makeText(applicationContext, "Battery optimization -> All apps -> $name -> Don't optimize", Toast.LENGTH_LONG).show()

            val intent = Intent(Settings.ACTION_IGNORE_BATTERY_OPTIMIZATION_SETTINGS)
            startActivity(intent)
        }
    }

[Xavron]

[SOLVED]

Maybe wake locks can do the job?

Looks like it no longer matters other than maybe to force not allow it lol.

Enabled is full wake lock and slow. Disabled is partial and fast. {SOLVED}

You were right on the pulse of it. Just the setting throws a mind bender :)

People are going to get confused on that one if the setting remains as it is.

[eakteam]

[SOLVED]

Maybe wake locks can do the job?

Looks like it no longer matters other than maybe to force not allow it lol.

Enabled is full wake lock and slow. Disabled is partial and fast. {SOLVED}

You were right on the pulse of it. Just the setting throws a mind bender :)

People are going to get confused on that one if the setting remains as it is.

I'm still unable to list directory /storage/emulated/0 or /storage/emulated/0/DCIM/ (Even after enabling Write External Storage)

Will attach screen recording below within 10 minutes :)

[Xavron]

I'm still unable to list directory /storage/emulated/0

At least this one Android no longer allows. Android's location picker [used with] write external storage will not allow that location directly. [That one is a lame part of Google's recent changes. I had to pull an app from that.]

[There's nothing to fix there unless one creates a new branch not to be used with the Play Store and enables All File Access.]

or /storage/emulated/0/DCIM/ (Even after enabling Write External Storage)

This one works for me immediately on Samsung Android 13 in FileZilla when it has been selected during write external storage location chooser picker. So I can't reproduce it.

[Xavron]

[SOLVED] Bytes sometimes not the same on FileZilla file transfer down.

FileZilla uses ASCII which adds some bytes where FileZilla settings > Transfer > FTP: File Types Binary choice transfer equals the same amount.

So nothing to fix there.

[eakteam]

I'm still unable to list directory /storage/emulated/0

At least this one Android no longer allows. Android's location picker [used with] write external storage will not allow that location directly. [That one is a lame part of Google's recent changes. I had to pull an app from that.]

or /storage/emulated/0/DCIM/ (Even after enabling Write External Storage)

This one works for me immediately on Samsung Android 13 in FileZilla when it has been selected during write external storage location chooser picker. So I can't reproduce it.

But it works very fine through this app -> https://github.com/wolpi/prim-ftpd on /storage/emulated/0 so i think something is missing?

[eakteam]

Also when adding new FTP user we are unable to change starting directory for the new user. Different users may use different starting location ?

[Xavron]

But it works very fine through this app -> https://github.com/wolpi/prim-ftpd on /storage/emulated/0 so i think something is missing?

https://github.com/wolpi/prim-ftpd/blob/d0ac8a2d3a4f398df730d9f25ce9d84fad6a3386/primitiveFTPd/src/org/primftpd/PrimitiveFtpdActivity.java#L561

Normally, ones not on Play Store such as this one there, use all files access. That's the only way to get that exact location used now.

I think dropping the target down low also may. But, I can't remember that one since I don't do things that way and the Play Store doesn't allow that anyway really.

FYI, its no longer scoped storage when that is used. It has access to everything. That's not what scoped storage is and is more of a security risk.

[Xavron]

Also when adding new FTP user we are unable to change starting directory for the new user. Different users may use different starting location ?

This one is that you pick the target location using write external storage. Then in each client app you set the path you want.

Eg:

write external storage set to "/storage/emulated/0/ftpFiles/"

Client A: uses path "/a"

Client B: uses path "/b"

or it could be:

write external storage set to "/storage/emulated/0/ftpFiles/"

Client A: uses path ""

Client B: uses path ""

and they both use the same location in that case.

FileZilla has that under manage sites > site > advanced and there you can work with the paths.

[eakteam]

After.Enabling.External.Storage.mp4

This video shows that even when enabling Write External Storage i'm unable to list folders/files inside DCIM

[Xavron]

This video shows that even when enabling Write External Storage i'm unable to list folders/files inside DCIM

Unfortunately, I can't reproduce it.

Scoped storage set to "/storage/emulated/0/DCIM/" gives that path back with permission. Its the exact path on all Samsung devices and works on my test Android 13 device immediately.

I can't even think of any reason why it wouldn't produce files there :'(

It could be possible that your FileZilla is dirty. Changing write external storage can cause FileZilla to mess up the paths. I have that noted way above. When you change write external storage you need to exit FileZilla. If it was a saved site then you'd also have to make sure the paths in there were correct too but it looks like you're using quick connect so only the former could be an issue. Although testing this now again and am not seeing an issue that way.

[eakteam]

This video shows that even when enabling Write External Storage i'm unable to list folders/files inside DCIM

Unfortunately, I can't reproduce it.

Scoped storage set to "/storage/emulated/0/DCIM/" gives that path back with permission. Its the exact path on all Samsung devices and works on my test Android 13 device immediately.

I can't even think of any reason why it wouldn't produce files there :'(

It could be possible that your FileZilla is dirty. Changing write external storage can cause FileZilla to mess up the paths. I have that noted way above. When you change write external storage you need to exit FileZilla. If it was a saved site then you'd also have to make sure the paths in there were correct too but it looks like you're using quick connect so only the former could be an issue. Although testing this now again and am not seeing an issue that way.

I'm able to reproduce the issue. So:

1. Enable Write External Storage to any folder and start FTP Server.
2. Connect via FileZilla (Directory listing successful/file listing successful).
3. Grab anything from the PC (Video, Picture etc. and upload it to the opened folder.
4. File upload successfully and also exist on the phone storage via its file manager BUT

After that FileZilla is unable to list folder and files again to the same folder where we upload the file from PC.
Is weird, but that's the steps to reproduce it :)

[eakteam]

Connecting to 192.168.192.209:2121...
Status: Connection established, waiting for welcome message...
Status: Insecure server, it does not support FTP over TLS.
Status: Logged in
Retrieving directory listing...
Status: Directory listing of "/storage/emulated/0/Kaptioned" successful ----------> Lists everything OK at this point
Status: Starting upload of C:\Users\EAKTEAM\Videos\After Enabling External Storage.mp4
Status: File transfer successful, transferred 2,938,558 bytes in 2 seconds
Status: Retrieving directory listing of "/storage/emulated/0/Kaptioned"...
Status: Directory listing of "/storage/emulated/0/Kaptioned" successful ----------> Doesn't work at this point

Also it doesn't work anymore even if we delete the previously uploaded file to that directory from phone file manager and that directory becomes unavailable forever.

[Xavron]

This video shows that even when enabling Write External Storage i'm unable to list folders/files inside DCIM

Unfortunately, I can't reproduce it.
Scoped storage set to "/storage/emulated/0/DCIM/" gives that path back with permission. Its the exact path on all Samsung devices and works on my test Android 13 device immediately.
I can't even think of any reason why it wouldn't produce files there :'(
It could be possible that your FileZilla is dirty. Changing write external storage can cause FileZilla to mess up the paths. I have that noted way above. When you change write external storage you need to exit FileZilla. If it was a saved site then you'd also have to make sure the paths in there were correct too but it looks like you're using quick connect so only the former could be an issue. Although testing this now again and am not seeing an issue that way.

I'm able to reproduce the issue. So:

1. Enable Write External Storage to any folder and start FTP Server.

2. Connect via FileZilla (Directory listing successful/file listing successful).

3. Grab anything from the PC (Video, Picture etc. and upload it to the opened folder.

4. File upload successfully and also exist on the phone storage via its file manager `BUT`

After that FileZilla is unable to list folder and files again to the same folder where we upload the file from PC. Is weird, but that's the steps to reproduce it :)

Thank you. But sadly, its not enough. I see the paths duplicating in your video but have no idea why.

It could be new a variation needing to be dealt with in inputPathToChrootedFile() but there's not enough to go on unless I just randomly upload test code and I can't do that here lol.

I will try yet another Samsung. If it still doesn't do it there then I got nothing. This one will be a Note 8.

[eakteam]

This video shows that even when enabling Write External Storage i'm unable to list folders/files inside DCIM

Unfortunately, I can't reproduce it.
Scoped storage set to "/storage/emulated/0/DCIM/" gives that path back with permission. Its the exact path on all Samsung devices and works on my test Android 13 device immediately.
I can't even think of any reason why it wouldn't produce files there :'(
It could be possible that your FileZilla is dirty. Changing write external storage can cause FileZilla to mess up the paths. I have that noted way above. When you change write external storage you need to exit FileZilla. If it was a saved site then you'd also have to make sure the paths in there were correct too but it looks like you're using quick connect so only the former could be an issue. Although testing this now again and am not seeing an issue that way.

I'm able to reproduce the issue. So:

1. Enable Write External Storage to any folder and start FTP Server.

2. Connect via FileZilla (Directory listing successful/file listing successful).

3. Grab anything from the PC (Video, Picture etc. and upload it to the opened folder.

4. File upload successfully and also exist on the phone storage via its file manager `BUT`

After that FileZilla is unable to list folder and files again to the same folder where we upload the file from PC. Is weird, but that's the steps to reproduce it :)

Thank you. But sadly, its not enough. I see the paths duplicating in your video but have no idea why.

It could be new a variation needing to be dealt with in inputPathToChrootedFile() but there's not enough to go on unless I just randomly upload test code and I can't do that here lol.

I will try yet another Samsung. If it still doesn't do it there then I got nothing. This one will be a Note 8.

Something goes wrong after we upload the file to the phone and it make the directory unaccessible forever.

Connecting to 192.168.192.209:2121...
Status: Connection established, waiting for welcome message...
Status: Insecure server, it does not support FTP over TLS.
Status: Logged in
Retrieving directory listing...
Status: Directory listing of "/storage/emulated/0/Kaptioned" successful ----------> `Lists everything OK at this point`
Status: Starting upload of C:\Users\EAKTEAM\Videos\After Enabling External Storage.mp4
Status: File transfer successful, transferred 2,938,558 bytes in 2 seconds
Status: Retrieving directory listing of "/storage/emulated/0/Kaptioned"...
Status: Directory listing of "/storage/emulated/0/Kaptioned" successful ----------> `Doesn't work anymore at this point`

[Xavron]

Alright. You have an MSLD error in the video. That does show up just before a listing of a directory. So does files to list, get working directory variable, violation of chroot, and a while lot more that could cause that failure there.

There's just not enough to go on and the note 8 is also fine. I have to be able to run it through the debugger.

Or you need to provide what's happening in logcat.

I did note the issue I've seen there previously in December though: which is the MLSD error string which is I did see it when write external storage wasn't being used.

[eakteam]

2023-03-27 23:58:40.568 32468-32468 FsService               com.example       D  Server is alive
2023-03-27 23:58:40.570 32468-32468 FsSettings              com.example       V  Using port: 2121
2023-03-27 23:58:40.570 32468-32468 NsdService              com.example       D  onCreate: Entered
2023-03-27 23:58:40.570 32468-32468 FsSettings              com.example       V  Using port: 2121
2023-03-27 23:58:40.571 32468-5088  NsdService              com.example       D  onCreate: Trying to get the NsdManager
2023-03-27 23:58:40.571 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.572 32468-32468 AccessibilityManager    com.example       D  accessibility service is not enabled
2023-03-27 23:58:40.572 32468-32468 AccessibilityManager    com.example       D  accessibility service is not enabled
2023-03-27 23:58:40.572 32468-32468 AccessibilityManager    com.example       D  accessibility service is not enabled
2023-03-27 23:58:40.573 32468-5088  NsdService              com.example       D  onCreate: Got the NsdManager
2023-03-27 23:58:40.577 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.577 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.577 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.578 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.578 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:40.578 32468-32468 NsdService              com.example       D  onStartCommand: Entered
2023-03-27 23:58:42.121 32468-5089  NsdService              com.example       D  onServiceRegistered: SM-N981N SwiFTP Server
2023-03-27 23:58:43.558 32468-5058  FA                      com.example       V  Inactivity, disconnecting from the service
2023-03-27 23:58:50.469 32468-32468 MessengerIpcClient      com.example       W  Received response for unknown request: 6
2023-03-27 23:58:52.841 32468-32482 InputTransport          com.example       D  Input channel destroyed: 'ClientS', fd=186
2023-03-27 23:58:52.841 32468-32482 JavaBinder              com.example       W  BinderProxy is being destroyed but the application did not call unlinkToDeath to unlink all of its death recipients beforehand.  Releasing leaked death recipient: com.google.android.play.core.splitinstall.internal.zzx
2023-03-27 23:58:52.841 32468-32482 BpBinder                com.example       I  onLastStrongRef automatically unlinking death recipients: <uncached descriptor>
2023-03-27 23:58:52.842 32468-32482 InputTransport          com.example       D  Input channel destroyed: 'ClientS', fd=169
2023-03-27 23:58:52.843 32468-32482 InputTransport          com.example       D  Input channel destroyed: 'ClientS', fd=166
2023-03-27 23:58:52.843 32468-32546 OpenGLRenderer          com.example       D  setSurface called with nullptr
2023-03-27 23:59:01.130 32468-5087  TrafficStats            com.example       D  tagSocket(157) with statsTag=0xffffffff, statsUid=-1
2023-03-27 23:59:01.131 32468-5087  TcpListener             com.example       I  New connection, spawned thread
2023-03-27 23:59:01.131 32468-5087  LocalDataSocket         com.example       D  State cleared
2023-03-27 23:59:01.139 32468-5087  FsService               com.example       D  Registered session thread
2023-03-27 23:59:01.140 32468-5183  SessionThread           com.example       I  SessionThread started
2023-03-27 23:59:01.147 32468-5183  FtpCmd                  com.example       D  Ignoring unrecognized FTP verb: AUTH
2023-03-27 23:59:01.152 32468-5183  FtpCmd                  com.example       D  Ignoring unrecognized FTP verb: AUTH
2023-03-27 23:59:01.164 32468-5183  CmdUSER                 com.example       D  USER executing
2023-03-27 23:59:01.165 32468-5183  FtpCmd                  com.example       D  Parsed argument: ftp
2023-03-27 23:59:01.172 32468-5183  CmdPASS                 com.example       D  Executing PASS
2023-03-27 23:59:01.173 32468-5183  CmdPASS                 com.example       I  User ftp password verified
2023-03-27 23:59:01.174 32468-5183  SessionThread           com.example       I  Authentication complete
2023-03-27 23:59:01.179 32468-5183  FtpCmd                  com.example       D  Parsed argument: UTF8 ON
2023-03-27 23:59:01.179 32468-5183  CmdOPTS                 com.example       D  Got OPTS UTF8 ON
2023-03-27 23:59:01.179 32468-5183  CmdOPTS                 com.example       D  Handled OPTS ok
2023-03-27 23:59:01.198 32468-5183  CmdPWD                  com.example       D  PWD executing
2023-03-27 23:59:01.370 32468-5183  CmdPWD                  com.example       D  PWD complete
2023-03-27 23:59:01.376 32468-5183  CmdTYPE                 com.example       D  TYPE executing
2023-03-27 23:59:01.376 32468-5183  FtpCmd                  com.example       D  Parsed argument: I
2023-03-27 23:59:01.377 32468-5183  CmdTYPE                 com.example       D  TYPE complete
2023-03-27 23:59:01.391 32468-5183  CmdPASV                 com.example       D  PASV running
2023-03-27 23:59:01.391 32468-5183  LocalDataSocket         com.example       D  State cleared
2023-03-27 23:59:01.391 32468-5183  TrafficStats            com.example       D  tagSocket(161) with statsTag=0xffffffff, statsUid=-1
2023-03-27 23:59:01.392 32468-5183  LocalDataSocket         com.example       D  Data socket pasv() listen successful
2023-03-27 23:59:01.392 32468-5183  CmdPASV                 com.example       D  PASV sending IP: 192.168.192.209
2023-03-27 23:59:01.392 32468-5183  CmdPASV                 com.example       D  PASV completed, sent: 227 Entering Passive Mode (192,168,192,209,174,185).
2023-03-27 23:59:01.404 32468-5183  CmdMLSD                 com.example       D  MLSD parameter: 
2023-03-27 23:59:01.518 32468-5183  CmdAbstractListing      com.example       D  Listing directory: com.example.utils.FileUtil$Gen@7223c0a
2023-03-27 23:59:01.534 32468-5183  TrafficStats            com.example       D  tagSocket(166) with statsTag=0xffffffff, statsUid=-1
2023-03-27 23:59:01.534 32468-5183  LocalDataSocket         com.example       D  onTransfer pasv accept successful
2023-03-27 23:59:01.536 32468-5183  LocalDataSocket         com.example       D  State cleared
2023-03-27 23:59:01.536 32468-5183  CmdAbstractListing      com.example       D  LIST/NLST done making socket
2023-03-27 23:59:01.537 32468-5183  CmdAbstractListing      com.example       D  Sent code 150, sending listing string now
2023-03-27 23:59:01.537 32468-5183  SessionThread           com.example       D  Using data connection encoding: UTF-8
2023-03-27 23:59:01.537 32468-5183  SessionThread           com.example       D  Closing data socket
2023-03-27 23:59:01.538 32468-5183  CmdAbstractListing      com.example       D  Listing sendViaDataSocket success
2023-03-27 23:59:01.538 32468-5183  CmdMLSD                 com.example       D  MLSD completed OK
2023-03-27 23:59:13.802 32468-4326  TrafficStats            com.example       D  tagSocket(138) with statsTag=0xffffffff, statsUid=-1

[Xavron]

I was expecting that to help but its only showing MLSD is fine there :\

Until I can reproduce it, I can't do anything to fix it. Sorry :(

Can't do certain other things here because of the branch being a pull request.

[I will keep thinking about it and trying to find something. If there's anything to find.]

[eakteam]

This is another view on how i can reproduce the bug:

1. Copied the project from -> https://github.com/Xavron/swiftp/tree/forBackups
2. Build the app and fresh install on my phone
3. Set write external storage to FTP FILES folder
4. Start FTP Server
5. Open FileZilla and please watch carefully the below attached video on how the bug is happening

Any idea how can i debug it to find the root cause of why is this happening?

BUG.mp4

[Xavron]

Any idea how can i debug it to find the root cause of why is this happening?

Thanks :) But, I just reproduced it as well. Had to push a file to DCIM folder to do it. Its not done it with any other folder [I've used and did a bunch of testing on internal in Dec] so I'm wondering if its the app or Android thing.

Have to debug it... will get back once I do that and determine what it actually is.

Edit: It won't take long lol. Already got it working. Indeed, though the Google way didn't work for some reason. Since that was something that should not be called anyway here, and yet it was, it caused an issue. Will commit it in some minutes (in a bit).

Thanks for your patience there and help :)

[Xavron]

Commit f3bb897

Fixed:

Some dirs getting empty file listings especially after transferring a file

Tests:

Internal DCIM folder and pushing files to it [SUCCESS]

Tested DCIM target location then moving into /Camera and transferring a file [SUCCESS]

Tested DCIM target location then making sub dirs /1/2/3 and transferring a file to /3 {SUCCESS]

Tested target location of /DCIM/Test/Test/Test/ and pushed a file to it [SUCCESS]

Tested for conflicts briefly with sd card and two client applications [SUCCESS]

_Not doing extensive testing on this one._

[Xavron]

Guess I should have tested further :\

Test:

Target location: /storage/emulated/0/Test/

Transfer swiftp source project.

1 transfer failure [FAILED]

1 wrong sub path created [FAILED]

FileZilla didn't refresh the folder at the end [FAILED]

Will look at that later. Can't do it right now.

[eakteam]

Another Bug(maybe).

Everything seems to works fine except that after uploading something to a directory it also creates a folder named storage after uploading is successful. Inside that directory is another one emulated and another one inside 0.

[Xavron]

Another Bug(maybe).

Everything seems to works fine except that after uploading something to a directory it also creates a folder named storage after uploading is successful. Inside that directory is another one emulated and another one inside 0.

Yeah, its all path stuff. Technically, its all fixed now (without conflict or further extensive testing anyway) but won't be committing it for some time because I no longer like the code and need to test a lot. That will take a while to properly do.

Update as an edit. Going great. Done in a short bit? I think so =)

[eakteam]

Another Bug(maybe).
Everything seems to works fine except that after uploading something to a directory it also creates a folder named storage after uploading is successful. Inside that directory is another one emulated and another one inside 0.

Yeah, its all path stuff. Technically, its all fixed now (without conflict or further extensive testing anyway) but won't be committing it for some time because I no longer like the code and need to test a lot. That will take a while to properly do.

Yeah, good point 😜

[Xavron]

commit 47efb9c

Simplified path code with return to some older code that's now working fine with it after things have been fixed in past commits and a couple more in this one. Didn't see that one coming.

Known issues not fixed:

  1. Ftp client already connected and then switching write external storage location causes chroot to not get updated.

      To reproduce:

        o Create and enter a dir, use write external storage and change locations eg sd card to internal, refresh, create dir and old chroot is still in use.

      Notes:

        o What happens on multiple clients connected? Disconnect them all?

        o Hot swap chroot location on all or temporarily keep the last UriPermission around until all clients finish?

  2. Seen the app hard crashing during debugging when FileZilla connects.

      o Not the app. Debugger issue. One workaround is by connecting without debugger attached then attach after.

Android 13:

  o Backups continual sd card two backups using two different clients [x]
  o Backups full restore over 25GB total sd card two backups using two different clients [x]

  o Multiple clients connected and transferring at the same time [x]

  o FileZilla:

    transfer tests up & down:

      swiftp source code transfer up and down plus clean and build:
        sd card /dir/1/ [x]
        local /DCIM/ [x] 
        local /Test/ [x]

      local /DCIM/ one file at chroot [x]
      local /Test/ one file at chroot [x]

    sd card:
      transfer up recursive [x]
      transfer down recursive [x]
      transfer up one  file [x]
      transfer down one  file [x]
      delete recursive [x]
      delete one [x]
      movement one dir up [x]
      movement one dir down [x]
      movement skip multi dirs [x]
      new dir [x]
      dir refresh [x]
      dir rename [x]
      9.8k files + 3.4k dirs u [x]
      9.8k files + 3.4k dirs d [x]

    local:
      transfer up recursive [x]
      transfer down recursive [x]
      transfer up one file [x]
      transfer down one file [x]
      delete recursive [x]
      delete one [x]
      movement one dir up [x]
      movement one dir down [x]
      movement skip multi dirs [x]
      new dir [x]
      dir refresh [x]
      9.8k files + 3.4k dirs u [x]
      9.8k files + 3.4k dirs d [x]

  ________________
  ALL SUCCESSFUL WITH THIS COMMIT - Files transferred over 47k times successfully in total on local and sd card.

[ppareit]

Lot of internal changes! Scoped storage will be the way forward.

[ppareit]

Also made some little changes to setup to make Android 13 work out of the box. See PreferenceFragment.onCreate.

[Xavron]

I'll be continuing to monitor for any unforeseen issues and fix them as I can.