Prototype generic process functions

[eve-mem]

Hello,

This draft PR is to add some experimental features to provide a generic way of getting basic information from processes regardless of operating system. The idea was discussed in the comments of this issue: #981

The idea being to provide a set of functions that can be used in plugins to make them easier to make (or volshell), and be consistent across the different operating systems so it's easier to switch between them.

It is very basic, just adding the following functions. Mostly just be shifting code out of various pslist plugins to the extensions so they can used from anywhere.

Function	Windows	Linux	Mac
get_pid	🟢	🟢	🟢
get_parent_pid	🟢	🟢	🟢
get_name	🟢	🟢	🟢
get_create_time	⚫	🔴	🟢
get_exit_time	⚫	🔴	🔴

(This is what I mean with the colours if it's not clear - 🟢: Added in this PR, ⚫: Function already existed, 🔴: Not added in this PR)

It was interesting to work on this. I noticed after that the linux module class also inherits from GenericIntelProcess and already has a get_name function.

Also that the windows EPROCESS already had the get_create_time and get_exit_time functions so I used those names in the generic part.

I've then modified the pslist plugins for windows/linux/mac to show how this could be used. If this is useful I'm happy to modify the existing plugins to use this, it shouldn't affect how they work - but might make them easier to read.

I'm not sure if this is the best (or even a good) way of doing this - so I'd be very interested in your thoughts.

Thanks!

[eve-mem]

@ikelos - I'd love your thoughts on this. The idea to have a simple to use way to get key information about a process for plugins. That way there can be a central way to do these lookups within plugins so that they're consistent and if an update needs to be done it only needs to happen in one place. It could be bulked out to include extra bits for a process like getting memory regions etc.

It should hopefully mean that we'd be able to make non-OS specific plugins - assuming it's done in the right way. Given it's above intel process that might be a good place for it given the arm work at the moment?

Thanks again.

[ikelos]

Yep, sorry, it is on my list, just a bit lower down at the moment, sorry! 5:S

[gcmoreira]

Hey @eve-mem ! It's a great idea to ensure better consistency across the framework.

I think this is also related to your other ticket Unify Linux plugins representation of pids which we should take a decision at some point.

[eve-mem]

Thanks both. I completely understand not having time to look at this given how little time i seem to have at the moment.

Thanks for those pointers @gcmoreira - that's almost exactly why I'd pike something like this PR to exist. Just a single place to have the 'right' answer so we can do it once for all plugins. Not necessarily this version in particular, but the idea at least.

[eve-mem]

Hey @ikelos and @gcmoreira,

I was working on the PR for #981 and it just felt to me that doing this first might make it all a bit easier.

Assuming that you both think that this is a good idea of course! What do you think?

I also tried to add an explanation of tgid/pid/tid into the readme docs. Not sure what you both make of that?

@gcmoreira thanks for the comments - I think I've got them all now. I'm a little unsure I got the datetime part correct in the mac side - could you check that a little closer?

Thanks both.
🦊

[gcmoreira]

Hey @eve-mem good job!
Yeah the aware datetime change is correct.

I think it's a solid idea, but we should keep refining the implementation. There are a few areas that could be improved, and some issues that still need fixing.

• GenericProcess:
  • Should include methods to return all TGID and PID (user PID and TID) and their respective parents. Not only the Thread Group Leader, otherwise we would be burying the threads behind that interface.
  • I don't think the method names do much to clear up the confusion between the kernel PID/TGID/TID/etc. In fact, it feels like we're just shifting the problem from one class to another. Even the docstrings are not helping at all to that purpose. We should be more explicit with the method names. We should choose a point of view and stick to it throughout the framework. If we decide to take the user-space's perspective, I think it should be better to use names like get_user_pid(), get_user_tid(), and so on. For instance, see below that with the current implementation (in red) even with the comment clarifying it, it's still confusing. This can be much clear if we use more explicit method names and include the user TID (green changes).

-    user_pid = task.get_pid()
-    user_tid = task.pid  # Note that pid is user_tid in Linux
-    user_ppid = task.get_parent_pid()
+    user_pid = task.get_user_pid()
+    user_tid = task.get_user_tid()
+    user_ppid = task.get_user_ppid() 
# or get_parent_user_pid()

• Filter: I strongly believe it's a mistake to filter by user_pid (TGID). Actually, with the proposed implementation we are changing the way Volatility3 is working now, since we are currently filtering by user_tid (PID). If we change to use user_pid (TGID) that makes impossible to filter by TID. Also, note that this affects not just the pslist plugin --filter argument, which can now be superseded by the more general --filters argument and maybe removed, but with the changes in create_pid_filter() here this is affecting other current and future plugins which may require to get a specific thread and not the thread group leader. With this change we are introducing a bug and no one in the framework will be able to find a specific thread. I explained this with some examples here. I believe it's quite clear, but if it's not, please let me know, and I'll do my best to clarify that explanation.
  • On the contrary, even for users unfamiliar with the meaning of "a pid" in the framework, using user_tid (PID) will yield the expected result, since user_pid == user_tid (or in kernel space task_struct.tgid == task_struct.pid) for a thread group leader (what is typically referred to as 'a process' in user-space).
• We should also extend these changes to the create_pid_filter() method name to stick to the proposed point of view. With the Volatility3's current behavior (filtering by user_tid) it should be renamed to create_user_tid_filter()
• Review all occurrences of the term pid in the framework and adjust accordingly to align with the proposed perspective.
• Lastly, we should avoid using the abbreviation upid for user_pid, as it has a specific meaning in the Linux kernel see here. I'm not saying you are already doing this, I just want to flag this in case we feel tempted to use it. I might be the first to slip up on my own words LOL! That's why peer reviews are crucial.

[eve-mem]

Thanks @gcmoreira - makes sense I'll work on it. Particularly the pid filtering, it makes sense to me now.

I was hoping that the functions would be the same across the different OSes, I'll change the get_pid to get_user_pid for all of them.

[eve-mem]

Hey @gcmoreira - I was updating the linux side of the generic processes as per your suggestion.

e.g.

-    user_pid = task.get_pid()
-    user_tid = task.pid  # Note that pid is user_tid in Linux
-    user_ppid = task.get_parent_pid()
+    user_pid = task.get_user_pid()
+    user_tid = task.get_user_tid()
+    user_ppid = task.get_user_ppid() 
# or get_parent_user_pid()

It means that then the linux side will have slightly different functions to the other OSes - meaning it's less generic again. I'm hoping it means a plugin could run on both linux/windows/macos without any changes.

What do you think about having the linux side having get_user_pid and get_user_tid as you described, but keeping get_pid as a wrapper simple wrapper that then simply calls get_user_pid.

It means there's still only one place for the actual logic for get_user_pid etc - so a single place for the logic if it ever needed to be updated. It let's people use the generic get_pid for generic plugins, and then still allows the use of get_user_pid in the actual linux plugins.

Would love to hear your thoughts on whether this simplifies things or adds unnecessary layers!

[ikelos]

I think an interface should be generic, with the common usage (so whilst there may be non-user pids and tids and ppids, we kind of all have an intuition of what we mean, and the docstring can explain precisely what's returned, and we can overrload OS specific ones with other OS specific functions if we really need (so people don't have to jump into and out of the handlers to do different things, but the only ones they can rely on across multiplatform methods are the ones on the generic interface. This will mean choosing what we expect to be "common" carefully, because changing it later will be nigh on impossible...

[atcuno]

Linux doesn't use a single task struct, threads have their own. The code reflects this but this part of the documentation is off:

"uses a single task structure to represent processes and the threads"

[atcuno]

this should be a renderer unavailable instance as 0 is misleading