Better handling non-XAS datatypes

[maurov]

@newville I have just realized that is not currently possible in xas_viewer to load a k, chi file and perform EXAFS analysis on it. I think this is a strong limitation and we should take action to solve it. Do you agree?

We could take this occasion to extend and better handle non-XAS datatypes in xas_viewer and larch in general. Here some ideas (random order). Please, feel free to comment them:

• definition of the DATATYPES should be moved out of the GUI (larch.wxlib.columnframe) and go into root larch. By the way, I think this should be valid for all globals that are not strictly related to GUI.
• add a chi datatype (or whatever naming) that when loaded will not allow to perform normalization and background removal. Ideally, the GUI should get all tabs disabled (visually grayed out, I do not know if this is possible with Wx) and leave only the EXAFS-related tabs
• add xas_norm datatype for already normalized XAS data -> the normalization panel should be disabled
• the raw datatype should not bring to the "XAS normalization" tab, but something more generic like a shell (I have in mind what is done in NexPy, all tabs should be disabled, apart peak fitting.
• one should be able to change the group datatype in the list of groups panel by right click.
• one could mix datatypes in the group list panel and each datatype get a different color for the group name.

So, the main question is: do we want to extend to other spectroscopy workflows rather than XAS? I think at least the Larch library should be able to know how to handle additional datatypes. Having the GUI also handling this would be great, but is much more work, I suppose.

[newville]

@maurov

@newville I have just realized that is not currently possible in xas_viewer to load a k, chi file and perform EXAFS analysis on it. I think this is a strong limitation and we should take action to solve it. Do you agree?

Well, I guess we haven't had a lot of complaints about it ;)

In fact, we don't even read chi(k) data from Athena Projects -- they aren't even shown as available when reading an Athena File.

I might say: "not no", but it might be some real work, and it might not be my highest priority. I guess I might ask for suggestions on how to set such priorities?

We could take this occasion to extend and better handle non-XAS datatypes in xas_viewer and larch in general.

OK. Maybe we should also aim to have this be a wider conversation? It's summer, and I have not tried to schedule a Larch Develepors meeting yet. Maybe this should be discussed there?

Here some ideas (random order). Please, feel free to comment them:
. definition of the DATATYPES should be moved out of the GUI (larch.wxlib.columnframe) and go into root larch. By the way, I think this should be valid for all globals that are not strictly related to GUI.

OK. Well, some things that are "globals" in a module are pretty specific to it. It is possible that "wxlib/columnframe.py" should be moved into "wxxas" -- it is pretty specific to XAS Viewer at this point.

. add a chi datatype (or whatever naming) that when loaded will not allow to perform normalization and background removal. Ideally, the GUI should get all tabs disabled (visually grayed out, I do not know if this is possible with Wx) and leave only the EXAFS-related tabs

I
think one can grey-out the contents of a Tabbed Panel, if not the Tabs themselves. But, parts of the EXAFS tab (say FT) would probably still want to be allowed?

add xas_norm datatype for already normalized XAS data -> the normalization panel should be disabled

I don't think I've heard that request before. Maybe we could just have a button that sets the edge_step to 1, or effectively
makes norm=mu. That would basically have the same effect.

the raw datatype should not bring to the "XAS normalization" tab, but something more generic like a shell (I have in mind what is done in NexPy, all tabs should be disabled, apart peak fitting.
one should be able to change the group datatype in the list of groups panel by right click.

Another option would be a separate GUI for peak and general data fitting for non-XAS data.

The fact that XAS Viewer can read non-XAS data has surprised several people ;)

one could mix datatypes in the group list panel and each datatype get a different color for the group name.

OK.

So, the main question is: do we want to extend to other spectroscopy workflows rather than XAS? I think at least the Larch library should be able to know how to handle additional datatypes. Having the GUI also handling this would be great, but is much more work, I suppose.

Well, I guess I might ask whether we should deliberately target other spectroscopies or aim for a more general-purpose tool?
Maybe targeting XES, including extraction of data from image data, would be a possible addition?

FWIW, I will also be continuing to work on XRF fitting and XRF mapping software within Larch, but not XAS Viewer. Those GUIs need work too ;).

I think going for a more general-purpose tool for curve fitting/peak fitting is a possibility too. That would basically be a data browser and LMFIT GUI that would be like XAS Viewer but not XAS-specific. So, more like a replacement for fityk than for Athena. But, maybe including the simple approaches to Linear-Combination and PCA in XAS Viewer for general-purpose data.

My sense is that:
a) we could definitely do this.
b) from what I see, this would be very popular.
c) it might be so popular that it swamps XAS/XAS/synchrotron work.

This is better than trying to push "general purpose data" into XAS Viewer. I think the "build the actual GUI" is not very hard -- much of it would be XAS Viewer. I might consider a separate project from all of Larch, but I think that would be "undecided" at the moment. There are pros and cons...
That tool would almost certainly need:
a general-purpose browser of HDF5
a general-purpose browser for CSV / Pandas data.
I would imagine that there would be further pressure to read other data sources, which could dominate the support time.

Do you have any thoughts on this? Like, imagine a variation of XAS Viewer that was really a replacement for Fityk -- would that be the right thing to do?

That is all sort of a non-linear set of thoughts. Maybe a wider discussion is needed?

[maurov]

@newville

Thanks for your comments. I agree that the overall topic should be discussed in a developers' meeting, probably after summer.

In the meantime, few things that could be very helpful and should not affect that much the current code:

• do you agree if we move out of the GUI the tuple DATATYPES? If yes, do you have a preference where? I was thinking in larch.symboltable
• can we add a chi datatype in order that the ASCII reader will correctly assign arrays to group.k and group.chi

If you agree, I can take care doing this.

Furthermore, the GUI should not reprocess the data each time passing from one group highlight to another, neither when simply doing a plot on current group or selected groups. Processing functions (e.g. pre_edge, autobk, etc.) on a group should be (re)called only when parameters are changed in the GUI. Do you agree with this?

PS:

I think one can grey-out the contents of a Tabbed Panel, if not the Tabs themselves.

If you know how to do this in wxPython, I would be interested in giving it a try.

[newville]

@maurov

do you agree if we move out of the GUI the tuple DATATYPES? If yes, do you have a preference where? I was thinking in larch.symboltable

At the moment, it is only used by the column file reader and XAS Viewer. So maybe it should be in wxxas/config.py.

can we add a chi datatype in order that the ASCII reader will correctly assign arrays to group.k and group.chi
If you agree, I can take care doing this.

I think so. I'm not entirely clear about is how to "install that group" into XAS Viewer. XAS Viewer basically started as "XANES/XAS data viewer", so several of the panels, dialogs, and to some extent the Controller sort of assume there are "energy" and "norm" arrays, typically also called "xdat" and "ydat".

For example, merging, LCF, and PCA only work on energy-arrayed data. If we support reading in chi(k) data from elsewhere, we will have to be explicit about whether these can work with chi(k) data or not.

So, I'm not opposed, but I think this could actually be some effort.

Are people asking for this? I have not heard of such a request.

PS:

I think one can grey-out the contents of a Tabbed Panel, if not the Tabs themselves.
If you know how to do this in wxPython, I would be interested in giving it a try.

I think you can use Widget.Disable() or Widget.Enable(boolean_value) . I don't know if that works on the whole panel, but it might. Each TaskPanel has an onPanelExposed method that is run when that tab panel is brought forward. That currently gets the "current group" and fills in the form. You could try checking the "datatype" and Enable the panel if the datatype is supported for that panel.

FWIW, I spent some time yesterday trying to figure out per-item coloring or font properties for the File List, and it does not look possible.

[gdoumy]

Hello,

I have a related request after trying to use XAS viewer to perform pre-edge peak analysis. I have a series of spectra that I import and would want to do the pre-edge fitting on that 'raw' data, but the only options are normalized, flattened and deconvolved.
Because of the nature of the data, the pre-edge is a bit noisy, and the normalization treatment does not work very well (and more importantly not consistently between spectra).
If the raw option (or way to normalized by doing nothing to the data) could be added that would be great!

Thanks,
-Gilles