Skip to content

Commit 3803d84

Browse files
smlptRuoshanLanskalarproduktraum
authored
Add eye tracking support to sciview. (#523)
Thanks to @smlpt and @RuoshanLan. This contains: * utilize eyetracking in sciview * a working version and adjust the size of some messages * code clean * let the volume show before starting the eye tracking * update bionic tracking * add headset for cell tracking * add test without VR headset * a working version of utilizing VRHeadset for cell tracking - part of sandbox function * add function of tracking with controllers * replace numerictype with realtype * fix issue in controller version * improve the VRHeadsSettTrackingDemo and fix a bug caused by UV coordinate * add VR interaction to VRHeadSetTrackingDemo and add nodetaggedevent * code clean * Fix line endings * Gradle: Add missing transitive dependencies * SciView: Revert failed merge for addVolume functions * clean and add comment * Gradle Build: update to scenery 0.9.0 * Gradle Build: fixing outdated code * SciView: bring back openDirTiff and openTrackFile * Gradle: fix directories dep issue * build: update gradle and make the build compatible with java 21 * improve volume loading from scene * fix filtering for volumes in the scene * SciView: fix VR toggling changes to stereo rendering * EyeTrackingDemo: catch if no volume is found * HedgehogAnalysis: add docs and comments * Add: test object for loading a volume and starting the eye tracking directly (using ugly hardcoded path) * Make newest sciview compatible with local scenery (stupid jackson) * Fix toggleVRRendering * Minor changes to fix finding local maxima, adding info logs * Increase volume size for testing dataset * Clean up logging * Change showMessage to centered, try to fix track instancing * Fix track cylinder radius * EyeTrackingDemo: update sciview API usage * Make eye tracking compatible with mastodon bridge * Change log to logger for consistency * Add: gaussian smoothing function * Change to sampleRayGridTraversal and add samplePos list to SpineMetadata * HedgehogAnalysis: get spine vertex position from grid traversal * Fix: ignore children in AABB intersection * EyeTrackingDemo: fix vertex positions * Create CellTrackingBase, let other classes inherit most of the (otherwise duplicated) methods * Make sure that cam is not inside volume when tracking * Add names to scene objects to make it easier to find and remove them again * CellTrackingBase: moved file to demo/advanced * Refactor the eye tracking logic into EyeTracking, so that EyeTrackingDemo is only a wrapper for command purposes * Add proper eyetracking shutdown handling * Unified naming scheme, extracted tracking functionality into separate classes * Use lazylogger instead of Logservice * StartEyeTrackingDirectly: Rename to EyeTrackingCommand * SciviewBridge: wrap eyeTracking into a separate thread to fix blank screen on toggleVRRendering * Add TimepointObserver to unify timepoint updates across VR and bridge * Fix monoscopic render rebuild after disabling VR * SciView: improve VR shutdown handling in toggleVRRendering * Improve scene cleanup after turning off VR * Cleaned up files for PR, removed comments, added documentation * Rename VR controller bindings using TrackerRole and OpenVRButton * push without content to get CI logs again * build: bump to scenery 0.12.0 * build: add mavenCentral repository * OpenN5: add Datatype.STRING for exhaustive when expression * remove legacy classes, not needed by manvr3d * updated Menuweights to remove legacy eyetracking command * SciView: removed legacy methods for loading tracks from files * added documentation * Move classes to commands.analysis * PR cleanup --------- Co-authored-by: ruoshan <[email protected]> Co-authored-by: Ulrik Günther <[email protected]>
1 parent b08d061 commit 3803d84

File tree

9 files changed

+1472
-21
lines changed

9 files changed

+1472
-21
lines changed
Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
/*-
2+
* #%L
3+
* Scenery-backed 3D visualization package for ImageJ.
4+
* %%
5+
* Copyright (C) 2016 - 2021 SciView developers.
6+
* %%
7+
* Redistribution and use in source and binary forms, with or without
8+
* modification, are permitted provided that the following conditions are met:
9+
*
10+
* 1. Redistributions of source code must retain the above copyright notice,
11+
* this list of conditions and the following disclaimer.
12+
* 2. Redistributions in binary form must reproduce the above copyright notice,
13+
* this list of conditions and the following disclaimer in the documentation
14+
* and/or other materials provided with the distribution.
15+
*
16+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
17+
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
19+
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
20+
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
21+
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
22+
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
23+
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
24+
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
25+
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
26+
* POSSIBILITY OF SUCH DAMAGE.
27+
* #L%
28+
*/
29+
package sc.iview.commands.file;
30+
31+
import org.scijava.command.Command;
32+
import org.scijava.io.IOService;
33+
import org.scijava.log.LogService;
34+
import org.scijava.plugin.Menu;
35+
import org.scijava.plugin.Parameter;
36+
import org.scijava.plugin.Plugin;
37+
import sc.iview.SciView;
38+
39+
import java.io.File;
40+
import java.io.IOException;
41+
42+
import static sc.iview.commands.MenuWeights.FILE;
43+
import static sc.iview.commands.MenuWeights.FILE_OPEN;
44+
45+
/**
46+
* Command to open a file in SciView
47+
*
48+
* @author Kyle Harrington
49+
*
50+
*/
51+
@Plugin(type = Command.class, menuRoot = "SciView", //
52+
menu = { @Menu(label = "File", weight = FILE), //
53+
@Menu(label = "Open Directory of tif...", weight = FILE_OPEN) })
54+
public class OpenDirOfTIFF implements Command {
55+
56+
@Parameter
57+
private IOService io;
58+
59+
@Parameter
60+
private LogService log;
61+
62+
@Parameter
63+
private SciView sciView;
64+
65+
// TODO: Find a more extensible way than hard-coding the extensions.
66+
@Parameter(style = "directory")
67+
private File file;
68+
69+
@Parameter
70+
private int onlyFirst = 0;
71+
72+
@Override
73+
public void run() {
74+
try {
75+
if(onlyFirst > 0) {
76+
sciView.openDirTiff(file.toPath(), onlyFirst);
77+
} else {
78+
sciView.openDirTiff(file.toPath(), null);
79+
}
80+
}
81+
catch (final IOException | IllegalArgumentException exc) {
82+
log.error( exc );
83+
}
84+
}
85+
}

src/main/kotlin/sc/iview/SciView.kt

Lines changed: 64 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -55,9 +55,9 @@ import graphics.scenery.utils.ExtractsNatives.Companion.getPlatform
5555
import graphics.scenery.utils.LogbackUtils
5656
import graphics.scenery.utils.SceneryPanel
5757
import graphics.scenery.utils.Statistics
58-
import graphics.scenery.utils.extensions.times
5958
import graphics.scenery.volumes.Colormap
6059
import graphics.scenery.volumes.RAIVolume
60+
import graphics.scenery.volumes.TransferFunction
6161
import graphics.scenery.volumes.Volume
6262
import graphics.scenery.volumes.Volume.Companion.fromXML
6363
import graphics.scenery.volumes.Volume.Companion.setupId
@@ -85,6 +85,7 @@ import net.imglib2.type.numeric.integer.UnsignedByteType
8585
import net.imglib2.view.Views
8686
import org.joml.Quaternionf
8787
import org.joml.Vector3f
88+
import org.joml.Vector4f
8889
import org.scijava.Context
8990
import org.scijava.`object`.ObjectService
9091
import org.scijava.display.Display
@@ -100,6 +101,7 @@ import org.scijava.thread.ThreadService
100101
import org.scijava.util.ColorRGB
101102
import org.scijava.util.Colors
102103
import org.scijava.util.VersionUtils
104+
import sc.iview.commands.demo.animation.ParticleDemo
103105
import sc.iview.commands.edit.InspectorInteractiveCommand
104106
import sc.iview.event.NodeActivatedEvent
105107
import sc.iview.event.NodeAddedEvent
@@ -110,14 +112,14 @@ import sc.iview.ui.CustomPropertyUI
110112
import sc.iview.ui.MainWindow
111113
import sc.iview.ui.SwingMainWindow
112114
import sc.iview.ui.TaskManager
113-
import ucar.units.ConversionException
114115
import java.awt.event.WindowListener
115116
import java.io.File
116117
import java.io.IOException
117118
import java.net.JarURLConnection
118119
import java.net.URL
119120
import java.nio.ByteBuffer
120121
import java.nio.FloatBuffer
122+
import java.nio.file.Path
121123
import java.time.LocalDate
122124
import java.time.format.DateTimeFormatter
123125
import java.util.*
@@ -134,6 +136,7 @@ import kotlin.concurrent.thread
134136
import javax.swing.JOptionPane
135137
import kotlin.math.cos
136138
import kotlin.math.sin
139+
import kotlin.system.measureTimeMillis
137140

138141
/**
139142
* Main SciView class.
@@ -784,6 +787,25 @@ class SciView : SceneryBase, CalibratedRealInterval<CalibratedAxis> {
784787
}
785788
}
786789

790+
@Throws(IOException::class)
791+
fun openDirTiff(source: Path, onlyFirst: Int? = null)
792+
{
793+
val v = Volume.fromPath(source, hub, onlyFirst)
794+
v.name = "volume"
795+
v.spatial().position = Vector3f(-3.0f, 10.0f, 0.0f)
796+
v.colormap = Colormap.get("jet")
797+
v.spatial().scale = Vector3f(15.0f, 15.0f,45.0f)
798+
v.transferFunction = TransferFunction.ramp(0.05f, 0.8f)
799+
v.metadata["animating"] = true
800+
v.converterSetups.firstOrNull()?.setDisplayRange(0.0, 1500.0)
801+
v.visible = true
802+
803+
v.spatial().wantsComposeModel = true
804+
v.spatial().updateWorld(true)
805+
addNode(v)
806+
}
807+
808+
787809
/**
788810
* Open a file specified by the source path. The file can be anything that SciView knows about: mesh, volume, point cloud
789811
* @param source string of a data source
@@ -1697,28 +1719,41 @@ class SciView : SceneryBase, CalibratedRealInterval<CalibratedAxis> {
16971719
val cam = scene.activeObserver as? DetachedHeadCamera ?: return
16981720
var ti: TrackerInput? = null
16991721
var hmdAdded = false
1700-
if (!hub.has(SceneryElement.HMDInput)) {
1701-
try {
1702-
val hmd = OpenVRHMD(false, true)
1703-
if (hmd.initializedAndWorking()) {
1704-
hub.add(SceneryElement.HMDInput, hmd)
1705-
ti = hmd
1706-
} else {
1707-
logger.warn("Could not initialise VR headset, just activating stereo rendering.")
1722+
1723+
if (vrActive) {
1724+
1725+
// VR activation logic
1726+
if (!hub.has(SceneryElement.HMDInput)) {
1727+
try {
1728+
val hmd = OpenVRHMD(false, true)
1729+
if (hmd.initializedAndWorking()) {
1730+
hub.add(SceneryElement.HMDInput, hmd)
1731+
ti = hmd
1732+
} else {
1733+
logger.warn("Could not initialise VR headset, just activating stereo rendering.")
1734+
}
1735+
} catch (e: Exception) {
1736+
logger.error("Could not add OpenVRHMD: $e")
17081737
}
17091738
hmdAdded = true
1710-
} catch (e: Exception) {
1711-
logger.error("Could not add OpenVRHMD: $e")
1739+
} else {
1740+
ti = hub.getWorkingHMD()
1741+
}
1742+
1743+
// Set tracker on the DetachedHeadCamera
1744+
if (ti != null) {
1745+
cam.tracker = ti
1746+
logger.info("tracker set")
17121747
}
17131748
} else {
1714-
ti = hub.getWorkingHMD()
1715-
}
1716-
if (vrActive && ti != null) {
1717-
cam.tracker = ti
1718-
} else {
1749+
// VR deactivation logic
1750+
// Convert back to normal Camera
1751+
logger.info("Shutting down VR")
17191752
cam.tracker = null
17201753
}
1721-
renderer.pushMode = false
1754+
1755+
// Enable push mode if VR is inactive, and the other way round
1756+
renderer.pushMode = !vrActive
17221757

17231758
// we need to force reloading the renderer as the HMD might require device or instance extensions
17241759
if (renderer is VulkanRenderer && hmdAdded) {
@@ -1732,8 +1767,17 @@ class SciView : SceneryBase, CalibratedRealInterval<CalibratedAxis> {
17321767
e.printStackTrace()
17331768
}
17341769
}
1735-
1736-
renderer.toggleVR()
1770+
}
1771+
logger.debug("Replaced renderer.")
1772+
renderer.toggleVR()
1773+
// Cleanup HMD after VR has been toggled off
1774+
if (!vrActive) {
1775+
if (hub.has(SceneryElement.HMDInput)) {
1776+
val hmd = hub.get(SceneryElement.HMDInput) as? OpenVRHMD
1777+
hmd?.close()
1778+
// TODO hub.remove(hmd)
1779+
logger.debug("Closed HMD.")
1780+
}
17371781
}
17381782
}
17391783

0 commit comments

Comments
 (0)