Skip to content

Boss_access tool refactor #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
diegoluna3 wants to merge 6 commits into
base: development
Choose a base branch
from
Open

Boss_access tool refactor #45

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d370ce1
Merge pull request #43 from aplbrain/development
erikjohnson24 Aug 27, 2020
5089460
refactor boss_access tools and Dockerfile
diegoluna3 Feb 10, 2021
7fa0a0b
enable user input for resolution
diegoluna3 Feb 11, 2021
45a53db
update boss access tool and documentation
diegoluna3 Feb 16, 2021
4ff9e34
add more user args and remove use of intern array
diegoluna3 Feb 23, 2021
3512190
pin versions for numpy and intern
diegoluna3 Feb 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions saber/boss_access/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,9 @@ FROM python:3.6

# Install any needed packages specified in requirements.txt
RUN pip install numpy
Copy link
Contributor

@erikjohnson24 erikjohnson24 Feb 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we pin version?

RUN pip install scikit-image
RUN pip install scipy boto3

# RUN git clone https://github.com/jhuapl-boss/intern.git && cd intern && git checkout RemoteExtension && git pull && python3 setup.py install --user
RUN pip install intern
Copy link
Contributor

@erikjohnson24 erikjohnson24 Feb 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we pin version?


RUN mkdir /app
COPY ./boss_access.py /app/
RUN chown -R 1000:100 /app/
ENV PATH /app:$PATH
WORKDIR /app

WORKDIR /app
29 changes: 29 additions & 0 deletions saber/boss_access/Dockerfile_legacy
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Copyright 2019 The Johns Hopkins University Applied Physics Laboratory
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#Use an official Python runtime as a parent image
#FROM python:3.6

# Install any needed packages specified in requirements.txt
#RUN pip install numpy
#RUN pip install scikit-image
#RUN pip install scipy boto3
#RUN pip install blosc==1.4.4
#RUN pip install intern

#RUN mkdir /app
#COPY ./boss_access.py /app/
#RUN chown -R 1000:100 /app/
#ENV PATH /app:$PATH
#WORKDIR /app
90 changes: 15 additions & 75 deletions saber/boss_access/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,21 @@ This Docker container contains the tools necessary to push and pull data from th

1. Navigate to this folder

```bash
cd saber/boss_access
```
cd saber-private/workflows/boss_access
```
1. Build the docker container
2. Build the docker image in your current directory

```
docker build -t boss-access
```bash
docker build -t <image_name> .
```

## Running

One can either run this docker container as a standalone tool, or you can launch an interactive terminal and access the tools via the command line. This is recommended, as you only have to attach volumes once.
To run this docker container

```
docker run -it -v ./data:/data/ boss-access /bin/bash
```bash
docker run --name <container_name> -it -v $(pwd)/data:/data/ <image_name> /bin/bash
```

This will launch the container as an interactive terminal and bind `./data` on your local system to `/data/` in the container.
Expand All @@ -31,8 +31,8 @@ This will launch the container as an interactive terminal and bind `./data` on y

#### Usage

```
boss_access.py pull (-c CONFIG | -t TOKEN) [-b BUCKET] --coll COLL --exp EXP --chan CHAN --coord COORD --dtype DTYPE --itype ITYPE [--res RES] [--xmin XMIN][--xmax XMAX] [--ymin YMIN] [--ymax YMAX] [--zmin ZMIN] [--zmax ZMAX] [--padding PADDING] -o OUTPUT [--s3-only]
```bash
boss_access.py pull (-c CONFIG | -t TOKEN) --coll COLL --exp EXP --chan CHAN [--res RES] [--xmin XMIN][--xmax XMAX] [--ymin YMIN] [--ymax YMAX] [--zmin ZMIN] [--zmax ZMAX] -o OUTPUT
```
#### Options

Expand All @@ -48,12 +48,6 @@ Type: String

Description: BOSS API Token

##### bucket

Type: String

Description: S3 Bucket to save to. If unset, only a local file will be saved.

##### coll

Type: String
Expand All @@ -66,24 +60,6 @@ Type: String

Description: BOSS experiment from which to download from.

##### coord

Type: String

Description: Coordinate system to use when downloading from the BOSS.

##### dtype

Type: numpy data type

Description: Datatype to recieve data in when downloading, or to use when uploading. Ex. uint8

##### itype

Type: "annotation" or "image"

Description: Image type to get.

##### res

Type: int
Expand All @@ -96,30 +72,18 @@ Type: int

Description: Bounds for volume.

##### padding

Type: int

Description: How much padding to use. Padding is applied to pulled images and stripped from pushed images.

##### Output

Type: File

Description: Output filename

##### s3-only

Type: Flag

Description: If set, no local file will be saved and the resulting cutout will be uploaded to S3.

### Pushing data

#### Usage

```
boss_access.py push (-c CONFIG | -t TOKEN) [-b BUCKET] --coll COLL --exp EXP --chan CHAN --coord COORD --dtype DTYPE --itype ITYPE [--res RES] [--xmin XMIN][--xmax XMAX] [--ymin YMIN] [--ymax YMAX] [--zmin ZMIN] [--zmax ZMAX] [--padding PADDING] -o OUTPUT [--s3-only]
```bash
boss_access.py push (-c CONFIG | -t TOKEN) --coll COLL --exp EXP --chan CHAN [--res RES] [--xmin XMIN][--xmax XMAX] [--ymin YMIN] [--ymax YMAX] [--zmin ZMIN] [--zmax ZMAX] --type TYPE -i INPUT
Copy link
Contributor

@dxenes1 dxenes1 Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add documentation that says what bossdb stack it defaults to. I'm assuming if you provide a token it defaults to 'api.bossdb.io' but maybe if you want to use 'api.theboss.io' you have to provide a config file with the host/protocol you want.

Copy link
Author

@diegoluna3 diegoluna3 Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unfamiliar with api.theboss.io, is there also a public token for that host?

Copy link
Contributor

@dxenes1 dxenes1 Feb 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah so thats the MICrONS stack so basically there are a couple different "bosses" out there. There is the public stack (api.bossdb.io), microns stack (api.theboss.io), test stack (api.bossdbtest.io), and personal dev stacks (api-xenesd1.thebossdev.io). All of them have separate data, user accounts, settings, etc. So to use this tool for another stack there should be two additional inputs:

protocol (optional, str): defaults to 'https'
host (optional, str): defaults to 'api.bossdb.io'

That way for regular use-cases I dont have to worry about remembering the public boss api domain but still have the option to use another stack if needed.

```
#### Options

Expand All @@ -135,12 +99,6 @@ Type: String

Description: BOSS API Token

##### bucket

Type: String

Description: S3 Bucket to load from. If unset, a local file specified by --input will be used.

##### coll

Type: String
Expand All @@ -153,24 +111,6 @@ Type: String

Description: BOSS experiment from which to upload to.

##### coord

Type: String

Description: Coordinate system to use when uploading to the BOSS.

##### dtype

Type: numpy data type

Description: Datatype to to use when uploading. Ex. uint8

##### itype

Type: "annotation" or "image"

Description: Image type to set.

##### res

Type: int
Expand All @@ -183,11 +123,11 @@ Type: int

Description: Bounds for volume.

##### padding
##### type

Type: int
Type: "annotation" or "image"

Description: How much padding to use. Padding is applied to pulled images and stripped from pushed images.
Description: Image type to set.

##### Input

Expand Down
Loading