Merge pull request #50 from catalogueglobal/dev

v3.0.0

Author: Landon Reed

docs/dev/deployment.md

@@ -2,9 +2,16 @@
 
 ## Prerequisites
 
-The application consists of two repositories: a [Spark-powered Java backend](https://github.com/conveyal/datatools-server) and a [Javascript frontend written with React and Redux](https://github.com/conveyal/datatools-ui). To install and deploy the application, you will need Java 8, Maven, Node/npm, yarn, and [mastarm](https://github.com/conveyal/mastarm).
+The application consists of two repositories: a [Spark-powered Java backend](https://github.com/conveyal/datatools-server)
+and a [Javascript frontend written with React and Redux](https://github.com/conveyal/datatools-ui).
+To install and deploy the application, you will need Java 8, Maven, Node/npm,
+yarn, and [mastarm](https://github.com/conveyal/mastarm).
 
-User authentication is done via [Auth0](http://auth0.com). You will need an Auth0 account and application to use the Data Manager.
+User authentication is done via [Auth0](http://auth0.com). You will need an
+Auth0 account and application to use the Data Manager.
+
+Two databases are required for the application: [MongoDB](https://www.mongodb.com/)
+and a SQL database ([PostgreSQL](https://www.postgresql.org/) is recommended).
 
 ## Installation and Basic Configuration
 
@@ -22,9 +29,13 @@ $ cp datatools-ui/configurations/default/env.yml.tmp datatools-ui/configurations
 $ cp datatools-server/configurations/default/env.yml.tmp datatools-server/configurations/default/env.yml
 ```
 
-You'll then need to supply Auth0 account information (see below) and API keys for various services used in the application.
+You'll then need to supply Auth0 account information (see below) and API keys
+for various services used in the application.
 
-The default `server.yml` (for `datatools-server`) and `settings.yml` (for `datatools-ui`) should work out of the box, but you may want to specify alternative settings files outside of these repositories. These can be specified as a directory during `datatools-ui` build with mastarm:
+The default `server.yml` (for `datatools-server`) and `settings.yml` (for
+`datatools-ui`) should work out of the box, but you may want to specify
+alternative settings files outside of these repositories. These can be specified
+as a directory during `datatools-ui` build with mastarm:
 
 ```bash
 $ mastarm build --config /path/to/configurations/dir
@@ -35,24 +46,56 @@ AND as individual file args for `datatools-server`:
 ```bash
 $ java -jar target/dt-v1.0.0.jar /path/to/env.yml /path/to/server.yml
 ```
-In `datatools-server:server.yml`, be sure to update the paths for where the databases will be stored:
+In `datatools-server:server.yml`, be sure to update the paths for where the
+databases will be stored:
 
 ```yaml
 application:
   data:
-    mapdb: /path/to/mapdb
-    gtfs: /path/to/gtfs
-    editor_mapdb: /path/to/editor
-    regions: /path/to/regions/geojson
+    gtfs: /path/to/gtfs/storage
 ```
+
+### Database setup
+
+#### GTFS data storage
+GTFS data storage is handled by whichever standard RDBMS you prefer. However,
+the application has been significantly tuned and optimized for PostgreSQL 9, so
+we highly recommend using PostgreSQL.
+
+Once PostgreSQL is installed and the service has been started, create the
+database:
+```bash
+$ createdb gtfs_storage_db
+```
+Pass the URL of the database in the server's `env.yml` (and optionally add
+additional connection variables):
+```yaml
+GTFS_DATABASE_URL: jdbc:postgresql://localhost/gtfs_storage_db
+# GTFS_DATABASE_USER:
+# GTFS_DATABASE_PASSWORD:
+```
+
+#### Application data storage
+Application data storage (i.e., where projects, feed sources, and feed versions
+are stored) is handled by MongoDB. There is no need to manually initialize a
+database in MongoDB (MongoDB will handle this automatically if you prefer).
+Connection details for MongoDB are also set in the server's `env.yml`:
+```yaml
+MONGO_URI: # defaults to localhost:27017 (MongoDB default) if empty
+MONGO_DB_NAME: application_db
+```
+
 ### Setting up Auth0
 
 #### Creating account and application (client)
 1. Create an [Auth0](https://auth0.com) account (free).
-2. Once you've created an Auth0 account, create an application (client) in Auth0 to use with the Data Manager with the following settings:
-    - enable only `Username-Password-Authentication` connections (i.e., turn off Google)
+2. Once you've created an Auth0 account, create an application (client) in Auth0
+   to use with the Data Manager with the following settings:
+    - enable only `Username-Password-Authentication` connections (i.e., turn off
+      Google)
     - set `Use Auth0 instead of the IdP to do Single Sign On` to true
-    - update the following application- and account-level settings to include `http://localhost:9000` (or the domain where the application will be hosted):
+    - update the following application- and account-level settings to include
+      `http://localhost:9000` (or the domain where the application will be hosted):
         - Account level (Account Settings > Advanced)
             - Allowed logout URLs
         - Application level
@@ -61,7 +104,10 @@ application:
     - keep all other default settings
 
 #### Creating your first user
-Create your first Auth0 user through Auth0 web console (Users > Create User). In addition to an email and password, you'll need to supply the user with the following default application admin `app_metadata` (`user_metadata` should remain empty):
+Create your first Auth0 user through Auth0 web console (Users > Create User). In
+addition to an email and password, you'll need to supply the user with the
+following default application admin `app_metadata` (`user_metadata` should
+remain empty):
 
 ```json
 {
@@ -78,7 +124,7 @@ Create your first Auth0 user through Auth0 web console (Users > Create User). In
 }
 ```
 
-#### Update `env.yml` for both
+#### Update `env.yml` for server and UI repos
 Update the following properties in **both** `env.yml` files to reflect the public Auth0 application settings:
 
 ```yaml
@@ -116,7 +162,6 @@ Build and deploy the frontend to s3 using npm script (which calls [mastarm](http
 
 ```bash
 $ npm run deploy -- s3://$S3_BUCKET_NAME/dist
->>>>>>> Stashed changes
 ```
 
 Package the application using Maven:
@@ -132,12 +177,21 @@ $ java -jar target/dt-v1.0.0.jar /path/to/env.yml /path/to/server.yml
 ```
 
 
-The application back-end should now be running at `http://localhost:9000` (or whatever port you specified in `server.yml`).
-The front-end assets are pointed to by the back end at whatever s3 bucket name is specified in `server.yml` at `application.assets_bucket`.
+The application back-end should now be running at `http://localhost:9000` (or
+whatever port you specified in `server.yml`). The front-end assets are pointed
+to by the back end at whatever s3 bucket name is specified in `server.yml` at
+`application.assets_bucket`.
 
 ## Configuring Modules
 
-The application contains a number of optional modules that each require their own configuration settings and resources. At minimum, each module must be set to `enabled: true` and may require additional configuration.
+The application contains a number of optional modules that each require their
+own configuration settings and resources. At minimum, each module must be set to
+`enabled: true` and may require additional configuration.
+
+**Note:** for `datatools-server` `v3.0.0`, the `editor` and `r5_network` should be
+disabled because they have not been refactored to handle updates to the loading
+of GTFS data into an RDBMS. Please use `v2.0.0` or wait for releases following
+`v3.0.0`.
 
 ### Editor
 
@@ -150,13 +204,16 @@ Enables the GTFS Editor module.
 - `MAPBOX_ACCESS_TOKEN`
 - `R5_URL` (optional parameter for r5 routing in editor pattern drawing)
 
-### Validator
+### R5 network validation
 
-While the application handles basic validation even without the validator module enabled, the validator allows for enhanced accessibility- and map-based validation.
+While the application handles basic validation even without the `r5_network`
+module enabled, this module allows for enhanced accessibility- and map-based
+validation.
 
 #### List of configuration settings
 
-- `OSM_VEX` - `datatools-server:env.yml` the validator requires the URL of a running instance of the [OSM vex server](https://github.com/conveyal/vanilla-extract).
+- `OSM_VEX` - `datatools-server:env.yml` the validator requires the URL of a
+running instance of the [OSM vex server](https://github.com/conveyal/vanilla-extract).
 
 
 ### Sign Configurations
@@ -189,26 +246,31 @@ Enables the OTP automated deployments module.
 
 #### List of configuration settings
 
-- `OSM_VEX` - `datatools-server:env.yml`  the validator requires the URL of a running instance of the [OSM vex server](https://github.com/conveyal/vanilla-extract).
+- `OSM_VEX` - `datatools-server:env.yml`  the validator requires the URL of a
+running instance of the [OSM vex server](https://github.com/conveyal/vanilla-extract).
 
 ### GTFS API
 
-Supports other modules with API endpoints for getting GTFS entities (routes, stops, etc.)
+Supports other modules with API endpoints for getting GTFS entities (routes,
+stops, etc.)
 
 #### List of configuration settings
 
 - `load_on_fetch` - whether to load GTFS feeds when new feed is detected
 - `load_on_startup` - whether to load GTFS feeds on application startup
 - `use_extension` - which extension to connect to
-- `update_frequency` - update frequency for GTFS API (in seconds). Comment out to disable updates.
+- `update_frequency` - update frequency for GTFS API (in seconds). Comment
+  out to disable updates.
 
 ## Configuring Extensions
 
-The application supports integration with several third-party resources for retrieving feeds.
+The application supports integration with several third-party resources for
+retrieving feeds.
 
 ### Integration with [transit.land](https://transit.land/)
 
-Ensure that the `extensions:transitland:enabled` flag is set to `true` in `config.yml`:
+Ensure that the `extensions:transitland:enabled` flag is set to `true` in
+`config.yml`:
 
 ```yaml
 extensions:
@@ -219,7 +281,8 @@ extensions:
 
 ### Integration with [TransitFeeds](http://transitfeeds.com/)
 
-Ensure that the `extensions:transitfeeds:enabled` flag is set to `true` in `config.yml`, and provide your API key:
+Ensure that the `extensions:transitfeeds:enabled` flag is set to `true` in
+`config.yml`, and provide your API key:
 
 ```yaml
 extensions:

lib/alerts/actions/alerts.js

@@ -3,6 +3,7 @@ import fetch from 'isomorphic-fetch'
 import { fetchStopsAndRoutes } from '../../gtfs/actions/general'
 
 import { secureFetch } from '../../common/actions'
+import {GTFS_API_PREFIX} from '../../common/constants'
 import { getAlertsUrl, getFeedId } from '../../common/util/modules'
 import { setErrorMessage } from '../../manager/actions/status'
 import {getActiveProject} from '../../manager/selectors'
@@ -129,7 +130,9 @@ export function editAlert (alert) {
 export function fetchEntity (entity, activeProject) {
   const feed = activeProject.feedSources.find(f => getFeedId(f) === entity.entity.AgencyId)
   const feedId = getFeedId(feed)
-  const url = entity.type === 'stop' ? `/api/manager/stops/${entity.entity.StopId}?feed=${feedId}` : `/api/manager/routes/${entity.entity.RouteId}?feed=${feedId}`
+  const url = entity.type === 'stop'
+    ? `${GTFS_API_PREFIX}stops/${entity.entity.StopId}?feed=${feedId}`
+    : `${GTFS_API_PREFIX}routes/${entity.entity.RouteId}?feed=${feedId}`
   return fetch(url)
   .then((response) => {
     return response.json()