From 4b4939a0f831fd20fbea93e9d2523cb12ded42ea Mon Sep 17 00:00:00 2001 From: JD Date: Tue, 3 Oct 2023 01:53:21 -0400 Subject: [PATCH 1/4] initial commit --- README.md | 62 ++---------- package.json | 15 +++ public/index.html | 73 +++++++++++++++ public/main.js | 234 ++++++++++++++++++++++++++++++++++++++++++++++ public/style.css | 92 ++++++++++++++++++ server.js | 6 ++ 6 files changed, 427 insertions(+), 55 deletions(-) create mode 100644 package.json create mode 100644 public/index.html create mode 100644 public/main.js create mode 100644 public/style.css create mode 100644 server.js diff --git a/README.md b/README.md index cc17a05..95815a5 100644 --- a/README.md +++ b/README.md @@ -1,61 +1,13 @@ -Assignment 4 - Creative Coding: Interactive Multimedia Experiences -=== +## Browser Instrument -Due: October 2nd, by 11:59 AM. +http://a4-jade-logan.glitch.me -For this assignment we will focus on client-side development using popular audio/graphics/visualization technologies. The goal of this assignment is to refine our JavaScript knowledge while exploring the multimedia capabilities of the browser. +The goal of the application is to provide a screen-based instrument that is played by clicking and dragging the mouse on the screen. The pitch and volume changes depending on the mouse position. The user is able to lock the volume so it isn't controlled by movement and adjust the volume dial. The user can change the waveform for the oscillator, and set a scale so that the screen is divided into regions corresponding to notes on the chosen scale rather than the frequency being totally dependant on mouse position. -[WebAudio / Canvas Tutorial](https://github.com/cs-4241-2023/cs4241-2023.github.io/blob/main/using.webaudio_and_canvas.md) -[SVG + D3 tutorial](https://github.com/cs-4241-2023/cs-4241-2023.github.io/blob/main/using.svg_and_d3.md) +I used Web Audio API for the sound, Canvas for the mouse movement visual and the scale bars, and Nexus UI for the interface. Pointer Events API was used for mouse/touch events. -Baseline Requirements ---- +It had initially been challenging to get the oscillator to play every time the mouse dragged. I hadn't realized once the oscillator stopped it couldn't be started again. It then took a bit of trial and error to get rid of the click noise every time the oscillator was stopped by fading the audio first. Getting the scales to work with the visual accurately representing which space would play which note also took trial and error. -Your application is required to implement the following functionalities: +When the page is first loaded, basic instructions about the mouse/touch controls are displayed. The instructions disappear when the user interacts with the interface or plays a note. -- A server created using Express. This server can be as simple as needed. -- A client-side interactive experience using at least one of the following web technologies frameworks. - - [Three.js](https://threejs.org/): A library for 3D graphics / VR experiences - - [D3.js](https://d3js.org): A library that is primarily used for interactive data visualizations - - [Canvas](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API): A 2D raster drawing API included in all modern browsers - - [SVG](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API): A 2D vector drawing framework that enables shapes to be defined via XML. - - [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API): An API for audio synthesis, analysis, processing, and file playback. -- A user interface for interaction with your project, which must expose at least four parameters for user control. [tweakpane](https://cocopon.github.io/tweakpane/) is highly recommended for this, but you can also use regular HTML `` tags (the `range` type is useful to create sliders). You might also explore interaction by tracking mouse movement via the `window.onmousemove` event handler in tandem with the `event.clientX` and `event.clientY` properties. Consider using the [Pointer Events API](https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events) to ensure that that both mouse and touch events will both be supported in your app. -- Your application should display basic documentation for the user interface when the application first loads. - -The interactive experience should possess a reasonable level of complexity. Some examples: -### Three.js -- A generative algorithm creates simple agents that move through a virtual world. Your interface controls the behavior / appearance of these agents. -- A simple 3D game... you really want this to be a simple as possible or it will be outside the scope of this assignment. -- An 3D audio visualization of a song of your choosing. User interaction should control aspects of the visualization. -### Canvas -- Implement a generative algorithm such as [Conway's Game of Life](https://bitstorm.org/gameoflife/) (or 1D cellular automata) and provide interactive controls. Note that the Game of Life has been created by 100s of people using ``; we'll be checking to ensure that your implementation is not a copy of these. -- Design a 2D audio visualizer of a song of your choosing. User interaction should control visual aspects of the experience. -### Web Audio API -- Create a screen-based musical instrument using the Web Audio API. You can use projects such as [Interface.js](http://charlie-roberts.com/interface/) or [Nexus UI](https://nexus-js.github.io/ui/api/#Piano) to provide common musical interface elements, or use dat.GUI in combination with mouse/touch events (use the Pointer Events API). Your GUI should enable users to control aspects of sound synthesis. If you want to use higher-level instruments instead of the raw WebAudio API sounds, consider trying the instruments provided by [Tone.js]() or [Gibber](https://github.com/charlieroberts/gibber.audio.lib). -### D3.js -- Create visualizations using the datasets found at [Awesome JSON Datasets](https://github.com/jdorfman/Awesome-JSON-Datasets). Experiment with providing different visualizations of the same data set, and providing users interactive control over visualization parameters and/or data filtering. Alternatively, create a single visualization with using one of the more complicated techniques shown at [d3js.org](d3js.org) and provide meaningful points of interaction for users. - -Deliverables ---- - -Do the following to complete this assignment: - -1. Implement your project with the above requirements. -3. Test your project to make sure that when someone goes to your main page on Glitch/Heroku/etc., it displays correctly. -4. Ensure that your project has the proper naming scheme `a4-firstname-lastname` so we can find it. -5. Fork this repository and modify the README to the specifications below. *NOTE: If you don't use Glitch for hosting (where we can see the files) then you must include all project files that you author in your repo for this assignment*. -6. Create and submit a Pull Request to the original repo. Name the pull request using the following template: `a4-firstname-lastname`. - -Sample Readme (delete the above when you're ready to submit, and modify the below so with your links and descriptions) ---- - -## Your Web Application Title - -your hosting link e.g. http://a4-charlieroberts.glitch.me - -Include a very brief summary of your project here. Images are encouraged when needed, along with concise, high-level text. Be sure to include: - -- the goal of the application -- challenges you faced in realizing the application -- the instructions you present in the website should be clear enough to use the application, but if you feel any need to provide additional instructions please do so here. +If the screen is resized, the page should be reloaded so that the canvas will be the right size. \ No newline at end of file diff --git a/package.json b/package.json new file mode 100644 index 0000000..75aca32 --- /dev/null +++ b/package.json @@ -0,0 +1,15 @@ +{ + "name": "a4-jade-logan", + "version": "0.1.0", + "description": "CS 4241 Assignment 4", + "author": "Jade Logan", + "scripts": { + "start": "node server.js" + }, + "dependencies": { + "express": "^4.18.2" + }, + "engines": { + "node": "16.x" + } +} \ No newline at end of file diff --git a/public/index.html b/public/index.html new file mode 100644 index 0000000..7ae6b94 --- /dev/null +++ b/public/index.html @@ -0,0 +1,73 @@ + + + + CS4241 Assignment 4 + + + + + + + + + +
+
+ Click and drag to make music + +
+
+
+

Volume

+
+
+
+

Lock Volume

+
+
+
+

Waveform

+
+
+
+

Scale

+
+
+
+
+
+
+ +
+

Click and drag to make music

+

up/down: change pitch

+

left/right: change volume

+
+
+ +
+

A

+

G#

+

F#

+

E

+

D

+

C#

+

B

+

A

+

G#

+

F#

+

E

+

D

+

C#

+

B

+

A

+
+ + + +
+
+ + + + \ No newline at end of file diff --git a/public/main.js b/public/main.js new file mode 100644 index 0000000..47f17bd --- /dev/null +++ b/public/main.js @@ -0,0 +1,234 @@ +const canvas = document.getElementById("canvas"); +const ctx = canvas.getContext("2d"); + +canvas.width = window.innerWidth; +canvas.height = window.innerHeight; + +//2 octaves? +const scales = [ + [220.00, 246.94, 277.18, 293.66, 329.63, 369.99, 415.30, 440.00, 493.88, 554.37, 587.33, 659.25, 739.99, 830.61, 880.00], //A major + [220.00, 246.94, 261.63, 293.66, 329.63, 349.23, 392.00, 440.00, 493.88, 523.25, 587.33, 659.25, 698.46, 783.99, 880.00], //A minor + [246.94, 277.18, 311.13, 329.63, 369.99, 415.30, 466.16, 493.88, 554.37, 622.25, 659.25, 739.99, 830.61, 932.33, 987.77], //B major + [246.94, 277.18, 293.66, 329.63, 369.99, 392.00, 440.00, 493.88, 554.37, 587.33, 659.25, 739.99, 783.99, 880.00, 987.77], //B minor + [261.63, 293.66, 329.63, 349.23, 392.00, 440.00, 493.88, 523.25, 587.33, 659.25, 698.46, 783.99, 880.00, 987.77, 1046.50], // C major + [261.63, 293.66, 311.13, 349.23, 392.00, 415.30, 466.16, 523.25, 587.33, 622.25, 698.46, 783.99, 830.61, 932.33, 1046.50], // C minor + [293.66, 329.63, 369.99, 392.00, 440.00, 493.88, 554.37, 587.33, 659.25, 739.99, 783.99, 880.00, 987.77, 1108.73, 1174.66], //D major + [293.66, 329.63, 349.23, 392.00, 440.00, 466.16, 523.25, 587.33, 659.25, 698.46, 783.99, 880.00, 932.33, 1046.50, 1174.66], //D minor + [164.81, 185.00, 207.65, 220.00, 246.94, 277.18, 311.13, 329.63, 369.99, 415.30, 440.00, 493.88, 554.37, 622.25, 659.25], //E major + [164.81, 185.00, 196.00, 220.00, 246.94, 261.63, 293.66, 329.63, 369.99, 392.00, 440.00, 493.88, 523.25, 587.33, 659.25], //E minor + [174.61, 196.00, 220.00, 233.08, 261.63, 293.66, 329.63, 349.23, 392.00, 440.00, 466.16, 523.25, 587.33, 659.25, 698.46], //F major + [174.61, 196.00, 207.65, 233.08, 261.63, 277.18, 311.13, 349.23, 392.00, 415.30, 466.16, 523.25, 554.37, 622.25, 698.46], //F minor + [196.00, 220.00, 246.94, 261.63, 293.66, 329.63, 369.99, 392.00, 440.00, 493.88, 523.25, 587.33, 659.25, 739.99, 783.99], //G major + [196.00, 220.00, 233.08, 261.63, 293.66, 311.13, 349.23, 392.00, 440.00, 466.16, 523.25, 587.33, 622.25, 698.46, 783.99] //G minor + ]; + +const scalesLetters = [ + ["A", "B", "C#", "D", "E", "F#", "G#", "A", "B", "C#", "D", "E", "F#", "G#", "A"], //A major + ["A", "B", "C", "D", "E", "F", "G", "A", "B", "C", "D", "E", "F", "G", "A"], //A minor + ["B", "C#", "D#", "E", "F#", "G#", "A#", "B", "C#", "D#", "E", "F#", "G#", "A#", "B"], //B major + ["B", "C#", "D", "E", "F#", "G", "A", "B", "C#", "D", "E", "F#", "G", "A", "B"], //B minor + ["C", "D", "E", "F", "G", "A", "B", "C", "D", "E", "F", "G", "A", "B", "C"], // C major + ["C", "D", "Eb", "F", "G", "Ab", "Bb", "C", "D", "Eb", "F", "G", "Ab", "Bb", "C"], // C minor + ["D", "E", "F#", "G", "A", "B", "C#", "D", "E", "F#", "G", "A", "B", "C#", "D"], //D major + ["D", "E", "F", "G", "A", "Bb", "C", "D", "E", "F", "G", "A", "Bb", "C", "D"], //D minor + ["E", "F#", "G#", "A", "B", "C#", "D#", "E", "F#", "G#", "A", "B", "C#", "D#", "E"], //E major + ["E", "F#", "G", "A", "B", "C", "D", "E", "F#", "G", "A", "B", "C", "D", "E"], //E minor + ["F", "G", "A", "Bb", "C", "D", "E", "F", "G", "A", "Bb", "C", "D", "E", "F"], //F major + ["F", "G", "Ab", "Bb", "C", "Db", "Eb", "F", "G", "Ab", "Bb", "C", "Db", "Eb", "F"], //F minor + ["G", "A", "B", "C", "D", "E", "F#", "G", "A", "B", "C", "D", "E", "F#", "G"], //G major + ["G", "A", "Bb", "C", "D", "Eb", "F", "G", "A", "Bb", "C", "D", "Eb", "F", "G"] //G minor + ]; + +const ball = { + x: 100, + y: 100, + vx: 5, + vy: 1, + radius: 20, + color: "#348feb", + draw() { + ctx.beginPath(); + ctx.arc(this.x, this.y, this.radius, 0, Math.PI * 2, true); + ctx.closePath(); + ctx.fillStyle = this.color; + ctx.fill(); + }, + }; + +function clear() { + ctx.fillStyle = "rgba(255, 255, 255, 0.2)"; + if (scale === -1) { + ctx.fillRect(0, 0, canvas.width, canvas.height); + } + else { + for (let i = 0; i < 15; i = i+2) { + ctx.fillStyle = "rgba(255, 255, 255, 0.2)"; + ctx.fillRect(0, i*(canvas.height/15), canvas.width, canvas.height/15); + ctx.fillStyle = "rgba(238, 238, 238, 0.2)"; + ctx.fillRect(0, i*(canvas.height/15)+(canvas.height/15), canvas.width, canvas.height/15); + } + } +} + +function realclear() { + for (let i = 0; i < 20; i++) { + setTimeout(clear, 10+i*20); + if (i === 19) { + setTimeout(background, 10+i*20); + } + } +} + +function background() { + if (scale === -1) { + ctx.fillStyle = "rgba(255, 255, 255)"; + ctx.fillRect(0, 0, canvas.width, canvas.height); + } + else { + for (let i = 0; i < 15; i = i+2) { + ctx.fillStyle = "rgba(255, 255, 255)"; + ctx.fillRect(0, i*(canvas.height/15), canvas.width, canvas.height/15); + ctx.fillStyle = "rgba(238, 238, 238)"; + ctx.fillRect(0, i*(canvas.height/15)+(canvas.height/15), canvas.width, canvas.height/15); + } + } +} + +let context = new window.AudioContext(); + +const gainNode = context.createGain(); +gainNode.gain.value = 0.1; +gainNode.connect(context.destination); + +const currentOsc = []; +let vol = 0.1; +let wave = 'sine'; +let scale = -1; +let lock = false; + +function playSound() { + let oscillator = context.createOscillator(); + oscillator.type = wave; + oscillator.connect(gainNode); + oscillator.frequency.value = 500; + return oscillator; +} + +document.onpointerdown = startDrag; + +function getSound(osc, e) { + if (scale === -1) { + osc.frequency.value = 1000 - e.clientY; + } + else { + //let freq = 0; + let scaleArea = Math.floor((e.clientY/canvas.height) * 15); + osc.frequency.value = scales[scale][14-scaleArea]; + } + if (lock === false) { + vol = e.clientX/canvas.width; + gainNode.gain.value = vol; + volume.value = vol; + } + clear(); + ball.x = e.clientX; + ball.y = e.clientY; + ball.draw(); + +} + +function startDrag(e) { + gainNode.gain.value = 0; + const osc = playSound(); + getSound(osc, e); + osc.start(); + gainNode.gain.setTargetAtTime(vol, context.currentTime, 0.01); + + //stop drag on pointer up + document.onpointerup = function() { + gainNode.gain.setTargetAtTime(0, context.currentTime, 0.01); + osc.stop(context.currentTime + 0.1); + document.onpointerup = null; + document.onpointermove = null; + realclear(); + } + document.onpointermove = function(e) { + getSound(osc, e); + } +} + +Nexus.colors.accent = "#348feb"; + +//volume +let volume = Nexus.Add.Dial('#volume',{ + 'size': [100,100], + 'value': vol + }); + +volume.on('change',function(v) { + vol = v; +}); + +//instrument/waveform +//radio button +var radiobutton = new Nexus.RadioButton('#waveform',{ + 'size': [120,25], + 'numberOfButtons': 4, + 'active': 0 +}); + +radiobutton.on('change',function(v) { + if (v === 0) { + wave = 'sine'; + } + else if (v === 1) { + wave = 'square' + } + else if (v === 2) { + wave = 'sawtooth' + } + else if (v === 3) { + wave = 'triangle' + } +}); + +//scale +//dropdown +let select = new Nexus.Select('#scale',{ + 'size': [100,30], + 'options': ['No Scale','A major', 'A minor','B major', 'B minor','C major', 'C minor','D major', 'D minor','E major', 'E minor','F major', 'F minor','G major', 'G minor'] +}); + +select.on('change',function(v) { + scale = select.selectedIndex - 1; + let notes = document.getElementById("notes"); + if (scale === -1) { + notes.style.display = "none"; + } + else { + notes.style.display = "block"; + for (let i = 0; i < scalesLetters.length + 1; i++) { + let text = document.getElementById("stripe" + i); + text.innerHTML = scalesLetters[scale][scalesLetters.length - i]; + } + } + background(); + }) + +//lock +//toggle +let toggle = new Nexus.Toggle('#lock',{ + 'size': [40,20], + 'state': false +}); + +toggle.on('change',function(v) { + lock = v; +}) + + +let oscilloscope = new Nexus.Oscilloscope('#oscilloscope',{ + 'size': [150,75] + }); + oscilloscope.connect(gainNode) + diff --git a/public/style.css b/public/style.css new file mode 100644 index 0000000..9a2f98a --- /dev/null +++ b/public/style.css @@ -0,0 +1,92 @@ +html { + font-family: 'M PLUS Code Latin'; + } + +*{font-family: inherit;} + +#canvas { + margin: 0; + padding: 0; + z-index: -1; + } + +html, body { + margin: 0; + padding: 0; + height : 100%; + width : 100%; +} + +p { + font-size: large; + color: #348feb; + text-align: center; +} + +h1 { + font-size: xx-large; + color: #348feb; + text-align: center; +} + +.flex { + position: absolute; + top:0; + left:0; + right:0; + bottom:0; + display: flex; +} + +.controlbar { + margin: 20px; + padding: 10px; + height: fit-content; + z-index: 1; + display: flex; + flex-direction: column; + border: 5px solid #348feb; + border-radius: 20px; + background-color: white; +} + +.controls { + margin: 10px; + padding: 10px; + display: flex; + flex-direction: column; + align-items: center; +} + +.intro { + z-index: -1; + margin:30px; + flex-grow: 1; + align-self: center; +} + +.notes { + z-index: 0; + position: absolute; + top: 0; + height: 100%; + margin-top: 1.5vh; + display: none; +} + +.stripe { + width: 100vw; + height: 6.666667vh; + text-align: right; + user-select: none; + overflow: hidden; +} + +.stripe p { + color: #348feb; + margin-right: 30px; + text-align: right; + padding: 0px; + margin-bottom: 0px; + margin-top: 0px; +} \ No newline at end of file diff --git a/server.js b/server.js new file mode 100644 index 0000000..f7c6eb0 --- /dev/null +++ b/server.js @@ -0,0 +1,6 @@ +const express = require("express"), + app = express(); + +app.use(express.static("public")); + +app.listen(process.env.PORT || 3000); \ No newline at end of file From 73aa0ddfab368cd9993d46c2f44fa04c763cb8ed Mon Sep 17 00:00:00 2001 From: JD Date: Tue, 3 Oct 2023 01:55:56 -0400 Subject: [PATCH 2/4] README fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 95815a5..e239707 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ## Browser Instrument -http://a4-jade-logan.glitch.me +https://a4-jade-logan.glitch.me The goal of the application is to provide a screen-based instrument that is played by clicking and dragging the mouse on the screen. The pitch and volume changes depending on the mouse position. The user is able to lock the volume so it isn't controlled by movement and adjust the volume dial. The user can change the waveform for the oscillator, and set a scale so that the screen is divided into regions corresponding to notes on the chosen scale rather than the frequency being totally dependant on mouse position. From 5457b847a3ac6c2e94ab50e4c5881cb40db10325 Mon Sep 17 00:00:00 2001 From: JD Date: Tue, 3 Oct 2023 15:29:38 -0400 Subject: [PATCH 3/4] div resizing --- public/style.css | 2 ++ 1 file changed, 2 insertions(+) diff --git a/public/style.css b/public/style.css index 9a2f98a..2862937 100644 --- a/public/style.css +++ b/public/style.css @@ -48,6 +48,8 @@ h1 { border: 5px solid #348feb; border-radius: 20px; background-color: white; + overflow: scroll; + max-height: 86.5%; } .controls { From 9e341f1c8318bac7f387c121a66ac6d1d05cf844 Mon Sep 17 00:00:00 2001 From: JD Date: Tue, 3 Oct 2023 19:29:07 -0400 Subject: [PATCH 4/4] updated instructions --- public/index.html | 1 + 1 file changed, 1 insertion(+) diff --git a/public/index.html b/public/index.html index 7ae6b94..b0ce311 100644 --- a/public/index.html +++ b/public/index.html @@ -42,6 +42,7 @@

Click and drag to make music

up/down: change pitch

left/right: change volume

+

lock volume to change volume dial manually