VIRTUAL FISH-TANK PROTOTYPE
September-December 2017
This is the primary project I completed for my internship at 1-10 inc. My task was to propose and prototype a digital installation that could be placed in Tsukiji’s outer market. Tsukiji Fish Market’s inner market(築地場内) was the largest fish market in the world and the main draw of customers to the community surrounding it, but was scheduled to move to a different location in October, 2018. As a result the Tsukiji community, which has a unique history and culture, would face a loss of income so the goal of my internship project was to prototype a digital installation that could help revitalize the area by encouraging young people and tourists to visit.
The prototype I ultimately chose to build was two virtual "fish tanks" that use sensors to respond to being shaken and being placed on top of other tanks. The physical tank, which was designed in Autodesk Inventor and then laser cut, was built to reflect images from a screen onto a slanted clear piece of acrylic in the center of the tank, creating an optical illusion known as the "Pepper's Ghost Effect" where the reflections look like they are floating in mid-air. The visuals were created using Autodesk Maya, SubstancePainter and Unity. The Unity app's camera displays were then cast to the screens wirelessly so the visuals could respond in real time. I used RFID Readers and an IMU to sense user interactions, and a wifi enabled micro-controller to communicate with the Unity app via Flask server.
 |  |  |  |  |
|---|---|---|---|---|
 |  |  |
CONCEPTUALIZING INSTALLATION
The goal of my project was to create a prototype of a digital installation that would encourage young people and tourists to visit Tsukijis outer market, but the specifics of the installation was left up to me. To understand what type of installation would be appropriate, I spent the first teo weeks researching Tsukiji and visiting both the inner and outer markets to interview workers there. I then used the research and Design Thinking Techniques to brainstorm various potential installations. My goal was to come up with an installation that would first excite and engage viewers, but then encourage them to learn more about Tsukiji's culture or visit local businesses. To help visualize how the installations would affect a space, I modeled the top five installations and created a VR visualization of them. Based on feedback from my coworkers, I ultimately chose to prototype the fish tank installation because of its potential versatility and how easily it could be adapted to any space. You can view more about my brainstorming process and how I built the VR visualization here:
 |  |  |  |
|---|
ROUGH PROTOTYPE
After some initial research into potential implementations, I decided to prototype responses to two types of user interactions: being stacked on top of other tanks and being moved by the user. I also decided that I want to use the “Pepper's Ghost Effect” to create the illusion that the fishes were floating inside the box. To rapidly prototype these concepts I used some square containers to build a very basic prototype of the tanks.
To experiment with boxes responding to touching other boxes, I made simple capacitive senors using aluminum tape and a circuit connected to an Arduino Uno. I placed a capacitive sensor on each of the boxes' corners. Because the aluminum tape was conductive, touching two capacitive sensors would change their capacitance significantly enough to be reflected in the Arduino's readings. By recording when two sensors' readings changed at roughly the same time, it was possible to tell which corners on the boxes were touching. I then had a Processing script read the Arduino’s output and visualize how the boxes were touching. While this illustrated how I wanted to use sensors to the tanks' orientation, I soon realized that capacitive sensors would not be effective in the final project because they couldn't distinguish what they were touching.
To track how individual boxes were moving, I used very high tech strategy of taping a Sparkfun MPU9250 onto one of the boxes and trying to use its output to calculate pitch, yaw and roll. I used Kris Winer's library for the MPU9250 as a starting point to some success, but unfortunately, I could not get the readings to be as consistent as I would have liked.
I also tested the Pepper's Ghost Effect using my phone and a slanted plastic sheet taped into the container, in a similar method to the image above. I primarily was testing how important the sheet being at a 45 degree angle was to its effectiveness.
 |  |  |  |  |
|---|---|---|---|---|
 |  |  |  |  |
 |
BUILDING THE "TANK"
After I understood how I needed the tank to work, I began creating a physical one. I first did some quick mock-ups of what the exterior could look like in Autodesk Maya and then moved on to modeling the box with accurate measurements in Autodesk Inventor. My primary goal was ensuring that the tank could hold a screen for projecting, sensors, a micro-controller and a battery, without getting in the way of the clear part of the tank. I also designed it so the battery and Arduino would be accessible via USB ports.
To create the Pepper's Ghost Effect, I designed the box to hold a screen in its lid so that the screen would be reflected on a slanted piece of acrylic in the body of the tank, resulting in the fish looking like they were floating. I chose a pyramid design for the slanted acrylic with the hope that it would allow the reflections to be viewed from the sides. I also designed the box to have a mirror on the bottom of the pyramid so that the screen in the top would be reflected in the mirror, which would then be reflected onto the back side of the clear slanted acrylic in the tank, allowing the projections to be viewed from the back of the tank. In retrospect, I should have only worried about projecting from the front and the back of the tank, since the shape of the screens my company gave me meant there wasn’t a lot of space to project on the side of the pyramids resulting in projections that are cut off.
Once the model was completed, I used Inventor’s engineering drawing tools to make 2D Illustrator files of the tank's parts and cut them out of acrylic using a laser cutter.
 |  |  |  |  |
|---|---|---|---|---|
 |
HARDWARE: RFID
To sense when boxes are touching, I decided to use an RFID RC522, which are RFID id readers similar to how a lot of access card readers. Because RFID cards can also store information that the readers can read, placing RFID tags in each corner of the boxes meant the it was relatively easy to tell which corner of which tank was touching a different tank. Unfortunately, the readers I got could only read one card at a time which meant only the bottom of the tank could hold readers and the top had to hold tags, limiting which parts of the tank could be recognized by the readers, but a different (and more expensive) type of reader would not have this issue.
To hook up 4 rfid readers to be powered and read by the same microntroller, I had to learn to sauter and getting them t run at the same time ended up being more time consuming than I would’ve liked.
 |  |  |  |
|---|
HARDWARE: IMU
I used a spark fun MPU9250 to get information about how the box was moving. Initially I was holp to get information about the tank’s rotation and while it’s easy to tell when the tank is rotating, I couldn’t get the readings to be as reliable as I would like. IMUs aren’t capable of accurately reading how far a box has moved, but I was able to tell when the box keeps changing direction over a short period of time by tracking how often the acceleration changes MORE INFO. I ultimately used this to allow the tanks to tell when they are being shaken
 |  |  |
|---|
HARDWARE/SOFTWARE COMMUNICATION
I initially planned to use an ESP8266 to communicate between the box and the unity part of the project, since they are capable of communicating over wifi. My company happened to have some WeMos D1 boards which have an ESP8266 in them and can be programmed using the Arduino IDE. I had my unity game run a flask server and the WeMos can JSONs information to It as long as they are on the same wifi. Unfortunately, the WeMos does not actually have enough pins that can act as slave selects to run 4 RFID readers simultaneously so I switched to having an Arduino due control the the WeMos and the readers as a quick solution. If I had more time I would have found a way to only use one microncontroller, but it was not a priority for completing the prototype.
To control the screens I used a AnyCast ( basically a ChromeCast knockoff) in the box to cast images from the unity game. In retrospect I think using tablets running as clients for a multiplayer unity app and having each tablet communicate with its microncontroller via Bluetooth would have been more effective, but I was given small monitors that had to be controlled by an hdmi cable by my company so for the prototype I chose to cast to them wirelessly.
 |  |  |  |  |
|---|
CREATING THE FISH
o To make the tanks fish, I first modeled and rigged a fish model and then used a SIN CONTROLLER ON IK HANDLE on the rig’s spine to animate the fish swimming and turning. I texture the fish in SubstancePainter and then imported it into Unity. From there I made a simple movement script for the fish which make swim forward, while occasionally turning or changing speed. The fish has a simple pill collider in the front half and will turn away whenever it collides with another collider.
| | | |
|---|
FISH TANKS IN UNITY
I then made simplified tanks which follow instructions that the game receives from the flask server. Upon the start of the unity app, a control script will create instances of a tank class, which then fish and tank prefab. I initially tried to make the fish stay in the tanks by building a literal tank model with colliders that would stop the fish. However, the tank had to be built out of multiple colliders so I chose to also have the fish check if they have moved outside of their boxes boundaries. If a tank gets an instruction saying two tank are touching each other then one of the tanks will move to the other tsnk and allow the fish to move freely between them. Each time the fish move between the conenected tank boundaries then the fish are reparented to a different tank. Each tank has a camera which creates the display for the the screens to project into the physical tank.