A few weeks ago I came to a part of a project where I needed to upload user’s files. I knew that these files are likely to be large and numerous, so using a cloud mass storage provider was the obvious solution – S3 was chosen as it offers cheap storage (and even free for a year).
The project was built using Nuxt, a framework built on top of VueJS which allows server side rendering as well as a strong organisational foundation to build from, talking to a Node + MongoDB API.
The actual part of the project in question was the process of a user creating a new project and uploading a 3D model to go with it. Sounds straight forward enough, right? What actually happened was a 2 day slog through patchy documentation, out of date components and some poor assumptions on my part. This post is a write up of the critical parts of that work so that you may not suffer like I did.
Here’s the logical process for creating a new project on our website and API;
- Create a new Project record in our API
- Return the document with it’s ID
- Add a new 3d Model record to the Project (as we could have multiple per project)
- Return the new Project document with the new Model record embedded
- Request an pre-signed url from AWS S3 via the API
- Send the files to the presigned location
- Return the final location of the files from S3 to save to the database
- Cry
I won’t go into the process of creating the database record on our own API here as there are already plenty of tutorials explaining how to make a Node API and instead jump straight into how to talk to AWS S3 and get an presigned upload URL.
Nuxt and Vue components
If you’ve used Nuxt before you might have run into a problem where components that are built for Vue don’t work due to Nuxt’s Server Side Rendering features. Specifically in the case of Vue-Dropzone, it uses JS features like the browser window which are obviously not available on the server. Fortunately Nuxt-Dropzone exists that essentially wraps inside if(process.browser) Vue-Dropzone to get around this problem.
So our front end upload vue component looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | <template lang="pug"> dropzone(id="uploader" ref="uploader" :awss3="awss3" :options="dropzoneOptions" :destroyDropzone="true" v-on:vdropzone-s3-upload-error="s3UploadError" v-on:vdropzone-s3-upload-success="s3UploadSuccess" :duplicateCheck="true") </template> <script> import Dropzone from 'nuxt-dropzone' import 'nuxt-dropzone/dropzone.css' export default { components: { Dropzone }, props:{ maxFiles:{ type:Number, default:null }, autoUpload:{ type:Boolean, default:false } }, methods: { s3UploadError(errorMessage){ console.log("error: " + errorMessage) }, s3UploadSuccess(s3ObjectLocation){ console.log("success! " + s3ObjectLocation) this.$emit('uploadSuccess',s3ObjectLocation) }, processQueue(){ this.$nextTick(()=>{ this.$refs.uploader.processQueue() }) } }, computed:{ dropzoneOptions(){ return { maxFiles:this.maxFiles, url:'/', addRemoveLinks:true, autoProcessQueue:this.autoUpload } }, awss3() { return{ signingURL:(f) => {return `${process.env.api_url}/v1/upload?filename=` + f.name}, headers: { Authorization:'Bearer ' + this.$store.state.authUser.token }, sendFileToServer : true, withCredentials: false } } } } </script> |
So they key thing to look at here is the computed data property awss3() the signingURL property returns a function that takes the supplied file name from dropzone and adds it to the endpoint query string. You can also see where we’re passing the Json Web Token to our API for authorisation.
So what do we do at the end point we’ve just called from Dropzone?
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | const AWS = require('aws-sdk') const moment = require('moment') const s3 = new AWS.S3() AWS.config.update({ region: 'eu-west-2' }) const myBucket = 'bucketName' const signedUrlExpireSeconds = 60 * 60 //this url will only work for this time exports.uploadEndPoint = (req, res, next) => { //ensure the filename is unique by appending a timestamp to it var filename = moment().format('YYYYMMDDHHmmss-') + req.query.filename //Creates the Presigned Post URL for Amazon s3.createPresignedPost({ Bucket: myBucket, Fields:{ Key: filename }, Expires: signedUrlExpireSeconds, Conditions:[ {'success_action_status': '201'}, {'acl':'public-read-write'}, {'Content-Type':''} ] },function(err,data){ if(err){ console.log(err) } data.fields["success_action_status"] = 201 data.fields['acl']='public-read-write' data.fields['Content-Type'] = '' var obj = {signature:data.fields,postEndpoint:data.url} return res.json(obj) }) } |
What’s not shown here are my AWS credentials. I’m storing them in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which the Node AWS SDK will automatically pick up for you. The biggest stumbling block here was adding the Conditions object AND then repeating them back into the data.fields object. You HAVE to do this for the AWS link to work.
The obj variable is sent back to the webpage and the Dropzone begins the upload. At this point it’s worth talking about the Dropzone callbacks;
1 2 3 4 5 6 7 | s3UploadError(errorMessage){ console.log("error: " + errorMessage) }, s3UploadSuccess(s3ObjectLocation){ console.log("success! " + s3ObjectLocation) this.$emit('uploadSuccess',s3ObjectLocation) } |
Vue Dropzone allows you to add functions for different events that Dropzone will invoke, the most important of them being the vdropzone-s3-upload-success event.
In this implementation I’m emitting the event back up my component stack so I can reuse this component anywhere, but you could take the returned s3ObjectLocation and save it to a database or log it somewhere.
So that’s it, a fairly short whistle stop tour of how I managed to get my Front End, API and AWS to talk to each other and play nicely. Let me know if it helps you, because I struggled to find anything.