Categories
Java

Spring Boot Upload File to S3 Asynchronously

In Spring Boot uploading files to S3 can be implemented using the REST API POST/PUT method very easily. This can be implemented using Spring WebMVC and was s3 SDK. I this example I used the following dependencies using Gradle.

Gradle dependencies

// https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web
	api 'org.springframework.boot:spring-boot-starter-web:2.4.1'

    // This dependency is exported to consumers, that is to say found on their compile classpath.
    api 'org.apache.commons:commons-math3:3.6.1'
    
    // https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3
	api 'com.amazonaws:aws-java-sdk-s3:1.11.924'
    
	// https://mvnrepository.com/artifact/org.apache.tika/tika-core
	api 'org.apache.tika:tika-core:1.25'
	
	// https://mvnrepository.com/artifact/commons-io/commons-io
	api 'commons-io:commons-io:2.8.0'
	
	// https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-hateoas
	api 'org.springframework.boot:spring-boot-starter-hateoas:2.4.1'

Spring boot config for async upload file to AWS S3

We need to define one bean for AmazonS3 and another for ThreadPoolTaskExecutor. See the below code for the definitions.

@Configuration
public class ServiceConfig {

	@Value("${AWSAccessKeyId}")
	private String accessKeyId;
	// Secret access key will be read from the application.properties file during the application intialization.
	@Value("${AWSSecretKey}")
	private String secretAccessKey;
	private final String region="ap-south-1";

	@Bean
	public AmazonS3 getAmazonS3Cient() {
		final BasicAWSCredentials basicAWSCredentials = new BasicAWSCredentials(this.accessKeyId, this.secretAccessKey);
		// Get AmazonS3 client and return the s3Client object.
		return AmazonS3ClientBuilder
				.standard()
				.withRegion(Regions.fromName(this.region))
				.withCredentials(new AWSStaticCredentialsProvider(basicAWSCredentials))
				.build();
	}

	@Bean(name="taskExecutor")
	public Executor taskExecutor() {
		final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
		executor.setCorePoolSize(10);
		executor.setMaxPoolSize(10);
		executor.setQueueCapacity(500);
		executor.setThreadNamePrefix("S3Uploader-");
		executor.initialize();
		return executor;
	}

}

Service for Async Upload to AWS S3

Below code is used to upload file to S3

@Service
public class S3UploadServiceImpl implements UploadService{

	@Autowired
	private AmazonS3 amazonS3;
	private final String bucket= "test.bimbim.in";

	//Change this with db
	private final ConcurrentMap<String, FileMeta> uploadTrackMap = new ConcurrentHashMap<String, FileMeta>();

	@Override
	public FileMeta get(final String id) {
		return this.uploadTrackMap.get(id);
	}

	@Async("taskExecutor")
	@Override
	public void  process(final byte[] bs, final FileMeta meta){
		this.upload(bs, meta);

	}

	@Override
	public void upload(final byte[] bs, final FileMeta meta) {
		this.uploadTrackMap.put(meta.getId(), meta);
		final String fileName = meta.getId()+meta.getFileName();
		final InputStream targetStream = new ByteArrayInputStream(bs);
		final PutObjectRequest putObjectRequest = new PutObjectRequest(this.bucket,fileName, targetStream, new ObjectMetadata());
		this.amazonS3.putObject(putObjectRequest);
		final String url = this.amazonS3.getUrl(this.bucket, fileName).toString();
		meta.setS3Path(url);
		meta.setUploaded(true);
	}

}

Controller to upload File

Below controller code used to received file buffered and involve service upload method to upload file aync.

@RestController
public class UploadController {

	@Autowired
	private UploadService uploadService;
	@RequestMapping(value = "/api/upload/{id}/", method = RequestMethod.GET)
	public ResponseEntity<Object> get(@PathVariable("id") final String id){
		final FileMeta meta = this.uploadService.get(id);
		if(meta==null) {
			return  ResponseEntity.notFound().build();
		}else if(meta.isUploaded()) {
			return ResponseEntity.ok(meta.add(linkTo(methodOn(UploadController.class).get(id)).withSelfRel()));
		}else {
			final ResourceStatus status = new ResourceStatus();
			status.setStatus("In Progress");
			return ResponseEntity.ok(status.add(linkTo(methodOn(UploadController.class).get(id)).withRel("delete")));
		}
	}



	@RequestMapping(value = "/api/upload/", method = RequestMethod.POST)
	public ResponseEntity<Object> upload(@RequestParam final MultipartFile file) throws IOException {
		if(file != null && !file.isEmpty()) {
			final FileMeta meta = new FileMeta();
			meta.setFileName(file.getOriginalFilename());
			meta.setLength(file.getSize());

			try {
				meta.setType( C4CFileUtils.getMediaType(file));
			} catch (final IOException e) {
				//No able to parse
				//Do exception handling
			}

			meta.setUploadedAt(Calendar.getInstance());
			final byte buffer[] = new byte[(int)file.getSize()];
			IOUtils.readFully(file.getInputStream(), buffer);

			this.uploadService.process(buffer, meta);

			return ResponseEntity.accepted().header("location", "/api/upload/"+meta.getId()+"/").build();
		}else {
			return new ResponseEntity<Object>(HttpStatus.BAD_REQUEST);
		}
	}
}

You can find the complete source on the GIT hub https://github.com/sheelprabhakar/spring-boot-examples/tree/main/spring-async-uploader-s3

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.