When creating an application (* server-side Java is assumed here), I think there is often a requirement to save and retrieve files. There are various candidates for storing files.
--Local file system
--Cloud storage such as S3 and Cloud Storage
--RDB (BLOB type)
here
--Define an interface that abstracts file storage that does not depend on a specific storage method, --The implementation on the back side can be easily switched depending on the environment etc.
I will show you how to do that.
The completed form can be found on GitHub.
We will implement the file storage as "file storage". "File storage" is like a key-value store that manages files.
--Key (file location) --Value (contents of file)
Suppose you manage files in the form of.
We will provide an interface called FileStorageService, through which files can be saved / retrieved.
There are three types of implementation of FileStorageService.
--LocalFileStorageService (Save the file in the file system of the local host. Assuming that it will be used during local development)
--S3FileStorageService (Save to S3. Assuming use in production environment)
--ʻInMemoryFileStorageService` (Keeps files in memory, assuming use during CI and automated testing)
Finally, set Spring Boot to replace these implementations depending on your environment.
This time, we will use commons-io and ʻaws-java-sdk-s3. Let's dive into dependencies. I usually use Lombok`, but I will not use it this time because it will be a relatively simple implementation.
dependencies {
implementation 'commons-io:commons-io:2.6'
implementation 'com.amazonaws:aws-java-sdk-s3:1.11.774'
}
First, we will make the "side" side. I will make three.
--FileStorageService ... The one who handles the processing such as saving / retrieving files. The leading role.
--FileLocation ... A value object that represents the location of a file on storage.
--FileStorageObject ... An object that represents the contents of a file
It is an image to use like this.
FileStorageService fileStorageService = ...;
//Save the file
fileStorageService.putFile(FileLocation.of("hoge/fuga/sample.txt"), file);
//Extract the saved file
FileStorageObject fileStorageObject = fileStorageService.getFile(FileLocation.of("hoge/fuga/sample.txt"));
InputStream is = fileStorageObject.getInputStream(); //The contents of the file can be obtained with InputStream
The key that represents the location of the file can be String, but let's prepare a value object called FileLocation that wraps String.
FileLocation.java
First, create an object that represents the location (key) of the file.
import java.util.Objects;
/**
*The location of the file on the file storage.
*/
public class FileLocation {
/**
*path."parent/child/file.txt"Assuming a value like.
*/
private final String value;
private FileLocation(String value) {
this.value = value.startsWith("/") ? value.substring(1) : value;
}
/**
*From a string{@link FileLocation}Create an instance of.
*
* @param value path
* @return instance
*/
public static FileLocation of(String value) {
return new FileLocation(value);
}
/**
*From multiple strings{@link FileLocation}Create an instance of.
*Each string is"/"It is connected by.
*
* @param parts Multiple strings that make up the path
* @return instance
*/
public static FileLocation of(String... parts) {
if (parts.length == 1) {
return new FileLocation(parts[0]);
}
return new FileLocation(String.join("/", parts));
}
@Override
public String toString() {
return value;
}
//Implement hashCode and equals
...
}
You can get the FileLocation object like this.
FileLocation fileLocation = FileLocation.of("key/to/file.txt");
This is the same. (I referred to the Java standard API Paths.get ())
FileLocation fileLocation = FileLocation.of("key", "to", "file.txt");
FileStorageObject.java
Next, create an object that represents the file retrieved from storage. It is used in the return value of FileStorageService # getFile ().
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import org.apache.commons.io.IOUtils;
/**
*An object that represents the contents of a file on storage.
*/
public interface FileStorageObject {
/**
*The contents of the file{@link InputStream}Get in.
*
* @return {@link InputStream}
*/
InputStream getInputStream();
}
Create as an interface. Here, only the method that returns ʻInputStream` is used, but it is OK to use various convenient methods.
FileStorageService.java
Finally, make a guy who works with files. It is the leading role.
import java.io.InputStream;
import java.nio.file.Path;
/**
*A service for manipulating files in file storage.
*/
public interface FileStorageService {
/**
*Save the file.
*
* @param fileLocation Destination on storage
* @Contents of the param inputStream file
*/
void putFile(FileLocation fileLocation, InputStream inputStream);
/**
*Save the file.
*
* @param fileLocation Destination on storage
* @param localFile File to save
*/
void putFile(FileLocation fileLocation, Path localFile);
/**
*Delete the file.
*If the file does not exist, do nothing.
*
* @param fileLocation Destination on storage
*/
void deleteFile(FileLocation fileLocation);
/**
*Get the file.
*
* @param fileLocation Location on storage
* @return File object. Null if it does not exist
*/
FileStorageObject getFile(FileLocation fileLocation);
}
It's like a map with FileLocation as the key and the contents of the file as the value.
The putFile () method provides putFile (InputStream) and putFile (Path),
You can save anything as long as you have ʻInputStream. If you want to specify the contents of the file with byte []`, it looks like this.
byte[] bytes = ...;
fileStorageService.putFile(FileLocation.of("hoge"), new ByteArrayInputStream(bytes));
In ↑, FileStorageService is created as an interface, so there is no content.
Here, we will implement three, LocalFileStorageService, S3FileStorageService, and ʻInMemoryFileStorageService`.
LocalFileStorageService.java
First, create an implementation of FileStorageService that stores files in your local file system. Receives the root directory of the file storage location as a constructor argument.
public class LocalFileStorageService implements FileStorageService {
private final Path rootDirPath;
public LocalFileStorageService(Path rootDirPath) {
this.rootDirPath = Objects.requireNonNull(rootDirPath);
}
@Override
public void putFile(FileLocation targetLocation, InputStream inputStream) {
Path target = rootDirPath.resolve(targetLocation.toString());
ensureDirectoryExists(target.getParent());
try (InputStream is = inputStream) {
Files.write(target, IOUtils.toByteArray(inputStream));
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
public void putFile(FileLocation targetLocation, Path localFile) {
Path target = rootDirPath.resolve(targetLocation.toString());
ensureDirectoryExists(target.getParent());
try {
Files.copy(localFile, target, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
public void deleteFile(FileLocation targetLocation) {
Path path = rootDirPath.resolve(targetLocation.toString());
if (!Files.exists(path)) {
return;
}
try {
Files.delete(path);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
public FileStorageObject getFile(FileLocation fileLocation) {
Path path = rootDirPath.resolve(fileLocation.toString());
if (!Files.exists(path)) {
return null;
}
return new LocalFileStorageObject(path);
}
private void ensureDirectoryExists(Path directory) {
if (!Files.exists(directory)) {
try {
Files.createDirectories(directory);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
}
private static class LocalFileStorageObject implements FileStorageObject {
private final Path path;
private LocalFileStorageObject(Path path) {
this.path = path;
}
@Override
public InputStream getInputStream() {
try {
return Files.newInputStream(path);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
}
}
If you do like ↓, a file will be created in /hoge/fuga/abc/efg.txt.
FileStorageService fileStorageService = new LocalFileStorageService(Paths.get("/hoge/fuga"));
fileStorageService.putFile(FileLocation.of("abc/efg.txt"), file);
S3FileStorageService.java
Next, create an implementation of FileStorageService that stores files in AWS S3.
public class S3FileStorageService implements FileStorageService {
private final AmazonS3 s3Client;
private final String bucketName;
public S3FileStorageService(AmazonS3 s3Client, String bucketName) {
this.s3Client = Objects.requireNonNull(s3Client);
this.bucketName = Objects.requireNonNull(bucketName);
}
@Override
public void putFile(FileLocation targetLocation, InputStream inputStream) {
Path scratchFile = null;
try (InputStream is = inputStream) {
//If you try to upload directly with InputStream, you have to set ContentLength, so write it to a file once
//PutFile if you care about performance(FileLocation, InputStream, int contentLength)Or maybe you can prepare
scratchFile = Files.createTempFile("s3put", ".tmp");
Files.copy(inputStream, scratchFile);
putFile(targetLocation, scratchFile);
} catch (IOException e) {
throw new UncheckedIOException(e);
} finally {
if (scratchFile != null) {
FileUtils.deleteQuietly(scratchFile.toFile());
}
}
}
@Override
public void putFile(FileLocation targetLocation, Path localFile) {
if (!Files.exists(localFile)) {
throw new IllegalArgumentException(localFile + " does not exists.");
}
s3Client.putObject(new PutObjectRequest(bucketName, targetLocation.toString(), localFile.toFile()));
}
@Override
public void deleteFile(FileLocation targetLocation) {
s3Client.deleteObject(bucketName, targetLocation.toString());
}
@Override
public FileStorageObject getFile(FileLocation fileLocation) {
S3Object s3Object = s3Client.getObject(new GetObjectRequest(bucketName, fileLocation.toString()));
if (s3Object == null) {
return null;
}
return new S3FileStorageObject(s3Object);
}
private static class S3FileStorageObject implements FileStorageObject {
private final S3Object s3Object;
private S3FileStorageObject(S3Object s3Object) {
this.s3Object = s3Object;
}
@Override
public InputStream getInputStream() {
return s3Object.getObjectContent();
}
}
}
InMemoryFileStorageService.java
Finally, create a FileStorageService that holds the files in memory.
public class InMemoryFileStorageService implements FileStorageService {
private final Map<FileLocation, byte[]> files = new ConcurrentHashMap<>();
@Override
public void putFile(FileLocation targetLocation, InputStream inputStream) {
try (InputStream is = inputStream) {
files.put(targetLocation, IOUtils.toByteArray(inputStream));
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
public void putFile(FileLocation targetLocation, Path localFile) {
try {
byte[] bytes = Files.readAllBytes(localFile);
files.put(targetLocation, bytes);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
@Override
public void deleteFile(FileLocation targetLocation) {
files.remove(targetLocation);
}
@Override
public FileStorageObject getFile(FileLocation fileLocation) {
byte[] bytes = files.get(fileLocation);
if (bytes == null) {
return null;
}
return new InMemoryFileStorageObject(bytes);
}
private static class InMemoryFileStorageObject implements FileStorageObject {
private final byte[] bytes;
private InMemoryFileStorageObject(byte[] bytes) {
this.bytes = bytes;
}
@Override
public InputStream getInputStream() {
return new ByteArrayInputStream(bytes);
}
}
}
At this point, the implementation of the library that handles "file storage" is complete. (Slightly improved ones are on GitHub.)
Let's replace the implementation of 3 types of FileStorageService depending on the environment. The side that uses the FileStorageService asks Spring to inject an instance of the FileStorageService so that it doesn't have to know which implementation is being used.
@Service
public class SampleService {
private final FileStorageService fileStorageService; //It is injected. You don't have to know which implementation is used
public SampleService(FileStorageService fileStorageService) { //Constructor injection
this.fileStorageService = fileStorageService;
}
public void doSomething() {
fileStorageService.getFile(...);
}
}
After that, it is perfect if the instance of FileStorageService to be injected is switched depending on the environment.
[Profiles](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-profiles] is a mechanism for switching settings for each environment in Spring. ). Let's use this to replace the implementation of FileStorageService for each environment.
Here, set three Profiles according to the environment.
--During local development: development (→ use LocalFileStorageService)
--During automatic testing: test (→ ʻInMemoryFileStorageServiceis used) --Production:production (→ Use S3FileStorageService`)
Make sure that the appropriate profile is enabled in each environment.
developmentAllows the profile development to be enabled by default.
If you write ↓ in src / main / resources / application.yml, this profile will be enabled when you start the Spring Boot application normally.
src/main/resources/application.yml
spring:
profiles:
active: development
testNext, make sure that the profile test is enabled when you run the test.
Write as ↓ in src / test / resources / application.yml. (At the time of testing, this is prioritized over ↑)
src/test/resources/application.yml
spring:
profiles:
active: test
productionWhen you start the Spring Boot application in production, start it with the option --spring.profiles.active = production.
Register the bean using Java Config.
You can use the annotation @Profile to generate & register a bean only when a specific Profile is valid.
FileStorageConfiguration.java
@Configuration
public class FileStorageConfiguration {
@Bean
@Profile("development")
FileStorageService localFileStorageService(
@Value("${app.fileStorage.local.rootDir}") String rootDir) {
return new LocalFileStorageServiceFactory(Paths.get(rootDir));
}
@Bean
@Profile("test")
FileStorageService inMemoryFileStorageService() {
return new InMemoryFileStorageService();
}
@Bean
@Profile("production")
FileStorageService s3FileStorageService(AmazonS3 amazonS3,
@Value("${app.fileStorage.s3.bucketName}") String bucketName) {
return new S3FileStorageService(amazonS3, bucketName);
}
@Bean
AmazonS3 amazonS3() {
return AmazonS3ClientBuilder.defaultClient();
}
}
In the part of ↑ where it is @Value ("$ {key} "), if you write the setting corresponding to ʻapplication- {profile} .yml`, the value will be injected automatically.
src/main/resources/application-development.yml
app:
fileStorage:
local:
rootDir: "/opt/app/file_storage"
src/main/resources/application-production.yml
app:
fileStorage:
s3:
bucketName: smaple_bucket
Recommended Posts