Featured image of post RustFS使用指南

RustFS使用指南

什么是 RustFS?

RustFS 是一个基于 Rust 实现的高性能 S3 兼容对象存储服务。在本项目中,我们使用 AWS SDK v2 访问 RustFS,享受与 AWS S3 完全兼容的 API。RustFS 提供了:

  • S3 兼容 API:使用标准的 S3 协议,可与现有 S3 工具无缝集成
  • 高性能:Rust 语言实现,保证低延迟和高吞吐量
  • 简单部署:轻量级部署,适合私有化部署场景
  • 成本效益:适合中小规模数据存储

项目集成

Maven 依赖

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
<properties>
    <aws-sdk.version>2.29.0</aws-sdk.version>
</properties>

<dependencies>
    <!-- AWS S3 SDK -->
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
        <version>${aws-sdk.version}</version>
    </dependency>
</dependencies>

S3 客户端配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.net.URI;

@Configuration
@RequiredArgsConstructor
public class S3Config {

    private final StorageConfigProperties storageConfig;

    @Bean
    public S3Client s3Client() {
        AwsBasicCredentials credentials = AwsBasicCredentials.create(
            storageConfig.getAccessKey(),
            storageConfig.getSecretKey()
        );

        return S3Client.builder()
            .endpointOverride(URI.create(storageConfig.getEndpoint()))
            .region(Region.of(storageConfig.getRegion()))
            .credentialsProvider(StaticCredentialsProvider.create(credentials))
            // 关键配置:使用路径风格访问,避免 DNS 解析问题
            .forcePathStyle(true)
            .build();
    }
}

配置文件

1
2
3
4
5
6
storage:
  endpoint: http://localhost:9000
  region: us-east-1
  bucket: my-bucket
  access-key: your-access-key
  secret-key: your-secret-key

基础使用

文件上传

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;

@Service
@RequiredArgsConstructor
public class FileStorageService {

    private final S3Client s3Client;
    private final StorageConfigProperties storageConfig;

    public String uploadFile(MultipartFile file, String prefix) {
        String fileKey = generateFileKey(file.getOriginalFilename(), prefix);

        PutObjectRequest putRequest = PutObjectRequest.builder()
            .bucket(storageConfig.getBucket())
            .key(fileKey)
            .contentType(file.getContentType())
            .contentLength(file.getSize())
            .build();

        try {
            s3Client.putObject(putRequest,
                RequestBody.fromInputStream(file.getInputStream(), file.getSize()));
            return fileKey;
        } catch (IOException e) {
            throw new BusinessException("文件上传失败");
        }
    }

    public String uploadResume(MultipartFile file) {
        return uploadFile(file, "resumes");
    }
}

文件下载

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
public byte[] downloadFile(String fileKey) {
    if (!fileExists(fileKey)) {
        throw new BusinessException("文件不存在: " + fileKey);
    }

    GetObjectRequest getRequest = GetObjectRequest.builder()
        .bucket(storageConfig.getBucket())
        .key(fileKey)
        .build();

    try {
        return s3Client.getObjectAsBytes(getRequest).asByteArray();
    } catch (S3Exception e) {
        throw new BusinessException("文件下载失败: " + e.getMessage());
    }
}

文件删除

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
public void deleteFile(String fileKey) {
    if (fileKey == null || fileKey.isEmpty()) {
        return;
    }

    if (!fileExists(fileKey)) {
        log.warn("文件不存在,跳过删除: {}", fileKey);
        return;
    }

    DeleteObjectRequest deleteRequest = DeleteObjectRequest.builder()
        .bucket(storageConfig.getBucket())
        .key(fileKey)
        .build();

    s3Client.deleteObject(deleteRequest);
}

文件是否存在检查

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
public boolean fileExists(String fileKey) {
    try {
        HeadObjectRequest headRequest = HeadObjectRequest.builder()
            .bucket(storageConfig.getBucket())
            .key(fileKey)
            .build();
        s3Client.headObject(headRequest);
        return true;
    } catch (NoSuchKeyException e) {
        return false;
    } catch (S3Exception e) {
        log.warn("检查文件存在性失败: {} - {}", fileKey, e.getMessage());
        return false;
    }
}

获取文件信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
public long getFileSize(String fileKey) {
    HeadObjectRequest headRequest = HeadObjectRequest.builder()
        .bucket(storageConfig.getBucket())
        .key(fileKey)
        .build();

    return s3Client.headObject(headRequest).contentLength();
}

public String getFileUrl(String fileKey) {
    return String.format("%s/%s/%s",
        storageConfig.getEndpoint(),
        storageConfig.getBucket(),
        fileKey);
}

文件键设计

按日期和 UUID 组织文件

1
2
3
4
5
6
7
private String generateFileKey(String originalFilename, String prefix) {
    LocalDateTime now = LocalDateTime.now();
    String datePath = now.format(DateTimeFormatter.ofPattern("yyyy/MM/dd"));
    String uuid = UUID.randomUUID().toString().substring(0, 8);
    String safeName = sanitizeFilename(originalFilename);
    return String.format("%s/%s/%s_%s", prefix, datePath, uuid, safeName);
}

生成的路径示例:resumes/2026/04/16/a1b2c3d4_张三简历.pdf

中文文件名处理

将中文文件名转换为拼音,保护特殊字符:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import net.sourceforge.pinyin4j.PinyinHelper;
import net.sourceforge.pinyin4j.format.HanyuPinyinCaseType;
import net.sourceforge.pinyin4j.format.HanyuPinyinOutputFormat;
import net.sourceforge.pinyin4j.format.exception.BadHanyuPinyinOutputFormatCombination;

private String sanitizeFilename(String filename) {
    if (filename == null || filename.isEmpty()) {
        return "unknown";
    }
    return convertToPinyin(filename);
}

private String convertToPinyin(String input) {
    HanyuPinyinOutputFormat format = new HanyuPinyinOutputFormat();
    format.setCaseType(HanyuPinyinCaseType.LOWERCASE);
    format.setToneType(HanyuPinyinToneType.WITHOUT_TONE);

    StringBuilder result = new StringBuilder();
    for (char ch : input.toCharArray()) {
        try {
            String[] pinyins = PinyinHelper.toHanyuPinyinStringArray(ch, format);
            if (pinyins != null && pinyins.length > 0) {
                // 首字母大写(大驼峰)
                result.append(capitalize(pinyins[0]));
            } else {
                result.append(sanitizeChar(ch));
            }
        } catch (BadHanyuPinyinOutputFormatCombination e) {
            result.append(sanitizeChar(ch));
        }
    }
    return result.toString();
}

private char sanitizeChar(char ch) {
    // 只保留字母、数字、点号、下划线、连字符
    if ((ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z')
            || (ch >= '0' && ch <= '9') || ch == '.' || ch == '_' || ch == '-') {
        return ch;
    }
    return '_';
}

存储桶操作

确保存储桶存在

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
public void ensureBucketExists() {
    try {
        HeadBucketRequest headRequest = HeadBucketRequest.builder()
            .bucket(storageConfig.getBucket())
            .build();
        s3Client.headBucket(headRequest);
        log.info("存储桶已存在: {}", storageConfig.getBucket());
    } catch (NoSuchBucketException e) {
        log.info("存储桶不存在,正在创建: {}", storageConfig.getBucket());
        CreateBucketRequest createRequest = CreateBucketRequest.builder()
            .bucket(storageConfig.getBucket())
            .build();
        s3Client.createBucket(createRequest);
        log.info("存储桶创建成功: {}", storageConfig.getBucket());
    } catch (S3Exception e) {
        log.error("检查存储桶失败: {}", e.getMessage());
    }
}

列出存储桶中的对象

1
2
3
4
5
6
7
8
9
public List<S3Object> listObjects(String prefix) {
    ListObjectsV2Request listRequest = ListObjectsV2Request.builder()
        .bucket(storageConfig.getBucket())
        .prefix(prefix)
        .build();

    ListObjectsV2Response response = s3Client.listObjectsV2(listRequest);
    return response.contents();
}

完整服务封装

以下是项目中实际使用的文件存储服务:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
@Slf4j
@Service
@RequiredArgsConstructor
public class FileStorageService {

    private final S3Client s3Client;
    private final StorageConfigProperties storageConfig;

    public String uploadResume(MultipartFile file) {
        return uploadFile(file, "resumes");
    }

    public String uploadKnowledgeBase(MultipartFile file) {
        return uploadFile(file, "knowledgebases");
    }

    public void deleteResume(String fileKey) {
        deleteFile(fileKey);
    }

    public void deleteKnowledgeBase(String fileKey) {
        deleteFile(fileKey);
    }

    public byte[] downloadFile(String fileKey) {
        if (!fileExists(fileKey)) {
            throw new BusinessException("文件不存在: " + fileKey);
        }

        GetObjectRequest getRequest = GetObjectRequest.builder()
            .bucket(storageConfig.getBucket())
            .key(fileKey)
            .build();
        return s3Client.getObjectAsBytes(getRequest).asByteArray();
    }

    private String uploadFile(MultipartFile file, String prefix) {
        String originalFilename = file.getOriginalFilename();
        String fileKey = generateFileKey(originalFilename, prefix);

        PutObjectRequest putRequest = PutObjectRequest.builder()
            .bucket(storageConfig.getBucket())
            .key(fileKey)
            .contentType(file.getContentType())
            .contentLength(file.getSize())
            .build();

        try {
            s3Client.putObject(putRequest,
                RequestBody.fromInputStream(file.getInputStream(), file.getSize()));
            log.info("文件上传成功: {} -> {}", originalFilename, fileKey);
            return fileKey;
        } catch (IOException e) {
            throw new BusinessException("文件读取失败");
        } catch (S3Exception e) {
            throw new BusinessException("文件存储失败: " + e.getMessage());
        }
    }

    private void deleteFile(String fileKey) {
        if (fileKey == null || fileKey.isEmpty()) {
            return;
        }

        if (!fileExists(fileKey)) {
            log.warn("文件不存在,跳过删除: {}", fileKey);
            return;
        }

        DeleteObjectRequest deleteRequest = DeleteObjectRequest.builder()
            .bucket(storageConfig.getBucket())
            .key(fileKey)
            .build();
        s3Client.deleteObject(deleteRequest);
        log.info("文件删除成功: {}", fileKey);
    }

    public boolean fileExists(String fileKey) {
        try {
            HeadObjectRequest headRequest = HeadObjectRequest.builder()
                .bucket(storageConfig.getBucket())
                .key(fileKey)
                .build();
            s3Client.headObject(headRequest);
            return true;
        } catch (NoSuchKeyException e) {
            return false;
        } catch (S3Exception e) {
            log.warn("检查文件存在性失败: {} - {}", fileKey, e.getMessage());
            return false;
        }
    }

    public String getFileUrl(String fileKey) {
        return String.format("%s/%s/%s",
            storageConfig.getEndpoint(),
            storageConfig.getBucket(),
            fileKey);
    }

    public void ensureBucketExists() {
        // 详见前文的实现
    }

    private String generateFileKey(String originalFilename, String prefix) {
        LocalDateTime now = LocalDateTime.now();
        String datePath = now.format(DateTimeFormatter.ofPattern("yyyy/MM/dd"));
        String uuid = UUID.randomUUID().toString().substring(0, 8);
        String safeName = sanitizeFilename(originalFilename);
        return String.format("%s/%s/%s_%s", prefix, datePath, uuid, safeName);
    }

    private String sanitizeFilename(String filename) {
        // 使用拼音转换实现
    }
}

错误处理

常见错误处理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
try {
    s3Client.putObject(putRequest, RequestBody.fromInputStream(inputStream, size));
} catch (NoSuchBucketException e) {
    // 存储桶不存在
    throw new BusinessException("存储桶不存在: " + storageConfig.getBucket());
} catch (NoSuchKeyException e) {
    // 文件不存在
    throw new BusinessException("文件不存在: " + fileKey);
} catch (S3Exception e) {
    // 其他 S3 错误
    log.error("S3 操作失败: {} - {}", e.awsErrorDetails().errorCode(),
              e.awsErrorDetails().errorMessage());
    throw new BusinessException("S3 操作失败: " + e.getMessage());
} catch (IOException e) {
    // IO 错误
    throw new BusinessException("文件读取失败");
}

AWS SDK v2 常用操作一览

上传下载

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
// 上传文件
s3Client.putObject(PutObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .contentType(contentType)
    .contentLength(size)
    .build(),
    RequestBody.fromInputStream(inputStream, size));

// 下载文件
byte[] data = s3Client.getObjectAsBytes(GetObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .build()).asByteArray();

// 删除文件
s3Client.deleteObject(DeleteObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .build());

存储桶操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
// 创建存储桶
s3Client.createBucket(CreateBucketRequest.builder()
    .bucket(bucketName)
    .build());

// 列出对象
s3Client.listObjectsV2(ListObjectsV2Request.builder()
    .bucket(bucket)
    .prefix(prefix)
    .build());

// 获取对象元数据
s3Client.headObject(HeadObjectRequest.builder()
    .bucket(bucket)
    .key(key)
    .build());

最佳实践

1. 使用路径风格访问

1
2
.builder()
.forcePathStyle(true) // 关键配置

2. 文件键设计

  • 使用日期路径便于清理和管理
  • 添加 UUID 防止文件名冲突
  • 中文文件名转拼音或 MD5

3. 异常处理

  • 始终处理 NoSuchBucketExceptionNoSuchKeyException
  • 对 IO 操作进行 try-catch
  • 记录详细日志便于排查

4. 性能优化

  • 使用流式上传避免大文件内存溢出
  • 合理设置 contentLength
  • 使用连接池(AWS SDK 默认使用连接池)

总结

RustFS 通过 S3 兼容 API 为我们提供了简单高效的对象存储能力。结合 AWS SDK v2,我们可以:

  • 轻松上传、下载、删除文件
  • 设计清晰的文件键结构
  • 处理中文文件名
  • 进行完整的错误处理

本指南覆盖了 RustFS 的基础使用、进阶操作和最佳实践,希望对你的项目开发有所帮助。

使用 Hugo 构建
主题 StackJimmy 设计

发布了 29 篇文章 | 共 67213 字