rocksdb连接java rocksdb java api

 

最近在做数据中台资产管理系统，主要核心功能是数据资产发布成api或者文档，客户通过api直接获取发布的数据资产。需要一下一下几点功能：

1  界面sql编辑数据产品

2. 自动发布数据产品到api

调研后一段时间主要技术有Rocket-api和Dataway(hasor) https://www.hasor.net/doc/display/dataway

简介：

"Rocket-API" 基于spring boot 的API敏捷开发框架，通过写SQL或者 mongodb原始执行脚本代替CRUD完成开发。提升开发效率。

特性：

1. api配置化开发，不再定义Controller,Service,Dao,Mybatis,xml,Entity,VO等对象和方法.

2. 可视化界面，将入参自动封装到可执行脚本上，支持sql执行语句

3. 完全基于spring boot 2.x作为springboot项目的starter方式集成

4. 在线动态编译，无需重新，立即生效，多数据源操作（静态配置）

5.版本控制，历史记录，回滚功能

6. 代码提示：sql提示，语法提示

7. 远程一建发布到生产环境

8. 线上POSTMAN调试,保存POSTMAN信息或三方文档的自动生成，历史调用记录存储，回塑

9. 用户控制管理，安全性管理

10. 动态数据源管理

工作原理：

1.  将api信息保存到数据库，调用springboot的RequestMapperHandlerMapping.registerMapping/unregisterMapping 实现动态管理RequestMapping。

2. 依赖java 1.8的ScriptEngineManager方法，调用Groovy引擎，赋于数据处理能力以及使代码逻辑能够实现动态编译，发布，而不用重启

搭建：

源码地址： https://gitee.com/alenfive/rocket-api

需要下载2.3.5.RELEAS版本，最新亲测有bug

带入idea，修改application.yml文件的数据库信息

datasource:
  url: jdbc:mysql://xxxxxxx:3317/dap?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
  username: myuser
  password: mypassword
  driver-class-name: com.mysql.cj.jdbc.Driver

配置数据源：

@Component
public class DefaultDataSourceManager extends DataSourceManager {

    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Autowired
    private MongoTemplate mongoTemplate;

    @PostConstruct
    public void init() {
        Map<String, DataSourceDialect> dialects = new LinkedHashMap<>();
        dialects.put("mysql",new MySQLDataSource(jdbcTemplate,true));
        //dialects.put("postgres",new PostgreSQLDataSource(jdbcTemplate,false));
        //dialects.put("oracle",new OracleDataSource(jdbcTemplate,true));
//        dialects.put("mongodb",new MongoDataSource(mongoTemplate,true));
        super.setDialectMap(dialects);
    }
}

创建数据库：

CREATE TABLE `api_info` (
  `id` varchar(45) NOT NULL,
  `method` varchar(45) DEFAULT NULL,
  `path` varchar(100) DEFAULT NULL,
  `type` varchar(5) DEFAULT NULL COMMENT '类型：CODE,QL',
  `service` varchar(45) DEFAULT NULL,
  `editor` varchar(45) DEFAULT NULL,
  `name` varchar(200) DEFAULT NULL,
  `datasource` varchar(45) DEFAULT NULL,
  `script` text,
  `options` text,
  `create_time` varchar(45) DEFAULT NULL,
  `full_path` varchar(200) DEFAULT NULL,
  `directory_id` varchar(45) DEFAULT NULL,
  `update_time` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `unique_path_method` (`service`,`full_path`,`method`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='路径明细';


CREATE TABLE `api_info_history` (
  `id` varchar(45) NOT NULL,
  `api_info_id` varchar(45) NOT NULL,
  `method` varchar(45) DEFAULT NULL,
  `path` varchar(100) DEFAULT NULL,
  `type` varchar(5) DEFAULT NULL COMMENT '类型：CODE,QL',
  `service` varchar(45) DEFAULT NULL,
  `editor` varchar(45) DEFAULT NULL,
  `name` varchar(200) DEFAULT NULL,
  `datasource` varchar(45) DEFAULT NULL,
  `script` text,
  `options` text,
  `create_time` varchar(45) DEFAULT NULL,
  `full_path` varchar(200) DEFAULT NULL,
  `directory_id` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='路径明细历史';


CREATE TABLE `api_example` (
  `id` varchar(45) NOT NULL,
  `api_info_id` varchar(45) NOT NULL,
  `method` varchar(45) DEFAULT NULL,
  `url` text,
  `request_header` text,
  `request_body` text,
  `response_header` text,
  `response_body` text,
  `status` varchar(10) DEFAULT NULL,
  `elapsed_time` int(11) DEFAULT NULL,
  `options` text,
  `editor` varchar(45) DEFAULT NULL,
  `create_time` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `index_api_id` (`api_info_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='模拟数据';

//在开启了spring.rocket-api.config-enabled=true才需要
CREATE TABLE `api_config` (
  `id` varchar(45) NOT NULL,
  `service` varchar(45) NOT NULL,
  `config_context` text,
  PRIMARY KEY (`id`),
  UNIQUE KEY `key_UNIQUE` (`id`),
  UNIQUE KEY `service_UNIQUE` (`service`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;


CREATE TABLE `api_directory` (
  `id` varchar(45) NOT NULL,
  `name` varchar(45) DEFAULT NULL,
  `path` varchar(200) DEFAULT NULL,
  `parent_id` varchar(20) DEFAULT NULL,
  `service` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

编译，重启，访问地址：http://localhost:8080/interface-ui

操作：

  创建request:

  文档同步:

     实现与 yapi 等文档中心的对接

  远程发布：

增量同步

对同步的数据存在更新，不存在新增，不会对其他API进行清除

全量同步

对目前库进行清空，导入同步的数据，会对其他API进行清除

数据流：

服务启动：

spring boot启动 -->

QLRequestMappingFactory.init() -->

注册脚本解析类dataSourceManager.setParseService(parseService)-->

加载数据库api dataSourceManager.listApiInfoByEntity(ApiInfo.builder().service(service).build()) -->

加载代码方式的API getPathListForCode(); -->

注册mapping registerMappingForApiInfo(apiInfo) -->
requestMappingHandlerMapping.registerMapping(mappingInfo,this, targetMethod);-->
 AbstractHandlerMethodMapping<T>.MappingRegistry.register(mapping, handler, method);

初始化界面:

/**
 * Api ui 页面显示
 */
@Controller
@RequestMapping("${spring.rocket-api.base-register-path:/interface-ui}")
@ConditionalOnProperty(name = "spring.rocket-api.view-enabled",havingValue = "true",matchIfMissing = true)
public class ViewController {
@GetMapping
    public String index(Model model, HttpServletRequest request){
        model.addAttribute("dataSourceList",dataSourceManager.getDialectMap().keySet());
        model.addAttribute("service", service);
        model.addAttribute("configEnabled",properties.isConfigEnabled());
        model.addAttribute("version", PackageUtils.getVersion());
        if (request.getRequestURI().endsWith("/")){
            return "redirect:"+properties.getBaseRegisterPath();
        }
        return "rocketapi/api-index";
    }
}

API 展现

/**
     * LOAD API LIST
     */
    @GetMapping("/api-list")
    public ApiResult getPathList(boolean isDb) throws Exception {
        return  ApiResult.success(mappingFactory.getPathList(isDb).stream().map(item->{
            Map<String,Object> newItem = new HashMap<>();
            newItem.put("id",item.getId());
            newItem.put("groupName",item.getGroupName());
            newItem.put("name",item.getName());
            newItem.put("method",item.getMethod());
            newItem.put("path",item.getPath());
            newItem.put("options",item.getOptions());
            newItem.put("datasource",item.getDatasource());
            return newItem;
        }).collect(Collectors.toList()));
    }

获取组件工具类信息： 前端缓存，自动提示

/**
     * 自动完成，类型获取
     */
    @GetMapping("/completion-items")
    public ApiResult provideCompletionTypes() throws Exception {
        String cacheKey = "completion-items-cache";
        CompletionResult result = null;
        if ((result = (CompletionResult) cache.get(cacheKey)) != null){
            return ApiResult.success(result);
        }

        result = new CompletionResult();
        Map<String,List<MethodVo>> clazzs = new LinkedHashMap<>();
        Map<String,String> variables = new HashMap<>();
        Map<String,String> syntax = new HashMap<>();
        Map<String,List<TableInfo>> dbInfos = new HashMap<>();
        result.setClazzs(clazzs);
        result.setVariables(variables);
        result.setSyntax(syntax);
        result.setDbInfos(dbInfos);

        //获取内置自定义函数变量
        Collection<IFunction> functionList = context.getBeansOfType(IFunction.class).values();
        functionList.forEach(item->{
            variables.put(item.getVarName(),item.getClass().getName());
        });

        //spring bean对象获取
        Map<String,Object> beans = context.getBeansOfType(Object.class);

        for (String key : beans.keySet()){
            buildClazz(clazzs,beans.get(key).getClass());
        }

        //本包JAVA类
        List<Class> classList = PackageUtil.loadClassByLoader(Thread.currentThread().getContextClassLoader());
        for (Class clazz : classList){
            buildClazz(clazzs,clazz);
        }

        //基础包 java.util java类
        List<String> classNames = PackageUtil.scan();
        for (String clazz : classNames){
            buildClazz(clazzs,clazz);
        }

        //常用语法提示
        syntax.put("foreach","for(item in ${1:collection}){\n\t\n}");
        syntax.put("fori","for(${1:i}=0;${1:i}<;${1:i}++){\n\t\n}");
        syntax.put("for","for(${1}){\n\t\n}");
        syntax.put("if","if(${1:condition}){\n\n}");
        syntax.put("ifelse","if(${1:condition}){\n\t\n}else{\n\t\n}");
        syntax.put("import","import ");
        syntax.put("continue","continue;");
        syntax.put("break","break;");

        //数据库信息获取
        Map<String, DataSourceDialect> dataSourceDialectMap = dataSourceManager.getDialectMap();
        dataSourceDialectMap.forEach((key,value)->{
            List<TableInfo> tableInfos = value.buildTableInfo();
            if (tableInfos != null){
                dbInfos.put(key,tableInfos);
            }
        });

        //常用工具类获取
        cache.put(cacheKey,result);
        return ApiResult.success(result);
    }

新建api：

/**
     * SAVE APIINFO
     * @param apiInfo
     */
    @PostMapping("/api-info")
    public ApiResult saveOrUpdateApiInfo(@RequestBody ApiInfo apiInfo,HttpServletRequest request) {

        String user = loginService.getUser(request);
        if(StringUtils.isEmpty(user)){
            return ApiResult.fail("Permission denied");
        }

        apiInfo.setEditor(user);
        try {
            if (!StringUtils.isEmpty(apiInfo.getScript())){
                apiInfo.setScript(scriptEncrypt.encrypt(apiInfo.getScript()));
            }
            return ApiResult.success(mappingFactory.saveOrUpdateApiInfo(apiInfo));
        }catch (Exception e){
            e.printStackTrace();
            return ApiResult.fail(e.getMessage());
        }

    }


@Transactional
    public String saveOrUpdateApiInfo(ApiInfo apiInfo) throws Exception {

        if (existsPattern(apiInfo)){
            throw new IllegalArgumentException("method: "+apiInfo.getMethod()+" path:"+apiInfo.getPath()+" already exist");
        }

        ApiInfo dbInfo = apiInfoCache.getAll().stream().filter(item->item.getId().equals(apiInfo.getId())).findFirst().orElse(null);

        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        apiInfo.setUpdateTime(sdf.format(new Date()));
        if (dbInfo == null){
            apiInfo.setType(ApiType.Ql.name());
            apiInfo.setCreateTime(sdf.format(new Date()));
            apiInfo.setService(service);
            apiInfo.setId(GenerateId.get().toHexString());
            dataSourceManager.saveApiInfo(apiInfo);
        }else{
            apiInfo.setType(dbInfo.getType());
            apiInfo.setCreateTime(dbInfo.getCreateTime());
            apiInfo.setService(dbInfo.getService());

            dataSourceManager.updateApiInfo(apiInfo);

            //取消mapping注册
            unregisterMappingForApiInfo(dbInfo);

            //清理缓存
            apiInfoCache.remove(dbInfo);
        }

        dbInfo = dataSourceManager.findApiInfoById(apiInfo);

        //入缓存
        apiInfoCache.put(dbInfo);

        //注册mapping
        this.registerMappingForApiInfo(dbInfo);

        //存储历史
        saveApiHistory(dbInfo);

        return dbInfo.getId();
    }

测试api:

/**
     * 脚本执行
     */
    @PostMapping("/api-info/run")
    public ApiResult runScript(@RequestBody RunApiReq runApiReq, HttpServletRequest request){

        String user = loginService.getUser(request);
        if(StringUtils.isEmpty(user)){
            return ApiResult.fail("Permission denied");
        }

        RunApiRes runApiRes = new RunApiRes();
        try {
            apiInfoContent.setIsDebug(runApiReq.isDebug());
            ApiInfo apiInfo = ApiInfo.builder()
                    .path(runApiReq.getPattern())
                    .options(runApiReq.getOptions())
                    .datasource(runApiReq.getDatasource())
                    .script(runApiReq.getScript())
                    .build();
            ApiParams apiParams = ApiParams.builder()
                    .header(decodeHeaderValue(runApiReq.getHeader()))
                    .pathVar(getPathVar(runApiReq.getPattern(),runApiReq.getUrl()))
                    .param(getParam(runApiReq.getUrl()))
                    .body(buildBody(runApiReq.getBody()))
                    .session(RequestUtils.buildSessionParams(request))
                    .build();
            Object value = scriptParse.runScript(apiInfo.getScript(),apiInfo,apiParams);
            runApiRes.setData(value);
            return ApiResult.success(runApiRes);
        }catch (Throwable e){
            e.printStackTrace();
            return ApiResult.fail(e.getMessage(),runApiRes);
        }finally {
            runApiRes.setLogs(apiInfoContent.getLogs());
            apiInfoContent.removeAll();
        }
    }

@Override
    @Transactional(rollbackFor=Exception.class)
    public Object runScript(String script, ApiInfo apiInfo, ApiParams apiParams) throws Throwable {

        Integer pageNo = buildPagerNo(apiParams);
        Integer pageSize = buildPagerSize(apiParams);
        apiParams.putParam(apiPager.getPageNoVarName(),pageNo);
        apiParams.putParam(apiPager.getPageSizeVarName(),pageSize);
        apiParams.putParam(apiPager.getIndexVarName(),apiPager.getIndexVarValue(pageSize,pageNo));

        try {

            //注入变量
            apiInfoContent.setApiInfo(apiInfo);
            apiInfoContent.setApiParams(apiParams);

            Bindings bindings = new SimpleBindings();

            apiInfoContent.setEngineBindings(bindings);
            for(IFunction function : functionList){
                bindings.put(function.getVarName(),function);
            }

            //注入属性变量
            buildScriptParams(bindings,apiParams);
            Object result = this.engineEval(script,bindings);
            return result;
        }catch (Exception e){
            if (e.getCause() != null && e.getCause().getCause() != null){
                throw e.getCause().getCause();
            }else{
                throw e;
            }
        }

    }

 @Override
    public Object engineEval(String script,Bindings bindings) throws Throwable {
        try {
            return engine.eval（script,bindings);
        }catch (Exception e){
            if (e.getCause() != null && e.getCause().getCause() != null){
                throw e.getCause().getCause();
            }else{
                throw e;
            }
        }
    }


javax.script.ScriptEngine.eval（String script, Bindings n)

文档同步：

/**
     * API DOC 同步
     */
    @GetMapping("/api-doc-push")
    public ApiResult apiDocPush(String apiInfoId) throws Exception {
        Collection<ApiInfo> apiInfos = mappingFactory.getPathList(false);
        String result = null;
        if (!StringUtils.isEmpty(apiInfoId)){
            ApiInfo apiInfo = apiInfos.stream().filter(item->item.getId().equals(apiInfoId)).findFirst().orElse(null);
            result = apiDocSync.sync(apiInfo,buildLastApiExample(apiInfo.getId()));
        }else{
            for(ApiInfo apiInfo : apiInfos){
                result = apiDocSync.sync(apiInfo,buildLastApiExample(apiInfo.getId()));
            }
        }
        return ApiResult.success(result);
    }

/**
 * 默认API信息接口同步，
 */
@Slf4j
@Component
public class DefaultApiDocSync implements IApiDocSync {

    @Override
    public String sync(ApiInfo apiInfo, ApiExample apiExample) {
        return "Successful push";
    }
}
啥也没做

 

 

 

源码：

 

总结

 

修改