ClassUtils调用引发的几处Bug解决过程记录

前言:
由于Java不像其他语言具有天生的自带的动态特性,为了满足AOP开发模式,我们需要大量用到反射技术,而反射则先需要找到对应的类,这里记录一下在使用ClassUtils进行扫包的时候遇到的问题


最近因为工作需要,需要对原Storm数据处理流程进行重构,将其更换为Spark计算框架,并需要考虑其通用性、拓展性以及开发简易程度,于是着手在原有的基础上进行更改。由于本身接触过POC框架、Spring框架、AOP编程、反射调用、Logstash等技术思路。
所以借鉴这些优秀的框架或技术,基本实现了一个用于Spark集群中的数据处理框架。

由于此项目需要多人协作开发,且解析器需要可配置,建立如下配置文件,其中解析器名对应相应的Class,由框架控制流程并进行统一反射调用执行。
(从配置文件可以看到Logstash的影子)


kafaka:
    kafkaTopics: 'test'
    brokers: 'localhost:9092'
    group: 'test'
etl:
    etlLogType: 'weblog'
    etlLogFormat: '正则解析'
    etlParser:
        -   etlDataEtlId: 1
            etlDataEtlName: 'GrokParser'
            etlDataEtlTarget: 'message'
            etlDataEtlDesc: '日志拆分解析器,日志的初步解析流程'

        -   etlDataEtlId: 2
            etlDataEtlName: 'ColumnCheckParser'
            etlDataEtlTarget: 'data'
            etlDataEtlDesc: '解析结果检查,未通过检查则标记为疑似漏报数据'

        -   etlDataEtlId: 3
            etlDataEtlName: 'UrlParser'
            etlDataEtlTarget: 'url'
            etlDataEtlDesc: 'Url解析器,将请求url进行页面与参数拆分'

        -   etlDataEtlId: 4
            etlDataEtlName: 'IpParser'
            etlDataEtlTarget: 'ip'
            etlDataEtlDesc: '信息增量解析器,增加对IP的解析,增加IP对应的运营商归属地'

        -   etlDataEtlId: 5
            etlDataEtlName: 'AtkInfoParser'
            etlDataEtlTarget: 'ua,url,message'
            etlDataEtlDesc: '攻击解析器,判断日志是否存在攻击特征,并添加相应的攻击信息'

        -   etlDataEtlId: 6
            etlDataEtlName: 'PointParser'
            etlDataEtlTarget: 'ab_attack_info'
            etlDataEtlDesc: '攻击行为点所在的空间维度坐标数据,攻击路径可视化所需数据'
output:
    etlOutput:
      esUrl: 127.0.0.1:9200
      esIndex: spark-weblog

由于所有的解析器都实现了BaseParser,所以我们只要扫包并使用isAssignableFrom判断是否是其超类或超接口即可找到所有解析器然后进行反射调用执行,代码如下:


package utils;

import com.anbai.common.BaseParser;
import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Enumeration;
import java.util.List;


@SuppressWarnings("rawtypes")
public class ClassUtil {

private Class clazz;
private String packagePath;

public ClassUtil(Class clazz, String packagePath) {
this.clazz = clazz;
this.packagePath = packagePath;
}

@SuppressWarnings("unchecked")
public static void main(String[] args) throws ClassNotFoundException, IOException, InstantiationException, IllegalAccessException {
List<Class> handlers = new ClassUtil(BaseParser.class, BaseParser.class.getName()).getAllClassByPackage();
}

/**
* 通过包名获取所有实现(可以将包名配置到统一配置文件中)
*
* @param
* @return
*/
@SuppressWarnings("unchecked")
public List<Class> getAllClassByPackage() {
ArrayList<Class> returnClassList = new ArrayList<Class>();
try {
List<Class> allClass = getClasses(packagePath);
// 判断是否是一个接口
for (int i = 0; i < allClass.size(); i++) {
if (clazz.isAssignableFrom(allClass.get(i))) {
if (!clazz.equals(allClass.get(i))) {
returnClassList.add(allClass.get(i));
}
}
}
} catch (Exception e) {
}
return returnClassList;
}

private List<Class> getClasses(String packageName) throws ClassNotFoundException, IOException {
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
String path = packageName.replace(".", "/");
Enumeration<URL> resources = classLoader.getResources(path);
List<File> dirs = new ArrayList<File>();
while (resources.hasMoreElements()) {
URL resource = resources.nextElement();
dirs.add(new File(resource.getFile()));
}
ArrayList<Class> classes = new ArrayList<Class>();
for (File directory : dirs) {
classes.addAll((Collection<? extends Class>) findClass(directory, packageName));
}
return classes;
}

private List<Class> findClass(File directory, String packageName) throws ClassNotFoundException {
List<Class> classes = new ArrayList<Class>();
if (!directory.exists()) {
return classes;
}
File[] files = directory.listFiles();
for (File file : files) {
if (file.isDirectory()) {
assert !file.getName().contains(".");
classes.addAll(findClass(file, packageName + "." + file.getName()));
} else if (file.getName().endsWith(".class")) {
classes.add((Class) Class.forName(packageName + "." + file.getName().substring(0, file.getName().length() - 6)));
}
}
return classes;
}
}

然而在真实场景下,却一直出现java.lang.NoClassDefFoundError错误,但是在IDE中直接本地调试却可以正常获取所有的解析器,不解之中yz丢来一个链接:https://github.com/javasec/javaweb/blob/master/javaweb-utils/src/main/java/org/javaweb/core/utils/ClassUtils.java

于是瞬间解惑,本地搜索的时候,搜索的是target/classes中的类,而真实环境中,所有的class都在jar包之中,所以开始的代码无法搜索到,想想这属于一个低级错误。

于是将ClassUtils down下来进行替换,经过稍微更改后,变成:


/*
 * Copyright yz 2016-01-14  Email:admin@javaweb.org.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package utils;

import org.apache.commons.beanutils.PropertyUtilsBean;
import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Opcodes;

import java.beans.PropertyDescriptor;
import java.io.File;
import java.io.IOException;
import java.lang.reflect.Field;
import java.net.*;
import java.util.*;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;

public class ClassUtil extends org.apache.commons.lang.ClassUtils {

/**
* 查找指定的文件夹下的获取所有的类文件
*
* @param f
* @param path
* @param ls
*/
public static void findAllClassFile(File f, String path, Set<String> ls) {
if (f.isDirectory()) {
File[] files = f.listFiles();
for (File file : files) {
findAllClassFile(file, path, ls);
}
} else {
String fileString = f.toString();

// 查找所有的class文件
if (fileString.endsWith(".class")) {
String classPath = fileString.substring(path.length(), fileString.length());
classPath = classPath.replaceAll("\\\\+", "/").replaceAll("/+", "/").replaceAll("^/", "");
ls.add(classPath.substring(0, classPath.length() - ".class".length()));
}
}
}

public static void getAllJarClass(File file, Set<String> classList) throws IOException {
if (file.isDirectory()) {
getAllJarClass(file, classList);
} else {
if (file.getName().endsWith(".jar")) {
getAllJarClass(file.toURI().toURL(), classList);
}
}
}

/**
* 获取jar包所有class
*
* @param url
* @param classList
* @throws IOException
*/
public static void getAllJarClass(URL url, Set<String> classList) throws IOException {
if (url != null) {
URLConnection    connection = url.openConnection();
JarURLConnection juc        = (JarURLConnection) connection;
if (connection instanceof JarURLConnection) {
JarFile               jf = juc.getJarFile();
Enumeration<JarEntry> je = jf.entries();

while (je.hasMoreElements()) {
JarEntry jar = je.nextElement();

if (jar.getName().endsWith(".class")) {
String classPath = jar.getName().replaceAll("\\\\", "/").replaceAll("/+", ".");
classList.add(classPath.substring(0, classPath.length() - ".class".length()));
}
}
}
}
}

/**
* 获取当前jar包或者class目录下所有的类文件
*
* @return classList 所有的类的完整包名加类名
* @throws IOException
*/
public static Set<String> getAllClass() throws IOException {
Set<String> classList  = new LinkedHashSet<String>();
ClassUtil   classUtils = new ClassUtil();

URL url = classUtils.getClass().getProtectionDomain().getCodeSource().getLocation();
try {
if (!"http".equalsIgnoreCase(url.toURI().getScheme()) && new File(url.toURI()).isDirectory()) {
File f = new File(url.toURI());
findAllClassFile(f, f.toString(), classList);
} else {
getAllJarClass(url, classList);
}
} catch (URISyntaxException e) {
e.printStackTrace();
} catch (IOException e) {
throw e;
}

return classList;
}

/**
* 类对象是否是接口
*
* @param cr
* @return
*/
public static boolean isInterface(ClassReader cr) {
return (cr.getAccess() & Opcodes.ACC_INTERFACE) != 0;
}

/**
* 获取java类编译版本
*
* @param cr
* @return
*/
public static int getClassVersion(ClassReader cr) {
return cr.readUnsignedShort(6);
}

private static boolean shouldComputeFrames(ClassReader cr) {
return getClassVersion(cr) >= 50;
}

public static ClassWriter getClassWriter(ClassReader cr, ClassLoader classLoader) {
int writerFlags = 1;

if (shouldComputeFrames(cr)) {
writerFlags = 2;
}

return new ClassWriter(cr, writerFlags);
}

/**
* 序列化java类成Map对象
*
* @param obj
* @return
*/
public static Map<String, Object> serializeClassToMap(Object obj) {
Map<String, Object> params = new HashMap<String, Object>(0);

try {
PropertyUtilsBean    propertyUtilsBean = new PropertyUtilsBean();
PropertyDescriptor[] descriptors       = propertyUtilsBean.getPropertyDescriptors(obj);

for (int i = 0; i < descriptors.length; i++) {
String name = descriptors[i].getName();

if (!"class".equals(name)) {
params.put(name, propertyUtilsBean.getNestedProperty(obj, name));
}
}
} catch (Exception e) {
e.printStackTrace();
}

return params;
}

/**
* 获取接口所有实现类
* @param clazz
* @return
*/
public static Map<String, Class> getAllImplementsClass(Class clazz) {
Map<String, Class> classMap = new HashMap<>();
if (clazz.isInterface()) {
try {
Set<String> allClass = ClassUtil.getAllClass();
for (String als : allClass) {
String path = als.replace("/", ".");
try {
//只加载接口类所在包
String white = clazz.getPackage().getName();
if(path.startsWith(white)){
Class clz = Class.forName(path, false, Thread.currentThread().getContextClassLoader());
if (clazz.isAssignableFrom(clz)) {
if (!clazz.equals(clz)) {
Class c = clz;
classMap.put(c.getSimpleName(), c);
}
}
}

} catch (Exception e) {
continue;
}
}
} catch (IOException e) {
}
}
return classMap;
}

public static void main(String[] args) throws Exception {

URL         url       = new URL("file:/Users/jeary/IdeaProjects/spark/target/spark-1.0-SNAPSHOT-jar-with-dependencies.jar");
Set<String> clazzList = new HashSet<>();
getAllJarClass(url, clazzList);

for (String clazz : clazzList) {
System.out.println(clazz);
}


}
}

在本地调试运行,结果出现错误:
```
Exception in thread "main" java.lang.ClassCastException: sun.net.www.protocol.file.FileURLConnection cannot be cast to java.net.JarURLConnection
at utils.ClassUtil.getAllJarClass(ClassUtil.java:80)
at utils.ClassUtil.main(ClassUtil.java:266)
```

错误行代码定位至:
JarURLConnection juc        = (JarURLConnection) connection;

错误原因为:无法将FileURLConnection转换为JarURLConnection,经过一番搜素得知是协议,便尝试将file更改为jar
```
file:/Users/jeary/IdeaProjects/spark/target/spark-1.0-SNAPSHOT-jar-with-dependencies.jar

to

jar:/Users/jeary/IdeaProjects/spark/target/spark-1.0-SNAPSHOT-jar-with-dependencies.jar
```

遇到错误:
```
Exception in thread "main" java.net.MalformedURLException: invalid url
```

结果一番查询,得知正确的协议格式为:
```
jar:file:/path/test.jar!/

```

遂添加函数:

* @param url
* @return
* @throws MalformedURLException
*/
public static URL formatFileUrl2JarUrl(URL url) throws MalformedURLException {
String        jar    = "jar:";
StringBuilder urlStr = new StringBuilder();
urlStr.append(jar);
urlStr.append(url.toString());
urlStr.append("!/");
System.out.println(urlStr.toString());
return new URL(urlStr.toString());
}

在URL进行强转前更改url协议,便可以成功搜到到jar中的类。

成功找到类以后,遇到了另外一个错误,因为此方法会寻找到所有的Class并执行Class.forName,其中的静态变量或者静态语句块,以及对应的依赖都会进行加载,于是乎遇到错误:找不到类 org.osgi.framework.BundleActivator,
一开始疑惑了很久,因为明面上,不曾记得哪里有使用到这个类,最开始的解决办法是在Class.forName传入flase表示不加载静态代码,然而错误依旧,最后使用临时的解决方法为,获取传入的接口类的包名,在类加载之前判断需要加载的类的包名是否为接口类的包,是则加载否则跳过,最后重新上传jar提交任务到集群,发现解析任务已经成功执行


总结:
1.在搜索Class时一定要考虑多种场景与情况,需要确认寻找的的Class在jar包还是文件夹或其他地方

2.file协议与jar协议虽然本质上都差不多,他们之间也能进行互相转换,但更优雅的方式还得进一步研究

3.在Class.forName时,设置false,如Class.forName(path, false, Thread.currentThread().getContextClassLoader());虽然可不加载静态语句块和初始化静态变量,但类里的依赖会进行加载,需注意运行环境下是否有所有的依赖环境或着是否可跳过加载某些类

发表评论