java实现新闻爬取并存入数据库

原创

mob64ca12d06991 2024-05-14 07:34:49 ©著作权

©著作权归作者所有：来自51CTO博客作者mob64ca12d06991的原创作品，请联系作者获取转载授权，否则将追究法律责任

Java实现新闻爬取并存入数据库

在互联网时代，新闻信息是我们获取资讯的重要渠道之一。为了方便我们对新闻信息进行管理和分析，我们可以编写一个Java程序来实现新闻的爬取，并将抓取到的新闻信息存入数据库中。本文将介绍如何使用Java编程语言实现这一功能。

程序设计

类图

classDiagram
class NewsSpider {
    -String url
    -Document doc
    +void fetchNews()
    +void parseNews()
}

class Database {
    +void saveNews(News news)
}

class News {
    -String title
    -String content
    +String getTitle()
    +String getContent()
}
NewsSpider --> Database
NewsSpider --> News

甘特图

gantt
    title 数据爬取与存储任务列表
    dateFormat  YYYY-MM-DD
    section 数据抓取
    网页抓取     :done, 2023-10-01, 2d
    数据解析     :done, 2023-10-03, 2d
    数据存储     :done, 2023-10-05, 2d

代码实现

NewsSpider.java

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public class NewsSpider {
    private String url;

    private Document doc;

    public NewsSpider(String url) {
        this.url = url;
    }

    public void fetchNews() {
        try {
            doc = Jsoup.connect(url).get();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void parseNews() {
        Elements newsList = doc.select(".news-list-item");
        for (Element news : newsList) {
            String title = news.select(".news-title").text();
            String content = news.select(".news-content").text();
            News news = new News(title, content);
            Database.saveNews(news);
        }
    }
}

News.java

public class News {
    private String title;
    private String content;

    public News(String title, String content) {
        this.title = title;
        this.content = content;
    }

    public String getTitle() {
        return title;
    }

    public String getContent() {
        return content;
    }
}

Database.java

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;

public class Database {
    public static void saveNews(News news) {
        String sql = "INSERT INTO news (title, content) VALUES (?, ?)";

        try (Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/newsdb", "root", "password");
             PreparedStatement pstmt = conn.prepareStatement(sql)) {
            pstmt.setString(1, news.getTitle());
            pstmt.setString(2, news.getContent());
            pstmt.executeUpdate();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }
}